Using the Z39.50 client “yaz-client”

If you have technical experience in libraries, you’ve probably heard of Z39.50 (https://en.wikipedia.org/wiki/Z39.50). It’s an old standard, but it’s something libraries still use.

Koha, the open source ILS/LMS, uses Indexdata’s Zebra as its search engine, and it communicates with it using Z39.50 to fetch records in MARC format (another old standard still in use).

MarcEdit is a program used for editing MARC records, and it integrates with some of Koha’s HTTP APIs for adding and updating bibliographic MARC records. It also uses Zebra for searching Koha’s database of MARC records.

I was reading Koha’s listserv when I encountered someone having troubles connecting to Zebra from MarcEdit. I had the same troubles years ago, so I decided to give it another shot now that I’m a lot more experienced with Koha, Zebra, Z39.50, MARC, networking, and really all things IT.

I followed the instructions in the Koha file for exposing Zebra over TCP, but MarcEdit was still failing to connect. Unfortunately, it didn’t really give me any useful information about why it was failing. So I decided to download a Z39.50 client and see if I could get down lower and see what was happening under the hood.

I’ve been using yaz-client (http://www.indexdata.com/yaz/doc/yaz-client.html) for years, as it comes bundled with the YAZ libraries used by Zebra. However, I’ve always been using it on the Linux servers running Koha. In this case, I wanted to connect from my Windows desktop…

Fortunately, you can download YAZ for Windows! I visited Indexdata’s page about installing YAZ on Windows at http://www.indexdata.com/yaz/doc/installation.win32.html, and downloaded the latest version of YAZ.

I still couldn’t connect to Zebra using yaz-client.exe, but now it was clear that it was a connection problem.

I tried using yaz-client on the Linux server that hosted Zebra, and it worked fine. It was clearly a networking issue… the default TCP port for Zebra (ie 9998) was probably being blocked by my network router. So I looked around for a free port using ‘netstat -ln | grep “3000”‘. Port 3000 wasn’t being used by anything else, so I changed the Zebra configuration, restarted my Zebra server, and tried to connect to Zebra using yaz-client.exe on Windows again.

This time the client connected successfully and I had anonymous read-only access to Zebra!

In hindsight, I should have tried port 210 which is the official port for Z39.50 communications or port 7090 which is also sometimes used for Z39.50 servers. But this was good enough for my temporary experiment. I tried again in MarcEdit, and it worked! I was now able to search for records in Koha, download them as MARC21, edit them in MarcEdit, and then update them in Koha using one of Koha’s HTTP APIs!

Now, I was connecting over a private local network. If you were to forward that port on your router so that it was Internet accessible, I would highly recommend changing the default username and password for Zebra!

Anyway, even if you’re not using Koha or MarcEdit or Zebra, you can still use yaz-client on Linux and yaz-client.exe on Windows to connect to all sorts of other Z39.50 servers like the ones at the Library of Congress, LibrariesAustralia, the National Library of Australia, and so on.

This can be useful when you’re trying to troubleshoot Z39.50 connection issues or if you just want to have more control over your Z39.50 requests than what you have baked into your ILS/LMS. Maybe you’re trying to fetch a record from the Library of Congress and you know it’s there… but your ILS/LMS doesn’t seem able to find it. Whip out yaz-client and check it out yourself!

Cronjobs in Debian

When I first started developing and managing Koha, the most mysterious things in my mind were cronjobs and Zebra indexing.

Well, I’ve learned a lot about Zebra indexing over the past year. I think I’ve already shared quite a bit on this blog and I’m sure there is more to come. Maybe I’ll even re-visit some old entries to give better explanations sometime.

However, right now I’d like to talk about cronjobs. Cronjobs are basically commands that are scheduled to occur at specific times. They can look scary: 15 17 * * * COMMAND.

That would translate as: “Run this ‘COMMAND’ every day of every month at 5:15pm.

Here is a graphic I stole from http://www.debian-administration.org/articles/56:

 

*     *     *     *     *  Command to be executed
-     -     -     -     -
|     |     |     |     |
|     |     |     |     +----- Day of week (0-7)
|     |     |     +------- Month (1 - 12)
|     |     +--------- Day of month (1 - 31)
|     +----------- Hour (0 - 23)
+------------- Min (0 - 59)

Another useful site I stumbled across was this: http://forums.hostsearch.com/showthread.php?2693-Crontab-explained

Now...if you use the koha-common packages for Koha...you should never ever have to touch anything related to cron or think about cronjobs. They'll just happen automagically (http://en.wiktionary.org/wiki/automagical). 

This is just for folks who either don't use the koha-common packages, or for those who just want to know how cron works.

In Debian, you'll notice that there are other areas than the crontab that can be used for scheduling cronjobs. These are cron.d, cron.daily, cron.hourly, cron.weekly, cron.monthly (http://www.debian-administration.org/articles/56). In all cases except cron.d, any scripts that you put in these directories will be executed daily, hourly, weekly, or monthly (the exact times being determined in the crontab). cron.d is there for your more specific cronjobs. For instance, maybe you want COMMAND X to run every 5 minutes. You would set up a special script (like koha-common or anacron) and type in: "*/5 * * * * COMMAND X"

Easy as pie! Easier, in fact.

There’s certainly more to cron than what I’ve laid out (you can get more sophisticated), but that’s what those other links are for ;).

Zebra indexing, Bib-1 Attributes, CCL, and more…

Unfortunately, I don’t have time to really elaborate today, but perhaps I will come back and explain later. Until then, here is a list of links…

Yaz-Client (for querying Z39.50 databases like Zebra)

http://www.indexdata.com/yaz/doc/yaz-client.html

Bib-1 Attributes

http://www.loc.gov/z3950/agency/defns/bib1.html

Bib-1 Attributes not supported in Zebra

http://www.indexdata.com/zebra/doc/querymodel-rpn.html#querymodel-bib1-nonuse

Zebra Query Model

http://www.indexdata.com/zebra/doc/querymodel-zebra.html

A bit of talk about how Koha and Zebra link together using CCL/PQF

http://koha.1045719.n5.nabble.com/Search-at-the-beginning-of-an-expression-td3364395.html

CCL Special Attribute Combos

http://www.indexdata.com/yaz/doc/tools.html#ccl.special.attribute.combos

In terms of adjusting Zebra/Koha settings, you’ll want to look at:

bib1.att

ccl.properties

record.abs

Bib1.att lists and maps bib-1 attributes (default from the library of congress spec and special added ones for Koha)
Ccl.properties maps CCL to BIB1/PQF
Record.abs maps MARC fields to BIB1

Also, since it’s really hard to find information about how to construct PQF queries using BIB1…

There are 6 types of Bib-1 attributes. Each type has a variety of codes within that type. To create a query, you would type something like the following into Yaz or whatever else you’re using that utilizes PQF:

f @attr 1=4 computer

f stands for find in yaz

@attr stands for attribute (you need to write one of these for each attribute you’re creating)
1=4 stands for type 1: use attributes with a code of 4 for title
1=4 -> use attribute=title

It’s actually quite straightforward, but it’s quite rare to find it spelled out for you on the Web!