[OPEN-ILS-DEV] "Better" Z39.50 support

[Dan Scott]

Just in time for the holidays, I worked out a solution for a problem
where we were seeing garbled characters instead of diacritics in the
records resulting from Z39.50 serarches. This has been a problem since
we went live, and I've had an on-again / off-again battle trying to
figure out why it works in some of the Z39.50 clients I try (yaz-client,
MarcEdit) but fails in RefWorks, VDX, and some other clients. 

The result of my head-bashing? 

Sadly, it looks like RefWorks, RACER, and many other library-oriented
applications still do not support the Unicode UTF-8 encoding for the
Z39.50 protocol. Never mind that UTF-8 was approved for use in library
systems back in 1998; or that the UTF-8 RFC was first published in 1996,
and has since been adopted as the defacto standard encoding in
information systems outside of the realm of libraries. 

The fix, therefore, was to force the server to return MARC-8 output. [1]
This horribly breaks Z39.50 clients that might request MARCXML records,
because MARC-8 isn't a valid XML encoding, but that doesn't really
matter; no sane XML-oriented applications would use Z39.50 anyway. Thus
the quotes around "better" in the subject; we become part of the problem
by perpetuating MARC-8 rather than being part of the solution. 

Oh, and our leader 09 positions will lie and say that these MARC-8,
because as far as I know there's no way to configure the yaz chain of
tools underlying the Z39.50 server to toggle the 'a' back to ' '.

Ah well. 

The end result, and the truly important part, is that you should find
that RefWorks, VDX, and other applications that use the Evergreen Z39.50
server should be happier.

Now, if there's a yaz configuration guru out there who can figure out
how to serve up both UTF-8 and MARC-8 encoded MARC21 and MARCXML
records, and toggle the LDR 09 position accordingly, that would be a
great enhancement. For now, I'm just going to be happy about making our
users' applications work better in the short term, and weep bitterly
about the damage we're doing to the adoption of UTF-8 in the library
world in the longer term.

1.
http://evergreen-ils.org/dokuwiki/doku.php?id=evergreen-admin:sru_and_z39.50&rev=1259004736&do=diff