[OPEN-ILS-DEV] ***SPAM*** Re: Enabling UTF8 data in SIP2 (patch and RFC)

[David Fiander]

Dan,

Do you have some unit tests to add to the SIP code to test this new
stuff? It would be great if we could test both configurations
automatically, but the test framework doesn't support changing the
config file on the fly, so we'd have to do something funky.

On Tue, Jan 5, 2010 at 14:19, Dan Scott <dan at coffeecode.net> wrote:
> On Tue, 2010-01-05 at 00:49 -0500, Dan Scott wrote:
>> On Mon, 2010-01-04 at 22:58 -0500, Mike Rylander wrote:
>> > On Mon, Jan 4, 2010 at 10:13 PM, Dan Scott <dan at coffeecode.net> wrote:
>> > > On Mon, 2010-01-04 at 21:18 -0500, Mike Rylander wrote:
>> > >> On Mon, Jan 4, 2010 at 8:34 PM, Dan Scott <dan at coffeecode.net> wrote:
>> > >> [snip]
>> > >>
>> > >> > So -- if I uncomment those two lines in OpenILS::SIP::clean_text() so
>> > >> > that the combining characters are stripped by default, would you find
>> > >> > this patch acceptable?
>> > >> >
>> > >> >
>> > >>
>> > >> Making the encoding a config file option (SIPServer.pm config, that
>> > >> is) and checking that inside O::S::clean_text(), defaulting to
>> > >> encode('ascii',$text) or the NFD()+s/\pM+//go, seems like it would be
>> > >> the most complete thing to do.  Is that reasonably doable?
>> > >>
>> > >
>> > > Yep, that's doable. To be clear about the SIPServer.pm config, you mean
>> > > a new element in
>> > > acsconfig/institutions/institution/implementation_config/ -- say,
>> > > <encoding> -- in OpenSRF/examples/oils_sip.xml.example, with "ascii" as
>> > > the default and "utf-8" as a commented out option?
>> >
>> > Indeed.
>> >
>> > >
>> > > The hard part is that we don't have any extra SIP client hardware, so
>> > > our production unit is also our testing platform. (Tiny violins, I
>> > > know). It just means that I have limited time to put into testing the
>> > > code because while I'm testing I'm depriving our patrons of the pleasure
>> > > of using the system :)
>> > >
>> >
>> > I don't see too much risk in the 'ascii' option -- it's what we're
>> > attempting to do right now -- but if we put that into trunk I bet we
>> > can find some sites to test it, or even vendors.  If the native test
>> > client is happy, and we can find at least one hardware client to test
>> > with, then it's backporting time, eh?
>> >
>>
>> Right-o. Please find the updated patch attached. I plan to give it a
>> trial run tomorrow in both ASCII and UTF-8 modes, and if all turns out
>> well and nobody spots a showstopper, I'll commit it.
>>
>> Thanks again, everyone.
>
> Okay, after testing the patch in both ASCII and UTF8 mode, all seems
> well. There are still some corners of UTF8 breakage in the 3M client
> (hello, renewals interface) but the UTF8 data is displayed cleanly in
> the core self-check interfaces and the printer.
>
> Also, the ASCII mangling works nicely now that the mangling has been
> extended to fields other than just the item title. Without the ASCII
> mangling of patron names, for example, the Renew button wouldn't even
> show up on the 3M client for a patron who had an accent in their name.
>
> Therefore I've committed the patch to trunk. Note that a required
> corollary for enabling UTF8 support is an update to the openncip code:
> http://sourceforge.net/support/tracker.php?aid=2925760 for now, until
> the patch gets merged (if it gets merged).
>
> Thanks for all the help, folks.
>
>