[OPEN-ILS-DEV] AddedContent.pm produces garbled encoding for application/json

Mon Jan 14 09:38:33 EST 2019

Linda,

Your plan sounds OK to me. I think it can wait for the new features,
since your site is the main (probably the only) user of the module.

I'll take a look at the Open Library bug you mentioned. We're not using
it, but maybe there is something that we can do generically to resolve
this in a way that doesn't require duplicated code.

Cheers,
Jason

On 1/14/19 9:19 AM, Linda Jansova wrote:
> Hi Jason,
> 
> There is not a Launchpad bug for this yet because at this point it has
> only been fixed locally (in ObalkyKnih.pm module in our Evergreen
> installation). Currently we are in the process of adding some more
> features to ObalkyKnih.pm module and testing these features. Of course,
> we will create a wishlist bug and submit the enriched code (which would
> also contain the encoding fix) to GIT. Should there be a wider interest
> in the corrected code at this point, we would submit a patch for the
> encoding only (and add the new features at a later date).
> 
> Maybe that this (or similar) solution would also work for encoding
> issues in added content coming from Open Library - for which a bug has
> been opened already: https://bugs.launchpad.net/evergreen/+bug/1610678.
> (Josh did some testing with Open Library data encoding.)
> 
> Linda
> 
> On 1/14/19 2:45 PM, Jason Stephenson wrote:
>> Hi, Linda.
>>
>> Good find! That sounds like a bug to me. Could you submit a Launchpad
>> bug (if there isn't one already) and a patch commit, please?
>>
>> Depending on which module this occurs in, the fix may not be as simple
>> as using utf8::decode. It could be that we need code to determine the
>> character set from the HTTP response.
>>
>> Cheers,
>> Jason
>>
>> On 1/14/19 2:38 AM, Linda Jansova wrote:
>>> Hi,
>>>
>>> Just letting you know that our developer has eventually changed the
>>> encoding of the data blob coming from Obalkyknih.cz from UTF-8 to Perl's
>>> internal representation:
>>>
>>> utf8::decode($response_content);
>>>
>>> Adding this step before further data processing takes place has
>>> successfully solved the problem of seeing gibberish characters in our
>>> catalog :-).
>>>
>>> Linda
>>>
>>> On 11/25/18 4:56 PM, Linda Jansova wrote:
>>>> Hi,
>>>>
>>>> We have encountered a problem in AddedContent.pm's get_url function
>>>> (both in Evergreen 3.1.4 and 2.12.6) - letters with diacritics from
>>>> Czech added content provider Obalkyknih.cz have started being
>>>> corrupted in our TPACs. Our added content provider has reported a
>>>> switch to application/json MIME type.
>>>>
>>>> After the switch we seem to be getting strange chars in summary (and
>>>> table of contents if available as text, not as an image). We have
>>>> tried to locate the problem in a separate Perl program and have come
>>>> to a conclusion that data only get corrupted when fetched by the
>>>> get_url function from AddedContent.pm.
>>>>
>>>> We have added additional logging to the following part of
>>>> AddedContent.pm:
>>>>
>>>> # returns an HTPP::Response object
>>>> sub get_url {
>>>>      my( $self, $url ) = @_;
>>>>
>>>>      $logger->info("added content getting [timeout=$net_timeout,
>>>> errors_remaining=$error_countdown] URL = $url");
>>>>      my $agent = LWP::UserAgent->new(timeout => $net_timeout);
>>>>
>>>>      my $res = $agent->get($url);
>>>>      $logger->info("added content request returned with code " .
>>>> $res->code);
>>>>
>>>>      #VJ
>>>>      $logger->info("added contet res is: " . $res->content);
>>>>
>>>>      die "added content request failed: " . $res->status_line ."\n"
>>>> unless $res->is_success;
>>>>
>>>>      return $res;
>>>> }
>>>>
>>>> And a corresponding sample log looks like this:
>>>>
>>>> [2018-11-24 22:22:58] /usr/sbin/apache2
>>>> [INFO:32231:AddedContent.pm:296:1543094555322319] added contet res is:
>>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"NÃ¡rodnÃ
>>>>
>>>> knihovna Ä<U+008C>eskÃ© Republiky
>>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>>>
>>>> obalkyknih.cz","html":"Encyklopedie sociÃ¡lnÃ prÃ¡ce (ESP)
>>>> pÅ<U+0099>inÃ¡Å¡Ã pÅ<U+0099>es 200 hesel ze vÅ¡ech oblastÃ
>>>> sociÃ¡lnÃ prÃ¡ce. ESP je postavena na interakÄ<U+008D>nÃm pojetÃ
>>>> sociÃ¡lnÃ prÃ¡ce. JedineÄ<U+008D>nost sociÃ¡lnÃ prÃ¡ce spoÄ
>>>> <U+008D>ÃvÃ¡ v tom, Å¾e operuje v poli mezi klientem a jeho
>>>> sociÃ¡lnÃm prostÅ<U+0099>edÃm; pracovnÃk je v obecnÃ©m smyslu
>>>> mediÃ¡torem mezi jednotlivcem a spoleÄ<U+008D>nostÃ. Jeho Ãºkolem je
>>>> napomÃ¡hat sociÃ¡lnÃmu fungovÃ¡nÃ klientÅ¯ a pomÃ¡hat
>>>> spoleÄ<U+008D>nosti, aby citlivÄ<U+009B> reagovala na potÅ<U+0099>eby
>>>> svÃ½ch Ä<U+008D>lenÅ¯. Tato dvojitÃ¡ mediaÄ<U+008D>nÃ role je role
>>>> angaÅ¾ovanÃ¡. Je zakotvenÃ¡ hodnotovÄ<U+009B> v nÃ¡boÅ¾enstvÃ nebo v
>>>> huma
>>>> [2018-11-24 22:22:58] /usr/sbin/apache2
>>>> [INFO:32231:ObalkyKnih.pm:228:1543094555322319] ObalkyKnih.cz for
>>>> books?isbn=9788026203667 response was
>>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"NÃ¡rodnÃ
>>>>
>>>> knihovna Ä<U+008C>eskÃ© Republiky
>>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>>>
>>>> obalkyknih.cz","html":"Encyklopedie sociÃ¡lnÃ prÃ¡ce (ESP)
>>>> pÅ<U+0099>inÃ¡Å¡Ã pÅ<U+0099>es 200 hesel ze vÅ¡ech oblastÃ
>>>> sociÃ¡lnÃ prÃ¡ce. ESP je postavena na interakÄ<U+008D>nÃm pojetÃ
>>>> sociÃ¡lnÃ prÃ¡ce. JedineÄ
>>>> <U+008D>nost sociÃ¡lnÃ prÃ¡ce spoÄ<U+008D>ÃvÃ¡ v tom, Å¾e operuje v
>>>> poli mezi klientem a jeho sociÃ¡lnÃm prostÅ<U+0099>edÃm; pracovnÃk
>>>> je v obecnÃ©m smyslu mediÃ¡torem mezi jednotlivcem a
>>>> spoleÄ<U+008D>nostÃ. Jeho Ãºkolem je napomÃ¡hat sociÃ¡lnÃmu
>>>> fungovÃ¡nÃ klientÅ¯ a pomÃ¡hat spoleÄ<U+008D>nosti, aby
>>>> citlivÄ<U+009B> reagovala na potÅ
>>>> <U+0099>eby svÃ½ch Ä<U+008D>lenÅ¯. Tato dvojitÃ¡ mediaÄ<U+008D>nÃ
>>>> role je role angaÅ¾ovanÃ¡. Je zakotvenÃ¡ hodnot
>>>>
>>>> Our Perl code used for testing purposes
>>>>
>>>> #!usr/bin/perl
>>>> use LWP::UserAgent;
>>>>
>>>> my $ua = LWP::UserAgent->new;
>>>> my $response = $ua->get(
>>>> 'http://cache.obalkyknih.cz/api/books?isbn=978-80-262-0366-7' );
>>>> my $content = $response->content;
>>>> print $content ;
>>>>
>>>> produces data with the correct encoding:
>>>>
>>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"Národní
>>>>
>>>> knihovna České Republiky
>>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>>>
>>>> obalkyknih.cz","html":"Encyklopedie sociální práce (ESP) přináší přes
>>>> 200 hesel ze všech oblastí sociální práce. ESP je postavena na
>>>> interakčním pojetí sociální práce. Jedinečnost sociální práce spočívá
>>>> v tom, že operuje v poli mezi klientem a jeho sociálním prostředím;
>>>> pracovník je v obecném smyslu mediátorem mezi jednotlivcem a
>>>> společností. Jeho úkolem je napomáhat sociálnímu fungování klientů a
>>>> pomáhat společnosti, aby citlivě reagovala na potřeby svých členů.
>>>> Tato dvojitá mediační role je role angažovaná. Je zakotvená hodnotově
>>>> v náboženství nebo v humanitních ideálech. V podobě tematicky
>>>> uspořádaných samostatných hesel poskytuje toto rozsáhlé dílo přehled
>>>> psychologických a sociologických teorií a přístupů s dopadem do
>>>> sociální práce, náboženský, filozofický a společenský kontext oboru.
>>>> Přináší přehled klíčových pojmů, technik a metod sociální práce,
>>>> ohrožených skupin a poskytovaných služeb. Samostatnou část tvoří hesla
>>>> charakterizující profesi sociálního pracovníka a hesla zabývající se
>>>> výzkumem v oblasti sociální práce. ESP reflektuje domácí vývoj oboru v
>>>> evropském kontextu a zohledňuje i širší mezinárodní zřetel. Hesla
>>>> popisují daný jev a jeho historii, hodnotová východiska, aplikační
>>>> možnosti a výzkum.","id":"2391500"},"csn_iso_690":"MATOUŠEK, Oldřich.
>>>> <i>Encyklopedie sociální práce. </i>Vyd. 1. Editor Alois KŘIŠŤAN.
>>>> Praha: Portál, 2013. 570
>>>> s.","nbn":"cnb002436000","rating_avg100":"100","orig_width":"346","rating_avg5":5,"toc_pdf_url":"https://cache.obalkyknih.cz/file/toc/362647/pdf","ean_other":[],"nbn_other":[],"rating_count":2,"cover_icon_url":"https://cache.obalkyknih.cz/file/cover/1830989/icon","dig_obj":{"BOA001":{"public":0,"url":"https://kramerius.mzk.cz/search/i.jsp?pid=uuid:6ec863b0-055e-11e6-a611-005056827e51","uuid":"uuid:6ec863b0-055e-11e6-a611-005056827e51"},"ABA001":{"public":0,"url":"http://kramerius4.nkp.cz/search/i.jsp?pid=uuid:6ec863b0-055e-11e6-a611-005056827e51","uuid":"uuid:6ec863b0-055e-11e6-a611-005056827e51"}},"bib_year":"2013","oclc":"(OCoLC)852382182","bib_title":"Encyklopedie
>>>>
>>>> sociální práce","succ_bib_count":"0","book_id":"112038753"}]
>>>>
>>>> It confirms that data from Obalkyknih.cz are in UTF-8.
>>>>
>>>> Basically, what happens when things go wrong is this (using letter á
>>>> as an example): the original character U+00E1
>>>> (http://www.fileformat.info/info/unicode/char/e1/index.htm) is encoded
>>>> as two letters Ã¡ (\u00c3\u00a1 as represented in memcached log). Ã is
>>>> U+00E3 (https://www.fileformat.info/info/unicode/char/00c3/index.htm)
>>>> while ¡ is U+00A1
>>>> (http://www.fileformat.info/info/unicode/char/00A1/index.htm).
>>>>
>>>> A description at
>>>> https://www.effectiveperlprogramming.com/2011/08/know-the-difference-between-character-strings-and-utf-8-strings/
>>>>
>>>> probably gives some ideas how encoding could get broken in Perl.
>>>>
>>>> However, we are not sure if the issue is in AddedContent.pm or in our
>>>> Apache configuration (because our test Perl code run from bash works
>>>> okay but AddedContent.pm called from Apache does not).
>>>>
>>>> Does anybody have any idea where to look next to fix it?
>>>>
>>>> Thank you in advance!
>>>>
>>>> Linda
>>>>
>>>>