[OPEN-ILS-DEV] AddedContent.pm produces garbled encoding for application/json

Mon Jan 14 09:19:33 EST 2019

Hi Jason,

There is not a Launchpad bug for this yet because at this point it has 
only been fixed locally (in ObalkyKnih.pm module in our Evergreen 
installation). Currently we are in the process of adding some more 
features to ObalkyKnih.pm module and testing these features. Of course, 
we will create a wishlist bug and submit the enriched code (which would 
also contain the encoding fix) to GIT. Should there be a wider interest 
in the corrected code at this point, we would submit a patch for the 
encoding only (and add the new features at a later date).

Maybe that this (or similar) solution would also work for encoding 
issues in added content coming from Open Library - for which a bug has 
been opened already: https://bugs.launchpad.net/evergreen/+bug/1610678. 
(Josh did some testing with Open Library data encoding.)

Linda

On 1/14/19 2:45 PM, Jason Stephenson wrote:
> Hi, Linda.
>
> Good find! That sounds like a bug to me. Could you submit a Launchpad
> bug (if there isn't one already) and a patch commit, please?
>
> Depending on which module this occurs in, the fix may not be as simple
> as using utf8::decode. It could be that we need code to determine the
> character set from the HTTP response.
>
> Cheers,
> Jason
>
> On 1/14/19 2:38 AM, Linda Jansova wrote:
>> Hi,
>>
>> Just letting you know that our developer has eventually changed the
>> encoding of the data blob coming from Obalkyknih.cz from UTF-8 to Perl's
>> internal representation:
>>
>> utf8::decode($response_content);
>>
>> Adding this step before further data processing takes place has
>> successfully solved the problem of seeing gibberish characters in our
>> catalog :-).
>>
>> Linda
>>
>> On 11/25/18 4:56 PM, Linda Jansova wrote:
>>> Hi,
>>>
>>> We have encountered a problem in AddedContent.pm's get_url function
>>> (both in Evergreen 3.1.4 and 2.12.6) - letters with diacritics from
>>> Czech added content provider Obalkyknih.cz have started being
>>> corrupted in our TPACs. Our added content provider has reported a
>>> switch to application/json MIME type.
>>>
>>> After the switch we seem to be getting strange chars in summary (and
>>> table of contents if available as text, not as an image). We have
>>> tried to locate the problem in a separate Perl program and have come
>>> to a conclusion that data only get corrupted when fetched by the
>>> get_url function from AddedContent.pm.
>>>
>>> We have added additional logging to the following part of
>>> AddedContent.pm:
>>>
>>> # returns an HTPP::Response object
>>> sub get_url {
>>>      my( $self, $url ) = @_;
>>>
>>>      $logger->info("added content getting [timeout=$net_timeout,
>>> errors_remaining=$error_countdown] URL = $url");
>>>      my $agent = LWP::UserAgent->new(timeout => $net_timeout);
>>>
>>>      my $res = $agent->get($url);
>>>      $logger->info("added content request returned with code " .
>>> $res->code);
>>>
>>>      #VJ
>>>      $logger->info("added contet res is: " . $res->content);
>>>
>>>      die "added content request failed: " . $res->status_line ."\n"
>>> unless $res->is_success;
>>>
>>>      return $res;
>>> }
>>>
>>> And a corresponding sample log looks like this:
>>>
>>> [2018-11-24 22:22:58] /usr/sbin/apache2
>>> [INFO:32231:AddedContent.pm:296:1543094555322319] added contet res is:
>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"NÃ¡rodnÃ
>>> knihovna Ä<U+008C>eskÃ© Republiky
>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>> obalkyknih.cz","html":"Encyklopedie sociÃ¡lnÃ prÃ¡ce (ESP)
>>> pÅ<U+0099>inÃ¡Å¡Ã pÅ<U+0099>es 200 hesel ze vÅ¡ech oblastÃ
>>> sociÃ¡lnÃ prÃ¡ce. ESP je postavena na interakÄ<U+008D>nÃm pojetÃ
>>> sociÃ¡lnÃ prÃ¡ce. JedineÄ<U+008D>nost sociÃ¡lnÃ prÃ¡ce spoÄ
>>> <U+008D>ÃvÃ¡ v tom, Å¾e operuje v poli mezi klientem a jeho
>>> sociÃ¡lnÃm prostÅ<U+0099>edÃm; pracovnÃk je v obecnÃ©m smyslu
>>> mediÃ¡torem mezi jednotlivcem a spoleÄ<U+008D>nostÃ. Jeho Ãºkolem je
>>> napomÃ¡hat sociÃ¡lnÃmu fungovÃ¡nÃ klientÅ¯ a pomÃ¡hat
>>> spoleÄ<U+008D>nosti, aby citlivÄ<U+009B> reagovala na potÅ<U+0099>eby
>>> svÃ½ch Ä<U+008D>lenÅ¯. Tato dvojitÃ¡ mediaÄ<U+008D>nÃ role je role
>>> angaÅ¾ovanÃ¡. Je zakotvenÃ¡ hodnotovÄ<U+009B> v nÃ¡boÅ¾enstvÃ nebo v
>>> huma
>>> [2018-11-24 22:22:58] /usr/sbin/apache2
>>> [INFO:32231:ObalkyKnih.pm:228:1543094555322319] ObalkyKnih.cz for
>>> books?isbn=9788026203667 response was
>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"NÃ¡rodnÃ
>>> knihovna Ä<U+008C>eskÃ© Republiky
>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>> obalkyknih.cz","html":"Encyklopedie sociÃ¡lnÃ prÃ¡ce (ESP)
>>> pÅ<U+0099>inÃ¡Å¡Ã pÅ<U+0099>es 200 hesel ze vÅ¡ech oblastÃ
>>> sociÃ¡lnÃ prÃ¡ce. ESP je postavena na interakÄ<U+008D>nÃm pojetÃ
>>> sociÃ¡lnÃ prÃ¡ce. JedineÄ
>>> <U+008D>nost sociÃ¡lnÃ prÃ¡ce spoÄ<U+008D>ÃvÃ¡ v tom, Å¾e operuje v
>>> poli mezi klientem a jeho sociÃ¡lnÃm prostÅ<U+0099>edÃm; pracovnÃk
>>> je v obecnÃ©m smyslu mediÃ¡torem mezi jednotlivcem a
>>> spoleÄ<U+008D>nostÃ. Jeho Ãºkolem je napomÃ¡hat sociÃ¡lnÃmu
>>> fungovÃ¡nÃ klientÅ¯ a pomÃ¡hat spoleÄ<U+008D>nosti, aby
>>> citlivÄ<U+009B> reagovala na potÅ
>>> <U+0099>eby svÃ½ch Ä<U+008D>lenÅ¯. Tato dvojitÃ¡ mediaÄ<U+008D>nÃ
>>> role je role angaÅ¾ovanÃ¡. Je zakotvenÃ¡ hodnot
>>>
>>> Our Perl code used for testing purposes
>>>
>>> #!usr/bin/perl
>>> use LWP::UserAgent;
>>>
>>> my $ua = LWP::UserAgent->new;
>>> my $response = $ua->get(
>>> 'http://cache.obalkyknih.cz/api/books?isbn=978-80-262-0366-7' );
>>> my $content = $response->content;
>>> print $content ;
>>>
>>> produces data with the correct encoding:
>>>
>>> [{"_id":"5bf904c905509b06182848ac","succ_toc_count":"0","cover_preview510_url":"https://cache.obalkyknih.cz/file/cover/1830989/preview510","ean":"9788026203667","uuid":["uuid:6ec863b0-055e-11e6-a611-005056827e51"],"cooperating_with":"https://www.cbdb.cz|CBDB.cz","succ_cover_count":"0","flag_bare_record":0,"csn_iso_690_source":"Národní
>>> knihovna České Republiky
>>> 18.11.2018","rating_url":"https://www.obalkyknih.cz/stars?value=100","rating_sum":200,"cover_thumbnail_url":"https://cache.obalkyknih.cz/file/cover/1830989/thumbnail","oclc_other":[],"reviews":[],"part_root":1,"orig_height":"510","backlink_url":"https://www.obalkyknih.cz/view?isbn=9788026203667","toc_thumbnail_url":"https://cache.obalkyknih.cz/file/toc/362647/thumbnail","cover_medium_url":"https://cache.obalkyknih.cz/file/cover/1830989/medium","annotation":{"source":"Web
>>> obalkyknih.cz","html":"Encyklopedie sociální práce (ESP) přináší přes
>>> 200 hesel ze všech oblastí sociální práce. ESP je postavena na
>>> interakčním pojetí sociální práce. Jedinečnost sociální práce spočívá
>>> v tom, že operuje v poli mezi klientem a jeho sociálním prostředím;
>>> pracovník je v obecném smyslu mediátorem mezi jednotlivcem a
>>> společností. Jeho úkolem je napomáhat sociálnímu fungování klientů a
>>> pomáhat společnosti, aby citlivě reagovala na potřeby svých členů.
>>> Tato dvojitá mediační role je role angažovaná. Je zakotvená hodnotově
>>> v náboženství nebo v humanitních ideálech. V podobě tematicky
>>> uspořádaných samostatných hesel poskytuje toto rozsáhlé dílo přehled
>>> psychologických a sociologických teorií a přístupů s dopadem do
>>> sociální práce, náboženský, filozofický a společenský kontext oboru.
>>> Přináší přehled klíčových pojmů, technik a metod sociální práce,
>>> ohrožených skupin a poskytovaných služeb. Samostatnou část tvoří hesla
>>> charakterizující profesi sociálního pracovníka a hesla zabývající se
>>> výzkumem v oblasti sociální práce. ESP reflektuje domácí vývoj oboru v
>>> evropském kontextu a zohledňuje i širší mezinárodní zřetel. Hesla
>>> popisují daný jev a jeho historii, hodnotová východiska, aplikační
>>> možnosti a výzkum.","id":"2391500"},"csn_iso_690":"MATOUŠEK, Oldřich.
>>> <i>Encyklopedie sociální práce. </i>Vyd. 1. Editor Alois KŘIŠŤAN.
>>> Praha: Portál, 2013. 570
>>> s.","nbn":"cnb002436000","rating_avg100":"100","orig_width":"346","rating_avg5":5,"toc_pdf_url":"https://cache.obalkyknih.cz/file/toc/362647/pdf","ean_other":[],"nbn_other":[],"rating_count":2,"cover_icon_url":"https://cache.obalkyknih.cz/file/cover/1830989/icon","dig_obj":{"BOA001":{"public":0,"url":"https://kramerius.mzk.cz/search/i.jsp?pid=uuid:6ec863b0-055e-11e6-a611-005056827e51","uuid":"uuid:6ec863b0-055e-11e6-a611-005056827e51"},"ABA001":{"public":0,"url":"http://kramerius4.nkp.cz/search/i.jsp?pid=uuid:6ec863b0-055e-11e6-a611-005056827e51","uuid":"uuid:6ec863b0-055e-11e6-a611-005056827e51"}},"bib_year":"2013","oclc":"(OCoLC)852382182","bib_title":"Encyklopedie
>>> sociální práce","succ_bib_count":"0","book_id":"112038753"}]
>>>
>>> It confirms that data from Obalkyknih.cz are in UTF-8.
>>>
>>> Basically, what happens when things go wrong is this (using letter á
>>> as an example): the original character U+00E1
>>> (http://www.fileformat.info/info/unicode/char/e1/index.htm) is encoded
>>> as two letters Ã¡ (\u00c3\u00a1 as represented in memcached log). Ã is
>>> U+00E3 (https://www.fileformat.info/info/unicode/char/00c3/index.htm)
>>> while ¡ is U+00A1
>>> (http://www.fileformat.info/info/unicode/char/00A1/index.htm).
>>>
>>> A description at
>>> https://www.effectiveperlprogramming.com/2011/08/know-the-difference-between-character-strings-and-utf-8-strings/
>>> probably gives some ideas how encoding could get broken in Perl.
>>>
>>> However, we are not sure if the issue is in AddedContent.pm or in our
>>> Apache configuration (because our test Perl code run from bash works
>>> okay but AddedContent.pm called from Apache does not).
>>>
>>> Does anybody have any idea where to look next to fix it?
>>>
>>> Thank you in advance!
>>>
>>> Linda
>>>
>>>