[Evergreen-dev] Empty OSRF gateway responses from missourievergreen.org and poisoned app cache

[Ken Cox]

[copying the list this time]

Jason,

Thanks for the info.  I hear that the server admins "did something"
and the problem is gone.  I don't have enough visibility to file a
useful bug.

Can you (or anyone?) tell me what should be done on the server to
fix the issue?  Did you see it on CW MARS at the time I reported
(2021-01-11 09:30:53.909 from the log above)?

That's a good idea to not cache empty results.  I will look into it.
Even if they are valid results, the cost of not caching them is much
smaller than the cost of caching them.

The library I use on Android is Volley, and I configure it to cache
for a maximum of one week (even though the cache headers from
Evergreen have a TTL of one year), up to a max of 10 MiB.

Ken

On Mon, Jan 11, 2021 at 10:26 AM Jason Boyer
<jboyer at equinoxinitiative.org> wrote:
>
> Hi Ken, missourievergreen.org is returning more consistent results this morning but I’ve seen indications that Jabber errors at certain points along the communication chain can cause this kind of result. That’s definitely an Evergreen bug.
>
> As for thoughts on caching, one potential option is to discard completely empty results like these (except for special cases where they make sense, perhaps.) But even for open-ils.search.biblio.record.mods_slim.retrieve the value returned could change if staff edit the record so entries shouldn’t live “forever." Are you using a cache that discards entries after a period of time, or at a certain storage size, or some other method? (I’m not really familiar with the options on Android, I assume it has some APIs that help manage cached values.)
>
> Jason
>
> --
> Jason Boyer
> Senior System Administrator
> Equinox Open Library Initiative
> phone:  +1 (877) Open-ILS (673-6457)
> email:  JBoyer at EquinoxInitiative.org
> web:  https://EquinoxInitiative.org/
>
> On Jan 10, 2021, at 3:54 PM, Ken Cox <kenstir at gmail.com> wrote:
>
> Hello devs, I have a pressing live issue and a request for advice.
>
>
> EMPTY OSRF GATEWAY RESPONSES CAUSING APP ERRORS
>
> For the last 4 days, some sizeable percentage of responses from
> https://missourievergreen.org/osrf-gateway-v1 are empty, meaning
>
>        {"payload":[],"status":200}
>
> What could cause this?  The OPAC appears to be working, it's just the
> OSRF gateway that is behaving this way.
>
> For example, this search request just returned an empty response:
>
>        [net] request
> https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.multiclass.query&param=%7B%22limit%22%3A200%2C%22offset%22%3A0%7D&param=%22keyword%3Aharry%20potter%20site(SRL-UN)%22&param=1&_ck=7&_sk=3-3-7
>        [net] recv 27: {"payload":[],"status":200}
>
> Sometimes the search response is empty, causing the app to error.
> Sometimes the search works but MODS responses are empty, causing the
> app to display search results with missing metadata, e.g.:
>
>        [net] cached:0
> url:https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.record.mods_slim.retrieve&param=1395189&_ck=7&_sk=3-3-7
>        [net] recv 27: {"payload":[],"status":200}
>
> This response is not expected; what is expected is an "mvr" object, as
> you will see if you paste that URL into your browser.
>
>
> POISONED CACHE
>
> The app is currently caching successful responses that I figured to be
> stable, e.g. open-ils.search.biblio.record.mods_slim.retrieve.
> Because the empty responses appear successful, the app cache is now
> poisoned.  Do you have any advice on how better to handle this
> situation?
>
> Notice in the requests above that I include 2 cache-busting
> parameters, _ck=7 (client app version 7) and _sk=3-3-7 (server version
> 3-3-7).  That way, either a client update or a server update will
> invalidate all cached responses.
>
> Thanks,
> Ken Cox
> _______________________________________________
> Evergreen-dev mailing list
> Evergreen-dev at list.evergreen-ils.org
> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev
>
>


-- 
-Ken