[Evergreen-dev] Empty OSRF gateway responses from missourievergreen.org and poisoned app cache

[Ken Cox]

Hello devs, I have a pressing live issue and a request for advice.


EMPTY OSRF GATEWAY RESPONSES CAUSING APP ERRORS

For the last 4 days, some sizeable percentage of responses from
https://missourievergreen.org/osrf-gateway-v1 are empty, meaning

        {"payload":[],"status":200}

What could cause this?  The OPAC appears to be working, it's just the
OSRF gateway that is behaving this way.

For example, this search request just returned an empty response:

        [net] request
https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.multiclass.query&param=%7B%22limit%22%3A200%2C%22offset%22%3A0%7D&param=%22keyword%3Aharry%20potter%20site(SRL-UN)%22&param=1&_ck=7&_sk=3-3-7
        [net] recv 27: {"payload":[],"status":200}

Sometimes the search response is empty, causing the app to error.
Sometimes the search works but MODS responses are empty, causing the
app to display search results with missing metadata, e.g.:

        [net] cached:0
url:https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.record.mods_slim.retrieve&param=1395189&_ck=7&_sk=3-3-7
        [net] recv 27: {"payload":[],"status":200}

This response is not expected; what is expected is an "mvr" object, as
you will see if you paste that URL into your browser.


POISONED CACHE

The app is currently caching successful responses that I figured to be
stable, e.g. open-ils.search.biblio.record.mods_slim.retrieve.
Because the empty responses appear successful, the app cache is now
poisoned.  Do you have any advice on how better to handle this
situation?

Notice in the requests above that I include 2 cache-busting
parameters, _ck=7 (client app version 7) and _sk=3-3-7 (server version
3-3-7).  That way, either a client update or a server update will
invalidate all cached responses.

Thanks,
Ken Cox