[Evergreen-dev] Empty OSRF gateway responses from missourievergreen.org and poisoned app cache

[Ken Cox]

And this morning, it has started happening with CW MARS:

2021-01-11 09:30:53.909 11996-12051/org.cwmars D/GatewayJsonRequest:
[net] cached:0 url:https://catalog.cwmars.org/osrf-gateway-v1?service=open-ils.actor&method=open-ils.actor.org_types.retrieve&_ck=91&_sk=3-2-10
2021-01-11 09:30:53.910 11996-12051/org.cwmars D/GatewayJsonRequest:
[net] recv 27: {"payload":[],"status":200}
2021-01-11 09:30:53.914 11996-12052/org.cwmars D/GatewayJsonRequest:
[net] cached:0 url:https://catalog.cwmars.org/osrf-gateway-v1?service=open-ils.actor&method=open-ils.actor.org_tree.retrieve&_ck=91&_sk=3-2-10
2021-01-11 09:30:53.915 11996-12052/org.cwmars D/GatewayJsonRequest:
[net] recv 27: {"payload":[],"status":200}

Is there some connection between these servers that I'm missing?

Ken

On Sun, Jan 10, 2021 at 3:54 PM Ken Cox <kenstir at gmail.com> wrote:
>
> Hello devs, I have a pressing live issue and a request for advice.
>
>
> EMPTY OSRF GATEWAY RESPONSES CAUSING APP ERRORS
>
> For the last 4 days, some sizeable percentage of responses from
> https://missourievergreen.org/osrf-gateway-v1 are empty, meaning
>
>         {"payload":[],"status":200}
>
> What could cause this?  The OPAC appears to be working, it's just the
> OSRF gateway that is behaving this way.
>
> For example, this search request just returned an empty response:
>
>         [net] request
> https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.multiclass.query&param=%7B%22limit%22%3A200%2C%22offset%22%3A0%7D&param=%22keyword%3Aharry%20potter%20site(SRL-UN)%22&param=1&_ck=7&_sk=3-3-7
>         [net] recv 27: {"payload":[],"status":200}
>
> Sometimes the search response is empty, causing the app to error.
> Sometimes the search works but MODS responses are empty, causing the
> app to display search results with missing metadata, e.g.:
>
>         [net] cached:0
> url:https://missourievergreen.org/osrf-gateway-v1?service=open-ils.search&method=open-ils.search.biblio.record.mods_slim.retrieve&param=1395189&_ck=7&_sk=3-3-7
>         [net] recv 27: {"payload":[],"status":200}
>
> This response is not expected; what is expected is an "mvr" object, as
> you will see if you paste that URL into your browser.
>
>
> POISONED CACHE
>
> The app is currently caching successful responses that I figured to be
> stable, e.g. open-ils.search.biblio.record.mods_slim.retrieve.
> Because the empty responses appear successful, the app cache is now
> poisoned.  Do you have any advice on how better to handle this
> situation?
>
> Notice in the requests above that I include 2 cache-busting
> parameters, _ck=7 (client app version 7) and _sk=3-3-7 (server version
> 3-3-7).  That way, either a client update or a server update will
> invalidate all cached responses.
>
> Thanks,
> Ken Cox



-- 
-Ken