[Evergreen-dev] OpenSRF / Redis Conf. Talk Continued

[Bill Erickson]

Hi All,

I presented a session at the Evergreen conference related to replacing
XMPP/Ejabberd with Redis.

https://github.com/berick/Presentations/blob/master/Evergreen-2022/osrf-redis.md

Following the session, Mike, Galen, and I had a good discussion in the dev
"room" about how best to leverage Redis features.  Mike suggested trying
Redis Streams and Consumer Groups in lieu of Redis Lists, which were used
in my proof of concept, because Streams have some benefits over lists.  In
short, you can collect a lot more information about the state of a stream
and those subscribed to the stream.

Based on that discussion, I've posted working branches that replace Lists
with Streams.

https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/lpxxx-redisrf-streams-v2

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lpxxx-redisrf-streams-v2

There's still a lot to do, but I wanted to share.  Also, I have follow up
thoughts/questions.

Streams support message acknowledgements (XACK message) and message
redelivery.  I'm curious if/when/how/where this might benefit us in
practice.  I'm having some trouble seeing the value.

A client has to actively request a Stream message, they're not just dropped
onto the network.  Presumably, if the client is requesting it, it has the
capacity to process it.  E.g. Listeners won't pull more messages if they
have reached max children.

The scenarios where an ACK would not get delivered are a service/worker
crash or non-graceful shutdown.  If a service crashes, there's a decent
chance it crashed because of the message, meaning we probably don't want to
reprocess it anyway.

ACK's and message reprocessing raise other questions about how long after a
message is sent does it make sense to reprocess it after a failure.
Similarly, I'm guessing we'd never attempt to reprocess messages sent
directly between clients and workers, since the messages would be
meaningless out of context.

Other thoughts?  Am I missing anything?

-b
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20220622/feeb8822/attachment.htm>