SPAM: Re: [OPEN-ILS-DEV] Full OpenSRF over HTTP
Bill Erickson
erickson at esilibrary.com
Thu Oct 18 09:04:15 EDT 2007
Mike Rylander wrote:
> Bill and I have been puzzling over how to give HTTP clients support
> for the more advanced features of OpenSRF, such as stateful,
> transactional sessions and streaming responses. The main problem is
> that HTTP is entirely stateless on its own, and OpenSRF is not. The
> second biggest problem is that most current HTTP clients do not lend
> themselves to the kind of connection control we'd like to have.
>
> Well, with the addition of (what really should be) a fairly trivial
> translator service written as an Apache module, and a new server-side
> setting that allows OpenSRF Applications to loosen their restrictions
> on who can talk to them and when, we think we have a way to provide
> advanced, robust, degradable OpenSRF services to all modern HTTP
> clients.
>
> Let us know what you think.
>
> And so, herein follows ...
>
>
> A modest proposal for streaming, stateful OpenSRF-over-HTTP
> -------------------------------------------------------------------------------
>
>
> Client Request
> ==============
>
> The method is POST, URL is that of the HTTP/XMPP translator. The body of the
> POST data should be a JSON encoded array of one or more osrfMessage objects.
> See the bottom of this document for both XMPP and HTTP examples of OpenSRF
> communication.
>
> OpenSRF HTTP headers from client to server:
>
> X-OpenSRF-service=<service-name> # for example, open-ils.search
> X-OpenSRF-to=<XMPP-address> # for example, open-ils.search
> X-OpenSRF-xid=<timestamp-ms> # for tracing messages to the client
> X-OpenSRF-thread=<guid-string> # uniquely identifies this conversation
>
> The X-OpenSRF-service and X-OpenSRF-to headers are mutually exclusive, and
> cannot be used together. There are exactly two instances where the
> X-OpenSRF-service header should be used:
>
> - At the beginning of a stateful session, when sending the CONNECT message
> - For stateless requests
>
> In all other instances, which is to say within any stateful session that is
> past the CONNECT phase, the client should use the X-OpenSRF-to header. The
> value of this header is set to the XMPP address returned in the by STATUS_OK
> message from the current session CONNECT request.
>
>
> If multipart/x-mixed-replace mode is supported by the client, then it should
> send the following header with every HTTP request:
>
> X-OpenSRF-multipart=true
>
> Otherwise, the translator must assume multipart is not supported by the client.
>
> Setting aside the underlying transport and supposing the complete equivalence
> of the HTTP headers to XMPP XML attributes as described above, all other data
> within the OpenSRF message remains entirely the same. In other words, The body
> of the HTTP request is exactly equivalent to the <body/> element with an XMPP
> <message/> element.
>
>
> Treament of these connection headers within the HTTP/XMPP translator
> --------------------------------------------------------------------
>
> * X-OpenSRF-service -- Should be mapped to the XMPP endpoint within the
> configured router, as described in the core config file for OpenSRF clients.
> Typically this will look something like "router at localhost/open-ils.search",
> where the service name provided in the header is used as the last component,
> the XMPP resource. This is used in the @to attr of the <message/> element on
> the XMPP network. This header is only used for the first message in a stateful
> connection or for stateless requests.
>
> * X-OpenSRF-to -- Within the client, he value for this header is taken by from
> value provided in the first X-OpenSRF-from header from the STATUS_OK message
> provided by the server in response to the initial CONNECT request. This is
> used in the @to attr of the <message/> element on the XMPP network.
>
> * X-OpenSRF-xid -- Typically, the time since the Unix epoc in milliseconds.
> Used to trace the path of messages through the OpenSRF network. If not
> supplied by the client, this should be created and supplied by the translator.
> This datum is used in the @osrf_xid attr of the <message/> ele ment on the XMPP
> network.
>
> * X-OpenSRF-thread -- A unique identifier for this request or session. Can be
> any string, though typically, a combination of the epoc time and originating
> process id are used. This is used by OpenSRF endpoints for identifying and
> continuing stateful sessions. If not supplied by the client, the translator
> should create and supply a value for this element. This datum is used as the
> text content of the <thread/> element on the XMPP network.
>
>
> Response from the server
> ========================
>
> Assuming no error was encountered, the OpenSRF service will respond with among
> other things several bits of data that are critical to the ongoing
> communication between the client and server. The most important of these is
> the XMPP @from attr of the <message/> element. This must be used by the client
> for all future requests within a stateful session, assuming a stateful session
> was requested and created. This datum should be mapped by the translator to an
> HTTP header called
>
> X-OpenSRF-from
>
> in the response to the client. The client will then use this value in the
>
> X-OpenSRF-to
>
> header of all future in-session communication.
>
> The body of the response, or responses if a multipart request is used and a
> streaming method called, will each be an array of one or more osrfMessage
> objects.
>
> If the client does not request multipart/x-mixed-replace mode using the
> X-OpenSRF-multipart header, as described above, then the translator will
> collect all response osrfMessage objects, up to and including the first
> COMPLETE message, and packaged them into a single array object as a single
> request response to the client.
>
>
> Changes to OpenSRF
> ==================
>
> Stateful communication with backend OpenSRF service instances will be made
> possible by giving each service the ability to accept "migratable" client
> remote IDs. If the configuration section for a given OpenSRF application
> has the setting
>
> <migratable-clients>true</migratable-clients>
>
Monty Python aside, how would you feel about "migratory-clients" (or
mobile-clients)? I'm pretty sure "migratable" isn't a word.
> then that service will ignore the requirement that the remote ID of a stateful
> client must match across all communication within a session, and instead rely
> only on the thread of the session for session tracking. It is normally, and
> currently always, an error condition for multiple client XMPP identifiers to
> be used with one thread and one session. This is to prevent session hijacking
> among other security concerns. However, loosening this restriction is required
> in order to support OpenSRF-over-HTTP, and concerns are largly mitigated by
> requiring the service to allow the feature explicitely in its configuration.
>
> There is no restriction on the type of osrfMessage objects that can be sent,
> and as long as their order (and associated succesful responses) would
> constitute a normal stateful OpenSRF session on an XMPP network, the use of
> the <migratable-clients/> setting will allow this same functionality over
> HTTP.
>
>
> Overview of the HTTP/XMPP translator
> ====================================
>
> The translator will be written as an Apache module. Its entire purpose is to
> smooth out the differences between the HTTP and XMPP transports with respect
> to OpenSRF communication. It will do as little interpretation and manipulation
> of data as possible, modifying only that data which must absolutely be changed
> in order to support certain sub-optimal client software, and to map concepts
> between HTTP and XMPP.
>
>
> Upstream HTTP-side handling
> ---------------------------
>
> For upstream (client to server) communication, no data modification need occur
> other than the mapping request headers to XMPP attributes, and the adding of
> XMPP-specific router information to the service initially requested by the
> client. No modification or parsing of the message body will be performed on
> upstream communication.
>
> Downstream HTTP-side handling
> -----------------------------
>
> For downstream (server to client) communication, there are to possible logic
> paths. The first, being both simpler to implement and more generally useful
> on the client end, is the case where the client supports
> multipart/x-mixed-replace mode, as is the case with the Mozilla browser from
> version 1.7 on and all version of the Firefox browser. We will call this
> "multipart mode". The second case contains all other browsers and clients, as
> no other browsers are known to support this mode at this time. We will call
> this "collected mode".
>
> - For multipart mode capable clients, no modification of any data will occur
> other than the mapping of the XMPP attributes to their respective HTTP
> counterparts. All data contained in the body of the OpenSRF message will be
> passed unaltered as a multipart chunk followed by a boundary marker. The
> translator will use this response mode when it receives a request that is
> accompanied by a "X-OpenSRF-multipart=true" header.
>
> - For collected mode clients, all osrfMessage objects sent by the server will
> be collected into a single JSON array, in the order they arrive, and delivered
> as a single HTTP response to the client. This will require parsing of the JSON
> in order to pull all osrfMessage objects out of each response packet and build
> the response array.
>
>
> XMPP-side communication
> -----------------------
>
> The translator will use standard OpenSRF libraries to communicate with the XMPP
> network-based services, just as the existing gateway and srfsh applications do.
>
>
> Example Data
> ============
>
> XMPP message example
> --------------------
>
> <message
> to='osrf at localhost/open-ils.search_listener_at_grendel.local_31319'
> from='gateway at localhost/1192540419__1192540420.166380_31367'
> router_command=''
> router_class=''
> osrf_xid='1192540419313673'>
> <thread>1192540427.567678.119254042731367</thread>
> <body>
> [
> {
> "__c":"osrfMessage",
> "__p":{
> "threadTrace":"1",
> "locale":"en-us",
> "type":"REQUEST",
> "payload":{
> "__c":"osrfMethod",
> "__p":{
> "method":"open-ils.search.biblio.record.mods_slim.retrieve",
> "params":[1487101]
> }
> }
> }
> }
> ]
> </body>
> </message>
>
>
> HTTP message example
> --------------------
>
> X-OpenSRF-to=open-ils.search
> X-OpenSRF-xid=1192540419313673
> X-OpenSRF-thread=1192540427.567678.119254042731367
> X-OpenSRF-multipart=true
>
> [
> {
> "__c":"osrfMessage",
> "__p":{
> "threadTrace":"1",
> "locale":"en-us",
> "type":"REQUEST",
> "payload":{
> "__c":"osrfMethod",
> "__p":{
> "method":"open-ils.search.biblio.record.mods_slim.retrieve",
> "params":[1487101]
> }
> }
> }
> }
> ]
>
>
> ---------------------------------------------------------------------------------------------------
>
> Thought, comments, suggestions, considerations?
>
>
Excellent writeup and breakdown of the various pieces...
I have an incomplete, yet, thus far, functionally equivalent prototype
running on dev.gapines.org. The example Javascript is at
http://dev.gapines.org/~erickson/multi/multi.html. (Warning, produces
several popups of raw JSON output and only works in Mozilla/FF). The
headers are slightly different (X-OSRF instead of X-OpenSRF) and I'm not
yet parsing the messages to see when a given request is complete, so
"collected mode" doesn't function yet, but it's close.
I think this opens up a lot of possibilities... One example that comes
to mind is having the ability to respond with OpenSRF "continue" status
messages. This will be great for long-running requests, in particular
the ILS collections API. This should help liven up some of the staff
client displays, as well. Current practice is to send along sets of IDs
from the various discovery API's, then go back and fetch the objects 1
by 1. With multipart, we can just return a stream of objects and the
interface can populate rows as the stream arrives, without having to
"lock up" on a singe batch of objects, all received at once. I'm
probably preaching to the choir, here, but I think this is pretty cool
stuff ;)
-bill
--
Bill Erickson
| VP, Software Development & Integration
| Equinox Software, Inc. / The Evergreen Experts
| phone: 877-OPEN-ILS (673-6457)
| email: erickson at esilibrary.com
| web: http://esilibrary.com
More information about the Open-ils-dev
mailing list