[Evergreen-dev] Thoughts on experimental OpenSRF config format change

[Bill Erickson]

Since we have potential plans for the contents of opensrf.xml, I'll set
that aside for now.  That simplifies the discussion quite a bit.

Just looking at opensrf_core.xml, then a YAML version that supports current
OpenSRF and a potential Redis-based future might look something like:

https://github.com/berick/OpenSRF/blob/user/berick/lpxxx-redisrf-streams-v3/examples/opensrf_core.yml.example

* Codify public vs. private subdomains.  This is explicit in the install
docs.  Making it explicit in the configs offers clarity and simplifies some
things.
* Link specific hosts to domains.
  ** We need to know both the specific hostname (for running services) and
its domain or some way to derive one from the other.
  ** One alternative to listing every host would be to pass both a domain
name and a hostname (or --localhost) to osrf_control during startup.
  ** Another alternative would be a way to specify a default domain to use
for clients connecting "here".  This is closer to how opensrf_core.xml
works now, but requires different configs per brick.
  ** A combo of the above could also work.
* Instead of specifying which services run on which subdomains (as in my
previous example), specify which services are permitted to register with
the router on each subdomain.  Which services actually run on a given host
are still controlled by the settings server.

I'm still experimenting, and thinking out loud, so this may evolve.
Thoughts appreciated.

-b


On Sat, Nov 5, 2022 at 1:09 PM Bill Erickson <berickxx at gmail.com> wrote:

>
>
> On Fri, Nov 4, 2022 at 5:46 PM Mike Rylander <mrylander at gmail.com> wrote:
>
>> On Fri, Nov 4, 2022 at 1:47 PM Bill Erickson <berickxx at gmail.com> wrote:
>> >
>> > On Fri, Nov 4, 2022 at 1:23 PM Mike Rylander <mrylander at gmail.com>
>> wrote:
>> >>
>> >> Hi Bill,
>> >>
>> >> First, thanks for moving this forward!
>> >>
>> >> (unfortunately?) Right off the bat, I'm /really/ not liking the idea
>> >> of merging application settings (opensrf.xml) into initial connection
>> >> settings (opensrf_core.xml).  If anything, I'd prefer to see the
>> >> application settings move farther away from the file system, somehow,
>> >> rather than have to spray that info all over the place -- right now
>> >> application config only /has/ to be colocated with the
>> >> opensrf.settings app, but it looks like this would require it
>> >> everywhere and in potentially only-very-slightly-different files
>> >> (differences of service assigned, or service settings, per host) that
>> >> would be difficult to troubleshoot.
>> >
>> >
>> > Not exactly.  I didn't dig into this too much in my previous message,
>> but the idea was to support a single file structure, without necessarily
>> requiring a single file.
>> >
>>
>> Hrm... maybe I'm misunderstanding, sorry if that's the case.  Let me
>> try to explain my thought process, concerns, and desires.
>>
>> After my first reading of your original message, I thought you were
>> basically only talking about yaml-izing opensrf_core.xml.  I'm fully
>> on board with making that smarter (reusable chunks, etc) and like you
>> I don't care much what the format is in the long run.  And, FTR, I'm
>> fully on board with making opensrf.xml smarter, or more dynamic, or
>> non-existent (as a filesystem object).
>>
>
> That was the original goal.  It quickly started evolving...
>
>
>>
>> But then I saw Blake's response that mentioned service defaults, and
>> looked at your linked example more closely, and that's when I came
>> away with the impression that you may /also/ be wanting to embed the
>> application config (what opensrf.settings reads and then distributes
>> to applications) in the connection config file /for the services to
>> look at directly/.  Is that correct?
>>
>
> Embed, yes. Look at directly, no.  More below.
>
>
>>
>> My fervent hope is that we retain the separation of concerns that the
>> two files embody today.  Right now, we just tell things how to connect
>> to the opensrf network via opensrf_core.xml, and then they ask the
>> opensrf.settings service for their config data.  I promise that's not
>> just an idle desire!  Because there are two "layers" (for lack of a
>> better term) of configuration, and human error is unfortunately still
>> a thing, it increases the risk that things will become broken in a
>> complicated environment when All The Things live in just one place,
>> instead of dedicated (and tool-enrobed) locations for different
>> configuration concerns and targets.
>>
>> To expand a little on my mention of opensrf.xml above, eventually I
>> would like to see the opensrf.settings service stop getting it's
>> pile-o-data from opensrf.xml (or any file, for that matter) and
>> instead read it from a structured data store.  Maybe a full-on
>> database, maybe a simple persistent document store, but something more
>> machine-actionable than a file that a human edits and SCP's around.
>>
>
> This.  As I'm reviewing the stock opensrf.xml file, I keep asking myself
> why so much of this stuff isn't stored in global flags, org settings, etc.
> (Of course, it's just easier to change a config file than create a database
> upgrade script with INSERTs, etc.).  I'm fully on board with moving as much
> of opensrf.xml as we can into a structured data store.
>
> In a world...  where OpenSRF relies on Redis, for example, persistent
> storage is an option with a lot of controls about what/when gets stored.
> This would be one way to do it w/o requiring, say, Postgres to run OpenSRF.
>
> (At the risk of fully derailing the conversation, I'd actually be fine
> with OSRF requiring PG or, for that matter, merging into Evergreen.  That's
> a substantial can of worms, though).
>
>
>>
>> Put another way, I don't want services to read a local file for
>> anything other than initial bootstrapping. If we do without even a
>> bootstrap config file and just have command line parameters, all the
>> better.  I don't see a way to do that right now, though.
>>
>> BUT WAIT!
>>
>> That all looks pretty negative, so let me offer a counter to myself.
>>
>> If what you meant (and what I misunderstood) was that opensrf.xml
>> could/would eventually turn into a yaml file which would have mutually
>> exclusive structural sections from opensrf_core.yml, and
>> opensrf.settings could be /actively/ told to read a yaml file that
>> just happens to be the same file on disk for both bootstrap and
>> settings data purposes  (or, I guess, be pointed to a section?) --
>> meaning, opensrf.settings would just look at different parts of the
>> file for each purpose and other services would just ignore all of the
>> non-bootstrap parts, and not try to use them directly without going to
>> the settings service -- then I retract my concerns.
>>
>
> Yes, that!  I didn't mean to suggest we deprecate opensrf.settings or
> point services directly at the service configuration part of the file.  I
> just like the idea of having one config file structure that can cover every
> purpose.
>
> Thanks!
>
> -b
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20221107/b68a9bf6/attachment.htm>