[Evergreen-dev] Thoughts on experimental OpenSRF config format change

[Bill Erickson]

On Fri, Nov 4, 2022 at 5:46 PM Mike Rylander <mrylander at gmail.com> wrote:

> On Fri, Nov 4, 2022 at 1:47 PM Bill Erickson <berickxx at gmail.com> wrote:
> >
> > On Fri, Nov 4, 2022 at 1:23 PM Mike Rylander <mrylander at gmail.com>
> wrote:
> >>
> >> Hi Bill,
> >>
> >> First, thanks for moving this forward!
> >>
> >> (unfortunately?) Right off the bat, I'm /really/ not liking the idea
> >> of merging application settings (opensrf.xml) into initial connection
> >> settings (opensrf_core.xml).  If anything, I'd prefer to see the
> >> application settings move farther away from the file system, somehow,
> >> rather than have to spray that info all over the place -- right now
> >> application config only /has/ to be colocated with the
> >> opensrf.settings app, but it looks like this would require it
> >> everywhere and in potentially only-very-slightly-different files
> >> (differences of service assigned, or service settings, per host) that
> >> would be difficult to troubleshoot.
> >
> >
> > Not exactly.  I didn't dig into this too much in my previous message,
> but the idea was to support a single file structure, without necessarily
> requiring a single file.
> >
>
> Hrm... maybe I'm misunderstanding, sorry if that's the case.  Let me
> try to explain my thought process, concerns, and desires.
>
> After my first reading of your original message, I thought you were
> basically only talking about yaml-izing opensrf_core.xml.  I'm fully
> on board with making that smarter (reusable chunks, etc) and like you
> I don't care much what the format is in the long run.  And, FTR, I'm
> fully on board with making opensrf.xml smarter, or more dynamic, or
> non-existent (as a filesystem object).
>

That was the original goal.  It quickly started evolving...


>
> But then I saw Blake's response that mentioned service defaults, and
> looked at your linked example more closely, and that's when I came
> away with the impression that you may /also/ be wanting to embed the
> application config (what opensrf.settings reads and then distributes
> to applications) in the connection config file /for the services to
> look at directly/.  Is that correct?
>

Embed, yes. Look at directly, no.  More below.


>
> My fervent hope is that we retain the separation of concerns that the
> two files embody today.  Right now, we just tell things how to connect
> to the opensrf network via opensrf_core.xml, and then they ask the
> opensrf.settings service for their config data.  I promise that's not
> just an idle desire!  Because there are two "layers" (for lack of a
> better term) of configuration, and human error is unfortunately still
> a thing, it increases the risk that things will become broken in a
> complicated environment when All The Things live in just one place,
> instead of dedicated (and tool-enrobed) locations for different
> configuration concerns and targets.
>
> To expand a little on my mention of opensrf.xml above, eventually I
> would like to see the opensrf.settings service stop getting it's
> pile-o-data from opensrf.xml (or any file, for that matter) and
> instead read it from a structured data store.  Maybe a full-on
> database, maybe a simple persistent document store, but something more
> machine-actionable than a file that a human edits and SCP's around.
>

This.  As I'm reviewing the stock opensrf.xml file, I keep asking myself
why so much of this stuff isn't stored in global flags, org settings, etc.
(Of course, it's just easier to change a config file than create a database
upgrade script with INSERTs, etc.).  I'm fully on board with moving as much
of opensrf.xml as we can into a structured data store.

In a world...  where OpenSRF relies on Redis, for example, persistent
storage is an option with a lot of controls about what/when gets stored.
This would be one way to do it w/o requiring, say, Postgres to run OpenSRF.

(At the risk of fully derailing the conversation, I'd actually be fine with
OSRF requiring PG or, for that matter, merging into Evergreen.  That's a
substantial can of worms, though).


>
> Put another way, I don't want services to read a local file for
> anything other than initial bootstrapping. If we do without even a
> bootstrap config file and just have command line parameters, all the
> better.  I don't see a way to do that right now, though.
>
> BUT WAIT!
>
> That all looks pretty negative, so let me offer a counter to myself.
>
> If what you meant (and what I misunderstood) was that opensrf.xml
> could/would eventually turn into a yaml file which would have mutually
> exclusive structural sections from opensrf_core.yml, and
> opensrf.settings could be /actively/ told to read a yaml file that
> just happens to be the same file on disk for both bootstrap and
> settings data purposes  (or, I guess, be pointed to a section?) --
> meaning, opensrf.settings would just look at different parts of the
> file for each purpose and other services would just ignore all of the
> non-bootstrap parts, and not try to use them directly without going to
> the settings service -- then I retract my concerns.
>

Yes, that!  I didn't mean to suggest we deprecate opensrf.settings or point
services directly at the service configuration part of the file.  I just
like the idea of having one config file structure that can cover every
purpose.

Thanks!

-b
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.evergreen-ils.org/pipermail/evergreen-dev/attachments/20221105/7256f820/attachment.htm>