[OPEN-ILS-DEV] Modifications to Circ Matrix Matchpoint checking
Mike Rylander
mrylander at gmail.com
Sat Sep 11 12:14:03 EDT 2010
On Fri, Sep 10, 2010 at 11:03 PM, Thomas Berezansky <tsbere at mvlc.org> wrote:
> I believe I have completed work on this particular project. For those
> interested here but not following launchpad, I submitted a patch there:
>
> https://bugs.launchpad.net/evergreen/+bug/635463
Thanks, Thomas.
Right off the bat I want to make it clear that because this is a big
feature addition, and as it stands contains large schema changes
(which I'd also like to discuss more -- there are parts that surprised
me), I can't see this getting into 2.0.
With that out of the way, here are some points that jumped out at me
looking at the patch (relevant sections pasted in-line), in no
particular order:
------------------------------------------------
** It looks like the circ mod test for inclusion now falls through **
@@ -182,15 +188,11 @@
CASE WHEN m.usr_age_upper_bound IS NOT NULL
THEN 0.5 ELSE 0 END DESC LOOP
IF current_mp.circ_modifier IS NOT NULL THEN
- CONTINUE WHEN current_mp.circ_modifier <>
item_object.circ_modifier OR item_object.circ_modifier IS NULL;
+ CONTINUE WHEN current_mp.circ_modifier <>
item_object.circ_modifier;
END IF;
IF current_mp.marc_type IS NOT NULL THEN
- IF item_object.circ_as_type IS NOT NULL THEN
- CONTINUE WHEN current_mp.marc_type <>
item_object.circ_as_type;
- ELSE
- CONTINUE WHEN current_mp.marc_type <>
rec_descriptor.item_type;
- END IF;
+ CONTINUE WHEN current_mp.marc_type <>
COALESCE(item_object.circ_as_type, rec_descriptor.item_type);
END IF;
IF current_mp.marc_form IS NOT NULL THEN
This wasn't discussed, and seems surprising to me. If the potential
matchpoint has a circ mod and the object either has none or it's is
different from what the matchpoint wants, then we should move on.
However, I like the COALESCE optimization of the marc_type bit! :)
It may be that you intended to simplify the test without changing the
semantics, but because of how NULL acts in a <> comparison (or any
comparison for that matter) it changed the behavior. Consider this
simplified example:
=# select case when 1 <> null then true else false end;
case
------
f
(1 row)
The equivalent of the above (the change in your patch) would /not/
trigger the CONTINUE (which it should), where as the equivalent of the
below (the original code) does, obviously.
=# select case when 1 <> null or null is null then true else false end;
case
------
t
(1 row)
------------------------------------------------
** Concern about conditional ordering and logic when breaking out of
the <<usergroup>> loop **
+ EXIT usergroup WHEN full_depth = false
+ AND matchpoint.circulate IS NOT NULL
+ AND matchpoint.duration_rule IS NOT NULL
+ AND matchpoint.recurring_fine_rule IS NOT NULL
+ AND matchpoint.max_fine_rule IS NOT NULL
+ AND hold_ratio_fallthrough OR (
+ matchpoint.total_copy_hold_ratio IS NOT NULL
+ AND
matchpoint.available_copy_hold_ratio IS NOT NULL
+ );
The OR'd parts there really need to be grouped together. But I'm also
concerned that it's not doing what is intended. IIUC, we want to
force looking for non-NULL ratios when hold_ratio_fallthrough is
enabled, else we just take what we have when we've filled in the four
required bits. That WHEN clause reads to me like, "we're done if: we
have all required parts AND ( either we have ratios OR we need to find
ratios )". Perhaps:
EXIT usergroup WHEN (
full_depth = false
AND matchpoint.circulate IS NOT NULL
AND matchpoint.duration_rule IS NOT NULL
AND matchpoint.recurring_fine_rule IS NOT NULL
AND matchpoint.max_fine_rule IS NOT NULL
) AND (
NOT hold_ratio_fallthrough
OR (matchpoint.total_copy_hold_ratio IS NOT NULL AND
matchpoint.available_copy_hold_ratio IS NOT NULL)
);
------------------------------------------------
** Result set coming from action.find_circ_matrix_matchpoint **
- RETURN matchpoint;
+ -- This is a successful matchpoint finding IF we have the
circulate button AND the rules
+ IF matchpoint.circulate IS NOT NULL AND matchpoint.duration_rule
IS NOT NULL AND matchpoint.recurring_fine_rule IS NOT NULL AND
matchpoint.max_fine_rule IS NOT NULL THEN
+ result.success = true;
+ result.matchpoint = matchpoint;
+ ELSE
+ result.success = false;
+ END IF;
+ -- For debug purposes, include the full considered row list regardless.
+ result.buildrows = row_list;
+ RETURN result;
The fact that we failed to find a complete constructed matchpoint is
certainly a failure case insofar as the database call is concerned.
However, we shouldn't simply fail to work. If there is no ultimate
"outer" matchpoint with all the information we need, the we need to
specify that somewhere. Either OU settings or global settings are
candidates for ultimate fall-back values. Another option, which
requires more magic in configuration but will require no changes to
the backend code you've presented, is to make a specific matchpoint
(with an id of, perhaps, 0) special and require it be tied to the
top-most OU, the top-most user group, and force it to have values for
the four fields we need.
------------------------------------------------
** Refactoring that complicates the code **
@@ -388,21 +435,16 @@
RETURN NEXT result;
END IF;
- -- Fail if the total copy-hold ratio is too low
- IF circ_test.total_copy_hold_ratio IS NOT NULL THEN
+ -- Fail if the total/available copy-hold ratio is too low
+ IF circ_test.total_copy_hold_ratio IS NOT NULL OR
circ_test.available_copy_hold_ratio IS NOT NULL THEN
SELECT INTO hold_ratio * FROM
action.copy_related_hold_stats(match_item);
- IF hold_ratio.total_copy_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio < circ_test.total_copy_hold_ratio THEN
+ IF circ_test.total_copy_hold_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio < circ_test.total_copy_hold_ratio THEN
result.fail_part :=
'config.circ_matrix_test.total_copy_hold_ratio';
result.success := FALSE;
done := TRUE;
RETURN NEXT result;
END IF;
- END IF;
-
- -- Fail if the available copy-hold ratio is too low
- IF circ_test.available_copy_hold_ratio IS NOT NULL THEN
- SELECT INTO hold_ratio * FROM
action.copy_related_hold_stats(match_item);
- IF hold_ratio.available_copy_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio < circ_test.available_copy_hold_ratio
THEN
+ IF circ_test.available_copy_hold_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio < circ_test.available_copy_hold_ratio
THEN
result.fail_part :=
'config.circ_matrix_test.available_copy_hold_ratio';
result.success := FALSE;
done := TRUE;
I don't see any functional change here, but it's certainly more
complicated to read and therefore maintain. I'd see this go back to
two blocks.
------------------------------------------------
** No changes to Circulate.pm included **
Certainly, because the return type of action.item_user_circ_test
changed, the Perl will need to lean how to use it. A simple diff
oversight, I'm sure.
------------------------------------------------
** No upgrade script **
That'll be critical to applying the change to trunk.
---------------------------------
Finally, will you describe your testing methodology? Did you set up
uninherited (fully filled) matchpoints on old and new DBs and confirm
that they work the same? Do you have some sample inherited
matchpoints that show the different features of the new design,
including the switch for ratio inheritance?
Note, not everything I mention above needs to have a complete
resolution before this goes into trunk. Getting it in to trunk will
mean more folks can poke at it, so I don't want to stall that any
longer than we must, but at least some discussion (if not significant
code changes) are warranted before commit, IMO.
Thanks again, Thomas. I'm very excited about this!
--miker
>
> Thomas Berezansky
> Merrimack Valley Library Consortium
>
>
> Quoting Mike Rylander <mrylander at gmail.com>:
>
>> There as been a /lot/ of discussion surrounding this functionality on
>> IRC over the last week, just to let everyone here know that it's not
>> being ignored.
>>
>> More below.
>>
>> On Tue, Sep 7, 2010 at 4:29 PM, Thomas Berezansky <tsbere at mvlc.org> wrote:
>>>
>>> My previous email included a patch that was, unfortunately, just the tip
>>> of
>>> the iceberg, so to speak. I have since figured out a lot of what needs to
>>> be
>>> done to do this on the backend.
>>>
>>> However, in looking at what needs to be done, I came up with several
>>> questions as to how to handle things.
>>>
>>> Currently the circulation flag, duration rule, recurring fine rule, and
>>> max
>>> fine rule are on my list of things to have "fall through".
>>>
>>> I am unsure as to whether or not to have the total copy hold ratio or the
>>> available copy hold ratio values fall through or not. They are not
>>> "endpoint" data, but at the same time may be considered to be an
>>> important
>>> part of the checks.
>>>
>>> In that regard, I see three logical options:
>>> 1 - Don't have them fall through. If they aren't set on the first
>>> matchpoint
>>> found, they aren't set.
>>> 2 - Have them fall through until we run out of rules or they are filled,
>>> like the four values above.
>>> 3 - Have them fall through, but only until we have filled the other four
>>> values. Once the other four values are filled we officially stop checking
>>> them.
>>>
>>> I am partial to 1 or 3 in this case.
>>>
>>
>> After discussion, option 3 seems sane and has gained general
>> consensus. One point I brought up was, how do you tell the system
>> that you do /not/ want to use an inherited test value and also don't
>> want to supply a value? IOW, what if you don't want to test a
>> copy-hold ratio and yet some broader rule does set a value for one of
>> these tests?
>>
>> The answer is that a value of 0 (zero) acts the same as "don't test"
>> since all possible values will pass.
>>
>>> The other main issue I am seeing right now is the circ mod test. In
>>> particular, the current system uses the matchpoint ID for the check. As
>>> my
>>> changes make it so that there is more than one matchpoint ID possibly
>>> involved there are several options:
>>>
>>> 1 - Negate the check when there is more than one matchpoint used in the
>>> final result
>>> 2 - Use only the first matchpoint found for the check
>>> 3 - Have it use all matchpoints that were considered for the check (were
>>> valid and before we filled in our values)
>>> 4 - Have it use all matchpoints that contributed to the check (were
>>> valid,
>>> before we filled in our values, AND set at least one value)
>>>
>>> I am partial to 3 in this case, as that would allow for creating a
>>> matchpoint that would be specifically for triggering that limit, without
>>> changing any of the resulting rules.
>>>
>>
>> This question has not been addressed yet. I think that without
>> broader input, and in fact outside requests, we need to sit on this
>> one. I understand the argument for the flexibility of 3, but these
>> matchpoints and rulesets can already be very complicated, and the
>> potential for SAAD is high. Consider:
>>
>> * matchpoint 1 (closest match) supplies all but max fine rule, and
>> has a circ-mod test for "book" that limits the user to 10 items out
>> * matchpoint 2 (fall-through by group) supplies the max fine rule,
>> and also has a circ-mod test for "book", but sets the limit at 5 items
>> out
>>
>> Which do we honor? And, like the question about copy-hold ratios, how
>> do we say "do NOT test per-circ-mod items out at all"? There are no
>> magic values here, and the benefit of flexibility IMO is outweighed by
>> the complexity of getting it right from a user perspective.
>>
>>> Any thoughts?
>>>
>>
>> All that said, I think this is some awesome work. Thanks, Thomas!
>>
>> --miker
>>
>>> Thomas Berezansky
>>> Merrimack Valley Library Consortium
>>>
>>>
>>> Quoting Thomas Berezansky <tsbere at mvlc.org>:
>>>
>>>> Theory:
>>>> Make the things returned (circulate, duration_rule,
>>>> recurring_fine_rule,
>>>> max_fine_rule) "fall through" when set to NULL.
>>>>
>>>> The attached patch attempts to make this happen on the back end, but
>>>> provides no front end interface changes for configuring it.
>>>>
>>>> IT HAS NOT BEEN TESTED, mainly because I don't want to screw with our
>>>> test system right now when others may be trying to test existing
>>>> functionality.
>>>>
>>>> It also adds in the ability to pass in one more (optional) boolean
>>>> parameter to the function to return the entire list of rules used to
>>>> create the final result, intended for "debug/test" front end
>>>> functionality
>>>> to show what rules were considered in the fall-through checking.
>>>>
>>>> Pros:
>>>>
>>>> You can override a subset of those fields with a specific rule while
>>>> allowing broader rules to fill in the holes.
>>>> This may result in less duplication of information across rules, making
>>>> things easier to maintain.
>>>> Thus, this may result in less rules in general, and thus less
>>>> processing
>>>> time on sorting them overall.
>>>>
>>>> Cons:
>>>>
>>>> Manually figuring out the specifics of what will happen will take more
>>>> time/effort.
>>>> Changing a single rule may have a greater unintended effect on other
>>>> rules.
>>>> Staff would need training for when to have a rule fall through and when
>>>> to set it.
>>>> More time to return from the DB for any rule that is "falling through"
>>>> to
>>>> broader rules.
>>>>
>>>> Examples for the following org tree:
>>>>
>>>> CONS
>>>> -SYSA
>>>> --LIBC
>>>> --LIBD
>>>> -SYSB
>>>> --LIBE
>>>> --LIBF
>>>>
>>>> Implementing the following "business" rules:
>>>>
>>>> At the CONS level:
>>>> By default, everything circulates, uses DFLT_DUR duration, DFLT_RFINE
>>>> recurring fine, and DFLT_MFINE max fine.
>>>> Circ Modifier "book" uses the duration BOOK_DUR
>>>> Reference flagged materials don't circulate
>>>>
>>>> At the SYSA level there are no special rules.
>>>>
>>>> At the SYSB level the max fine should be SYSB_MFINE.
>>>>
>>>> At the LIBC level the recurring fine is LIBC_RFINE
>>>>
>>>> At the LIBD level circ modifier "book" uses the DFLT_DUR duration
>>>> instead
>>>> of "BOOK_DUR"
>>>>
>>>> At the LIBE level reference flagged materials circulate.
>>>>
>>>> At the LIBF level there are no special rules.
>>>>
>>>> The current method would require the following circ rules to implement
>>>> those business rules:
>>>>
>>>> CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
>>>> CONS NULL NULL TRUE DFLT_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> CONS NULL TRUE FALSE DFLT_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> CONS book NULL TRUE BOOK_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> CONS book TRUE FALSE BOOK_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> SYSB NULL NULL TRUE DFLT_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>> SYSB NULL TRUE FALSE DFLT_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>> SYSB book NULL TRUE BOOK_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>> SYSB book TRUE FALSE BOOK_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>> LIBC NULL NULL TRUE DFLT_DUR LIBC_RFINE
>>>> DFLT_MFINE
>>>> LIBC NULL TRUE FALSE DFLT_DUR LIBC_RFINE
>>>> DFLT_MFINE
>>>> LIBC book NULL TRUE BOOK_DUR LIBC_RFINE
>>>> DFLT_MFINE
>>>> LIBC book TRUE FALSE BOOK_DUR LIBC_RFINE
>>>> DFLT_MFINE
>>>> LIBD book NULL TRUE DFLT_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> LIBD book TRUE FALSE DFLT_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> LIBE NULL NULL TRUE DFLT_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>> LIBE book NULL TRUE BOOK_DUR DFLT_RFINE
>>>> SYSB_MFINE
>>>>
>>>> 16 circ rules total.
>>>>
>>>> The new method would require the following circ rules to implement
>>>> those
>>>> business rules:
>>>>
>>>> CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
>>>> CONS NULL NULL TRUE DFLT_DUR DFLT_RFINE
>>>> DFLT_MFINE
>>>> CONS book NULL NULL BOOK_DUR NULL NULL
>>>> CONS NULL TRUE FALSE NULL NULL NULL
>>>> SYSB NULL NULL NULL NULL NULL
>>>> SYSB_MFINE
>>>> LIBC NULL NULL NULL NULL LIBC_RFINE NULL
>>>> LIBD book NULL NULL DFLT_DUR NULL NULL
>>>> LIBE NULL TRUE TRUE NULL NULL NULL
>>>>
>>>> 7 circ rules total.
>>>>
>>>> Starting with the above, lets assume that SYSA wants to change their
>>>> recurring fine to SYSA_RFINE.
>>>> LIBC's recurring fine is to be unchanged.
>>>>
>>>> The current method requires the following changes:
>>>>
>>>> ADD the following entries:
>>>> CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
>>>> SYSA NULL NULL TRUE DFLT_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>> SYSA NULL TRUE FALSE DFLT_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>> SYSA book NULL TRUE BOOK_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>> SYSA book TRUE FALSE BOOK_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>>
>>>> UPDATE the LIBD entries:
>>>> CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
>>>> LIBD book NULL TRUE DFLT_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>> LIBD book TRUE FALSE DFLT_DUR SYSA_RFINE
>>>> DFLT_MFINE
>>>>
>>>> 4 rules added, 2 changed, total is now 20 rules.
>>>>
>>>> The new method would require the following changes:
>>>>
>>>> ADD the following entry:
>>>> CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
>>>> SYSA NULL NULL NULL NULL SYSA_RFINE NULL
>>>>
>>>> 1 rule added, 0 changed, total is now 8 rules.
>>>>
>>>> Thomas Berezansky
>>>> Merrimack Valley Library Consortium
>>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Mike Rylander
>> | VP, Research and Design
>> | Equinox Software, Inc. / The Evergreen Experts
>> | phone: 1-877-OPEN-ILS (673-6457)
>> | email: miker at esilibrary.com
>> | web: http://www.esilibrary.com
>>
>
>
--
Mike Rylander
| VP, Research and Design
| Equinox Software, Inc. / The Evergreen Experts
| phone: 1-877-OPEN-ILS (673-6457)
| email: miker at esilibrary.com
| web: http://www.esilibrary.com
More information about the Open-ils-dev
mailing list