<div dir="ltr"><div>Hi Jon,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 7, 2024 at 11:19 PM JonGeorg SageLibrary via Evergreen-general <<a href="mailto:evergreen-general@list.evergreen-ils.org" target="_blank">evergreen-general@list.evergreen-ils.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Greetings. I have two libraries that have noticed that hold emails have not been going out the last couple days. One of the libraries was able to send me specific patron info and I was able to query the db and verify that no hold emails went out for those patrons. I do see other emails going out from the system via the mail.log on the server, so the mail server is up and working.</div></blockquote><div><br></div><div>The table to pay attention to is action_trigger.event and you'll want to filter that by the event_def (ids are in action_trigger.event_definition).  Any event rows in a state other than "pending" or "invalid" should be investigated. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I did have an issue where the log server ran out of space and I had to reboot it after clearing space, however I have been able to verify that hold emails went out after that, so that should not be an issue, but it is the only unusual recent event for our system that I'm aware of. Email receipts are working as expected, included among the patrons I know did not receive hold email notifications.<br><div><br></div><div>I've checked syslog for CRON jobs, for such things as <a href="http://hold_targeter.pl" target="_blank">hold_targeter.pl</a>, <a href="http://action_trigger_runner.pl" target="_blank">action_trigger_runner.pl</a>, thaw_expired_frozen_holds.srfsh, and I don't see any errors and all CRON jobs appear to be running and finishing successfully. <br></div></div></div></blockquote><div><br></div><div>How are you checking this?  I enable a mail client on the server and crons should generate mail to the crontab owner if they fail (or generate any output at all).  You can also relay these to something external (like an <a href="mailto:admin@yourlibrary.org">admin@yourlibrary.org</a> email address).</div><div><br></div><div>I would start with the theory that <a href="http://action_trigger_runner.pl">action_trigger_runner.pl</a> is indeed failing.  This can happen if the server running it has insufficient RAM.  Other possibilities we've encountered are 1) there's a bad template, and the OpenSRF logs should tell you that and/or 2) the <a href="http://action_trigger_runner.pl">action_trigger_runner.pl</a> process generates a "too large" message for EJabberD to handle and EJabberD dies off (and/or the OpenSRF listener dies off).<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br>I've also checked the mail.log file and do not see any entries for these patrons regarding their holds which makes sense as there is no entry in the db, but I do see email receipts. I do see a ton of 'Network is unreachable' errors related to gmail and other providers, but that is a separate issue and likely a reason to consider a 3rd party application to handle notifications.</div></div></div></blockquote><div><br></div><div>Yes, I think eventually our notification methods will have to change or we're all going to have to go to third-party services. :-/<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Is it possible that there is a backlog of stuck hold emails like what happens when the reporter gets stuck? Is there a way to force the hold email function to stop and restart, perhaps in the database itself? I've restarted all services on all servers [minus the db servers] without resolution.</div></div></blockquote><div><br></div><div>More likely that this is happening at the Evergreen (Action/Trigger) level.  If other emails are working as expected, I would work under the assumption that the problem is not with email itself.</div><div><br></div><div><br></div><div>Hope that's helpful!<br></div></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr" style="margin-left:30.75pt" align="left"><table style="border:medium;border-collapse:collapse"><colgroup><col width="76"><col width="7"><col width="64"><col width="437"></colgroup><tbody><tr style="height:74.25pt"><td colspan="2" style="vertical-align:top;overflow:hidden"><p dir="ltr" style="line-height:1.2;text-align:right;margin-top:0pt;margin-bottom:0pt"><a href="https://georgialibraries.org/" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial,sans-serif;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><span style="border:medium;display:inline-block;overflow:hidden;width:75px;height:75px"><img alt="logo with link to Georgia Public Library Service website" src="https://lh7-us.googleusercontent.com/pM15Sp3i-lw8e92bKCiBkBeM6HEQBzyODg5D9Bg9drqqunCrhxNPa7o6RWYfa7lXmatlwiStCvDFLlNKuTT31-tKlI-kNpyuWmsVIWTsBmisImWnGykHRCdYl_bOsH7RFg3eilPres7HT6tglCAqKbA" width="75" height="75" style="margin-left:0px;margin-top:0px"></span></span></a></p></td><td colspan="2" style="vertical-align:top;padding:5pt;overflow:hidden"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13pt;font-family:Verdana,sans-serif;color:rgb(30,67,155);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Chris Sharp, PINES System Administrator</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"></p><hr><p></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Verdana,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"></span><span style="font-size:12pt;font-family:Verdana,sans-serif;color:rgb(30,67,155);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Georgia Public Library Service</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Verdana,sans-serif;color:rgb(30,67,155);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">2872 Woodcock Blvd, Suite 250 | Atlanta, GA 30341</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12pt;font-family:Verdana,sans-serif;color:rgb(30,67,155);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">(404) 235-7147 | <a href="mailto:csharp@georgialibraries.org" target="_blank">csharp@georgialibraries.org</a></span></p></td></tr><tr style="height:36pt"><td colspan="2" style="vertical-align:top;overflow:hidden"><br></td><td colspan="2" style="vertical-align:top;padding:5pt;overflow:hidden"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="https://www.facebook.com/georgialibraries" style="text-decoration:none" target="_blank"><span style="font-size:9pt;font-family:Verdana,sans-serif;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><span style="border:medium;display:inline-block;overflow:hidden;width:36px;height:36px"><img alt="logo with link to Georgia Public Library Service Facebook page" src="https://lh7-us.googleusercontent.com/dbDlZYB2rcEmRmoj2-mHPOUe0ZTCj1cah_ZpTri_Gz1fD0Q4fJ8k_VlDB0nwtnow-AVzA-qLjfw71Sh1zsvnd2yn0bJ6B4rlD5lqA3KUsLwXnWrsYcPu_3pSABeL_ptWs_kL336KSmWYZWarzpo9kPg" width="36" height="36" style="margin-left:0px;margin-top:0px"></span></span></a><a href="https://www.instagram.com/georgialibraries/" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Verdana,sans-serif;color:rgb(17,85,204);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><span style="border:medium;display:inline-block;overflow:hidden;width:34px;height:34px"><img alt="logo with link to Georgia Public Library Service Instagram page" src="https://lh7-us.googleusercontent.com/39EAj7rwrQ91HPeGH9KrK7s5SYbeu8TNBE1oT4mgn3Sq18sV2ntFcXaS_Ap7OBQPskuL2aiqs3Q2To2SaiuAH42GGjuHTxCq99u2o1TbLGrxFTlcP_DpUnff3UC50byZIwrvp_sU_d6IZz5chmYqeZM" width="34" height="34" style="margin-left:0px;margin-top:0px"></span></span></a><a href="https://www.linkedin.com/company/georgia-public-library-service/" style="text-decoration:none" target="_blank"><span style="font-size:9pt;font-family:Verdana,sans-serif;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><span style="border:medium;display:inline-block;overflow:hidden;width:36px;height:36px"><img alt="logo with link to Georgia Public Library Service LinkedIn page" src="https://lh7-us.googleusercontent.com/EUyyO6DxbAch9ZlHlZYHG80dIcS0XKqbAZkMX8IBACic9eOCoFIyIiOj-TtwbLEpVZ6tjn3ArAUiq5reBZW8u2ueosIyjTMuJbTwS2tRphiHE5S6V_Yn5h-_co5WNxmk8lqjHJ6WW5K6TQG9_j3tqZY" width="36" height="36" style="margin-left:0px;margin-top:0px"></span></span></a><a href="https://www.threads.net/@georgialibraries" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Verdana,sans-serif;color:rgb(17,85,204);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><span style="border:medium;display:inline-block;overflow:hidden;width:34px;height:34px"><img alt="logo with link to Georgia Public Library Service Threads page" src="https://lh7-us.googleusercontent.com/XdtkF0Dx0M_t8rpMpVk9HyZkhYwYjNG0lEE55aan3ClbHoFa_YvkRVVu0tZjnphpym3gfvWt9sNRRqyJe8UgkXrWLzo8UrgENFy3eJ1wZBXW1_a_NMKpcruxICERZX6dzPPrYmgO6xJz61FfBZ5mmWQ" width="34" height="34" style="margin-left:0px;margin-top:0px"></span></span></a></p></td></tr><tr style="height:16.9873pt"><td colspan="2" style="vertical-align:top;overflow:hidden"><br></td><td colspan="2" style="vertical-align:top;padding:5pt;overflow:hidden"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><a href="http://georgialibraries.org/subscription" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Verdana,sans-serif;color:rgb(17,85,204);background-color:rgb(255,255,255);font-weight:400;font-style:italic;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">Join our email list</span></a><span style="font-size:11pt;font-family:Verdana,sans-serif;color:rgb(0,0,0);background-color:rgb(255,255,255);font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Verdana,sans-serif;color:rgb(67,67,67);background-color:rgb(255,255,255);font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">for stories of Georgia libraries making an impact in our communities.</span></p></td></tr></tbody></table></div></div></div></div>