<div dir="ltr">Hi Linda,<div><br></div><div>It's not for Evergreen, but my colleague <a href="https://github.com/pulibrary/princeton_ansible/commit/6f9009249a168442391d90e2b75028d40a8a9e91">recently blocked claudebot using fail2ban on our load balancer</a>.  Essentially, fail2ban is configured to watch Nginx's access log, and if more than 10 claudebot requests appear within the past minute from a particular IP, it automatically blocks all requests from that IP for the next 24 hours.  I would think that something similar could work for Apache's access log.</div><div><br></div><div>Good luck with the bots!</div><div><br></div><div>  -Jane</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El vie, 19 abr 2024 a la(s) 3:42 a.m., Linda Jansová via Evergreen-general (<a href="mailto:evergreen-general@list.evergreen-ils.org">evergreen-general@list.evergreen-ils.org</a>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear all,<br>
<br>
Have any of you encountered an extensive crawling by Bytespider and <br>
Bytedance (see e.g., <br>
<a href="https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/" rel="noreferrer" target="_blank">https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/</a>), <br>
Claudebot or other AI bots?<br>
<br>
If so, do you have any secret recipe how to disable the crawler from <br>
accessing the site?<br>
<br>
Thank you very much for sharing your experience!<br>
<br>
Linda<br>
<br>
_______________________________________________<br>
Evergreen-general mailing list<br>
<a href="mailto:Evergreen-general@list.evergreen-ils.org" target="_blank">Evergreen-general@list.evergreen-ils.org</a><br>
<a href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general" rel="noreferrer" target="_blank">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general</a><br>
</blockquote></div>