<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
All,<br>
<br>
I almost replied with the arstechnica article that Josh linked when
the thread was started. But I decided not to put it out there until
I had setup a test system to see if I could get that code working. A
tarpit, I think, serves them right. And, of course, the whole issue
is destined to receive the fate of spam and spam filters forever and
ever.<br>
<br>
It was a serendipitous timed article. It's existence at this moment
in time signals to me that this isn't a "just us" problem. It's the
entire planet.<br>
<br>
<pre class="moz-signature" cols="72">-Blake-
Conducting Magic
Will consume any data format
MOBIUS
</pre>
<div class="moz-cite-prefix">On 2/13/2025 3:10 PM, Josh Stompro via
Evergreen-dev wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGOQQftvG72nky1Okk-_-Gee3tVSW6kQFRE=NA5E_9BMMmFoyw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">Jeff, thanks for bringing this up on the list.
<div><br>
</div>
<div>We are seeing a lot of requests like</div>
<div> "GET
/eg/opac/mylist/delete?anchor=record_184821&record=184821"
from never seen before IPs, and they make 1-12 requests and
then stop.</div>
<div><br>
</div>
<div>And they seem like they usually have a random out of date
chrome version in the user agent string. </div>
<div><span style="color:rgb(0,0,0)">Chrome/88.0.4324.192</span></div>
<div><span style="color:rgb(0,0,0)">Chrome/86.0.4240.75</span><font
color="#000000"><br>
</font>
<div><br>
</div>
<div>I've been trying to slow down the bots by
collecting logs and grabbing all the obvious patterns and
blocking netblocks for non US ranges. <a
href="http://ipinfo.io" moz-do-not-send="true">ipinfo.io</a>
offers a free country & ASN database download that
I've been using to look up the ranges and countries. (<a
href="https://ipinfo.io/products/free-ip-database"
moz-do-not-send="true" class="moz-txt-link-freetext">https://ipinfo.io/products/free-ip-database</a>)
I would be happy to share a link to our current blocklist
that has 10K non US ranges. </div>
<div><br>
</div>
<div>I've also been reporting the non US bot activity to <a
href="https://www.abuseipdb.com/" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.abuseipdb.com/</a>
just to bring some visibility to these bad bots. I
noticed initially that many of the IPs that we were
getting hit from didn't seem to be listed on any
blocklists already, so I figured some reporting might
help. I'm kind of curious if Evergreen sites are getting
hit from the same IPs, so an evergreen specific blocklist
would be useful. If you look up your bot IPs on <a
href="http://abuseipdb.com" moz-do-not-send="true">abuseipdb.com</a>
you can see if I've already reported any of them.</div>
<div><br>
</div>
<div>I've also been making use of block lists from <a
href="https://iplists.firehol.org/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://iplists.firehol.org/</a></div>
<div>Such as </div>
<div><a
href="https://iplists.firehol.org/files/cleantalk_30d.ipset"
moz-do-not-send="true" class="moz-txt-link-freetext">https://iplists.firehol.org/files/cleantalk_30d.ipset</a><br>
<div><span
style="color:rgb(51,51,51);font-family:Roboto,sans-serif;font-size:14px"><a
href="https://iplists.firehol.org/files/botscout_7d.ipset"
moz-do-not-send="true" class="moz-txt-link-freetext">https://iplists.firehol.org/files/botscout_7d.ipset</a></span></div>
<div><a
href="https://iplists.firehol.org/files/firehol_abusers_1d.netset"
moz-do-not-send="true" class="moz-txt-link-freetext">https://iplists.firehol.org/files/firehol_abusers_1d.netset</a></div>
<div><br>
</div>
</div>
<div>We are using HAProxy so I did some looking into the
CrowdSec HAProxy Bouncer (<a
href="https://docs.crowdsec.net/u/bouncers/haproxy/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://docs.crowdsec.net/u/bouncers/haproxy/</a>)
but I'm not sure that would help since these IPs don't
seem to be on blocklists. But I may just not quite
understand how CrowdSec is supposed to work.</div>
<div><br>
</div>
<div>HAProxy Enterprise has a ReCaptcha module that I think
would allow us to feed any non-us connections that haven't
connected before through a recaptcha, but the price for
HAProxy Enterprise is out of our budget. <a
href="https://www.haproxy.com/blog/announcing-haproxy-enterprise-3-0#new-captcha-and-saml-modules"
moz-do-not-send="true" class="moz-txt-link-freetext">https://www.haproxy.com/blog/announcing-haproxy-enterprise-3-0#new-captcha-and-saml-modules</a></div>
<div><br>
</div>
<div>There is also a fairly up to date project for adding
Captchas through haproxy at </div>
<div><a href="https://github.com/ndbiaw/haproxy-protection"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/ndbiaw/haproxy-protection</a>,
This looks promising as a transparent method, requires new
connections to perform a javascript proof of work
calculation before allowing access. Could be a good
transparent way of handling it. </div>
<div><br>
</div>
<div>We were taken out by ChatGTP bots back in December,
which were a bit easier to block the netblocks since they
were not as spread out. I recently saw this article about
how some people are fighting back against bots that ignore
robots.txt, <a
href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</a></div>
</div>
<div><br>
</div>
<div>Josh</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Mon, Jan 27, 2025 at
6:33 PM Jeff Davis via Evergreen-dev <<a
href="mailto:evergreen-dev@list.evergreen-ils.org"
moz-do-not-send="true" class="moz-txt-link-freetext">evergreen-dev@list.evergreen-ils.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi
folks,<br>
<br>
Our Evergreen environment has been experiencing a
higher-than-usual volume of unwanted bot traffic in recent
months. Much of this traffic looks like webcrawlers hitting
Evergreen-specific URLs from an enormous number of different
IP addresses. Judging from discussion in IRC last week, it
sounds like other EG admins have been seeing the same thing.
Does anyone have any recommendations for managing this
traffic and mitigating its impact?<br>
<br>
Some solutions that have been suggested/implemented so far:<br>
- Geoblocking entire countries.<br>
- Using Cloudflare's proxy service. There's some trickiness
in getting this to work with Evergreen.<br>
- Putting certain OPAC pages behind a captcha.<br>
- Deploying publicly-available blocklists of "bad bot"
IPs/useragents/etc. (good but limited, and not EG-specific).<br>
- Teaching EG to identify and deal with bot traffic itself
(but arguably this should happen before the traffic hits
Evergreen).<br>
<br>
My organization is currently evaluating CrowdSec as another
possible solution. Any opinions on any of these approaches?<br>
-- <br>
Jeff Davis<br>
BC Libraries Cooperative<br>
_______________________________________________<br>
Evergreen-dev mailing list<br>
<a href="mailto:Evergreen-dev@list.evergreen-ils.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Evergreen-dev@list.evergreen-ils.org</a><br>
<a
href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev</a><br>
</blockquote>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
Evergreen-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Evergreen-dev@list.evergreen-ils.org">Evergreen-dev@list.evergreen-ils.org</a>
<a class="moz-txt-link-freetext" href="http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev">http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>