K_N

The problem with bots is that they try to hide what they are and show as little information as possible. Which is also how most privacy focused browsers appear.

Another source just fills the gap. It's mostly people that don't know what they're doing, running AI training that just scrapes every website they find. The one I blocked was a datacenter for a specific cloud computing company that had been responsible for a lot of the last 24 hours, but that was just one.

For those of you that have static IPs, I should be able to collect a list from the admin panel and exclude you guys from the verification. If we have to leave it on longer than a couple days I'll try to do that.

Yeah, sorry guys. I can try reducing the frequency of the verification screens at least. But at least its better than no website.

I have employed a couple of different anti-bot mitigations. You may see some security verification screens every now and then. I also identified the source of a lot of the bot traffic from the last 24 hours and blocked their entire ASN.

I will be attempting to deploy anubis tomorrow/sunday. We will see how it goes.

They are absolutely milking our shit raw, man. I need to spend some time looking at the request paths and figuring out how they're causing so many cache misses. The light blue part of that graph hits the server directly and causes these issues - the dark blue is served from cloudflare and doesn't affect us.

Thank fuck I did not want to have to go through installing Anubis in front of this site.

As far as I can tell, the mitigations seem to be helping. Is anyone still experiencing slowdowns?

There's been a massive spike in traffic over the last few days - from around 40k unique sessions per day up to 1.2 million on the 14th. Even behind Cloudflare, the server this site runs on and the way it's set up isn't built to handle that kind of traffic. From what I can tell, this is almost entirely AI crawlers - 47% of that traffic came from OpenAI, and another 31% from Anthropic. For the mean time, I've taken a couple easy measures: Gave the database server a little more memory Disabled bot crawling at the domain level (not all of them will respect this) Enabled a bot challenge mode through Cloudflare I am hoping this is enough to mitigate the issue for now, but if you feel like firebombing a datacenter... well, I don't condone terrorism but I would not consider it a demerit to your personal character. I will monitor the effects of these changes for a few days and hopefully it helps. Also, If the Cloudflare bot challenge hits you guys too much and gets annoying let me know, I can try turning it off.

only like 2 months late to notice but thanks yall

Installed an update today as well. Let me know if you're seeing it again.

Still not sure what's causing this but given that Cloudflare keeps changing settings without my consent I am leaning towards blaming them. Might need to move us onto a different WAF/edge provider.

Looks like it's related to the theme we're using. You can avoid it by burning out your retinas with the blazing white Default theme for now.

Going to briefly take the site down after midnight tonight and run some updates to see if fixes the JSON redirect issue some people have started seeing.