The IA web advance bots are Internet cockroaches, many software developers believe. Some developers have begun to fight again, often humorous.
While any website can be cursed behavior target, sometimes demolishing the site, open source developers are “disproportionately” affected, writing Niccolò Venerandi, developer of a Linux desk known as Plasma and owner of the Blog Librenews.
By their nature, sites that organize free and open -source code projects (Foss) share more of their infrastructure publicly, and also tend to have less resources than commercial products.
The problem is that many AI bots do not honor the Robot.txt Robot.txt Robot Exclusion file, the tool that tells BSS what does not crawl, originally created for search engine bots.
In a “Help cry” Blog In January, Foss XE IASO developer described how Amazonbot hit a git server website to the point of causing ddos interruptions. Git servers organize Foss projects so that anyone who wants to download the code or contribute to it.
But this bot ignored Iaso’s robot.txt, hid behind other IP addresses and pretended to be other users, Iaso said.
“It is useless to block the crawler’s bots because they lie, they change their user agent, use residential IP addresses such as proxies and more,” Iaso lamented.
“They will scratch their place until they fall, and then they will scrape it a little more. They will click on each link in each link of each link, seeing the same pages again and again. Some of them will even click on the same link several times in the same second,” the developer wrote in the publication.
Enter the tombs God
Then, Iaso defended itself with intelligence, building a tool called Anubis.
Anubis is A Verification of Proxy Inverse Test That must be approved before requests can press a git server. Block the bots but let the browsers operated by humans.
The funny part: Anubis is the name of a God in Egyptian mythology that leads the dead to judgment.
“Anubis weighed your soul (heart) and if it was heavier than a pen, your heart was eaten and you, like, Mega died,” Iaso told britcommerce. If a web application passes the challenge and determines that it is human, A nice anime photo Announces success. The drawing is “My opinion about Anubis Anubis,” says Iaso. If it is a bot, the application refuses.
The project ironically has spread like the wind between the community of Foss. Oaso shared it in Github On March 19, and in just a few days, he collected 2,000 stars, 20 taxpayers and 39 holders.
Revenge as a defense
Anubis’s instant popularity shows that Iaso’s pain is not unique. In fact, Venerandi shared history after history:
- CEO of founder of Sourcehut Drew Devault described Spend “from 20 to 100% of my time in any week that mitigates the trackers of LLM hypergrestesivos on scale” and “experiencing dozens of short interruptions per week.”
- Jonathan Corbet, a famous Foss developer who runs the news site of the Linux LWN industry, warned that his site was being slowed by traffic at the ddos level “Of the Hasper Bots.”
- Kevin Fenzi, the Sysadmin of the huge Fedora project of Linux, Ai Raper’s bots said He had become so aggressive that he had to block access to the whole country in Brazil.
Venerandi tells britcommerce that he knows about other projects that experience the same problems. One of them “had to temporarily prohibit all Chinese IP addresses in a moment.”
Let that sink for a moment, that developers “even have to resort to the prohibition of entire countries” only to defend against the bots that ignore the robot.txt files, says Venerandi.
Beyond weighing the soul of a web applicant, other developers believe that revenge is the best defense.
A few days ago in Hacker newsuser XYZAL Suggested load robot. Prohibited pages of TXT with “a load load of articles on the benefits of whitening drink” or “articles on the positive effect of measles capture on bed performance”.
“I think we must aim at the bots to obtain the utility value _Egative_ to visit our traps, not just a zero value,” Xyzal explained.
As, in January, an anonymous creator known as “Aaron” launched a tool called Nepenthes That aims to do that exactly. It catches the trackers in an endless maze of false content, an objective that the developer admitted Ars Technica It is aggressive if not frankly malicious. The tool is named after a carnivorous plant.
And Cloudflare, perhaps the largest commercial player who offers several tools to defend against Ai Crawlers, last week he launched a similar tool called Ai Labyrinth.
It intends to “reduce speed, confuse and waste the resources of the AI and other bots that do not respect ‘unrection’ directives”, described Cloudflare In its blog post. Cloudflare said that the bad behavior of the “irrelevant content instead of extracting the legitimate data from the website” feeds badly.
Sourcehut’s Devault told britcommerce that “Nepenthes has a satisfactory sense of justice, since it feeds meaningless to the trackers and poisons its wells, but ultimately, Anubis is the solution that worked” for its site.
But Devault also issued a public and sincere plea for a more direct solution: “Please stop legitimizing LLMS or generators of Ia or co -pilot of github or any of this garbage. I beg you to stop using them, stop talking about them, stop making new, just stop.”
Since the probability of that is Zilch, developers, particularly in Foss, are fighting with intelligence and a touch of humor.