{"id":106128,"date":"2025-03-25T21:36:58","date_gmt":"2025-03-25T21:36:58","guid":{"rendered":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/"},"modified":"2025-03-25T21:36:58","modified_gmt":"2025-03-25T21:36:58","slug":"open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries","status":"publish","type":"post","link":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/","title":{"rendered":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries"},"content":{"rendered":"<div>\n<p>Software developer Xe Iaso <a href=\"https:\/\/xeiaso.net\/notes\/2025\/amazon-crawler\/\">reached a breaking point<\/a> earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures\u2014adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic\u2014Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies.<\/p>\n<p>Desperate for a solution, Iaso eventually resorted to moving their server behind a VPN and creating &#8220;Anubis,&#8221; a custom-built proof-of-work challenge system that forces web browsers to solve computational puzzles before accessing the site. &#8220;It&#8217;s futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more,&#8221; Iaso wrote in a <a href=\"https:\/\/xeiaso.net\/notes\/2025\/amazon-crawler\/\">blog post<\/a> titled &#8220;a desperate cry for help.&#8221; &#8220;I don&#8217;t want to have to close off my Gitea server to the public, but I will if I have to.&#8221;<\/p>\n<p>Iaso&#8217;s story highlights a broader crisis rapidly spreading across the open source community, as what appear to be aggressive AI crawlers increasingly overload community-maintained infrastructure, causing what amounts to persistent distributed denial-of-service (DDoS) attacks on vital public resources. According to a <a href=\"https:\/\/thelibre.news\/foss-infrastructure-is-under-attack-by-ai-companies\/\">comprehensive recent report<\/a> from LibreNews, some open source projects now see as much as 97 percent of their traffic originating from AI companies&#8217; bots, dramatically increasing bandwidth costs, service instability, and burdening already stretched-thin maintainers.<\/p>\n<p><a href=\"https:\/\/arstechnica.com\/ai\/2025\/03\/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/\">Read full article<\/a><\/p>\n<p><a href=\"https:\/\/arstechnica.com\/ai\/2025\/03\/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/#comments\">Comments<\/a><\/p>\n<\/div>\n<p class=\"wpematico_credit\"><small>Powered by <a href=\"http:\/\/www.wpematico.com\" target=\"_blank\">WPeMatico<\/a><\/small><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures\u2014adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic\u2014Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","tve_updated_post":"","tve_custom_css":"","tve_user_custom_css":"","tve_globals":{},"tcb2_ready":0,"tcb_editor_enabled":0,"tve_landing_page":"","_tve_header":"","_tve_footer":""},"categories":[241],"tags":[],"class_list":["post-106128","post","type-post","status-publish","format-standard","hentry","category-technology"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com\" \/>\n<meta property=\"og:description\" content=\"Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures\u2014adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic\u2014Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/\" \/>\n<meta property=\"og:site_name\" content=\"UshopWell.com\" \/>\n<meta property=\"article:published_time\" content=\"2025-03-25T21:36:58+00:00\" \/>\n<meta name=\"author\" content=\"UShopWell\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"UShopWell\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/\"},\"author\":{\"name\":\"UShopWell\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/person\\\/6fd1f9e0ff932e534c86c70d5acff0fc\"},\"headline\":\"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries\",\"datePublished\":\"2025-03-25T21:36:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/\"},\"wordCount\":250,\"publisher\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\"},\"articleSection\":[\"Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/\",\"name\":\"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#website\"},\"datePublished\":\"2025-03-25T21:36:58+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#website\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\",\"name\":\"UshopWell.com\",\"description\":\"The Premiere Online Marketplace\",\"publisher\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#organization\",\"name\":\"UshopWell\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/wp-content\\\/uploads\\\/2018\\\/01\\\/pandaSwea.png\",\"contentUrl\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/wp-content\\\/uploads\\\/2018\\\/01\\\/pandaSwea.png\",\"width\":365,\"height\":359,\"caption\":\"UshopWell\"},\"image\":{\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/#\\\/schema\\\/person\\\/6fd1f9e0ff932e534c86c70d5acff0fc\",\"name\":\"UShopWell\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g\",\"caption\":\"UShopWell\"},\"url\":\"https:\\\/\\\/ushopwell.com\\\/ublog\\\/author\\\/kburnettu\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/","og_locale":"en_US","og_type":"article","og_title":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com","og_description":"Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures\u2014adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic\u2014Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted...","og_url":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/","og_site_name":"UshopWell.com","article_published_time":"2025-03-25T21:36:58+00:00","author":"UShopWell","twitter_card":"summary_large_image","twitter_misc":{"Written by":"UShopWell","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/#article","isPartOf":{"@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/"},"author":{"name":"UShopWell","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/person\/6fd1f9e0ff932e534c86c70d5acff0fc"},"headline":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries","datePublished":"2025-03-25T21:36:58+00:00","mainEntityOfPage":{"@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/"},"wordCount":250,"publisher":{"@id":"https:\/\/ushopwell.com\/ublog\/#organization"},"articleSection":["Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/","url":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/","name":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries - UshopWell.com","isPartOf":{"@id":"https:\/\/ushopwell.com\/ublog\/#website"},"datePublished":"2025-03-25T21:36:58+00:00","breadcrumb":{"@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ushopwell.com\/ublog\/open-source-devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ushopwell.com\/ublog\/"},{"@type":"ListItem","position":2,"name":"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries"}]},{"@type":"WebSite","@id":"https:\/\/ushopwell.com\/ublog\/#website","url":"https:\/\/ushopwell.com\/ublog\/","name":"UshopWell.com","description":"The Premiere Online Marketplace","publisher":{"@id":"https:\/\/ushopwell.com\/ublog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ushopwell.com\/ublog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ushopwell.com\/ublog\/#organization","name":"UshopWell","url":"https:\/\/ushopwell.com\/ublog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/logo\/image\/","url":"https:\/\/ushopwell.com\/ublog\/wp-content\/uploads\/2018\/01\/pandaSwea.png","contentUrl":"https:\/\/ushopwell.com\/ublog\/wp-content\/uploads\/2018\/01\/pandaSwea.png","width":365,"height":359,"caption":"UshopWell"},"image":{"@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ushopwell.com\/ublog\/#\/schema\/person\/6fd1f9e0ff932e534c86c70d5acff0fc","name":"UShopWell","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4adb372cadd43b4d4c57964dab95b0f69618bf960d131c4acf49d96d6bbc9c6e?s=96&d=mm&r=g","caption":"UShopWell"},"url":"https:\/\/ushopwell.com\/ublog\/author\/kburnettu\/"}]}},"_links":{"self":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts\/106128","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/comments?post=106128"}],"version-history":[{"count":0,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/posts\/106128\/revisions"}],"wp:attachment":[{"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/media?parent=106128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/categories?post=106128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ushopwell.com\/ublog\/wp-json\/wp\/v2\/tags?post=106128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}