Creepy crawlies
Konstantin Ryabitsev 指出,从 Linux 官方 Git 仓库 git.kernel.org 角度看,恶意爬虫产生的“背景辐射”已严重恶化。目前该服务在五个地理分布节点上,有 14 个 CPU 核心完全用于将 Git 提交渲染为 HTML,其消耗的 CPU 周期已超过所有其他合法访问(包括 Git 克隆)的总和。Ryabitsev 担忧此现象对 Datasette 的影响,后者服务于大量可爬取的网页。
EVENT DOSSIER
On September 7, 2026, Simon Willison published an article titled “Creepy Crawlies,” which systematically analyzed the hidden risks in the technology sector. The article noted that traditional web crawlers have evolved into automated data theft tools, capturing public information and exploiting API vulnerabilities and proxy pools to bypass anti-crawling mechanisms. In addition, AI models are used to build covert monitoring networks, masquerading as normal user behavior to collect data. Willison emphasized that although these technologies are not completely legal, they are widely present in the gray area, posing substantial threats to privacy and data security.
Konstantin Ryabitsev 指出,从 Linux 官方 Git 仓库 git.kernel.org 角度看,恶意爬虫产生的“背景辐射”已严重恶化。目前该服务在五个地理分布节点上,有 14 个 CPU 核心完全用于将 Git 提交渲染为 HTML,其消耗的 CPU 周期已超过所有其他合法访问(包括 Git 克隆)的总和。Ryabitsev 担忧此现象对 Datasette 的影响,后者服务于大量可爬取的网页。