Grass founder discusses challenges of web scraping for AI data
Summary
Grass founder Andrej Radonjic, known as @0xdrej, recently discussed the complexities of web scraping and its impact on AI data during an episode of the New Delphi Podcast. He emphasized that while anyone can deploy a crawler to scrape vast amounts of web pages, much of the content available online is of low quality and not useful for training AI. Radonjic noted that the rise of AI has transformed the search landscape by making indexing less costly, yet has simultaneously resulted in websites implementing stricter blocks against crawlers, complicating the process of retrieving live data. This shift highlights that AI labs increasingly prioritize high-quality, valuable data over mere volume when sourcing their training material.
Tokens
$GRASS
Analysis
Grass: Grass is a decentralized web data platform powered by millions of opted-in residential nodes that crawl the internet while appearing as normal user traffic to websites. It supplies exabytes of web data for AI training to frontier labs and multiple Mag7 companies, establishing training data as a core profitable business. The project is extending beyond data delivery with a live Contents API, upcoming search features, and multimodal grounding capabilities. Yan Liberman: Yan Liberman hosts the Delphi Podcast and conducts in-depth interviews with AI and crypto leaders, including the episode featuring Grass founder Andrej Radonjic on web data, scraping, and AI applications. Andrej Radonjic: Andrej Radonjic, known online as @0xdrej, is the founder of Grass and brings a background in nuclear physics to the development of advanced web scraping and data systems. He leads efforts to address the shift in AI data challenges, where acquiring live web pages has become harder due to site blocking while indexing grows easier. Data Quality Focus: AI labs prioritize valuable, high-signal web data over sheer volume or low-quality content when sourcing training material. Search Industry Shift: AI tools have inverted traditional search economics, making indexing inexpensive while website blocking has elevated the difficulty of reliably fetching live pages.
Categories
aitechmachine_learning