From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet

arXiv:2511.18354 · cs.NI, cs.AI, cs.IR · Submitted 2025-11-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "From Crawled to Published".

Jane: Here's the gist of the paper. The gist The rise of Generative AI Search is fundamentally transforming how users and intelligent systems interact with the Internet, yet the web remains optimized for human browsing rather than AI-driven semantic retrieval, resulting in wasted network bandwidth, lower information quality, and unnecessary complexity for developers.

Tom: AI-Native Internet Concept The paper introduces the concept of an AI-Native Internet as a web architecture where servers expose semantically relevant information chunks instead of full documents This paradigm shifts the focus from document fetching to semantic access, proposing three key architectural directions These directions include Restructured Web Sources, a Web-Native Semantic Resolver, and First-Class Semantic Abstractions Key Architectural Directions The proposed architecture aims to address the inefficiencies of current HTML-based retrieval by focusing on semantic primitives Restructured Web Sources: Servers could expose semantically chunked units of information, such as text blocks, claims, tables, and metadata, already vectorized and indexed in vector databases This approach is intended to reduce redundancy and bandwidth waste by serving only the most relevant sources and fine-grained semantic chunks rather than entire pages Web-Native Semantic Resolver: Instead of serving only URLs and documents through Domain Name System (DNS), the web could provide a semantic discovery API that returns the most relevant sources for a given query This allows AI applications to obtain “semantic pointers” before retrieving fine-grained chunks, optimizing retrieval pipelines for reasoning rather than rendering First-Class Semantic Abstractions: The architecture should elevate semantic retrieval to a first-class abstraction, allowing developers to focus on reasoning and orchestration rather than bespoke scraping and parsing This abstraction would enable applications to discover relevant sources and retrieve fine-grained information chunks directly from a standardized semantic substrate of the web Design Goals and Components The design goals for the AI-Native Internet center on Data Efficiency, Improved Response Quality Through Scoping, First-Class Semantic Abstractions, and Democratized Access to Semantic Retrieval Query processing involves steps such as De-anonymization, Rephrasing, and Decomposition to optimize queries for semantic resolution The web servers themselves are envisioned as being AI-native, capable of receiving a query and replying with the relevant top-k pieces of information across all data types Experimental Findings Experiments comparing centralized and decentralized models show that semantic retrieval avoids substantial waste by returning only the most relevant sources and fine-grained semantic chunks rather than entire pages, demonstrating a 74%-87% reduction in data transfer needed for comparable accuracy In the decentralized model, more chunks need to be retrieved to achieve the same performance compared to the centralized setting, suggesting that retrieving information from many sources rather than just the top sources can be beneficial Governance and Challenges Key challenges remain regarding Governance of the Semantic Resolver, which introduces new issues around trust, transparency, and ranking bias Trust interacts with both stages of retrieval: the semantic resolver returns sources with provenance metadata, while LLM applications apply their own re-ranking across both sources and chunks Furthermore, semantic chunks must carry verifiable provenance, including lineage and cryptographic signatures, to allow LLMs to track origins and licenses Publisher incentives require simple tooling that allows website operators to export chunked representations without a full redesign of existing content Coexistence with Existing Web The architecture must integrate smoothly with current HTTP infrastructure and CDNs, allowing publishers to expose semantic chunks alongside existing HTML without disrupting legacy traffic The vision is for an AI-Native Internet to coexist with the human-facing web, where HTML pages remain the primary representation for human browsing while semantic manifests provide a parallel view optimized for LLMs and agents The paper argues that this vision paper sketches an architectural direction rather than presenting a complete system, inviting deeper exploration across protocols, incentives, provenance, and deployment.

Jane: The work posits that semantic retrieval should become a foundational network primitive rather than an inefficient layer retrofitted atop HTML.

Tom: Page 6 --- Muhammad Bilal, Zafar Qazi, and Marco Canini 3.

Paper summary: Jane: Looking at this paper, "From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet," the authors are essentially proposing that we need to rethink how the web is built for AI interaction.

Tom: They claim that the current setup, optimized for human browsing, creates inefficiencies like wasted network bandwidth and lower information quality when AI agents are trying to find things semantically.

Lu: The main proposal is an AI-Native Internet where servers expose semantically relevant information chunks instead of full documents, supported by a semantic resolver that helps agents discover sources before retrieving the fine-grained content.

Meng: It’s about moving from fetching whole pages to accessing specific, meaningful units like claims or tables directly through controlled publisher interfaces.

Lalam: The implication for people who only listen to this is that the search process moves away from blindly crawling everything toward a more targeted, meaning-based discovery system.

Jane: They show experimental results where their model achieves up to four point two times higher accuracy and saves significant data compared to centralized retrieval pipelines <ref:2511.18354#pg2>.

Tom: So, the paper’s main message is that the web lacks a native layer for publisher-controlled semantic discovery and retrieval, and they are proposing this as a way forward.

Lu: They laid out architectural directions like Restructured Web Sources and First-Class Semantic Abstractions to guide how we evolve this document-centric web into an AI-oriented substrate.

Meng: The key challenge they flagged is getting the governance right around the semantic resolver, ensuring trust and transparency when ranking sources is involved.

Lalam: They also made it clear that for this to work, publishers need simple tooling to export those semantically chunked representations without needing a total overhaul of their existing content systems.

Conclusion: Jane: So, we've been looking at this paper from arXiv called "From Crawled to Published: A Publisher-Controlled Web Architecture for an AI-Native Internet."

Tom: Yeah, it’s about a structural problem with how the web is built for AI agents. The authors argue that the current setup isn't serving what those agents actually need.

Lu: Basically, they say we need a native layer for publisher-controlled discovery and retrieval. It’s not just about better chunking algorithms, right?

Meng: Exactly. They propose an architecture where publishers expose semantic units directly—like specific claims or tables—through interfaces they control. That’s different from what we see now where everything is just scraped content.

Lalam: And they suggest a web-native semantic resolver, kind of like DNS, but instead of location, it points to meaning. It lets agents find the right sources before pulling the fine-grained data.

Tom: The experimental results are pretty solid. They show their end-to-end prototype matches or beats centralized retrieval pipelines in answer quality on some specialized documents.

Jane: That's impressive, but what does this actually mean for us as users? I mean, if publishers can control how their content is represented...

Tom: It means attribution and usage policies travel with the content because it terminates at publisher servers. That realigns the whole incentive structure across the web ecosystem.

Lu: They’re aiming for data efficiency too. By only returning relevant chunks instead of entire pages, they see a significant reduction in data transfer needed for comparable accuracy—around seventy-four to eighty-seven percent less data moved.

Meng: But there are hurdles on the governance side. How do you get publishers to adopt this? They need simple tooling to export these chunked representations without needing a complete overhaul of their existing systems, and they’re worried about trust in that semantic resolver.

Lalam: I'm confident in the direction they're pushing. The core idea is moving from fetching documents to accessing meaning directly. It’s about making the web native for AI interaction instead of just scraping it for us.

Jane: It feels like a big step toward making the web work *for* intelligence, rather than forcing intelligence to squeeze through a system built for human eyes.

Muhammad Bilal, Zafar Qazi, Marco Canini

King Abdullah University of Science and Technology (KAUST) · LUMS

cs.NI, cs.AI, cs.IR

Submitted: 2025-11-23

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist The rise of Generative AI Search is fundamentally transforming how users and intelligent systems interact with the Internet, yet the web remains optimized for human browsing rather than

Key concepts

AI-Native Internet
A web architecture where servers expose semantically meaningful information chunks instead of full documents. This shifts the focus from fetching entire pages to accessing specific pieces of information, making the internet optimized for AI agents.
Restructured Web Sources
This involves web servers exposing semantically chunked units like text blocks, claims, and tables that are already vectorized and indexed. This reduces bandwidth waste by serving only the most relevant parts of a document instead of the whole page.
Web-Native Semantic Resolver
A discovery API analogous to DNS but operating over meaning rather than location. It returns relevant sources to AI applications before they retrieve fine-grained content, optimizing retrieval pipelines for reasoning.

Terminology

Summary

The gist The rise of Generative AI Search is fundamentally transforming how users and intelligent systems interact with the Internet, yet the web remains optimized for human browsing rather than AI-driven semantic retrieval, resulting in wasted network bandwidth, lower information quality, and unnecessary complexity for developers.

AI-Native Internet Concept

The paper introduces the concept of an AI-Native Internet as a web architecture where servers expose semantically relevant information chunks instead of full documents This paradigm shifts the focus from document fetching to semantic access, proposing three key architectural directions These directions include Restructured Web Sources, a Web-Native Semantic Resolver, and First-Class Semantic Abstractions

Key Architectural Directions

The proposed architecture aims to address the inefficiencies of current HTML-based retrieval by focusing on semantic primitives

** Restructured Web Sources: Servers could expose semantically chunked units of information, such as text blocks, claims, tables, and metadata, already vectorized and indexed in vector databases This approach is intended to reduce redundancy and bandwidth waste by serving only the most relevant sources and fine-grained semantic chunks rather than entire pages **

** Web-Native Semantic Resolver: Instead of serving only URLs and documents through Domain Name System (DNS), the web could provide a semantic discovery API that returns the most relevant sources for a given query This allows AI applications to obtain “semantic pointers” before retrieving fine-grained chunks, optimizing retrieval pipelines for reasoning rather than rendering **

** First-Class Semantic Abstractions: The architecture should elevate semantic retrieval to a first-class abstraction, allowing developers to focus on reasoning and orchestration rather than bespoke scraping and parsing This abstraction would enable applications to discover relevant sources and retrieve fine-grained information chunks directly from a standardized semantic substrate of the web **

Design Goals and Components

The design goals for the AI-Native Internet center on Data Efficiency, Improved Response Quality Through Scoping, First-Class Semantic Abstractions, and Democratized Access to Semantic Retrieval Query processing involves steps such as De-anonymization, Rephrasing, and Decomposition to optimize queries for semantic resolution The web servers themselves are envisioned as being AI-native, capable of receiving a query and replying with the relevant top-k pieces of information across all data types

Experimental Findings

Experiments comparing centralized and decentralized models show that semantic retrieval avoids substantial waste by returning only the most relevant sources and fine-grained semantic chunks rather than entire pages, demonstrating a 74%-87% reduction in data transfer needed for comparable accuracy In the decentralized model, more chunks need to be retrieved to achieve the same performance compared to the centralized setting, suggesting that retrieving information from many sources rather than just the top sources can be beneficial

Governance and Challenges

Key challenges remain regarding Governance of the Semantic Resolver, which introduces new issues around trust, transparency, and ranking bias Trust interacts with both stages of retrieval: the semantic resolver returns sources with provenance metadata, while LLM applications apply their own re-ranking across both sources and chunks Furthermore, semantic chunks must carry verifiable provenance, including lineage and cryptographic signatures, to allow LLMs to track origins and licenses Publisher incentives require simple tooling that allows website operators to export chunked representations without a full redesign of existing content

Coexistence with Existing Web

The architecture must integrate smoothly with current HTTP infrastructure and CDNs, allowing publishers to expose semantic chunks alongside existing HTML without disrupting legacy traffic The vision is for an AI-Native Internet to coexist with the human-facing web, where HTML pages remain the primary representation for human browsing while semantic manifests provide a parallel view optimized for LLMs and agents

The paper argues that this vision paper sketches an architectural direction rather than presenting a complete system, inviting deeper exploration across protocols, incentives, provenance, and deployment. The work posits that semantic retrieval should become a foundational network primitive rather than an inefficient layer retrofitted atop HTML.

--- Page 6 ---

Muhammad Bilal, Zafar Qazi, and Marco Canini

  1. Return Source Addresses

  2. Find Sources(Query)

  3. Retrieve Under Utility Constraints(Query)

  4. Return Data Chunks

Semantic Resolver Data Sources

Users... Chunked, and Vectorized Multimodal Data 1. Query Processing AI Agents / LLMs Apps 2 AI-NATIVE INTERNET Figure 1: Basic architecture for AI-Native Internet

--- Page 7 ---

2.1 Design Goals Below, we discuss the key design goals for AI-Native Internet: • Data Efficiency: Minimize unnecessary crawling and data transfer by retrieving only semantically relevant sources and finegrained chunks rather than full documents • Improved Response Quality Through Scoping: Increase answer accuracy by narrowing retrieval to relevant domains, time periods, usage licenses, and trusted publishers • First-Class Semantic Abstractions: Developers increasingly want to focus on reasoning and orchestration rather than bespoke scraping and parsing • Democratized Access to Semantic Retrieval: Provide Internetwide access to semantic retrieval capabilities so that developers, agents, and applications can benefit without relying on proprietary crawlers or closed APIs from a few large providers

--- Page 8 ---

2.2 Components of AI-Native Internet In this section, we discuss the main components of an AI-Native Internet and the design choices that exist Figure 1 shows the basic architecture of how we envision an AI-Native Internet

--- Page 9 ---

2.2.1 Query Processing As shown in Figure 1, the user query is first received by the AI Agent or LLM-based web search app This Agent/App could run locally or as a third-party service The user query can be used directly for search but for efficiency and effectiveness, it must be processed before it can be used for semantic resolution and to query web sources The query processing includes one or more of the following steps, depending on the requirements • De-anonymization: Remove any PII from the query • Rephrasing: Rephrase the query to remove ambiguity and optimize for search • Decomposition: Decompose a complex query into individual, simpler queries that are optimized for efficiently searching relevant sources and information chunks

--- Page 9 ---

Figure 2: Queries (out of 200) that were rephrased, decomposed, or left unchanged by GPT-5-mini

--- Page 9 ---

Top 1 Top 5 Top 10 Top 30 Top 120 Full Context Chunk Configuration Context Present Count hotpotqa triviaqa truthfulqa

--- Page 9 ---

2.2.2 Web Servers/Web sources One of the main things we are proposing is that the web servers serving webpages, files, and data have to become AI-native We envision that these web servers will have the following characteristics instead of or in addition to serving webpages • Receive a query and reply with the relevant top-k pieces of information • Make all types of information searchable, such as images, files, audio, video, databases etc.

Improvements for AI systems

  1. Restructured Web Sources: Servers could expose semantically chunked units of information, such as text blocks, claims, tables, and metadata already vectorized and indexed in vector databases. This reduces redundancy and bandwidth waste by serving only semantically relevant information chunks rather than full documents.

  2. Web-Native Semantic Resolver: The web could provide a semantic discovery API that returns the most relevant sources for a given query instead of just URLs. AI applications would obtain semantic pointers before retrieving fine-grained chunks, enabling retrieval pipelines optimized for reasoning rather than rendering.

  3. First-Class Semantic Abstractions: Developers can focus on reasoning and orchestration rather than scraping and parsing by interacting with a standardized semantic substrate of the web. This involves exposing semantic retrieval abstractions such as: given a query and intent constraints (e.g., exclude copyrighted material, require signed provenance), return the set of semantically relevant sources.

Abstract

AI agents, not humans, are becoming the web's primary consumers, yet the web still ships content structured for human browsing, with no native way for publishers to expose the semantic units within it. Access is therefore mediated by a handful of AI providers that crawl, chunk, and index content unilaterally, leaving publishers little control over how their content is represented, retrieved, or attributed. We argue this is a structural problem, not one better chunking or retrieval algorithms can solve. The web lacks a native layer for publisher-controlled semantic discovery and retrieval. We propose AISTRAL, an AI-Native Internet architecture that supplies this layer. Publishers expose semantically meaningful units (e.g., passages, claims, tables) through explicit, policy-governed interfaces they control, and a web-native semantic resolver, analogous to DNS but operating over meaning rather than location, lets agents discover relevant sources before retrieving fine-grained content directly from publishers. Our end-to-end prototype matches or exceeds representative centralized retrieval pipelines in answer quality, with up to 4.2x higher accuracy on large, specialized document collections. Against a public web-search agent, it answers up to 4.2x more questions correctly, while moving 2.9-41x less data per query. Because retrieval terminates at publisher-operated servers, attribution, provenance, and usage policies travel with the content, realigning incentives across the web ecosystem.

Sources

Related papers