TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings

Comprehensive Research Summary of TILE (Table-Interfaced LLM Embeddings)

As a diligent researcher, I have thoroughly analyzed the provided excerpts from the paper detailing TILE (Table-Interfaced LLM Embeddings). This architecture represents a significant advancement in bridging the gap between powerful Large Language Models (LLMs) and traditional Table Foundation Models (TFMs), specifically addressing the challenge of heterogeneous real-world tables that combine numerical features with free-form string columns.

Core Problem Addressed

The fundamental limitation identified is that existing methods struggle to effectively utilize LLM semantic knowledge for tabular predictions when dealing with tables containing strings. While LLMs possess rich semantic understanding, they are not inherently designed for structured tabular prediction tasks. Current approaches often rely on using LLMs as simple plug-in encoders for cell strings, which fails to extract the most effective representations needed for downstream tabular learning. The core challenge is how to extract meaningful, context-aware representations from these heterogeneous string columns without losing the structural information of the table or introducing numerical imprecision into string embeddings.

TILE Architecture: The Four Key Ingredients

TILE introduces a novel framework that combines four specific architectural choices to effectively interface LLMs with tabular learning:

1. Row Serialization (Context Provision): To combat cell isolation—where individual cell encoders fail due to insufficient context—TILE serializes the table by concatenating descriptive strings for every cell across all M columns (e.g., The is ). This serialization provides crucial row-level context to every cell representation, ensuring that the LLM has a holistic view of the data structure.

2. Intermediate-Layer Extraction (Semantic Focus): Instead of using the final layers specialized for next-token prediction, TILE extracts representations from an intermediate layer of the LLM. This choice is deliberate: these earlier layers are argued to carry more transferable semantic information relevant to general contextual understanding than the output layer.

3. Columnar Pooling (Structural Integrity): Row serialization inherently flattens the table structure into a long sequence, which can be detrimental for tabular learning. TILE employs columnar pooling, which allows each string column to generate its own distinct vector representation (j). This restores the essential column structure and mitigates the length bias associated with global mean pooling, where verbose columns can disproportionately dominate the representation.

4. By-pass Features (Numerical Precision): To correct for numerical imprecision and potential representation drift across rows stemming from string embeddings, TILE incorporates by-pass features (x num, x cat, x te). These features route the exact numerical values and categorical identities directly around the main LLM processing path, ensuring their precise information is preserved and injected into the final learning stage without being distorted by the LLM's string encoding.

Performance and Impact

The synergy of these four ingredients allows TILE to achieve superior performance:

• State-of-the-Art Results: Across 100 diverse datasets, TILE has demonstrated state-of-the-art performance, beating the strongest baselines on 89% of splits. It ranks first on the external STRABLE benchmark (a 108-dataset test), even surpassing an oracle that selects the best of 51 baselines per dataset.

• Scalability Advantage: A key finding is that Scale pays off only through the right interface. TILE's advantage over a baseline LLM grows significantly with model size—from 4.2 points for Qwen3-0.6B up to 16.7 points for Qwen3-8B—demonstrating that the LLM gains are maximized when paired with TILE's specific interface design.

• Versatility: TILE is highly versatile, showing consistent gains across a wide spectrum of models, including various LLMs (Llama families and Qwen family) and diverse tabular learners (TFMs like TabPFN-2.5/3, RealMLP, XGBoost, Ridge regression).

• Efficiency: In terms of hardware requirements, TILE is efficient; the LLM stage is the primary constraint on memory usage, with truncation nearly halving the memory footprint compared to a cell-wise LLM approach.

Conclusion: The Paradigm Shift

The central thesis of TILE is that what has held back LLMs on tables is their interface to the table. By providing row-level context, leveraging intermediate layers, preserving column structure via columnar pooling, and carefully routing numerical data via by-pass features, TILE successfully transforms LLMs into powerful contextual encoders for tabular data. This results in representations that are not only informative due to structural context but also robust against representation drift. TILE establishes a new benchmark for how LLMs can be effectively adapted to the complex, heterogeneous reality of real-world tabular data.

1. textbfRow Serialization for Contextual Understanding of Strings in Tables (TILE Interface): This allows each row to become a sequence of blocks 'The is,' so that the LLM’s own attention conditions each cell on the column headers and on the rest of the row. This resolves cell isolation that leaves cell-wise encoders with strings too short to disambiguate, enabling LLMs to resolve ambiguous string entries like E.T. by providing row-level context.

2. textbfIntermediate-Layer Extraction for Transferable Semantics: Instead of relying on the output layer, TILE implements this by reading one layer per LLM, chosen once (14 of 32 for Llama-3.1-8B; Section 5.2) because intermediate layers carry more transferable semantics than either end. This ensures the TFM learns from richer, more general representations rather than outputs specialized for next-token prediction.

3. textbfColumnar Pooling to Preserve Table Structure: To restore the structure lost during serialization, TILE uses columnar pooling where the cell embedding is cj = 1/Ij X i∈Ij hi, which restores it by giving each string column its own vector. This avoids the bias of mean pooling, which dilutes the signal of short but informative ones.

4. textbfBy-pass Features for Numerical and Categorical Information: TILE implements by-pass features to route numbers and categories past the LLM, ensuring that numerical columns are passed directly to the TFM for exact access, and categorical features bypass the embedding process where they can be target-encoded or passed as native features. This corrects numerical imprecision and mitigates representation drift where a string’s embedding depends on its row.

Real-world tables are often heterogeneous, combining numerical features with free-form string columns. Table foundation models (TFMs) excel on numerical tables but require strings to be encoded numerically, while large language models (LLMs) hold semantic knowledge but are not designed for tabular predictions. Most existing bridges use LLMs as plug-in encoders for cell strings, yet how to extract representations effectively for tabular learning remains underexplored. We show that the interface between LLMs and tables matter as much as the LLM itself. Applied cell-wise, larger LLMs do not give better features. With our interface, the same LLMs improve steadily with size. TILE (Table-Interfaced LLM Embeddings) combines four ingredients: row serialization to provide cross-column context, intermediate-layer extraction to avoid layers specialized for next-token prediction, columnar pooling to remove mean pooling's length bias, and by-pass features to route numbers and categories past the LLM. Paired with TFMs, TILE sets a new state of the art for tables with strings. Across 100 datasets, it beats the strongest baseline on 89% of splits, with gains across 7 LLMs and 10 tabular learners, and ranks first on the 108-dataset STRABLE benchmark.