Controlling Output Rankings in LLM-Based Search via Generative Engine Optimization

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Controlling Output Rankings in LLM-Based Search via Generative Engine Optimization".

Jane: Here's the gist of the paper. The gist The CORE method proposes an optimization technique to control output rankings in LLM-based search by appending strategically designed content to retrieved items, which has been shown to achieve high promotion success rates across various LLMs and product categories.

Tom: Problem Definition The core problem addressed is that the final recommendations Alice sees are still largely determined by the initial order of results returned by the search engines, even after an LLM summarizes and reranks them The ranking objective is defined as L(q, I; T(i⋆)) = − log pθ z(T(i⋆)) q, I where i⋆ is the designated target item with associated text T(i⋆).

Jane: Optimizing T(i⋆) to minimize L increases the likelihood that the generative engine ranks the target item more favorably.

Tom: Optimization Strategy CORE incorporates three types of optimization content to steer the ranking of outputs: string-based, reasoning-based, and review-based.

Paper summary: Tom: So we're talking about this paper called Controlling Output Rankings in gEnerative Engines for LLM-based search. We started by saying that customers are using LLMs for product recommendations, and that these recommendations rely heavily on the initial order of results from those search engines Jane The central problem is that this initial ordering disadvantages small businesses and independent creators because their products can get buried in the retrieval results Lu

Lu: To solve this, they propose CORE, an optimization method for Controlling Output Rankings in gEnerative Engines for LLM-based search Tom The main idea is that since we can't control the LLM or the retrieval process, we focus on manipulating the text of those search results themselves Jane

Meng: They claim that CORE optimizes retrieved content by appending strategically designed optimization content to steer the ranking of outputs Tom Specifically, they introduce three types of optimization content: string-based, reasoning-based, and review-based Jane

Lu: The paper shows that these three types of added content are effective at shaping the output rankings Meng They show that these methods can be used in two main ways depending on computational resources available Tom One is a shadow-model solution where they use another model to mimic the synthesizing LLM and compute rankings directly using gradients Jane

Jane: And there’s the second way, which is a query-based solution that uses LLMs to approximate gradient propagation without needing access to the model’s internals Lu It sets up an iterative loop where you start with a draft ranking and then iteratively append content and ask an optimizer for feedback Tom

Meng: So it’s essentially a generator initializing a draft, appending text, asking the LLM to rank it, and then using an optimizer to refine that draft until it matches the target ranking Jane It sounds like a lot of moving parts when you think about shipping this to production Lu

Tom: Yeah, but they found some pretty solid performance numbers on ProductBench Meng They tested this on four LLMs: GPT-4o, Gemini-two point five, Claude-four and Grok-three Jane Under the best optimization strategy, CORE achieved an average promotion success rate of ninety-one point four percent at Top-five across all fifteen product categories Lu

Jane: And they also saw results of eighty-six point six percent at Top-three and eighty point three percent at Top-one Tom That’s a pretty impressive spread across different LLMs and product types, showing the method is quite robust Meng

Lu: The paper also compares CORE against prior baselines like STS, TAP, SRP, and RAF across models and ranking positions Tom They found that CORE's reasoning and review strategies consistently outperformed those existing methods in terms of promotion success rate Jane For example, GPT-4o achieved a Top-one PSR of ninety-one point four percent with CORE-Reasoning Meng

Tom: It’s good to see that the mechanism they designed actually works against established methods when you look at the specific content they appended Lu But we still have to be careful because those results are on a specific benchmark Jane

Meng: That’s true. The paper mentions that in practice, one of the challenges is that developers choose which search engines to use and those choices are outside end-user control Tom So, we can't just assume this works everywhere without knowing the backend setup Jane

Lu: And they also point out that widely deployed LLMs operate in black-box settings where users only interact through queries and outputs Meng This is why their approach focuses on what we can actually influence: the text of the items shown to us Tom

Jane: So, to summarize this paper, it’s about proposing CORE as an optimization method for Controlling Output Rankings in gEnerative Engines for LLM-based search by focusing on appending specific types of content like string-based, reasoning-based, and review-based ones Lu It’s about using these content types to steer the ranking outcomes Tom

Meng: That sounds like a lot of work to get implemented, but if the results hold up when we scale it up, it could be a useful tool for anyone trying to improve their visibility in these systems Jane

Lu: The authors also provide theoretical analysis on how this works and show that the performance gap between models scales linearly with the model mismatch delta Tom This suggests there’s a predictable relationship between the text additions and the final ranking outcome Jane

Conclusion: Tom: So, wrapping up this paper Controlling Output Rankings in gEnerative Engines for LLM-based search, we learned that they developed CORE to control output rankings by treating ranking as an optimization problem Jane They focused on influencing the search results themselves rather than trying to control the LLM’s internal workings Lu

Lu: The authors created a method with shadow models and query-based solutions to steer how LLMs synthesize retrieved content Tom And they demonstrated that this approach can yield promotion success rates of ninety-one point four percent at Top-five eighty-six point six percent at Top-three and eighty point three percent at Top-one across fifteen product categories Jane

Meng: It’s a concrete result showing the effectiveness of adding reasoning and review content to steer rankings Tom This is something we can actually use to try and improve visibility for our own products Lu But we still have some caveats, like the fact that review content is still the strongest performer under defense mechanisms Jane

Tom: So, while they explored defense methods like filtering by perplexity or pattern-based filters, CORE still performed well against them Lu The main implication is that in LLM-based search systems, where final recommendations are heavily influenced by the initial retrieval order Jane controlling the content we feed into those systems is a viable way to fight against being ranked late in the list Meng

Jane: It shows that these LLM-based search systems are mediating product discovery more than just listing items for us Tom So, this paper on Controlling Output Rankings in gEnerative Engines for LLM-based search gives us a method to intervene in that mediation process Lu