Paper Reference

OneSearch — reading notes on search-specific Semantic IDs, the Stage-3 generative retriever, and preference alignment via DPO.

TL;DR

1. Search SID is different from RecSys SID

The core problem in search is not only representing items semantically, but representing queries and items in the same relevance-aware space.

For recommendation, SID construction can mostly focus on:

[ \boxed{ item \rightarrow embedding \rightarrow SID } ]

where the embedding primarily captures item semantics and collaborative relationships.

For search, OneSearch needs:

[ \boxed{ query \leftrightarrow item } ]

alignment before quantization. Both query and item representations are mapped into the same embedding space and then the same SID/codebook space:

[ q \rightarrow e_q \rightarrow SID_q ]

[ i \rightarrow e_i \rightarrow SID_i ]

so a relevant query and item can occupy nearby / structurally related regions in both continuous and discrete spaces.

Search also needs to preserve finer distinctions than recommendation. Two items may be almost interchangeable for recommendation but very different for a strict search query:

Women's black Nike Pegasus
vs
Men's white Nike Pegasus

Therefore OneSearch uses:

[ \boxed{ RQ + OPQ } ]

where RQ captures the shared hierarchical semantic structure and OPQ preserves the remaining residual / distinctive information that may be query-critical.

Another important experimental observation from the paper is that the SID space appears relatively stable under item shifting and query shifting. Their experiments suggest that moderate changes in the item/query distribution have limited impact on the existing SID space, so the tokenizer/codebooks do not necessarily need to be retrained every time the catalog or query distribution shifts.

My takeaway is:

[ \boxed{ \text{Search SID}

\text{query-item aligned embedding} + \text{shared discrete SID space} + \text{fine-grained distinctive information} } ]

rather than simply applying a RecSys item tokenizer to search.

2. What DPO is doing in OneSearch

Stage-3 SFT already makes OneSearch a functional generative search model:

[ x=(user,query,history) \rightarrow P_\theta(SID_i|x) ]

Beam search converts those sequence probabilities into a ranked list:

[ [1,2,3,4,5]. ]

A stronger personalized ranking/reward model may prefer:

[ [2,4,1,3,5]. ]

DPO does not modify the reward model and does not modify the beam-search algorithm.

Instead, DPO fine-tunes the OneSearch model itself so its SID sequence probabilities increasingly reflect the teacher’s preference ordering:

[ P_\theta(SID_2|x) \uparrow ]

relative to:

[ P_\theta(SID_1|x),; P_\theta(SID_3|x),\dots ]

OneSearch uses listwise preference information rather than only one independent positive-negative pair.

So the most useful mental model is:

[ \boxed{ \text{personalized ranker / reward model} \rightarrow \text{listwise preference order} \rightarrow \text{DPO} \rightarrow \text{OneSearch} } ]

DPO is therefore effectively distilling ranking-order knowledge back into the generative OneSearch model.

After preference training, the desired serving path is still:

query + user + history
        ↓
OneSearch
        ↓
beam search
        ↓
final top-K items

rather than:

OneSearch
        ↓
reward model
        ↓
reranking

The reward model is primarily a training-time teacher.

1. The Mental Model I Ended Up With

The most useful way for me to understand OneSearch is to separate it into four layers:

[ \boxed{ \text{Representation / SID} \rightarrow \text{Generative Search} \rightarrow \text{Reward Model} \rightarrow \text{Preference Alignment} } ]

More concretely:

query / item information
        ↓
search-aware embedding
        ↓
Semantic ID (SID)
        ↓
Stage-3 generative model
        ↓
beam search → top-K item SIDs
        ↓
reward model reranking / preference construction
        ↓
DPO / listwise preference fine-tuning
        ↓
OneSearch itself produces a better top-K ordering

The important conceptual distinction is:

• SFT teaches the model which items are plausible.
• DPO teaches the model how to prefer/rank plausible items.
• The reward model is primarily a training-time teacher, not necessarily an online serving component.

2. Search SID: Why It Is Different from Recommendation SID

2.1 Query SID and Item SID Need a Shared Representation Space

For search, relevance is conditioned on the query:

[ P(i \mid q, u) ]

rather than only:

[ P(i \mid u) ]

as in recommendation.

Therefore, query and item representations should live in a shared search-aware embedding space:

[ q \rightarrow e_q ]

[ i \rightarrow e_i ]

such that a relevant query-item pair satisfies approximately:

[ e_q \approx e_i ]

The point is not simply semantic similarity. The representation should capture search relevance, including behavioral relationships and fine-grained attributes.

Then query and item should also reuse the same quantization/codebook space:

[ SID_q = Q(e_q) ]

[ SID_i = Q(e_i) ]

This gives search a shared continuous and discrete representation space.

2.2 Keyword / Facet Enhancement

A Transformer encoder can already convert a full query or item description into one contextual embedding:

[ e_{\text{global}} = Encoder(\text{full text}) ]

This embedding already contains:

• word information,
• contextual relationships,
• word order,
• global sentence meaning.

Therefore, keyword enhancement is not needed because the Transformer cannot understand individual words.

The purpose is to explicitly emphasize search-critical attributes that might otherwise be diluted in the global embedding.

For an item such as:

2026 hot-selling Nike Pegasus women's breathable sports shoes,
black, official flagship, discount...

the system may extract structured facets such as:

brand: Nike
model: Pegasus
audience: Women
color: Black
function: Running

A useful mental model is:

[ \boxed{ \text{raw contextual representation} + \text{structured facet enhancement} } ]

rather than:

[ \text{replace raw text with independently averaged words} ]

Conceptually:

[ e_{\text{global}} = Encoder(\text{full item/query}) ]

[ e_{\text{facet}}

\frac{1}{K}\sum_{j=1}^{K} Encoder(k_j) ]

and then:

[ e^o = Combine(e_{\text{global}}, e_{\text{facet}}) ]

The facet branch loses some word-order/context information if attributes are encoded independently, but it does not replace the global contextual embedding. Its job is to bias the final representation toward exact attributes that matter for search relevance.

3. How a Continuous Embedding Becomes an SID List

Suppose the final search-aware representation is:

[ e \in \mathbb{R}^d ]

OneSearch uses residual quantization-style codebooks.

Assume three RQ codebooks:

[ C^{(1)}, C^{(2)}, C^{(3)} ]

with codebook sizes:

[ 4096,\ 1024,\ 512 ]

respectively.

Level 1

Find the nearest centroid:

[ z_1

\arg\min_j |e-c_j^{(1)}|^2 ]

Suppose:

[ z_1=137 ]

Then the first SID token is:

[ SID_1=137 ]

Compute the residual:

[ r_1=e-c_{137}^{(1)} ]

Level 2

Quantize the residual:

[ z_2

\arg\min_j |r_1-c_j^{(2)}|^2 ]

Suppose:

[ z_2=52 ]

Then:

[ SID=[137,52,\dots] ]

and:

[ r_2=r_1-c_{52}^{(2)} ]

Level 3

Again:

[ z_3

\arg\min_j |r_2-c_j^{(3)}|^2 ]

Suppose:

[ z_3=301 ]

Then:

[ SID_{RQ}=[137,52,301] ]

and approximately:

[ e \approx c_{137}^{(1)} + c_{52}^{(2)} + c_{301}^{(3)} ]

The SID integers are simply indices into learned codebooks.

They should not be interpreted literally as:

137 = footwear
52  = running shoes
301 = women

Those semantics are emergent, not manually assigned.

4. Why Search Needs More Than Standard RQ SID

Suppose two items are extremely similar:

Item A:
Nike Pegasus 41
Women's
Black
Size 7

Item B:
Nike Pegasus 41
Men's
White
Size 11

Their embeddings may be close:

[ e_A \approx e_B ]

and standard RQ might give both:

[ SID^{RQ}_A=[137,52,301] ]

[ SID^{RQ}_B=[137,52,301] ]

For recommendation, this may not be disastrous. Both represent roughly the same product family.

For search, however, a query may be:

women black nike pegasus size 7

Now:

women vs men
black vs white
size 7 vs size 11

are not minor preference differences. They can determine relevant vs irrelevant.

This is the key search-specific problem:

[ \boxed{ \text{search SID must preserve finer distinctions than generic RecSys SID} } ]

5. RQ + OPQ: Preserve the Remaining Distinctive Information

After all RQ levels, there is still residual information:

[ r_{\text{final}}

e

\sum_{l=1}^{L} c_{z_l}^{(l)} ]

Standard RQ-style systems may effectively discard this remaining error.

OneSearch instead treats it as valuable because it may contain exactly the attributes that distinguish highly similar items.

The residual is quantized again using OPQ.

Conceptually:

[ r_{\text{final}} \overset{OPQ}{\longrightarrow} [o_1,o_2] ]

For example:

[ SID_A=[137,52,301,17,204] ]

[ SID_B=[137,52,301,83,71] ]

The first part:

[ [137,52,301] ]

captures shared hierarchical semantics.

The second part:

[ [17,204] ]

or:

[ [83,71] ]

preserves distinctive residual information.

Thus:

[ \boxed{ SID_{\text{OneSearch}}

[ \underbrace{z_1,z_2,z_3}{\text{shared hierarchy}}, \underbrace{o_1,o_2}{\text{distinctive residual}} ] } ]

My main takeaway:

Recommendation often benefits from grouping similar items together. Search must additionally distinguish fine-grained attributes because those attributes may be explicit query constraints.

6. SID Stability Under Query and Item Shifting

A practical question is whether the SID tokenizer/codebooks must be retrained whenever:

• the item catalog changes,
• the query distribution changes,
• new traffic patterns appear.

OneSearch reports experiments around item shifting and query shifting.

The important takeaway from those experiments is that moderate shifts have relatively limited impact on the SID space. In other words, the learned semantic/codebook space appears stable enough that new or shifted queries/items can often be projected into the existing SID space without rebuilding the tokenizer from scratch.

This matters operationally because otherwise a generative-retrieval system would suffer from a major maintenance problem:

query/catalog distribution changes
        ↓
retrain SID tokenizer
        ↓
all item SIDs change
        ↓
retrain downstream generative model

The paper’s results suggest this full retraining loop is not required for every moderate shift.

A useful practical takeaway is:

[ \boxed{ \text{SID tokenizer should behave like a relatively stable representation layer} } ]

rather than a component that needs constant retraining.

7. A Question I Still Have About User-ID Construction

OneSearch constructs a behavior-based user representation using recency-weighted aggregation of behavior SIDs.

Conceptually:

[ SID_u

\left\lceil \sum_i \lambda_i SID_i \right\rceil ]

where more recent behavior gets larger weight.

This is interesting, but it also raises an important concern.

A K-means cluster index is normally categorical, not ordinal.

For example:

[ SID=35 ]

and:

[ SID=42 ]

do not imply that 35 and 42 are semantically close.

Cluster IDs could be arbitrarily permuted without changing the clustering.

Therefore:

[ 0.5 \times 35 + 0.5 \times 42 ]

does not naturally have the same geometric meaning as averaging the actual centroid vectors:

[ 0.5c_{35}+0.5c_{42} ]

A geometrically cleaner method would be:

[ e_u

\sum_i \lambda_i c_{SID_i} ]

then re-quantize:

[ SID_u = Q(e_u) ]

So one question I would keep for further reading is:

How is direct arithmetic over categorical SID indices justified? Is there an implementation-side reordering/remapping step not fully explained in the paper?

8. Stage 3: Personalized Generative Search

After SID pretraining/alignment stages, Stage 3 learns the actual personalized search task.

The input is approximately:

[ x= [ uid,, q,, SID_q,, Seq_q,, Seq_{short},, Seq_{long}^{emb} ] ]

The output label is the SID of a target positive item:

[ y = SID_i ]

Suppose:

[ SID_i=[42,19,27,83,71] ]

The deployed architecture is an encoder-decoder Transformer.

The encoder consumes the full context.

The decoder predicts the SID autoregressively:

[ P(42|x) ]

then:

[ P(19|x,42) ]

then:

[ P(27|x,42,19) ]

and so on.

Therefore:

[ P(SID_i|x)

\prod_t P(s_t|x,s_{<t}) ]

and the SFT loss is:

[ L_{\text{SFT}}

-\sum_t \log P_\theta(s_t|x,s_{<t}) ]

Stage 3 is therefore essentially learning:

[ \boxed{ P(item\ SID \mid query,user,history) } ]

9. How Stage-3 Training Data Differs from CTR/CVR Ranking Data

A traditional ranking model learns:

[ f(q,u,i) \rightarrow P(click), P(ATC), P(CVR) ]

It can use impression-level binary labels:

item A → click=1
item B → click=0
item C → click=0

Stage-3 generative training is different.

Its task is closer to next-positive-item prediction.

Suppose:

query q1 → click A
query q2 → click B
query q3 → purchase C

Possible training examples become:

[ [q_1, history] \rightarrow SID_A ]

[ [q_2, history + A] \rightarrow SID_B ]

[ [q_3, history + A + B] \rightarrow SID_C ]

The paper also uses sliding-window augmentation over short behavior sequences so that later behaviors can become prediction targets.

Thus:

[ \boxed{ \text{ranking model}: (q,u,i)\rightarrow binary / continuous engagement target } ]

versus:

[ \boxed{ \text{Stage-3 OneSearch}: (q,u,history)\rightarrow positive item SID } ]

10. How One Target Item Becomes a Top-K List at Serving Time

Training has one target SID per example, but inference uses beam search.

At decoder step 1, the model outputs a distribution:

42 : 0.40
17 : 0.30
91 : 0.15
...

Instead of greedily choosing only 42, beam search with beam size (B) keeps several prefixes.

For (B=3):

[ [42], [17], [91] ]

At the next step, expand each prefix.

For example:

[ P(19|x,42)=0.50 ]

[ P(33|x,42)=0.30 ]

Then:

[ score([42,19])

\log 0.40 + \log 0.50 ]

Continue through all SID positions.

Eventually beam search produces complete SID sequences:

[ SID_1 ]

[ SID_2 ]

[ SID_3 ]

…

Each sequence receives:

[ score(SID_i)

\sum_t \log P_\theta(s_{i,t}|x,s_{i,<t}) ]

Then the top-K sequences are ranked by this sequence score.

So:

[ \boxed{ [1,2,3,4,5]

\text{top-5 complete SID sequences by model probability} } ]

This ranking exists before DPO.

11. Constrained Beam Search

Because SID sequences correspond to catalog items, many arbitrary token combinations may not represent any real item.

A SID trie can encode valid catalog paths:

root
├── 42
│    ├── 19
│    │    ├── 27
│    │    │    ├── 17
│    │    │    └── 83
│    │    └── 31
│    └── 33
├── 17
│    └── ...

If the current prefix is:

[ [42,19] ]

and only:

[ {27,31} ]

are valid next tokens, all other decoder logits can be masked:

[ P(z_3\notin{27,31})=0 ]

This ensures beam search stays on valid catalog paths.

Serving therefore looks like:

user + query + history
        ↓
Transformer encoder
        ↓
autoregressive SID decoder
        ↓
constrained beam search
        ↓
top-K valid SIDs
        ↓
catalog lookup
        ↓
items

12. Why Stage-3 SFT Is Not Enough for Ranking

Stage-3 SFT says:

[ x \rightarrow i^+ ]

and optimizes:

[ -\log P(i^+|x) ]

This teaches:

Make this observed positive item likely.

But it does not directly teach:

[ i_1 > i_2 > i_3 > i_4 ]

among several plausible candidates.

This creates a training/evaluation mismatch.

SFT is naturally good at:

[ \boxed{ \text{which items belong in the plausible set?} } ]

while search metrics care strongly about:

[ \boxed{ \text{how should those plausible items be ordered?} } ]

This is where preference optimization enters.

13. Reward Model: A Traditional MTML Ranker Used as a Teacher

The reward model is conceptually much more conventional.

Its input looks like:

[ (user,\ query,\ history,\ candidate\ item) ]

and it predicts engagement signals such as:

[ CTR,\ CVR,\ CTCVR ]

A useful abstraction is:

[ h=f(user,q,history,item) ]

followed by multiple heads:

[ \hat y_{CTR}=\sigma(f_{CTR}(h)) ]

[ \hat y_{CVR}=\sigma(f_{CVR}(h)) ]

[ \hat y_{CTCVR}=\sigma(f_{CTCVR}(h)) ]

Each tower can use BCE-style training:

[ L_{CTR}

-y\log\hat y -(1-y)\log(1-\hat y) ]

The final reward/ranking score combines engagement predictions and a strong relevance term.

The main architectural takeaway:

[ \boxed{ \text{Reward model} \approx \text{ordinary search MTML ranker} } ]

It does not need to be another generative Transformer.

Technically, an existing Transformer-based global search ranker could also serve as this teacher if computational cost is acceptable.

14. Adaptive Reward vs Binary MTML Labels

The MTML towers can still be trained with binary labels.

For example:

clicked     → CTR label = 1
not clicked → CTR label = 0
converted   → CVR label = 1

The adaptive reward is a separate scalar used to encode preference strength.

Behavior levels can have different base rewards, for example:

search purchase        > recommendation purchase
                       > click
                       > impression/no click
                       > unshown
                       > random

Then engagement statistics such as CTR/CVR can further modify the magnitude.

Therefore two purchase items do not necessarily need to have exactly the same preference strength.

The important distinction is:

[ \boxed{ \text{MTML labels} = supervision for training reward-model heads } ]

versus:

[ \boxed{ \text{adaptive reward scalar} = strength/order used for preference construction } ]

15. How the Initial [1,2,3,4,5] Is Produced

After Stage-3 SFT, OneSearch is already a functional generative retrieval model.

Beam search produces:

[ [1,2,3,4,5] ]

ranked according to:

[ \log P_\theta(SID_i|x) ]

For example:

[ P(1|x)=0.30 ]

[ P(2|x)=0.25 ]

[ P(3|x)=0.20 ]

[ P(4|x)=0.15 ]

[ P(5|x)=0.10 ]

Thus the initial ordering is:

[ 1>2>3>4>5 ]

This does not come from the reward model.

It comes directly from the Stage-3 generative model.

16. Reward-Model Reranking

The reward model evaluates the same candidate set.

Suppose:

[ R(1)=0.62 ]

[ R(2)=0.91 ]

[ R(3)=0.55 ]

[ R(4)=0.83 ]

[ R(5)=0.20 ]

Then the teacher prefers:

[ 2>4>1>3>5 ]

This reranking can be done offline.

The system does not need to first expose [2,4,1,3,5] to users before the first DPO training phase.

This avoids a cold-start loop.

The existing production ranking model and historical logs can bootstrap the preference-training stage.

17. Positive and Negative Preference Samples

OneSearch can construct preference examples such as:

[ i_w \succ {i_{l1},i_{l2},i_{l3}} ]

A preferred item can come from signals such as:

• clicked item,
• purchased item,
• item moved upward by the teacher/reward model.

Negatives can include:

• items moved downward,
• low-ranked items,
• lower-engagement items.

An important nuance:

A low-ranked item that actually receives strong engagement should not automatically be treated as negative.

For example, if item 5 was originally low-ranked but clicked/purchased:

[ 5 \rightarrow positive\ signal ]

This is why preference construction must combine ranking position with real behavioral evidence.

18. DPO Does Not Update the Reward Model

There are two different models:

[ R_\phi(u,q,i) ]

the reward/teacher ranker, and:

[ \pi_\theta(SID_i|x) ]

the OneSearch generative model.

During DPO:

[ \boxed{ \phi\ \text{is fixed} } ]

[ \boxed{ \theta\ \text{is updated} } ]

Thus DPO is effectively:

[ \boxed{ \text{distilling ranking preferences from the teacher into OneSearch} } ]

The final goal is not:

OneSearch → reward model → rerank

but:

OneSearch → beam search → final items

The teacher can disappear from online serving after the knowledge has been transferred.

19. What DPO Actually Changes

Suppose Stage-3 SFT currently gives:

[ P_\theta(1|x)=0.30 ]

[ P_\theta(2|x)=0.25 ]

so beam search prefers:

[ 1>2 ]

But the reward model says:

[ 2\succ1 ]

DPO does not modify beam search itself.

Instead it modifies the parameters of the generative model so that the underlying sequence probabilities move toward:

[ P_{\theta’}(2|x)

P_{\theta’}(1|x) ]

Then the same beam-search algorithm naturally produces the new ordering.

This distinction is important:

[ \boxed{ \text{DPO changes model probabilities} } ]

not:

[ \boxed{ \text{DPO changes the beam-search algorithm} } ]

Beam search is only the decoder that exposes the probability distribution as a top-K list.

20. DPO as a Training Objective, Not a Model Architecture

DPO is not a specific architecture such as:

• DNN,
• Transformer,
• encoder-decoder,
• decoder-only model.

It is a preference-training objective.

The core setup is:

[ \pi_\theta(y|x) ]

for the trainable policy/model, plus a frozen reference:

[ \pi_{\text{ref}}(y|x) ]

For preferred output (y_w) and rejected output (y_l), define:

[ \hat r_\theta(x,y)

\beta \log \frac{ \pi_\theta(y|x) }{ \pi_{\text{ref}}(y|x) } ]

Then DPO encourages:

[ \hat r_\theta(x,y_w)

\hat r_\theta(x,y_l) ]

For OneSearch:

[ y=SID_i ]

and:

[ \log \pi_\theta(SID_i|x)

\sum_t \log P_\theta(s_t|x,s_{<t}) ]

So DPO increases the relative sequence probability of preferred item SIDs.

21. Why DPO Can Improve Ranking Beyond SFT

SFT gives pointwise/generative supervision:

[ x\rightarrow i^+ ]

and says:

[ P(i^+|x)\uparrow ]

It does not explicitly compare:

[ i^+ \text{ vs } i_1,i_2,i_3 ]

DPO adds comparative supervision:

[ i^+\succ i_1 ]

[ i^+\succ i_2 ]

[ i^+\succ i_3 ]

This moves the learning problem from:

[ \boxed{ \text{pointwise likelihood learning} } ]

toward:

[ \boxed{ \text{pairwise/listwise preference learning} } ]

which is much closer to the ranking objective used at evaluation and serving time.

22. Why Listwise DPO?

Ordinary pairwise preference learning might independently train:

[ A\succ B ]

[ A\succ C ]

[ A\succ D ]

OneSearch instead tries to use a positive item together with multiple negatives:

[ A\succ{B,C,D} ]

The objective aggregates comparisons across the list, giving stronger emphasis to difficult negatives.

Conceptually:

[ \boxed{ \text{make the winner beat all relevant negatives, especially hard negatives} } ]

This is more aligned with metrics such as:

• MRR,
• NDCG@K,
• top-K ranking quality.

23. Preference Gap: Two Different Quantities

There are two notions that are easy to mix together.

External behavior/reward gap

For a positive and negative item:

[ r(q,i_{pos}) ]

[ r(q,i_{neg}) ]

The difference:

[ r(q,i_{pos})-r(q,i_{neg}) ]

represents the strength of the external preference signal.

This can affect the weight of the training example.

Model-internal DPO gap

DPO also has:

[ \hat r_\theta(x,i_w)

\hat r_\theta(x,i_l) ]

where:

[ \hat r_\theta(x,i)

\beta \log \frac{ \pi_\theta(i|x) }{ \pi_{\text{ref}}(i|x) } ]

This measures how much the trainable model has shifted relative to the reference model.

These are different:

[ \boxed{ \text{external reward gap} \neq \text{DPO model preference gap} } ]

24. Why Keep an NLL / Generation Term Together with DPO?

Pure preference training mainly requires:

[ P(i_w)>P(i_l) ]

It could theoretically improve this inequality partly by decreasing the losers.

But generative retrieval needs the winner to actually appear in beam search.

Therefore it is useful to also retain:

[ -\log P(i_w|x) ]

so training has both:

[ \boxed{ \text{relative ranking improvement} + \text{absolute generation probability} } ]

This is especially important for generative retrieval.

25. The Reward Model Is Basically Teacher-Student Distillation

Suppose Stage-3 OneSearch generates:

[ [1,2,3,4,5] ]

The existing personalized ranker says:

[ [2,4,1,3,5] ]

Preference training modifies OneSearch until its own beam search increasingly produces:

[ [2,4,1,3,5] ]

Thus:

[ \boxed{ \text{strong personalized ranker} \rightarrow \text{preference labels} \rightarrow \text{generative model} } ]

This resembles teacher-student distillation, but instead of distilling exact logits, DPO distills relative preference/order information.

26. Can an Existing Transformer Ranking Model Be the Reward Model?

Yes.

If an existing production ranker already predicts:

[ CTR,\ ATC,\ CVR ]

using:

[ f_\phi(u,q,i) ]

then it can technically serve as the teacher.

The reward model does not have to be a simple DNN.

The important requirement is that it provides a useful ranking/preference signal over candidates.

The reason OneSearch can use a lighter reward model is mostly engineering efficiency, not a conceptual requirement.

27. Could DPO Be Used in a Traditional Retrieval → Ranking Cascade?

Yes, but the benefit is less automatic.

A conventional ranker already outputs:

[ s_\theta(q,u,i) ]

and can directly optimize pairwise ranking loss:

[ L_{\text{pair}}

-\log\sigma ( s_\theta(i^+)-s_\theta(i^-) ) ]

or listwise softmax/ranking objectives.

Therefore, if the model already directly optimizes candidate ranking, classic pairwise/listwise ranking losses may be simpler than DPO.

DPO becomes more interesting when we specifically want:

[ \boxed{ \text{preference adaptation while staying close to an existing strong model} } ]

because the reference model provides an explicit anchor.

This suggests a possible use case for domain adaptation.

28. My Agentic Search Connection

I currently have:

• a well-trained global search ranking model,
• a new agentic search flow,
• little/no dedicated personalization modeling for agentic search,
• only around 10k agentic-search traffic per day,
• roughly 30 days of traffic so far.

That means only about:

[ 10k \times 30 \approx 300k ]

total impressions, and the number of actual positive click/ATC/CVR examples is substantially smaller.

This is likely insufficient for training a new agentic ranker from scratch.

Therefore the more natural framing is:

[ \boxed{ \text{strong global search ranker} \rightarrow \text{agentic-domain adaptation} } ]

rather than:

[ \boxed{ \text{small agentic dataset} \rightarrow \text{new standalone ranker} } ]

29. Why Agentic Search Is a Distribution-Shift Problem

Global search may mostly see conventional queries such as:

running shoes
high protein food
nike pegasus

Agentic search may see queries more like:

I want something healthy but filling for dinner,
high protein, preferably not too spicy.

or:

something I can wear for long city walks
but that doesn't look too much like a running shoe

The existing global ranker already knows a huge amount about:

• item quality,
• relevance,
• popularity,
• CTR/CVR,
• broad semantic relationships.

I do not want 300k agentic examples to relearn all of this.

What I want to learn is:

[ \boxed{ \Delta_{\text{agentic}}

\text{how ranking preferences change under agentic-query patterns} } ]

Conceptually:

[ score_{\text{agentic}} \approx score_{\text{global}} + \Delta_{\text{agentic}} ]

30. Potential Adaptation Experiments

I would compare several approaches while holding retrieval fixed.

A. Existing global ranker

Baseline:

[ R_{\text{global}} ]

B. MTML fine-tuning on agentic traffic

Initialize from the global ranker and continue training on agentic impressions:

[ L

L_{CTR} + \lambda_1L_{ATC} + \lambda_2L_{CVR} ]

This is the simplest target-domain adaptation baseline.

C. Pairwise preference fine-tuning

Construct high-confidence agentic preferences such as:

[ purchase > ATC > click > examined\text{-}no\text{-}click ]

Then optimize pairwise ranking loss.

D. DPO-style fine-tuning

Freeze the original global ranker as:

[ \pi_{\text{ref}} ]

Initialize:

[ \pi_\theta \leftarrow \pi_{\text{ref}} ]

Then use agentic preference data to modify the ranker while remaining anchored to the global model.

This is attractive because agentic traffic is small relative to the data used to train the global model.

31. Why DPO-Style Adaptation Might Be Useful Here

With a small target-domain dataset, naïve full fine-tuning can overfit and destroy useful global-search knowledge.

DPO-style adaptation asks something closer to:

Relative to what my global ranker already believes, what ranking changes are consistently supported by agentic traffic?

That is almost a residual correction:

[ \boxed{ \text{new model}

\text{global knowledge} + \text{agentic preference correction} } ]

This is more appealing than relearning the entire search problem from 300k impressions.

32. Preference Data Quality Matters More Than Raw Row Count

For agentic search, I should be careful not to treat every non-click as a strong negative.

Potential problems include:

• position bias,
• incomplete examination,
• conversational abandonment,
• multiple acceptable results,
• sparse conversion signals,
• different UI behavior from normal search.

A safer preference hierarchy might use strong signals:

[ purchase > ATC > click ]

and only use impression/no-click negatives when examination probability is credible.

Thus:

[ \boxed{ \text{high-confidence preference groups}

\text{large numbers of noisy binary pairs} } ]

For an initial POC, tens of thousands of high-confidence preference groups may already be enough to determine whether preference-based adaptation is promising.

33. Experimental Design for Agentic Search

Hold retrieval fixed first.

For the same retrieved candidate set, compare:

Evaluate on held-out agentic traffic.

Useful slices:

short keyword-like queries
long natural-language queries
multi-constraint queries
context-dependent conversational queries

Metrics could include:

• NDCG@K
• MRR
• Recall@K
• CTR / ATC / CVR proxies
• calibration / global-search regression checks

The hypothesis should not be framed as simply:

“DPO should improve the ranker.”

A better research question is:

[ \boxed{ \textbf{Can preference-based fine-tuning adapt a strong global-search ranker to the agentic-query distribution while preserving its global knowledge?} } ]

34. Final Mental Model

Search SID

[ \boxed{ \text{query-item alignment} + \text{shared embedding/SID space} + \text{fine-grained residual preservation} } ]

Stage-3 OneSearch

[ \boxed{ (user,query,history) \rightarrow P(SID_i|x) } ]

Beam Search

[ \boxed{ P(SID_i|x) \rightarrow topK\ item\ SIDs } ]

Reward Model

[ \boxed{ (user,query,history,item) \rightarrow personalized\ ranking\ preference } ]

DPO

[ \boxed{ \text{teacher/listwise ranking preference} \rightarrow \text{update OneSearch probabilities} } ]

Final Serving

[ \boxed{ query + user + history \rightarrow OneSearch \rightarrow beam search \rightarrow final items } ]

The most important shift in my understanding is that OneSearch is not simply replacing item IDs with semantic IDs.

It is trying to turn the whole search problem into:

[ \boxed{ \text{learn a relevance-aware discrete search space} + \text{generate item IDs directly} + \text{distill ranking preferences back into generation} } ]