Paper Reference

Recommender Systems with Generative Retrieval — Rajput et al.

Model: TIGER

Main idea: represent items using learned Semantic IDs, then use an encoder-decoder Transformer to generate the next item.

Map

Recommender Systems with Generative Retrieval
├── Item Representation
│   ├── Random Item IDs
│   ├── Continuous Item Embeddings
│   └── Semantic IDs
├── Semantic ID Learning
│   ├── Autoencoder
│   ├── Vector Quantization
│   ├── Residual Quantization
│   └── RQ-VAE
├── Generative Retrieval
│   ├── User Interaction Sequence
│   ├── Encoder-Decoder Transformer
│   ├── T5
│   ├── T5X
│   └── Autoregressive Decoding
└── Follow-up Topics
    ├── Codebook Collapse
    ├── Hourglass Phenomenon
    ├── Product Quantization
    ├── Large-Scale Semantic ID Training
    └── Continuous Embeddings vs. Discrete IDs

1. Motivation

Traditional recommendation systems commonly represent each item using a unique random ID.

Pizza A  -> 183729
Burger B -> 912874
Coffee C -> 381922

These IDs are useful for identifying items, but they do not contain semantic meaning.

For example, suppose two items are semantically similar:

Spicy Chicken Sandwich -> Item 1024
Crispy Chicken Burger  -> Item 8749

The two integer IDs do not indicate that the items are related. A model using only ID-based representations must learn their relationship indirectly from interaction data. This creates several problems:

• semantically similar items may have unrelated representations
• the model must learn each item almost independently
• cold-start and low-frequency items are difficult to learn
• a generative model may need an extremely large output vocabulary
• the item representation contains little structure before entering the Transformer

The paper therefore proposes learning a structured discrete representation for each item, called a Semantic ID, before the item sequence is passed into the Transformer encoder-decoder.

The overall pipeline is:

Item Content / Item Features
        ↓
Initial Item Embedding
        ↓
      RQ-VAE
        ↓
   Semantic ID
        ↓
User Sequence of Semantic IDs
        ↓
Transformer Encoder-Decoder
        ↓
Generate the Next Semantic ID
        ↓
    Retrieve Item

The complete recommendation architecture proposed in the paper is called TIGER.

2. Overview

TIGER contains two main training stages.

Stage 1: Learn Semantic IDs

An RQ-VAE converts each continuous item embedding into a sequence of discrete codebook indices. For an item embedding $x$, the RQ-VAE produces a Semantic ID:

$$\mathrm{RQ}(x) = (c_1, c_2, \dots, c_m),$$

where $c_i$ is the selected codeword index from quantization level $i$. For example:

Item A -> (12, 83, 5)
Item B -> (12, 81, 19)
Item C -> (74, 2, 41)

Items with similar semantics may share some earlier codewords. For example:

Spicy Chicken Sandwich -> (12, 83, 5)
Crispy Chicken Burger  -> (12, 83, 17)

The shared prefix may indicate that the two items belong to a similar semantic region.

Stage 2: Train the Generative Retriever

After Semantic IDs are generated, every item in a user’s interaction history is replaced by its Semantic ID.

Original sequence:      Item A -> Item B -> Item C
Semantic ID sequence:   (12, 83, 5) -> (41, 7, 9) -> (18, 64, 2)

The Transformer learns to generate the Semantic ID of the next item.

3. RQ-VAE Details

3.1 Initial Item Embedding

Each item is first represented using an existing continuous embedding:

$$x \in \mathbb{R}^{d_x}.$$

In the paper, the initial item embedding dimension is $d_x = 32$. The initial embedding may come from item metadata, text, image features, collaborative signals, or another pretrained representation model. The RQ-VAE does not directly operate on a random integer item ID — it operates on this continuous item representation.

3.2 RQ-VAE Architecture

RQ-VAE stands for Residual-Quantized Variational Autoencoder. Its main components are: an encoder, multiple codebooks, a residual quantizer, and a decoder. The high-level forward pass is:

$$x \xrightarrow{E} z_e \xrightarrow{Q} z_q \xrightarrow{D} \hat{x},$$

where $x$ is the original item embedding, $E$ is the encoder, $z_e$ is the continuous encoder output, $Q$ is the residual quantization operation, $z_q$ is the quantized latent representation, $D$ is the decoder, and $\hat{x}$ is the reconstructed item embedding.

3.3 Encoder

The encoder maps the original item embedding into a latent representation:

$$z_e = E(x).$$

The encoder architecture described in the paper is an MLP with hidden dimensions similar to 32 -> 512 -> 256 -> 128 -> 32:

$$h_1 = \sigma(W_1 x + b_1), \quad h_2 = \sigma(W_2 h_1 + b_2), \quad h_3 = \sigma(W_3 h_2 + b_3), \quad z_e = W_4 h_3 + b_4,$$

where $W_i$ and $b_i$ are learnable parameters, $\sigma$ is a nonlinear activation function, and $z_e \in \mathbb{R}^{32}$. The encoder output is still continuous — residual quantization converts this continuous vector into a discrete Semantic ID.

3.4 Codebooks

RQ-VAE uses $m$ quantization levels. Each level has a codebook:

$$C_i = { e_{i,1}, e_{i,2}, \dots, e_{i,K} },$$

where $i \in {1,\dots,m}$ is the quantization level, $K$ is the number of vectors in each codebook, and $e_{i,k} \in \mathbb{R}^{d_z}$ is the $k$-th codeword at level $i$. Each codebook can be represented as a matrix $C_i \in \mathbb{R}^{K \times d_z}$.

The final Semantic ID contains one selected index from each codebook: $\mathrm{RQ}(x) = (c_1, c_2, \dots, c_m)$. The total number of possible Semantic IDs is approximately $K^m$. For example, if $K=256, m=3$, then the number of possible code combinations is $256^3 = 16{,}777{,}216$.

The model therefore obtains a large structured item space using relatively small token vocabularies.

3.5 Codebook Initialization

The codebooks are initialized using k-means, not KNN. (KNN is a nearest-neighbor lookup algorithm; k-means is a clustering algorithm and can be used to initialize codebook vectors.)

Suppose a set of encoder outputs is $Z = {z_e^{(1)}, z_e^{(2)}, \dots, z_e^{(N)}}$. K-means divides these points into $K$ clusters $S_1,\dots,S_K$. The initial codeword for cluster $k$ is its centroid:

$$e_k = \frac{1}{|S_k|}\sum_{z \in S_k} z.$$

The k-means objective is:

$$\min_{{S_k},{e_k}} \sum_{k=1}^{K} \sum_{z \in S_k} |z - e_k|_2^2.$$

This gives the codebooks a more meaningful initialization than purely random vectors.

3.6 Residual Quantization

Residual quantization decomposes the encoder output into a sum of codewords. Start with $r_0 = z_e$. At level $i$, select the nearest codeword to the current residual:

$$c_i = \arg\min_{k \in {1,\dots,K}} |r_{i-1} - e_{i,k}|_2^2.$$

The selected codeword is $q_i = e_{i,c_i}$. Then subtract that codeword from the current residual:

$$r_i = r_{i-1} - q_i = r_{i-1} - e_{i,c_i}.$$

At the first level, $r_1 = z_e - e_{1,c_1}$. At the second level, $r_2 = r_1 - e_{2,c_2}$; substituting $r_1$ gives $r_2 = z_e - e_{1,c_1} - e_{2,c_2}$. After $m$ levels:

$$r_m = z_e - \sum_{i=1}^{m} e_{i,c_i}.$$

Define the final quantized representation as $z_q = \sum_{i=1}^{m} e_{i,c_i}$. Therefore $r_m = z_e - z_q$, i.e. $z_e = z_q + r_m$. The selected codeword indices form the Semantic ID: $\mathrm{RQ}(x) = (c_1,c_2,\dots,c_m)$.

Intuition: the first codebook approximates the largest/coarsest part of the latent vector, the second approximates the error left by the first, the third approximates the remaining error, and so on:

$$z_e \approx e_{1,c_1} + e_{2,c_2} + \dots + e_{m,c_m}.$$

3.7 Decoder

The decoder maps the quantized latent vector back into the original embedding space, $\hat{x} = D(z_q)$. A decoder MLP may be written as:

$$\hat h_1 = \sigma(W’_1 z_q + b’_1), \quad \hat h_2 = \sigma(W’_2 \hat h_1 + b’_2), \quad \hat{x} = W’_3 \hat h_2 + b’_3.$$

The decoder tries to reconstruct the original input embedding $x$. If the reconstruction is accurate, the Semantic ID preserves useful information from the original item embedding.

3.8 Reconstruction Loss

$$L_{\text{recon}} = |x-\hat{x}|2^2, \qquad L{\text{recon}} = \frac{1}{B}\sum_{n=1}^{B} | x^{(n)}-\hat{x}^{(n)} |_2^2 \text{ (batch of } B \text{ items)}.$$

The reconstruction loss trains the decoder directly, and the encoder through the straight-through estimator.

4. Why Quantization Is Hard to Train

The codeword index is selected using $c_i = \arg\min_k |r_{i-1}-e_{i,k}|2^2$. The $\arg\min$ operation returns a discrete integer index. A small change in $r{i-1}$ usually does not change the selected index, but at a boundary the selected index may suddenly switch from one codeword to another. Therefore the quantization operation is not normally differentiable — its gradient is not useful for ordinary gradient descent. Without an approximation, the reconstruction loss cannot propagate through the hard codeword selection and update the encoder.

4.1 Straight-Through Estimator (STE)

The straight-through estimator uses different behavior in the forward and backward passes.

Forward pass — use the actual quantized vector: $z_q = \sum_{i=1}^{m} e_{i,c_i}$. The decoder receives $z_q$, not $z_e$: $\hat{x} = D(z_q)$.

Backward pass — pretend quantization behaves approximately like the identity function: $\frac{\partial z_q}{\partial z_e} \approx I$.

A common implementation is:

$$z_{\text{STE}} = z_e + \mathrm{sg}(z_q - z_e),$$

where $\mathrm{sg}(\cdot)$ denotes stop-gradient. Because stop-gradient does not change the forward value, $z_{\text{STE}} = z_e + (z_q - z_e) = z_q$ — the decoder sees the quantized vector. Since the stop-gradient term has zero derivative, $\frac{\partial z_{\text{STE}}}{\partial z_e} = I$ — gradients flow into the encoder as though quantization were the identity.

4.2 Stop-Gradient

$$\mathrm{sg}(x) = x \text{ (forward)}, \qquad \frac{\partial}{\partial x}\mathrm{sg}(x) = 0 \text{ (backward)}.$$

Stop-gradient does not happen after applying the learning rate. The training order is:

Forward pass
    ↓
Compute loss
    ↓
Backpropagation
    ↓
Stop-gradient blocks selected gradient paths
    ↓
Optimizer reads gradients
    ↓
Learning-rate-based parameter update

Stop-gradient controls which parameters receive gradients from each loss term.

4.3 Codebook Loss

The selected codeword should move toward the residual it is intended to represent. At level $i$:

$$L_{\text{codebook}}^{(i)} = | \mathrm{sg}(r_{i-1}) - e_{i,c_i} |2^2, \qquad L{\text{codebook}} = \sum_{i=1}^{m} | \mathrm{sg}(r_{i-1}) - e_{i,c_i} |_2^2.$$

Because the residual is stop-gradient, this loss updates the codeword but not the encoder through $r_{i-1}$. The gradient for one selected codeword is:

$$\frac{\partial L_{\text{codebook}}}{\partial e_{i,c_i}} = 2(e_{i,c_i} - r_{i-1}),$$

and a gradient descent update moves the codeword toward the residual: $e_{i,c_i} \leftarrow e_{i,c_i} - \eta \cdot 2(e_{i,c_i} - r_{i-1})$.

4.4 Commitment Loss

The encoder should also commit to the selected codewords. At level $i$:

$$L_{\text{commit}}^{(i)} = \beta| r_{i-1} - \mathrm{sg}(e_{i,c_i}) |2^2, \qquad L{\text{commit}} = \beta\sum_{i=1}^{m} | r_{i-1} - \mathrm{sg}(e_{i,c_i}) |_2^2,$$

where $\beta$ controls the strength of the commitment loss. Because the codeword is stop-gradient, this loss updates the encoder-side residual but not the selected codeword:

$$\frac{\partial L_{\text{commit}}}{\partial r_{i-1}} = 2\beta(r_{i-1} - e_{i,c_i}).$$

The encoder is therefore encouraged to produce latent representations that remain close to available codewords. Without commitment loss, the encoder output may move rapidly while the codebooks struggle to follow.

Because RQ-VAE has multiple residual levels, commitment loss is applied at every level rather than only on the final term $|z_e - \mathrm{sg}(z_q)|2^2$. This explicitly trains each codebook to approximate its corresponding residual — but since residuals depend on earlier selected codewords ($r_i = r{i-1} - e_{i,c_i}$), implementations must carefully control gradient flow so later-level losses don’t unintentionally update earlier codebooks through the residual computation (e.g. by detaching the residual before using it as the target for the next codebook loss).

4.5 Total RQ-VAE Loss

$$L_{\text{RQVAE}} = L_{\text{recon}} + L_{\text{codebook}} + \beta L_{\text{commit}},$$

expanded:

$$L_{\text{RQVAE}} = |x-\hat{x}|2^2 + \sum{i=1}^{m} |\mathrm{sg}(r_{i-1}) - e_{i,c_i}|2^2 + \beta\sum{i=1}^{m} |r_{i-1} - \mathrm{sg}(e_{i,c_i})|_2^2.$$

4.6 Backpropagation Flow

Input embedding x
      ↓
   Encoder E
      ↓
Continuous latent z_e
      ↓
Residual quantization ── Selected codewords ←── Codebook loss
      ↓
Quantized latent z_q
      ↓
Straight-through estimator
      ↓
   Decoder D
      ↓
Reconstruction x_hat

• Reconstruction loss ($L_{\text{recon}} = |x - D(z_q)|2^2$) updates the decoder directly, and — through the STE approximation $\frac{\partial L{\text{recon}}}{\partial z_e} \approx \frac{\partial L_{\text{recon}}}{\partial z_q}$ — the encoder as well.
• Codebook loss updates selected codewords ($\frac{\partial L_{\text{codebook}}}{\partial e_{i,c_i}} \neq 0$) but not the encoder, since the residual is detached ($\frac{\partial L_{\text{codebook}}}{\partial r_{i-1}} = 0$).
• Commitment loss updates the encoder-side representation ($\frac{\partial L_{\text{commit}}}{\partial r_{i-1}} \neq 0$) but not the codeword, since the codeword is detached ($\frac{\partial L_{\text{commit}}}{\partial e_{i,c_i}} = 0$).

4.7 Codebook Usage and Entropy

A possible failure mode is that only a small number of codewords are selected. Let $p_k$ be the empirical probability that codeword $k$ is used across the dataset:

$$p_k = \frac{\text{count}(k)}{N}.$$

The codebook usage entropy is:

$$H(p) = -\sum_{k=1}^{K} p_k \log p_k.$$

If all codewords are used uniformly, $p_k = 1/K$ and entropy is maximized: $H(p) = \log K$. If every item uses the same codeword, $p_1=1, p_{k\neq 1}=0$, so $H(p) = 0$.

For dataset-wide codebook usage, higher entropy usually indicates more balanced utilization — this is different from per-example prediction entropy, where low entropy indicates confidence. For codebook usage across the entire dataset, very low entropy may instead indicate codebook collapse.

A regularization term may maximize usage entropy (minimize $L_{\text{entropy}} = -H(p)$), or equivalently encourage the usage distribution to match a uniform distribution $u_k = 1/K$ via $L_{\text{uniform}} = D_{\text{KL}}(p | u)$.

5. Semantic ID Generation and Collisions

After the RQ-VAE is trained, each item is passed through the encoder and residual quantizer. For item $x_j$: $z_e^{(j)} = E(x_j)$, and at each level $c_i^{(j)} = \arg\min_k |r_{i-1}^{(j)} - e_{i,k}|_2^2$. The final Semantic ID is $\mathrm{RQ}(x_j) = (c_1^{(j)}, c_2^{(j)}, \dots, c_m^{(j)})$. The RQ-VAE decoder is no longer needed when training the generative retriever — only the discrete code sequence matters.

Two items may still receive the same Semantic ID, $\mathrm{RQ}(x_a) = \mathrm{RQ}(x_b)$ — a collision. Even though the total number of code combinations may be large, collisions are still possible because the item catalog may be very large, semantically similar embeddings may quantize to the same code, and the quantizer is optimized for reconstruction rather than unique identification. A practical system may append an additional disambiguation token, $\widetilde{\mathrm{RQ}}(x) = (c_1,\dots,c_m,c_{\text{dedup}})$, preserving semantic structure while ensuring unique item identification.

6. TIGER Generative Retriever

6.1 Input Representation

After Semantic IDs are assigned, an interaction sequence $(i_1,i_2,\dots,i_T)$ is converted into $(\mathrm{RQ}(i_1), \mathrm{RQ}(i_2), \dots, \mathrm{RQ}(i_T))$. If each Semantic ID contains $m$ tokens, $\mathrm{RQ}(i_t) = (c_{t,1},\dots,c_{t,m})$, then the flattened token sequence is:

$$(c_{1,1},\dots,c_{1,m},\ c_{2,1},\dots,c_{2,m},\ \dots,\ c_{T,1},\dots,c_{T,m}).$$

Special tokens may be used to indicate item boundaries, task type, user sequence structure, and prediction position.

6.2 Encoder-Decoder Transformer

TIGER uses an encoder-decoder Transformer implemented with the T5X framework. The encoder receives the user’s historical interaction sequence $X = (x_1,\dots,x_n)$ and produces contextualized hidden states $H = \mathrm{Enc}(X) = (h_1,\dots,h_n)$.

The decoder generates the next item’s Semantic ID autoregressively. Suppose the target Semantic ID is $Y=(y_1,\dots,y_m)$. The probability factorizes as:

$$P(Y|X) = \prod_{t=1}^{m} P(y_t \mid y_{<t}, X), \qquad y_{<t} = (y_1,\dots,y_{t-1}).$$

The decoder predicts one Semantic ID token at a time.

6.3 Training Objective

The negative log-likelihood loss is:

$$L_{\text{gen}} = -\sum_{t=1}^{m} \log P(y_t \mid y_{<t}, X), \qquad L_{\text{gen}} = -\frac{1}{B}\sum_{n=1}^{B}\sum_{t=1}^{m} \log P!\left(y_t^{(n)} \mid y_{<t}^{(n)}, X^{(n)}\right)$$

for a batch of $B$ examples — ordinary token-level cross-entropy. If the target token is one-hot $q$ and the predicted distribution is $p$, then $L = -\sum_{k=1}^K q_k \log p_k = -\log p_{y_t}$.

6.4 Teacher Forcing

During training, the decoder receives the correct previous target tokens. For target $(y_1,y_2,y_3)$:

Input to decoder:  <BOS>              → Predict: y_1
Input to decoder:  <BOS>, y_1         → Predict: y_2
Input to decoder:  <BOS>, y_1, y_2    → Predict: y_3

During inference, the true future tokens are unavailable, so the model must use its own previous predictions:

Input:  <BOS>                          → Generate: ŷ_1
Input:  <BOS>, ŷ_1                     → Generate: ŷ_2
Input:  <BOS>, ŷ_1, ŷ_2                → Generate: ŷ_3

6.5 Inference

At inference time, TIGER estimates $\hat{Y} = \arg\max_Y P(Y|X)$. Since enumerating all possible sequences is infeasible, approximate decoding is used.

• Greedy decoding: at every step, $\hat{y}t = \arg\max_k P(y_t = k \mid \hat{y}{<t}, X)$.
• Beam search: maintains the top $B$ partial Semantic ID sequences, commonly scored as $s(Y) = \sum_{t=1}^{|Y|} \log P(y_t \mid y_{<t}, X)$.

After the Semantic ID is generated, it is mapped back to the corresponding item or candidate set.

6.6 T5 and T5X References

T5 is an encoder-decoder Transformer architecture that formulates tasks in a text-to-text format (T5 paper). T5X is a JAX-based framework for training T5-style models at scale (T5X repository) — TIGER uses it as the implementation framework, but T5X is a training/evaluation framework, not itself a new Transformer architecture.

7. Implementation Sketches

RQ-VAE (PyTorch)

RQ-VAE
├── Encoder
├── Residual Quantizer
│   ├── Codebook 1
│   ├── Codebook 2
│   └── Codebook m
├── Straight-Through Estimator
├── Decoder
├── Reconstruction Loss
├── Codebook Loss
├── Commitment Loss
└── Training Loop

Suggested interface:

z_e = encoder(x)

z_q, semantic_ids, quantization_losses = quantizer(z_e)

x_hat = decoder(z_q)

loss = (
    reconstruction_loss(x_hat, x)
    + codebook_loss
    + beta * commitment_loss
)

Important implementation details:

• use .detach() for stop-gradient
• compute nearest codewords using squared Euclidean distance
• update the residual after each quantization level
• return both z_q and the selected code indices
• monitor codebook usage and dead/rarely-selected codewords
• verify whether gradients flow to the intended components

Nearest-codeword distance: for residuals $R \in \mathbb{R}^{B\times d}$ and codebook $E \in \mathbb{R}^{K\times d}$, the pairwise squared distance can be computed as:

$$|r-e|_2^2 = |r|_2^2 + |e|_2^2 - 2r^\top e, \qquad D = \mathrm{diag}(RR^\top)\mathbf{1}^\top + \mathbf{1},\mathrm{diag}(EE^\top)^\top - 2RE^\top,$$

then $c = \arg\min_k D_{:,k}$. This avoids explicitly materializing a $(B\times K\times d)$ tensor.

TIGER Transformer (PyTorch)

TIGER Transformer
├── Semantic ID Dataset
├── Token Embeddings
├── Positional Embeddings
├── Transformer Encoder
├── Transformer Decoder
├── Output Projection
├── Cross-Entropy Loss
└── Autoregressive Inference

Potential implementation choices: torch.nn.Transformer, Hugging Face T5, or a custom encoder-decoder Transformer.

Encoder input:   User interaction history represented as Semantic IDs
Decoder input:   <BOS> + shifted target Semantic ID
Decoder target:  Target Semantic ID + <EOS>

8. Open Questions

8.1 Why not feed LLM item embeddings directly into the Transformer?

Suppose the original item representation is already a high-quality LLM embedding $x_i \in \mathbb{R}^d$. Why not directly use $(x_1,\dots,x_T)$ as the input to a sequential Transformer? For sequence encoding this is possible — the Transformer can consume continuous item embeddings after a projection $h_i = W_p x_i + b_p$.

The difficulty is on the output side. A generative retrieval model must produce an item. With Semantic IDs, item generation is token classification, $P(c_t \mid c_{<t}, X)$. With continuous embeddings, the model would need to generate a vector $\hat{v}{T+1} \in \mathbb{R}^d$ and then perform nearest-neighbor search, $\hat{i} = \arg\min_i |\hat{v}{T+1} - x_i|_2^2$:

Transformer → Predicted continuous embedding → ANN retrieval over all items → Retrieved item

Semantic IDs provide: discrete autoregressive generation, smaller per-step vocabulary, compatibility with cross-entropy training, structured sharing across items, direct use of language-model-style decoding, and no final global nearest-neighbor lookup in the same form. However, continuous embeddings may preserve more information and avoid quantization error. So the real comparison isn’t “semantic vs. non-semantic representation” — it’s continuous semantic representation vs. discrete semantic representation, which is a genuine design question rather than an obvious win for Semantic IDs.

8.2 What information is lost during quantization?

RQ-VAE approximates $z_e \approx z_q$; the quantization error is $\epsilon_q = z_e - z_q$, with squared magnitude $|\epsilon_q|_2^2$. If the codebooks are too small or the number of levels is insufficient, quantization may remove fine-grained item information — two distinct items may map to nearby or identical Semantic IDs. This may improve generalization but reduce item-level precision.

8.3 What are the risks of training RQ-VAE?

• Codebook collapse — only a small subset of codewords is used
• Dead codewords — some codewords are never selected and receive little/no gradient
• Quantization error — the discrete representation may fail to preserve important item information
• Semantic ID collisions — multiple items may obtain the same code sequence
• Unstable code assignments — after retraining, an item may receive a completely different Semantic ID
• Semantic drift — as catalog content and user behavior change, old codebooks may no longer represent current item semantics
• Uneven codebook utilization — some codewords represent a huge number of items, others very few
• Optimization instability — encoder and codebooks may move at different speeds
• Non-differentiable assignment — the model relies on STE, a biased gradient approximation

8.4 Is RQ-VAE scalable to 86M items and 800K stores?

Suppose $N = 86{,}000{,}000$ items, $m$ quantization levels, $K$ codewords per level, and latent dimension $d$. A naive assignment cost is approximately $O(N \cdot m \cdot K \cdot d)$ — for every item and every level, the residual must be compared with all $K$ codewords. This is large but highly parallelizable using matrix multiplication on GPUs or distributed workers.

The codebook parameter size itself is small, $m \cdot K \cdot d$ — for $m=3, K=256, d=32$, that’s only $3 \times 256 \times 32 = 24{,}576$ scalars. So codebook storage is not the primary scalability issue. The main challenges are: encoding all 86M items, performing codebook assignment, storing and versioning Semantic IDs, retraining when embeddings change, updating new/modified items, ensuring stable IDs across model versions, handling collisions, and rebuilding downstream training datasets. A practical architecture may look like:

Offline embedding generation
        ↓
Distributed RQ-VAE training
        ↓
Batch Semantic ID assignment
        ↓
Versioned Semantic ID table
        ↓
Incremental assignment for new items
        ↓
Periodic full refresh

8.5 Can new items receive Semantic IDs without retraining?

Given a trained encoder and fixed codebooks, a new item $x_{\text{new}}$ can be assigned a Semantic ID using only inference: $z_e^{\text{new}} = E(x_{\text{new}})$, then run residual quantization as usual. This does not require retraining the RQ-VAE. However, if new items come from a significantly different distribution, the existing codebooks may no longer provide good coverage.

8.6 How often should the RQ-VAE be retrained?

Possible triggers: major changes to the upstream embedding model, significant catalog distribution shift, high quantization error, increased collision rate, degraded codebook utilization, retrieval quality degradation, or a large volume of newly added items.

Retraining creates a versioning problem, since Semantic IDs may change between model versions ($\mathrm{RQ}^{(v)}(x) \neq \mathrm{RQ}^{(v+1)}(x)$), requiring coordinated updates to: Semantic ID lookup tables, training sequences, Transformer vocabulary, generative retrieval checkpoints, serving indexes, and evaluation datasets. Stable Semantic IDs may therefore be an important production requirement not fully captured by offline reconstruction metrics.

8.7 Why use RQ-VAE rather than one large codebook?

A single codebook with $K$ vectors can represent only $K$ discrete values. Residual quantization with $m$ codebooks can represent up to $K^m$ combinations, while the parameter count remains $m \cdot K \cdot d$ rather than $K^m \cdot d$. For example, $K=256, m=4$ gives $256^4 = 4{,}294{,}967{,}296$ combinations from only $m \cdot K = 1024$ total codewords. RQ-VAE therefore provides combinatorial capacity without requiring a gigantic output vocabulary.

8.8 Does a shared Semantic ID prefix always imply semantic similarity?

Not necessarily. Residual quantization is optimized primarily for reconstruction, $\min |x-\hat{x}|_2^2$ — it does not explicitly guarantee that a shared first token implies a shared human-interpretable semantic category. A shared prefix means the items selected the same early codeword, suggesting their latent vectors share a similar coarse approximation, but the meaning may not align with categories such as cuisine, product type, price, dietary preference, brand, or store identity. The semantics depend entirely on the information contained in the original embedding and the RQ-VAE objective.

9. Discussion

Semantic IDs as a Representation Layer

The most important architectural idea is that TIGER inserts a learned discrete representation layer between item embeddings and the Transformer.

Without Semantic IDs:   Item ID or embedding → Sequential model → Item prediction

With TIGER:             Item embedding → RQ-VAE → Semantic ID → Encoder-decoder Transformer
                         → Generated Semantic ID → Item

RQ-VAE and the Transformer solve different problems. RQ-VAE learns how should an item be represented as discrete tokens? The Transformer learns given the user’s history, which discrete item-token sequence should come next?

Benefits

Structured item vocabulary, semantic sharing between items, smaller token vocabulary per decoding step, compatibility with autoregressive generation, possible improvement for sparse/long-tail items, combinatorial representation capacity, and easier use of language-model architectures for recommendation.

Limitations

Information loss from quantization, collisions, codebook collapse, unstable code assignments, complex offline pipelines, retraining/versioning cost, longer generated sequences (one item becomes multiple tokens), autoregressive decoding latency, and possible mismatch between reconstruction quality and recommendation quality.

Production Questions for Food Delivery

For a food-delivery catalog with approximately 86M items and 800K stores:

• Should stores and items use separate Semantic ID spaces?
• Should the Semantic ID represent only item semantics, or also store context?
• Can identical menu items from different stores share a semantic prefix?
• Should geographic availability be encoded in the Semantic ID?
• How should frequently changing prices and availability be handled?
• How frequently should item embeddings and Semantic IDs be refreshed?
• Can new items be encoded online?
• How should ID collisions be resolved?
• How stable must Semantic IDs remain across model versions?
• Does autoregressive generation meet serving-latency requirements?
• Should TIGER retrieve stores, items, or both?
• Should Semantic IDs be used for retrieval, ranking, or only as auxiliary features?

Follow-Up Reading

The next topic is the hourglass phenomenon in residual quantization — kept as a separate reading note rather than folded into this one.

Potential follow-ups:

• Breaking the Hourglass Phenomenon of Residual Quantization
• VQ-VAE
• Product Quantization / Optimized Product Quantization
• Hierarchical Semantic IDs
• Semantic ID stability
• Generative retrieval latency
• HSTU
• T5 and T5X internals