[논문리뷰] TPPO: Tracking-Value PPO For Language Model Reasoning

링크: 논문 PDF로 바로 열기

The paper "TPPO: Tracking-Value PPO For Language Model Reasoning" has been browsed. Now I need to extract the required information and format it as requested.

Part 1: Summary Body (Markdown)

Metadata:

• Authors: Futing Wang, Chenxi Li, Jianhao Yan, Yun Luo, Zhi Wang, Yizhuo Li, Runzhe Zhan, Yu Cheng, Tao Lin
• Keywords: I'll need to read the abstract and introduction to identify 5-8 relevant academic keywords.
  • Potential keywords: Reinforcement Learning, Proximal Policy Optimization (PPO), Value Function, Credit Assignment, Language Model Reasoning, Log-Odds Space, Bayesian Update, Generative AI, LLMs. I will refine these after reading the full text.

Section 1: Key Terms & Definitions I will identify 3-5 core technical terms.

• PPO (Proximal Policy Optimization): A widely used reinforcement learning algorithm for policy optimization.
• Value Evolution: The change in the state-value function (probability of success) as a language model generates successive tokens in a reasoning trajectory.
• Credit Assignment: The process of determining how much each token-level action contributes to the final reward in a reinforcement learning setting, especially challenging in long reasoning chains with sparse rewards.
• Log-Odds Space: A mathematical transformation (logit function) where probabilities are converted into a scale suitable for additive updates, as identified in the Bayesian structure of value evolution.
• RLVR (Reinforcement Learning with Verifiable Rewards): A specific reinforcement learning setup where rewards for language models are verifiable, often binary (e.g., correct/incorrect solution).

Section 2: Motivation & Problem Statement

• The paper highlights the challenge of Credit Assignment in LLM reasoning with long Chain-of-Thought (CoT), especially under sparse terminal rewards.
• Existing PPO critics predict each prefix value independently and fail to explicitly model value evolution, leading to inaccurate token-level credit assignment.
• Figure 1(a) empirically demonstrates this failure, where a standard PPO critic mispredicts the sign of value evolution (e.g., from a correct step to an incorrect claim), leading to reversed GAE advantage ordering.
• Figure 1(b) further shows that the sign accuracy of these predictions remains near chance and even declines during training, indicating a systematic problem.
• This motivates the need for a critic that can better model and learn value evolution across successive states.

Section 3: Method & Key Results

• The authors propose Tracking-Value PPO (TPPO), a novel critic parameterization that explicitly models value evolution.
• TPPO leverages a Bayesian decomposition of successive prefix values, showing that their updates become additive in log-odds space (Proposition 4.1, Equation 8).
• This additive structure is used to design a cumulative value parameterization where position-wise scalar outputs are summed into value logits, which are then transformed by a sigmoid to reconstruct prefix values [Figure 2].
• The critic is trained using terminal-outcome supervision on these reconstructed values, without requiring additional rollouts or process-level annotations.
• Key Quantitative Results:
  • On mathematical reasoning, TPPO significantly improves aggregate scores over PPO, achieving 7.0 percentage points gain on Qwen3-4B-Base and 8.4 percentage points gain on Qwen3-8B-Base (Table 2).
  • These gains extend to out-of-distribution generalization and other tasks like countdown and scientific coding (Table 2, Table 4, Figure 4).
  • TPPO consistently achieves higher validation accuracy and Pass@4 earlier in training, maintaining a lead over PPO, with smaller actor gradient norms indicating more stable updates [Figure 3].
  • Diagnostic analysis confirms that TPPO better tracks value evolution with improved sign agreement (3.9–6.8 percentage points for τ=0 and 4.9–7.4 for τ=1 across the last four checkpoints) [Figure 5].
  • It also leads to more reliable advantage estimates, showing higher cosine similarity and sign agreement (15.3–19.9 percentage points improvement in advantage sign accuracy across the last four checkpoints) with MC-based reference advantages [Figure 6].
  • Ablation studies show that the cumulative parameterization is the primary driver of these gains, with BCE loss providing additional benefit (Figure 8a).

Section 4: Conclusion & Impact

• The paper concludes that TPPO effectively addresses the challenge of accurately tracking value evolution and credit assignment in LLM reasoning by introducing a cumulative parameterization grounded in a Bayesian decomposition in log-odds space.
• This research significantly advances reinforcement learning for LLMs by enabling more stable and effective policy optimization without relying on additional data or complex supervision.
• The consistent performance improvements across various models and tasks, coupled with enhanced diagnostic metrics, suggest that TPPO provides a more robust and reliable mechanism for LLM reasoning, paving the way for more capable and generalizable language models in complex problem-solving domains.

Part 2: Important Figure Information (JSON) I need to find up to 3 important figures. Looking at the HTML content, I see:

• Figure 1: Critics mispredict how value evolves along reasoning. (Contains (a) and (b))
• Figure 2: Difference between standard and TPPO critic parameterizations. (Contains (a) and (b))
• Figure 3: Training dynamics comparing PPO and TPPO on Qwen3-8B-Base.
• Figure 4: Countdown task on Qwen2.5-Math-1.5B.
• Figure 5: Value evolution tracking. Sign agreement between critic predictions and MC-estimated value changes. This is probably the same data as Figure 1 (b) but for TPPO, or a more detailed version of the results referred in RQ2.
• Figure 6: Credit assignment. Cosine similarity and sign agreement with GAE advantages. (Results referred in RQ3)
• Figure 7: Value estimates on test data. Error and accuracy against MC estimates at different reasoning positions.
• Figure 8: Critic ablations and actor GAE coefficients. (Contains (a) and (b))
• Figure 9: Additional results on Llama3.2-3B-Instruct.
• Figure 10: Value-change sign accuracy by rollout source.
• Figure 11: Shared and separate output heads.

The most critical figures would be:

1. Figure 2: Shows the core proposed method (TPPO critic architecture) vs. standard. This is key to understanding "Method".
2. Figure 1: Shows the problem TPPO is trying to solve (PPO critics struggle). This is key to "Motivation".
3. Figure 3: Shows the training dynamics and overall performance (validation accuracy, Pass@4), which are key results. Or Figure 6, which shows the credit assignment quality, directly supporting RQ3. Figure 3 seems more general for "Key Results" section, showing overall training improvements. Figure 6 is more specific for credit assignment quality. I will choose Figure 3 for overall performance in training and Figure 6 for credit assignment quality as it is a direct comparison to MC reference. Let's choose Figure 2 (method), Figure 1 (problem/motivation), and Figure 6 (key result: credit assignment quality).

Now I need to get the image_url for these figures. I'll search for <img> tags for "Figure 1", "Figure 2", and "Figure 6".

From the browsed content:

• Figure 1: <img src="2610.16794v1/x1.png" alt="Figure 1"> -> https://arxiv.org/html/2610.16794v1/x1.png
• Figure 2: <img src="2610.16794v1/x2.png" alt="Figure 2"> -> https://arxiv.org/html/2610.16794v1/x2.png
• Figure 6: <img src="2610.16794v1/x6.png" alt="Figure 6"> -> https://arxiv.org/html/2610.16794v1/x6.png

Looks good. Now I will compile the final output.저자: Futing Wang, Chenxi Li, Jianhao Yan, Yun Luo, Zhi Wang, Yizhuo Li, Runzhe Zhan, Yu Cheng, Tao Lin, et al.

1. Key Terms & Definitions (핵심 용어 및 정의)

• PPO (Proximal Policy Optimization): 대규모 언어 모델(LLM)의 추론 능력 향상을 위해 검증 가능한 보상(Verifiable Rewards)을 사용하는 Reinforcement Learning (RL)에서 널리 활용되는 알고리즘입니다.
• Value Evolution: LLM이 추론 궤적(Reasoning Trajectory)을 따라 연속적인 토큰(Token)을 생성할 때, 각 Prefix의 기대 성공 확률인 상태 가치(State Value)가 어떻게 변화하는지를 나타내는 개념입니다.
• Credit Assignment: RL 환경에서 각 토큰 수준(Token-level)의 행동이 최종 보상(Terminal Reward)에 얼마나 기여했는지 측정하는 과정으로, 특히 희소한 보상(Sparse Rewards)과 긴 추론 체인(Long Reasoning Chains)에서 어렵습니다.
• Log-Odds Space: 확률(Probability) 값을 로그-오즈(Log-odds) 형태로 변환한 공간으로, 본 논문에서는 Value Evolution이 이 공간에서 가산적인(Additive) 업데이트 구조를 가진다는 점을 밝혀냅니다.
• RLVR (Reinforcement Learning with Verifiable Rewards): LLM 추론에서 결과의 정답 여부와 같이 명확하게 검증 가능한 보상 신호를 활용하는 Reinforcement Learning 설정입니다.

2. Motivation & Problem Statement (연구 배경 및 문제 정의)

본 논문은 LLM reasoning 과정에서 sparse terminal rewards 환경 하의 credit assignment 문제와 PPO critics의 한계를 해결하는 것을 목표로 합니다. 기존 PPO critics는 각 prefix value를 독립적으로 예측하며, reasoning trajectory를 따라 발생하는 value evolution을 명시적으로 모델링하지 못합니다 [Figure 1]. 이러한 한계는 token-level credit assignment의 정확성을 저하시키고, 특히 긴 Chain-of-Thought (CoT) 추론에서 문제가 됩니다. 경험적으로, PPO critic은 정확한 단계에서 오답으로 넘어갈 때 발생하는 Monte Carlo (MC) estimated value drop을 예측하지 못하고 오히려 반대 부호를 예측하는 등, value evolution의 방향성을 제대로 추적하지 못하는 것으로 나타났습니다 [Figure 1]. 또한, 이러한 예측의 부호 정확도(Sign Accuracy)는 학습 과정 내내 약 50% 수준에 머무르거나 심지어 감소하여, 기존 PPO critics가 value evolution을 안정적으로 학습하지 못하고 noisy credit assignment를 유발함을 보여줍니다 [Figure 1]. 이 문제점은 LLM의 추론 성능 향상을 저해하는 핵심 요인으로 작용하며, critic이 연속적인 상태에 걸쳐 value evolution을 더 잘 모델링하고 학습할 필요성을 제기합니다.

3. Method & Key Results (제안 방법론 및 핵심 결과)

저자들은 LLM reasoning에서 value evolution을 명시적으로 모델링하기 위해 Tracking-Value PPO (TPPO)를 제안합니다. TPPO는 연속적인 prefix value 간의 관계를 분석하여, log-odds space에서 가산적인(Additive) 업데이트를 갖는 Bayesian decomposition 구조를 밝혀냅니다 [Proposition 4.1, Equation 8]. 이 가산적 구조를 활용하여 TPPO는 cumulative value parameterization을 설계합니다. 이는 위치별(Position-wise) 스칼라 출력(Scalar Outputs)을 value logits으로 누적(Accumulate)한 다음, 시그모이드(Sigmoid) 함수를 적용하여 prefix value를 재구성하는 방식입니다 [Figure 2]. 제안된 critic은 표준 PPO와 동일한 terminal-outcome supervision을 사용하여 학습되며, 추가적인 롤아웃(Rollouts)이나 과정 수준(Process-level)의 어노테이션(Annotations)을 요구하지 않습니다. 재구성된 state values는 GAE (Generalized Advantage Estimation) 및 표준 PPO objective에 직접 통합됩니다.

실험 결과, TPPO는 다양한 추론 태스크에서 PPO 대비 상당한 성능 향상을 보였습니다. 수학적 추론 태스크에서 TPPO는 Qwen3-4B-Base 모델에서 평균 정확도(Average Accuracy)를 7.0 percentage points 개선했으며, Qwen3-8B-Base에서는 8.4 percentage points 개선하여 모든 베이스라인을 능가했습니다 [Table 2]. 이러한 성능 향상은 countdown 및 scientific coding tasks 등 다른 태스크와 Qwen2.5-Math-1.5B 같은 다른 모델에도 일관되게 적용됩니다 [Figure 4, Table 4]. TPPO는 학습 초기부터 validation accuracy 및 Pass@4에서 PPO 대비 지속적인 우위를 유지했으며, 더 작은 actor gradient norms와 느린 엔트로피(Entropy) 감소를 통해 더 안정적인 학습 업데이트를 시사했습니다 [Figure 3]. 진단 분석 결과, TPPO는 value evolution을 더 정확하게 추적하며, 특히 MC-estimated value changes와의 부호 일치도(Sign Agreement)가 τ=0에서 3.9~6.8 percentage points, τ=1에서 4.9~7.4 percentage points 향상되었습니다 [Figure 5]. 또한, MC-based reference advantages와 비교했을 때, TPPO의 advantage estimates는 더 높은 cosine similarity와 sign agreement (마지막 네 체크포인트에서 15.3~19.9 percentage points 개선)를 보여, credit assignment의 품질이 크게 개선되었음을 입증했습니다 [Figure 6]. 이러한 결과는 cumulative parameterization이 value evolution을 효과적으로 모델링하여 PPO의 한계를 극복했음을 나타냅니다.

4. Conclusion & Impact (결론 및 시사점)

본 논문은 LLM reasoning을 위한 reinforcement learning에서 value evolution을 모델링하는 새로운 cumulative parameterization인 TPPO를 제시합니다. 저자들은 연속적인 prefix value가 log-odds space에서 가산적인 업데이트를 통해 Bayesian decomposition을 이룬다는 것을 밝혀냈으며, TPPO는 이 구조를 활용하여 위치별 스칼라 출력을 누적하고 재구성된 값에 대한 terminal-outcome supervision을 통해 학습됩니다. 실험 결과, TPPO는 수학적 추론, 카운트다운, 과학 코딩 태스크 등 다양한 시나리오에서 PPO 대비 일관된 성능 향상을 보였으며, 이는 두 가지 Qwen3 모델 크기뿐만 아니라 다른 모델군에서도 나타났습니다. Critic 분석은 TPPO가 Monte Carlo values와 더 강한 일치도를 보이고 MC-based reference advantages와 더 잘 정렬되는 것을 입증하여 credit assignment의 품질 향상을 시사합니다. 이러한 연구 결과는 reasoning trace를 따라 value evolution을 모델링하는 것이 추가적인 롤아웃 샘플링(Rollout Sampling)이나 과정 수준 라벨(Process-level Labels) 없이도 credit assignment를 개선할 수 있음을 보여줍니다. TPPO는 LLM의 복잡한 문제 해결 능력과 일반화 가능성을 향상시켜, 해당 분야의 발전에 크게 기여할 것입니다.

⚠️ 알림: 이 리뷰는 AI로 작성되었습니다.