Efficiency of On-Policy Distillation
Making on-policy distillation cheaper and faster while preserving teacher quality and student gains.
Salesforce AI Research · Our north star
Agents that get better with use
We aim for agents that learn from the work they do, bring useful lessons to unfamiliar tasks and environments, and keep improving as experience accumulates.
What progress looks like
TransferBetter on unfamiliar tasks
PersistenceGains that last beyond one episode
AutonomyLess task-specific guidance over time
A cycle of self-improvement
Our approach links fast adaptation through memory with lasting model improvement. Better models then produce richer experience for the next round.
Memory loopAdapts between tasks
Memory v1Experience
Memory v2Guidance
Keep trajectories and reusable strategies. Curate what the next task needs.
Model loopImproves through training
Model v1Student policy
Model v2Stronger policy
Learn from verified outcomes and memory-guided teaching.
Four questions that guide our research
Each addresses a bottleneck between raw experience and transferable capability.
01 · What persists?
Across the cycle · What earns reinforcement?
Agent memory
Which experiences should we keep, and when should we turn them into guidance for a new task?
Explore papers 02 · What helps this student?Adaptive teaching
What should a teacher reveal, and how can feedback improve the guidance it gives?
Explore papers 03 · What becomes capability?On-policy distillation
How do we use the student's own attempts to turn useful guidance into model behavior?
Explore papersRL post-training
Study rewards, environments, and training stages so the behaviors we reinforce transfer beyond one task.
The peopleMeet the team
ICs
Interns
Managers
Skill-Evolution Playground
🚧 Playground under construction
Task
❄️
Executor
Trajectory
Curator
Memory bank
Retriever
Self-judge
No memory path — experience ends with the episode.
Overview
Environment success across every available setting. Epoch 0 is the canonical no-skill baseline.
Mechanisms
Model pair
curator / self-judge
Task types
Loading experiment summary…
Config
Show more configHide extra config
- Agent loopReAct · <think> · <action>
- Historylast 3 steps
Evolution replay
No completed production run is available for this configuration.
Saved trajectories
Trajectory Diff Checker
Publications
Papers on self-improving agents, newest first.
Themes
Sep 2026
arXiv:2609.27334
JitMem
Learning to Curate Task-Adaptive Memory for LLM Agents
Keeps past experience whole and curates a short, task-adaptive briefing at read time — when the next task is known — instead of compressing memory too early.
+16.2ALFWorld
+16.3WebShop
+3.9τ²-bench
Sep 2026
arXiv:2609.05295
RISE
Recursive Improvement via Self-Extrapolating Policy Distillation
Builds a synthetic future teacher from the model’s own RLVR trajectory via self-extrapolation, then distills dense token-level targets — no external teacher required.
No externalteacher
1.3–1.6×GRPO wall time
Math→Agentstask suite
Aug 2026
arXiv:2608.22631
RIVER
Learning Generalizable Behaviors for Terminal Agents
Filters noisy synthetic terminal environments and shapes rewards so RL reinforces transferable multi-turn behaviors — using <30% of TMax data for larger gains.
<30%TMax envs used
+106%RL gain · TB-Lite
19.4River-8B avg
Jul 2026
NeurIPS 2026
arXiv:2607.01480
PMD
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Converts cross-episode signals into reusable procedural memory and distills it into the policy’s weights during training. Memory is a training scaffold, so inference is memory-free.
3.8–5.5%SciKnowEval
7.9–13.6%LiveCodeBench
Memory-freeinference
May 2026
NeurIPS 2026
arXiv:2605.15113
VPD
Learning from Language Feedback via Variational Policy Distillation
Reframes on-policy self-distillation as variational EM: the teacher is refined on trajectory outcomes, then distilled into the student’s rollouts, with a dynamic trust region inside a shared-weight network.
49.6%LiveCodeBench v6
+2.7–4.7%SciKnowEval AVG
Shared-weightadaptive trust region
PMD+
Next-generation procedural memory distillation with richer memory abstractions and stronger co-evolution.
RL, SFT, and On-Policy Distillation Interplay
How reinforcement learning, supervised fine-tuning, and on-policy distillation interact — and when to combine them.