Efficiency of On-Policy Distillation
Making on-policy distillation cheaper and faster while preserving teacher quality and student gains.
Salesforce AI Research · Research program
Self-Evolving Agents
We study agents that improve from their own experience — memory, language feedback, and policy updates that accumulate over time. The goal is self-improvement; distillation is the means.
Evolution Playgroundtest-time evolution
The model weights stay frozen. What changes is the experience the agent carries: each attempt is checked, curated into memory, gated for what generalizes, and reused on the next task. That loop is test-time evolution — improvement without retraining.
Taskthe problem the agent faces right now
Memoryskills, notes, or traces that survived gating
Executorfrozen policy that acts — weights never update
Trajectorythe attempt: what was seen and done
Verifieroutcome signal — success, failure, or reward
Curatorturns the attempt into a candidate memory update
Gatekeeps what generalizes; drops what doesn’t
Run dashboardMemCurator on ALFWorld
Scrub the timeline across tasks: trajectory bank, memory curator, success-rate charts, and a fail→success case study. Full page: memcurator.html.
Publications
Papers on self-improving agents, newest first.
Themes
PMD+
Next-generation procedural memory distillation with richer memory abstractions and stronger co-evolution.
RL, SFT, and On-Policy Distillation Interplay
How reinforcement learning, supervised fine-tuning, and on-policy distillation interact — and when to combine them.
Draft · Sep 2026
Interactive demo
MemCurator
Just-in-Time Memory for LLM Agents
Keeps past experience whole and curates a short, task-adaptive briefing at read time — when the next task is known — instead of compressing memory too early.
77.4%ALFWorld · RL
3benchmarks
Read-timecuration
Sep 2026
arXiv:2609.05295
RISE
Recursive Improvement via Self-Extrapolating Policy Distillation
Builds a synthetic future teacher from the model’s own RLVR trajectory via self-extrapolation, then distills dense token-level targets — no external teacher required.
No externalteacher
1.3–1.6×GRPO wall time
Math→Agentstask suite
Aug 2026
arXiv:2608.22631
RIVER
Learning Generalizable Behaviors for Terminal Agents
Filters noisy synthetic terminal environments and shapes rewards so RL reinforces transferable multi-turn behaviors — using <30% of TMax data for larger gains.
<30%TMax envs used
+106%RL gain · TB-Lite
19.4River-8B avg
Jul 2026
arXiv:2607.01480
PMD
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Converts cross-episode signals into reusable procedural memory and distills it into the policy’s weights during training. Memory is a training scaffold, so inference is memory-free.
3.8–5.5%SciKnowEval
7.9–13.6%LiveCodeBench
Memory-freeinference
May 2026
arXiv:2605.15113
VPD
Learning from Language Feedback via Variational Policy Distillation
Reframes on-policy self-distillation as variational EM: the teacher is refined on trajectory outcomes, then distilled into the student’s rollouts, with a dynamic trust region inside a shared-weight network.
49.6%LiveCodeBench v6
+2.7–4.7%SciKnowEval AVG
Shared-weightadaptive trust region