Notes on latent world models and planning
Last updated: September 2026
A probe is not a planner
High in-slice probe scores did not establish a planning advantage for factorization in this comparison. That distinction sounds obvious, but it is easy to lose when a probe score is the cleanest number in an experiment.
The public Ego-World JEPA project provides a small controlled example. It compares factored and monolithic action-conditioned latent world models on PushT. Both variants use the same detector and MPPI controller; the main pair changes the latent factorization while keeping the downstream evaluation protocol fixed.
The appealing hypothesis
Suppose a linear probe can recover task-relevant state from a latent with an \(R^2\) close to one. It is tempting to read that as evidence that the latent contains what planning needs. A factored representation may make the story even more appealing: separate object-relevant components, decode state cleanly, and plan more effectively.
That chain contains a hidden jump. A probe asks whether information can be decoded on the sampled evaluation slice. The probe does not evaluate action-conditioned rollouts or the cost used by MPPI. These are different tests.
The controlled comparison
The main comparison evaluates 50 planning episodes for each representation and training seed. MPPI uses 512 samples, six optimization iterations, and an eight-step horizon.
The table reports an in-slice ridge probe of absolute block pose on the 96 px g1/g2 checkpoints. At seed 0, the displacement probe on the same checkpoints reached \(R^2=0.205\) for the factored model and \(R^2=0.240\) for the monolithic model, reversing the absolute-pose ordering. At seed 1, the displacement scores were \(R^2=0.225\) and \(R^2=0.201\), with the factored model ahead on both probes.
| Model | Training seed | Absolute-pose probe \(R^2\) | Planning successes |
|---|---|---|---|
| Factored | 0 | 0.997 | 6 / 50 |
| Monolithic | 0 | 0.992 | 2 / 50 |
| Factored | 1 | 0.996 | 2 / 50 |
| Monolithic | 1 | 0.993 | 2 / 50 |
Seed 0 appears to favor the factored model, but the reported Fisher test does not support that difference (\(p=0.269\)), and the gap disappears on seed 1. The result does not prove that the models are equivalent—two seeds and low success counts are far too weak for that claim. It does show that near-perfect in-slice decodability was not sufficient for a robust planning improvement in this setup.
The broader 13-run screening set points in the same cautious direction. The reported Spearman association between probe \(R^2\) and planning success is \(\rho=0.273\) with \(p=0.367\). This is a small, incomplete grid, so the absence of a significant association is not evidence of a universal absence. It is a reason not to promote the probe into a planning metric.
What the probe leaves unresolved
Three limitations matter when interpreting the high \(R^2\) values.
First, the probe is evaluated in-slice rather than on a genuinely held-out distribution. Second, auxiliary supervision is directly aligned with some of the decoded properties, so a high score partly verifies that the auxiliary task worked. Third, a one-step readout does not test multi-step rollout behavior or closed-loop distribution shift.
A planning-oriented representation evaluation therefore needs several layers:
- Decodability: is task-relevant state recoverable at all?
- Dynamics: do predicted displacements and action effects remain accurate over the planning horizon?
- Control: does the representation improve closed-loop success under a shared controller and compute budget?
- Stability: does the conclusion survive training seeds and reasonable protocol changes?
Each layer can falsify a stronger claim made from the previous one. Passing the first layer is useful evidence, but not a shortcut through the remaining three.
The useful negative result
The seed-0 planning advantage for factorization did not replicate at seed 1 despite high in-slice probe scores. That narrows the next question: evaluate representations through rollout and decision quality, not only through static recoverability.
The reported metrics and limitations are documented in the multiseed statistics and the project’s results manifest.