Causal keyframe detection for robot manipulation from complementary visual and proprioceptive predictive surprise.
Keyframing provides a compact temporal abstraction for long-horizon robot manipulation by retaining only task-relevant transitions from otherwise redundant trajectories. We present PULSE, an online and unsupervised keyframe detector that identifies manipulation boundaries from predictive surprise without boundary supervision or future observations. PULSE learns visual and proprioceptive dynamics independently using two causal predictive branches. Visual surprise is measured as local novelty in a compact forecast-trained latent space, while proprioceptive surprise is measured from future-state prediction error. Each modality is robustly calibrated and thresholded independently, and only the resulting peak lists are fused, allowing either branch to detect events that are salient in one modality but weak in the other.
We evaluate PULSE on manually annotated frame-level boundaries from RoboMME, RMBench, and MIKASA-Robo. The fused detector achieves 0.719, 0.681, and 0.749 F1 at ±10 frames, outperforming every evaluated baseline on all three benchmarks and the average of the top three baselines by 15.4%, 15.4%, and 22.5%, respectively. Task-level analysis further shows that visual surprise is especially effective for environment-driven events with weak kinematic signatures, whereas proprioceptive surprise dominates on contact- and robot-state-driven transitions.
A frozen DINOv3 ViT-S/16 produces visual features that are compressed by a learned bottleneck and trained with short-horizon latent forecasting. At inference, the forecast head is discarded and visual surprise is measured as local novelty in the whitened latent trajectory.
A causal GRU forecasts the robot state K steps ahead. Surprise is the residual between the state predicted earlier and the state that actually arrives, capturing motion, contact, and robot-state transitions.
Temporal detector constants are defined in seconds rather than fixed frame counts: W_win = 2.0 s for visual novelty and NMS = 0.65 s for peak suppression, then converted to frames using the dataset FPS.
Visual and proprioceptive signals are robustly calibrated and thresholded separately. Raw scores are never averaged. Only candidate peaks are merged, preserving events that are salient in one modality but absent in the other.
PULSE-Union is the strongest method at ±10 frames on RoboMME, RMBench, and MIKASA-Robo.
| Method | RoboMME | RMBench | MIKASA-Robo |
|---|---|---|---|
| AWE offline | 0.671 | 0.562 | 0.739 |
| KEMO-style hybrid online | 0.585 | 0.564 | 0.417 |
| VQ-Memory online | 0.410 | 0.422 | 0.329 |
| Ours-Proprio online | 0.687 | 0.653 | 0.690 |
| CoSeg offline | 0.613 | 0.645 | 0.678 |
| WorthRemembering online | 0.498 | 0.468 | 0.415 |
| Ours-Visual online | 0.667 | 0.641 | 0.730 |
| Ours-Union online | 0.719 | 0.681 | 0.749 |
Baseline tuning. AWE, KEMO, VQ-Memory, and CoSeg receive per-task hyperparameter selection on labeled validation episodes. PULSE uses no validation labels for detector tuning: its temporal constants are fixed in seconds and converted using dataset frame rate. On MIKASA-Robo, tuning CoSeg changes F1 from 0.216 to 0.678, while tuning PULSE's suppression window changes 0.749 to 0.755.
When the scene changes but the manipulator barely moves, conventional kinematic and proprioceptive keyframing can miss the event entirely.
On the visually driven RoboMME subset, Ours-Visual reaches 0.755 mean F1 versus 0.606 for Ours-Proprio.
Examples include object appearance, unmasking, and scene-state changes with weak robot-motion signatures.
On the contact-dominant RoboMME subset, Ours-Proprio reaches 0.729 mean F1 versus 0.593 for Ours-Visual.
Examples include insertion, engagement, and transitions primarily expressed through robot state.
No single event statistic dominates across both modalities. Local novelty is stronger for vision on RoboMME and RMBench, while prediction residual is stronger for proprioception on RoboMME and MIKASA-Robo.
The bottleneck reduces trainable parameters from 17.1M to 4.0M and substantially improves MIKASA-Robo, where the unbottlenecked model over-fires on sparse episodes.
MSE-only training causes severe variance collapse: 83.9–100% of latent dimensions fall below the variance target, reducing valid surprise peaks and recall.
We presented PULSE, an online and unsupervised keyframe detector for robot manipulation that identifies task-relevant boundaries from predictive surprise in visual and proprioceptive signals. By modeling, calibrating, and thresholding the two modalities independently before fusing only their detected peaks, PULSE preserves events that are salient in one modality but weak in the other. Across RoboMME, RMBench, and MIKASA-Robo, the fused detector achieves 0.719, 0.681, and 0.749 F1 at ±10 frames and outperforms every evaluated baseline. Task-level analysis shows that visual surprise is particularly effective for environment-driven transitions with weak kinematic signatures, while proprioceptive surprise is stronger for contact- and robot-state-driven events.
@inproceedings{pulse2026,
title = {PULSE: Surprise-Based Online Unsupervised Keyframe Detection},
author = {Anonymous Authors},
booktitle = {[Workshop], NeurIPS},
year = {2026}
}