ICRA 2026 Workshop · From Data to Decisions · 1st place / 36 teams

Per-Group Error, Not Total MSE

Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

Pau Montagut Bofi^1,2 · Mario García Blasco^1,2 · Tessa Pulli¹ · Markus Vincze¹

¹Vision for Robotics Lab (V4R), ACIN, TU Wien · ²Universitat Jaume I

Paper Code Model Poster

Robot demos

The policy on the Toyota HSR

Three trials from the workshop track, each clip start to finish. Head camera, hand camera and third-person view, side by side, at 2× speed.

HSR picks the mug not at a corner and places it at the missing one · 3 cameras · 2× speed

HSR fails to place the spoon in the empty mug failure case · 3 cameras · 2× speed

HSR opens the microwave, picks up the plate, places it inside, and closes the door 4-step task · highlights · 3 cameras · 2× speed

Abstract

A single MSE can pick the wrong checkpoint

Vision-Language-Action models are usually evaluated offline with a single mean-squared error over the full action vector. For the Toyota Human Support Robot, with 11 degrees of freedom split across four functionally different joint groups (arm, gripper, head, base), that single number hides which group is actually driving the error.

We decompose the offline MSE into the four joint groups and use the decomposition as a checkpoint-selection signal. In a 60-trial real-robot evaluation against three models, the arm-MSE ranking matched the robot ranking, while the total-MSE ranking did not. The π₀.₅ fine-tune built on this analysis won the AIRoA VLA Pipeline Competition at the ICRA 2026 workshop.

The lowest total MSE does not pick the best policy. Per-group MSE is a better signal, and on the Toyota HSR the arm-MSE alone reproduces the robot ranking.

Platform

The Toyota Human Support Robot

The HSR has 11 degrees of freedom split across four functionally different joint groups. A 5-DoF arm, a 1-DoF parallel gripper, a 2-DoF head, and a 3-DoF holonomic base. The four groups have very different scales and dynamics, which is what motivates the per-group decomposition.

11 degrees of freedom, 4 joint groups

Hover or tap a marker on the robot to see what each group does and how hard it is to learn.

Method

Two-phase fine-tuning, per-group MSE for selection

We start from the released SmolVLA backbone and run a generalist phase on the public AIRoA MoMa dataset, which teaches the model the HSR observation layout and the 11-DoF action space. We then continue with a task-specific top-up on a private subset of the AIRoA ICRA 2026 dataset distributed only to the competing teams. For π₀.₅ we use an expert-only fine-tune on top of the workshop baseline.

All offline evaluation is done with our per-group MSE script, which slices the squared error into arm, gripper, head and base rather than averaging the four blindly.

The vision-language backbone stays frozen during fine-tuning. Only the action head is trained.

Per-group MSE

What the aggregate metric hides

The four joint groups converge at very different rates. The gripper falls fast, the head is small and stable, the arm carries most of the manipulation signal, and the base converges last and sets the ceiling on total MSE. Aggregating the four together blends them into a single number whose ranking does not reflect the on-robot ranking.

Per-joint-group MSE across the SmolVLA training schedule, animated. The total draws first, then splits into the four joint groups. Hover the chart for exact values, click a legend label to hide that curve.
Checkpoint naming. PT is the generalist checkpoint after pretraining on AIRoA MoMa. B8 and B32 are task-specific fine-tuning runs with batch size 8 and 32, and the number that follows counts training steps, so B32 40k is the batch-32 run after 40,000 steps.

Results

Arm-MSE matches the robot ranking, total MSE does not

Model	Total MSE (×10⁻³)	Arm MSE (×10⁻³)	Robot score
π₀.₅ baseline (80k)	1.04	0.30	4.00 / 4
π₀.₅ (ours)	0.95	0.59	3.75 / 4
HSR-SmolVLA (40k)	1.61	0.88	3.50 / 4

Robot score is the mean over 20 trials per model on a 4-point rubric. Mann-Whitney U (one-sided), p ≤ 0.010 between the π₀.₅ baseline and either fine-tuned model. The lowest total MSE is not the best policy on the robot, while the arm-MSE column matches the robot ranking. The 60 raw scores and the script that recomputes these statistics are in the repository.

Future work

From diagnostic to training signal

The per-group decomposition is not only a diagnostic. The same analysis can be brought into the training loop itself, for example by weighting the joint groups in the action loss so the optimizer spends its capacity where the robot actually needs it, or by scheduling that weighting as the groups converge at different rates. We are actively investigating this direction and it will be the subject of upcoming work.

Team

V4R Lab, ACIN, TU Wien

The paper is the result of a collaboration at the Vision for Robotics Lab between Pau Montagut Bofi, Mario García Blasco, Tessa Pulli and Markus Vincze. The V4R lab provided access to the Toyota HSR and the GPU infrastructure used for training and offline evaluation.

Citation

BibTeX

@inproceedings{montagut2026pergroup,
  title  = {Per-Group Error, Not Total {MSE}: Fine-Tuning Vision-Language-Action
            Models for 11-{DoF} Mobile Manipulation},
  author = {Montagut Bofi, Pau and Garc\'ia Blasco, Mario and
            Pulli, Tessa and Vincze, Markus},
  booktitle = {ICRA 2026 Workshop on From Data to Decisions},
  year   = {2026},
  eprint = {2606.00253},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url    = {https://arxiv.org/abs/2606.00253}
}