Newshot

Flatness Preserves Instruction Following in Vision-Language-Action Models

Summary

arXiv:2606.23641v2 Announce Type: replace Abstract: Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations. This leads to brittle action policies that ignore language instructions in favor of visual shortcuts, despite the base VLM having this knowledge, which is a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We analyze this effect, then show that regularization through explicit flatness-preserving optimization is a simple yet effective technique while finetuning on the exact same data. Specifically, we demonstrate that applying sharpness-aware minimization (SAM) during VLA finetuning significantly improves instruction following by over 55% across…

Original text(via arXiv cs.RO)

Computer Science > Robotics

Title:Flatness Preserves Instruction Following in Vision-Language-Action Models

Abstract:Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations. This leads to brittle action policies that ignore language instructions in favor of visual shortcuts, despite the base VLM having this knowledge, which is a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We analyze this effect, then show that regularization through explicit flatness-preserving optimization is a simple yet effective technique while finetuning on the exact same data. Specifically, we demonstrate that applying sharpness-aware minimization (SAM) during VLA finetuning significantly improves instruction following by over 55% across multiple simulation and real-world benchmarks without additional data, architectural modification, or pretraining. Explicitly preserving flatness better preserves pretrained knowledge compared to other regularization techniques that simply anchor model weights or constrain learning. We further analyze the effect of selective sharpness, quantify its effects, and show that such approaches are complementary to existing guidance techniques.
Comments:This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
Subjects:Robotics (cs.RO)
Cite as:arXiv:2606.23641 [cs.RO]
(or arXiv:2606.23641v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.23641 arXiv-issued DOI via DataCite

Submission history

Access Paper:

  • HTML (experimental)

Additional Features

  • Audio Summary

Current browse context:

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by arXiv cs.RO
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗