Tactile VLA fine-tuning and real-robot control with π0.5-DROIDπ0.5-DROID 触觉 VLA 微调与真机控制
Adapting π0.5-DROID to visual–tactile demonstrations: reproducible data preparation, shared image adapters and three completed LoRA fine-tunes.基于视觉触觉示教适配 π0.5-DROID，建立可追溯的数据处理与共享图像接口，完成三组 LoRA 微调。
I lead the Bristol-VLA project and its VLA workstream, working with colleagues on hardware and demonstration collection. My focus is turning multimodal observations into auditable training inputs and deployable policy interfaces for Franka manipulation.我负责 Bristol-VLA 项目及其 VLA 路线，与团队协作开展硬件集成和示教采集，重点将多模态观测转为可核验的训练输入，以及面向 Franka 操作的策略接口。
Explore the implementation & experiments查看技术实现与实验对照- Role项目角色
- Project Lead · VLA Development项目负责人 · VLA 研发
- Period项目时间
- Jun 2026–present
- Support项目资助
- NVIDIA Developer Program · 8× H100 training accessNVIDIA Developer Program · 8 张 H100 训练资源
Build & progress具体工作与进展
What is in place目前做到哪一步
Teleoperation & demonstrations遥操与操作示教
A native libfranka leader–follower setup records front, wrist and two tactile RGB streams with robot state and accepted gripper intent. The dataset contains 30 demonstrations labelled successful by the operator.使用原生 libfranka 主从遥操，采集前视、腕部及双路触觉 RGB，并记录机械臂状态与已接受的夹爪指令。数据集包含 30 段由操作员标注成功的示教。
Completed已完成
Tactile input adaptation & LoRA fine-tuning触觉输入适配与 LoRA 微调
Completed visual, visual–tactile and reviewed-export experiments, each independently initialised from native π0.5-DROID and trained for 5,000 LoRA steps. Paired tactile RGB images share the third model image slot.完成视觉基线、视觉触觉和人工审核导出三组实验，均从原生 π0.5-DROID 独立初始化并进行 5,000 步 LoRA 微调；双路触觉 RGB 合成为模型的第三路图像输入。
Completed已完成
Local inference setup本地推理环境
Set up OpenPI / π0.5 inference on an RTX 5090. Model deployment must keep each checkpoint paired with its image adapter and normalisation assets; the repository records the inference setup separately from robot evaluation.搭建 RTX 5090 上的 OpenPI / π0.5 推理环境。部署时需将 checkpoint 与对应图像适配器、归一化文件配套使用；仓库分别记录推理环境与真机评测状态。
Completed已完成
Closed-loop comparison真机闭环对比
Preparing a common red-cube pick-up protocol with fixed setup, success criteria and per-trial records. A comparable closed-loop trial set has not yet been completed for these three checkpoints.正在准备统一的红方块抓取评测，固定环境、成功判据并逐次记录结果。三组 checkpoint 尚未完成可直接比较的真机闭环试验集。
In progress进行中
01 / problem问题
Bringing contact observations into a VLA policy让接触观测参与 VLA 策略
The research question is whether tactile observations can help a manipulation policy respond to contact conditions that are hard to infer from external vision. The current implementation studies red-cube pick-up with front, wrist and fingertip imagery.研究关注触觉观测能否帮助操作策略判断外部视觉难以捕捉的接触状态。当前实现以抓取红方块为任务，结合前视、腕部和指尖触觉图像。
02 / contribution工作
My contribution我负责的工作
I lead the project and VLA route. Hardware development and demonstration collection are team efforts; my work connects data quality, model adaptation and deployment contracts.我担任项目负责人并负责 VLA 路线，硬件开发与示教采集由团队协作推进；我的重点是打通数据质量、模型适配与部署接口。
- Data: defined state/action interfaces, validity checks and continuous-segment filtering; prepared episode-level splits and training-only statistics for the reviewed export.数据：定义状态与动作接口、有效性检查及连续片段筛选；为人工审核导出建立 episode 级划分和仅基于训练数据的统计量。
- Policy: integrated two tactile RGB streams as an additional model image and completed three independently initialised LoRA experiments.策略：将双路触觉 RGB 合成额外的模型图像输入，完成三组独立初始化的 LoRA 实验。
- Deployment: made image preprocessing consistent between training and inference, and documented checkpoint, adapter and normalisation compatibility.部署：统一训练与推理的图像预处理，记录 checkpoint、适配器与归一化文件之间的配套关系。
03 / approach方法
Demonstrations → validation → fine-tuning → deployment示教 → 校验 → 微调 → 部署
The baseline and tactile experiments use the same 8,116 selected frames from 30 demonstrations. The tactile variant adds a composed A/B image while preserving visual pixels, robot states, actions and training settings. A separate reviewed export introduces an episode-level holdout and training-only normalisation.视觉基线与触觉实验使用相同的 30 段示教、8,116 个筛选后帧。触觉版本增加 A/B 合成图，同时保持视觉像素、机器人状态、动作和训练设置一致。另设人工审核导出版本，采用 episode 级验证集及仅基于训练数据的归一化。
- A ZED 2i provides the front view and a RealSense D435 the wrist view; two USB cameras provide vision-based tactile RGB.ZED 2i 提供前视，RealSense D435 提供腕部视角，两路 USB 相机提供视觉触觉 RGB。
- The state and physical action supervision each have eight dimensions. Training uses an action horizon of 15; this does not prescribe executing all 15 steps open-loop.状态与物理动作监督均为 8 维，训练动作窗口为 15 步；这不等于部署时必须开环执行全部 15 步。
- The archived libfranka controller includes relative joint following, command limits and stale-state checks, with a DROID impedance-control interface.归档的 libfranka 控制代码包含相对关节跟随、指令限幅与状态过期检查，并调用 DROID 阻抗控制接口。
04 / outcome结果
Training completed; robot comparison is next已完成训练，下一步是真机对比
All three runs completed 5,000 steps and saved their final checkpoints. Data and preprocessing checks establish input consistency. They do not establish better grasp stability or task success: the shared closed-loop robot evaluation is still being prepared.三组实验均完成 5,000 步训练并保存最终 checkpoint。数据与预处理检查验证了输入一致性，但不能据此得出抓取更稳定或成功率更高的结论；统一的真机闭环评测仍在准备中。
Bristol-VLA / engineering notesBristol-VLA / 工程实现
Inside the VLA implementationVLA 的具体实现
Native PI0.5-DROID, adapted with LoRA and a tactile image interface. The work connects demonstration validation, multimodal observations, and a consistent action contract.基于原生 PI0.5-DROID，通过 LoRA 与触觉图像接口进行适配，贯通演示校验、多模态观测和统一动作契约。
- 01
4 → 3
Camera inputs相机输入
A ZED front view, D435 wrist view, and two tactile RGB views. The tactile pair shares one image slot.ZED 前视、D435 腕部视角与双路触觉 RGB 图像；两路触觉图共享一个图像槽。
- 02
224 × 224
Policy observations策略观测
Three RGB image slots, a task prompt, seven measured joint positions, and measured gripper state. The visual baseline masks out the third image.三个 RGB 图像槽、任务文本、七维实测关节位置与夹爪状态。纯视觉基线关闭第三图像槽。
- 03
H15 × 8
Action supervision动作监督
15-step chunks with eight physical action values per step: seven normalized joint-target deltas and accepted binary gripper intent.每个动作块包含 15 步，每步 8 维物理动作：七维归一化关节目标增量与一维已接受的二值夹爪意图。
Three independent fine-tunes三个独立微调实验
Each starts from native pi05_droid · 5,000 steps · batch 32 · seed 42 · H15均从原生 pi05_droid 独立初始化 · 5,000 步 · batch 32 · seed 42 · H15
Scroll sideways to compare all columns →左右滑动，查看完整实验对照 →
| Experiment实验 | Inputs输入 | Data数据 | Split划分 | Training训练 |
|---|---|---|---|---|
lora30 | Front + wrist · 2 image slots前视 + 腕部 · 2 个图像槽 | Rule-filtered · 8,116 frames / 253 continuous segments规则筛选 · 8,116 帧 / 253 个连续片段 | All 30 episodes for training; no held-out split30 条演示全部用于训练；无独立验证划分 | Completed已完成5,000 steps |
lora30_tactile | Front + wrist + tactile pair · 3 image slots前视 + 腕部 + 双路触觉 · 3 个图像槽 | Same 8,116 frames and 253 segments; tactile images added相同 8,116 帧与 253 个片段；增加触觉图像 | Same 30 training episodes; no held-out split相同 30 条训练演示；无独立验证划分 | Completed已完成5,000 steps |
lora30_export | Front + wrist + tactile pair · 3 image slots前视 + 腕部 + 双路触觉 · 3 个图像槽 | Human-reviewed · 11,080 total frames; 3,432 train / 1,092 validation H15 windows人工审核 · 共 11,080 帧；3,432 个训练 / 1,092 个验证 H15 窗口 | 24 training / 6 validation episodes; split by source episode24 条训练 / 6 条验证演示；按原始演示划分 | Completed已完成5,000 steps |
Deterministic data filtering确定性数据筛选
The automatic pipeline checks file hashes, readable images, recorder validity flags, and finite state/action values. It requires image age ≤200 ms and a sampling interval within ±25 ms of the nominal 66.7 ms (15 Hz), splits at invalid frames or gaps, and keeps runs of at least 16 frames. Of 12,974 raw frames, 8,116 remain. This uses explicit rules, without model-generated labels or action interpolation.自动流程检查文件哈希、图像可读性、采集有效标记及状态与动作数值。图像时效不超过 200 ms，采样间隔须在名义 66.7 ms（15 Hz）的 ±25 ms 内；遇到无效帧或间断即分段，只保留至少 16 帧的连续段。从 12,974 个原始帧保留 8,116 帧，使用明确规则，不生成模型标签或插值动作。
A controlled visual–tactile input comparison保持一致的视觉与触觉输入对照
For lora30 and lora30_tactile, all 8,116 selected frames were compared: front/wrist images, states, and actions match. Training settings and native DROID normalization also stay the same. The intended change is the tactile pair in the third image slot. This establishes input consistency; robot performance remains to be measured.lora30 与 lora30_tactile 逐帧核对了全部 8,116 帧，前视与腕部图像、状态、动作均一致；训练设置及原生 DROID 归一化也保持相同。输入变化是第三图像槽中的双路触觉图；这些检查确认数据一致性，机器人性能仍待评测。
Shared preprocessing, two composition variants共享预处理与两种触觉合成方式
In lora30_tactile, each tactile image is aspect-fitted into a 224×112 region before stacking. Training and inference share an adapter that rejects missing sensors; preprocessing tests check pixel consistency and verify that both inputs contribute. lora30_export instead stacks the pair at equal source resolution, then resizes the whole image with padding. These are different compositions. The implementation uses the native image interface.lora30_tactile 将两路触觉图分别等比缩放至 224×112 区域后上下拼接。训练与推理共用适配器，缺失传感器时报错；预处理测试检查像素一致性及两路图像是否都参与输入。lora30_export 则先按相同源分辨率上下拼接，再将整幅图等比缩放并补边；两版合成方式不同。当前实现复用原生图像接口。
Action contract and train-only normalization动作契约与仅使用训练集的归一化
Joint supervision is (target joint position − measured joint position) / 0.2, bounded to [−1, 1], plus accepted binary gripper intent. The eight physical action values are padded internally by the model. The reviewed export samples complete H15 windows without crossing gaps or padding endpoints, and recomputes normalization using only the 24 training episodes. Validation episodes are excluded from these statistics.关节监督为（目标关节位置 − 实测关节位置）/ 0.2，范围限制在 [−1, 1]，另加已接受的二值夹爪意图。8 维物理动作在模型内部补齐维度。人工审核导出只采样完整 H15 窗口，不跨越间断或补齐端点；归一化仅根据 24 条训练演示重新计算，排除验证集。
Evidence论文与材料