How to use the `lerobot_eval` command

Hmm… In that case, seems this may not be a single issue:

---

Short version

I would split this into two different layers :

1. Thelerobot-eval problem

   • This may be an installation / optional-dependency issue.
   • lerobot-eval can require extras such as lerobot[evaluation], the policy extra, and, for PushT simulation, the pusht environment extra.

2. The camera resolution/FPS problem inrecord / train

   • This sounds like a real camera configuration/backend problem.
   • If 640x480 is not being applied even in record / train, I would not treat that as only an eval problem.
   • On Linux/OpenCV cameras, I would specifically check whether OpenCV is using FFMPEG instead of V4L2, and whether MJPG / width / height / FPS settings are actually being applied.

So my current guess is:

Issue A: `lerobot-eval` became unusable because the required optional dependencies were missing or the environment got into a partially-installed state.

Issue B: the camera settings not applying in `record` / `train` is probably a separate real-camera backend/configuration issue.

Why I would separate these

Your latest detail changes the triage.

You said:

First, when I ran the eval command, I got an error indicating that some dependencies were missing.
After that, the eval command became unusable.

Then I tried using the record and train commands, and I noticed that the camera settings were incorrect.
Even after changing the resolution setting to 640x480, the changes were not applied.

That sounds less like one single SmolVLA problem, and more like two failures that happened close together:

Symptom	Likely layer	First thing to check
lerobot-eval says dependencies are missing	install / optional extras / environment state	lerobot[evaluation], lerobot[pusht], policy extras, clean environment
record / train camera settings are wrong	real camera config / OpenCV backend / device mode	lerobot-find-cameras opencv, V4L2 vs FFMPEG, MJPG, supported camera modes
640x480 is configured but not applied	camera backend or camera mode negotiation	v4l2-ctl, OpenCV backend, fourcc, actual stream profile
SmolVLA runs but behaves badly	policy/dataset/robot observation mismatch	camera keys, camera count, camera order, dataset metadata

Layer 1: lerobot-eval probably needs optional extras

The lerobot-eval script itself says it evaluates a policy on an environment, and its docstring lists requirements like:

pip install 'lerobot[evaluation]'

plus the policy extra, and the environment extra such as:

lerobot[pusht]

when evaluating in simulation.

Reference:

• lerobot_eval.py

The installation docs also say that the base PyPI package:

pip install lerobot

installs only the default dependencies, and that additional functionality uses extras such as:

pip install 'lerobot[all]'
pip install 'lerobot[aloha,pusht]'
pip install 'lerobot[feetech]'

Reference:

• LeRobot Installation

So if the exact missing-dependency error is gone, I would still suspect this area first.

Minimal checks for the eval side

which python
python --version
which lerobot-eval
lerobot-info
pip show lerobot
pip freeze | grep -E "lerobot|gym|gymnasium|pusht|torch|opencv|av|ffmpeg"
lerobot-eval --help

If you are trying to evaluate PushT specifically, I would also check whether the PushT extra is installed in the same environment where lerobot-eval is being run.

A clean reinstall may be easier than repairing a partially modified environment:

conda create -n lerobot-clean python=3.12 -y
conda activate lerobot-clean

# choose the install form that matches the docs/version you are using
pip install 'lerobot[pusht]'
# or, from source:
# git clone https://github.com/huggingface/lerobot.git
# cd lerobot
# pip install -e ".[pusht]"

If lerobot-eval still fails, the most useful thing to post is the full error traceback , not just the summary.

Layer 2: simulation eval vs real robot rollout

I would also keep the command-path distinction explicit.

There are at least three different-looking paths:

Goal	Typical command family	Main config object
PushT / simulation benchmark evaluation	lerobot-eval --env.type=pusht	environment config
Real SO101 policy deployment, current docs	lerobot-rollout	robot config + policy config
Real SO101 policy execution in older examples/releases	lerobot-record --policy.path ...	robot config + camera config + policy config

--env.type=pusht points toward the simulation/environment-evaluation path. That can be valid if you are evaluating a PushT policy in the PushT environment.

But if your real goal is:

SO101 + SmolVLA + real physical cameras

then I would not use --env.type=pusht as the main mental model.

Current docs describe lerobot-rollout as the CLI for deploying trained policies on real robots:

• Policy Deployment / lerobot-rollout

The cheat sheet also says that inference means running the trained policy on a robot, and notes that older v0.5.1 workflows use lerobot-record instead of lerobot-rollout:

• LeRobot Cheat Sheet

So I would first decide which branch you are in:

A. I want PushT simulation evaluation
   -> use `lerobot-eval --env.type=pusht`
   -> install the evaluation + pusht extras
   -> debug environment/policy compatibility

B. I want real SO101 robot evaluation
   -> use the real-robot path
   -> current docs: `lerobot-rollout`
   -> older release examples: `lerobot-record --policy.path ...`
   -> pass cameras through `--robot.cameras`

C. I am training/recording and the camera setting is wrong
   -> this is probably not an `eval` issue
   -> debug camera backend / camera mode / device path

Layer 3: camera resolution/FPS not applying

The camera part sounds like a separate issue because you said the wrong settings also appear in record and train.

For that, I would start with:

lerobot-find-cameras opencv

Then compare:

expected width/height/fps
actual width/height/fps
backend API
camera path or index
default stream profile

LeRobot camera docs:

• LeRobot Cameras

If you are on Linux with USB/OpenCV cameras, I would also check the actual camera modes through V4L2:

v4l2-ctl --list-devices
v4l2-ctl --list-formats-ext -d /dev/video0
v4l2-ctl --list-formats-ext -d /dev/video2

Replace /dev/video0 and /dev/video2 with your actual camera devices.

Why V4L2 / FFMPEG / MJPG may matter

There is a very relevant LeRobot issue:

• OpenCVCameraConfig settings fail under backend=ANY on Linux; the same settings work with backend=V4L2

The reported pattern is very close to a “resolution/FPS setting does not apply” situation:

backend=ANY
-> OpenCV selects FFMPEG
-> fourcc="MJPG" does not apply
-> width/height/fps setting fails or is not reliably applied

backend=V4L2 + fourcc="MJPG"
-> the same requested camera settings work

There is also a related Hugging Face forum post from an SO101 user:

• Lerobot Camera Backend Issues

That user saw errors like:

OpenCVCamera(/dev/video0) failed to set fourcc=MJPG
OpenCVCamera(/dev/video0) failed to set capture_width=640

and temporarily fixed the issue by forcing OpenCV to use Video4Linux2:

cv2.VideoCapture(self.index_or_path, cv2.CAP_V4L2)

I am not saying this is definitely your exact bug, but it is a strong candidate if:

- you are on Linux
- you use OpenCV cameras
- `lerobot-find-cameras opencv` reports FFMPEG backend
- width/height/fps settings do not apply
- 640x480 works in another camera tool but not through LeRobot

Camera config shape to verify

For real SO101 camera configuration, I would expect something like this shape, adjusted to your installed LeRobot version:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=<your_policy_path_or_hub_repo> \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}}" \
  --task="<your_task>" \
  --duration=60

For older versions, the equivalent may use lerobot-record instead of lerobot-rollout.

The important point is not this exact command. The important point is:

real robot camera settings should be passed through the robot/camera config path,
not through `--env.type=pusht`.

SmolVLA-specific thing to keep in mind

If the camera settings are eventually applied correctly but the policy still behaves badly, then I would check a different layer:

Do the camera names/count/order match what the SmolVLA policy expects?

SmolVLA uses camera observations, robot state, and language instruction. So a mismatch in image keys can matter even when the camera itself works.

Reference:

• SmolVLA docs

Related issues/examples:

• Policy image features vs robot cameras mismatch #1620
• Best camera setup/order for SmolVLA + SO101 #1763
• SmolVLA / camera-count related debugging #2753
• lerobot-eval with SmolVLA and env mismatch #2246

This is probably not the first explanation for “640x480 is not applied,” but it may become relevant after the camera backend/config issue is fixed.

What I would post next for debugging

If you ask in the LeRobot Discord or continue here, I would include this exact information:

OS:
LeRobot version:
Install method: pip / source / uv / conda
Python version:
Policy path:
Robot type:
Camera type: OpenCV / RealSense / other
Camera model:
Full command used for eval:
Full command used for record:
Full command used for train:
Expected resolution/FPS:
Actual resolution/FPS:

And these outputs:

lerobot-info
pip show lerobot
pip freeze | grep -E "lerobot|opencv|av|ffmpeg|torch|gym|gymnasium|pusht"
lerobot-eval --help
lerobot-record --help
lerobot-rollout --help
lerobot-find-cameras opencv
v4l2-ctl --list-devices
v4l2-ctl --list-formats-ext -d /dev/video0

If you have multiple cameras, also run:

v4l2-ctl --list-formats-ext -d /dev/video2
v4l2-ctl --list-formats-ext -d /dev/video4

using your actual devices.

Practical triage order

I would debug in this order:

Step	Question	Why
1	Is lerobot-eval failing because optional dependencies are missing?	The original eval error was a dependency error.
2	Are you trying to evaluate PushT simulation or a real SO101 robot?	--env.type=pusht and real robot rollout are different paths.
3	Which LeRobot version are you using?	CLI names changed; current docs use lerobot-rollout, older examples may use lerobot-record.
4	Are camera settings explicitly passed through --robot.cameras?	Real camera settings need to be in the robot/camera config.
5	What does lerobot-find-cameras opencv report?	This shows actual camera backend and default profile.
6	Does V4L2 list 640x480 at the desired FPS?	The camera may not support the requested mode in the current format/backend.
7	Is OpenCV using FFMPEG instead of V4L2?	This is a known suspect on Linux/OpenCV webcam setups.
8	Do policy/dataset/robot camera keys match?	SmolVLA can fail or behave badly if observation keys/count/order differ.

My best current guess

Based on the new details, I would not assume there is one root cause.

My best guess is:

1. `lerobot-eval` failed because the environment did not have the required optional dependencies for evaluation / PushT simulation.

2. The camera resolution/FPS problem in `record` / `train` is a separate real-camera backend/config issue, likely involving OpenCV backend selection, V4L2 vs FFMPEG, MJPG, or device-mode negotiation.

3. If the camera backend is fixed but the SmolVLA policy still behaves incorrectly, then check camera names/count/order and dataset-policy observation compatibility.

So I would first restore a clean LeRobot environment with the correct extras, then independently debug the camera mode with lerobot-find-cameras opencv and v4l2-ctl.