{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigqab4zckfb64gtn2wid5ykssbcbovj2x56lble7e2uhvjklzrbge",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnyy556w6ey2"
  },
  "path": "/t/how-to-use-the-lerobot-eval-command/176555#post_4",
  "publishedAt": "2026-06-11T09:56:20.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "lerobot_eval.py",
    "LeRobot Installation",
    "Policy Deployment / lerobot-rollout",
    "LeRobot Cheat Sheet",
    "LeRobot Cameras",
    "OpenCVCameraConfig settings fail under backend=ANY on Linux; the same settings work with backend=V4L2",
    "Lerobot Camera Backend Issues",
    "SmolVLA docs",
    "Policy image features vs robot cameras mismatch #1620",
    "Best camera setup/order for SmolVLA + SO101 #1763",
    "SmolVLA / camera-count related debugging #2753",
    "lerobot-eval with SmolVLA and env mismatch #2246"
  ],
  "textContent": "Hmm… In that case, seems this may not be a single issue:\n\n* * *\n\n## Short version\n\nI would split this into **two different layers** :\n\n  1. **The`lerobot-eval` problem**\n\n     * This may be an installation / optional-dependency issue.\n     * `lerobot-eval` can require extras such as `lerobot[evaluation]`, the policy extra, and, for PushT simulation, the `pusht` environment extra.\n  2. **The camera resolution/FPS problem in`record` / `train`**\n\n     * This sounds like a real camera configuration/backend problem.\n     * If `640x480` is not being applied even in `record` / `train`, I would not treat that as only an `eval` problem.\n     * On Linux/OpenCV cameras, I would specifically check whether OpenCV is using `FFMPEG` instead of `V4L2`, and whether `MJPG` / width / height / FPS settings are actually being applied.\n\n\n\nSo my current guess is:\n\n\n    Issue A: `lerobot-eval` became unusable because the required optional dependencies were missing or the environment got into a partially-installed state.\n\n    Issue B: the camera settings not applying in `record` / `train` is probably a separate real-camera backend/configuration issue.\n\n\n## Why I would separate these\n\nYour latest detail changes the triage.\n\nYou said:\n\n\n    First, when I ran the eval command, I got an error indicating that some dependencies were missing.\n    After that, the eval command became unusable.\n\n    Then I tried using the record and train commands, and I noticed that the camera settings were incorrect.\n    Even after changing the resolution setting to 640x480, the changes were not applied.\n\n\nThat sounds less like one single SmolVLA problem, and more like two failures that happened close together:\n\nSymptom | Likely layer | First thing to check\n---|---|---\n`lerobot-eval` says dependencies are missing | install / optional extras / environment state | `lerobot[evaluation]`, `lerobot[pusht]`, policy extras, clean environment\n`record` / `train` camera settings are wrong | real camera config / OpenCV backend / device mode | `lerobot-find-cameras opencv`, V4L2 vs FFMPEG, MJPG, supported camera modes\n`640x480` is configured but not applied | camera backend or camera mode negotiation | `v4l2-ctl`, OpenCV backend, `fourcc`, actual stream profile\nSmolVLA runs but behaves badly | policy/dataset/robot observation mismatch | camera keys, camera count, camera order, dataset metadata\n\n## Layer 1: `lerobot-eval` probably needs optional extras\n\nThe `lerobot-eval` script itself says it evaluates a policy on an environment, and its docstring lists requirements like:\n\n\n    pip install 'lerobot[evaluation]'\n\n\nplus the policy extra, and the environment extra such as:\n\n\n    lerobot[pusht]\n\n\nwhen evaluating in simulation.\n\nReference:\n\n  * lerobot_eval.py\n\n\n\nThe installation docs also say that the base PyPI package:\n\n\n    pip install lerobot\n\n\ninstalls only the default dependencies, and that additional functionality uses extras such as:\n\n\n    pip install 'lerobot[all]'\n    pip install 'lerobot[aloha,pusht]'\n    pip install 'lerobot[feetech]'\n\n\nReference:\n\n  * LeRobot Installation\n\n\n\nSo if the exact missing-dependency error is gone, I would still suspect this area first.\n\n### Minimal checks for the eval side\n\n\n    which python\n    python --version\n    which lerobot-eval\n    lerobot-info\n    pip show lerobot\n    pip freeze | grep -E \"lerobot|gym|gymnasium|pusht|torch|opencv|av|ffmpeg\"\n    lerobot-eval --help\n\n\nIf you are trying to evaluate PushT specifically, I would also check whether the PushT extra is installed in the same environment where `lerobot-eval` is being run.\n\nA clean reinstall may be easier than repairing a partially modified environment:\n\n\n    conda create -n lerobot-clean python=3.12 -y\n    conda activate lerobot-clean\n\n    # choose the install form that matches the docs/version you are using\n    pip install 'lerobot[pusht]'\n    # or, from source:\n    # git clone https://github.com/huggingface/lerobot.git\n    # cd lerobot\n    # pip install -e \".[pusht]\"\n\n\nIf `lerobot-eval` still fails, the most useful thing to post is the **full error traceback** , not just the summary.\n\n## Layer 2: simulation eval vs real robot rollout\n\nI would also keep the command-path distinction explicit.\n\nThere are at least three different-looking paths:\n\nGoal | Typical command family | Main config object\n---|---|---\nPushT / simulation benchmark evaluation | `lerobot-eval --env.type=pusht` | environment config\nReal SO101 policy deployment, current docs | `lerobot-rollout` | robot config + policy config\nReal SO101 policy execution in older examples/releases | `lerobot-record --policy.path ...` | robot config + camera config + policy config\n\n`--env.type=pusht` points toward the simulation/environment-evaluation path. That can be valid if you are evaluating a PushT policy in the PushT environment.\n\nBut if your real goal is:\n\n\n    SO101 + SmolVLA + real physical cameras\n\n\nthen I would not use `--env.type=pusht` as the main mental model.\n\nCurrent docs describe `lerobot-rollout` as the CLI for deploying trained policies on real robots:\n\n  * Policy Deployment / lerobot-rollout\n\n\n\nThe cheat sheet also says that inference means running the trained policy on a robot, and notes that older `v0.5.1` workflows use `lerobot-record` instead of `lerobot-rollout`:\n\n  * LeRobot Cheat Sheet\n\n\n\nSo I would first decide which branch you are in:\n\n\n    A. I want PushT simulation evaluation\n       -> use `lerobot-eval --env.type=pusht`\n       -> install the evaluation + pusht extras\n       -> debug environment/policy compatibility\n\n    B. I want real SO101 robot evaluation\n       -> use the real-robot path\n       -> current docs: `lerobot-rollout`\n       -> older release examples: `lerobot-record --policy.path ...`\n       -> pass cameras through `--robot.cameras`\n\n    C. I am training/recording and the camera setting is wrong\n       -> this is probably not an `eval` issue\n       -> debug camera backend / camera mode / device path\n\n\n## Layer 3: camera resolution/FPS not applying\n\nThe camera part sounds like a separate issue because you said the wrong settings also appear in `record` and `train`.\n\nFor that, I would start with:\n\n\n    lerobot-find-cameras opencv\n\n\nThen compare:\n\n\n    expected width/height/fps\n    actual width/height/fps\n    backend API\n    camera path or index\n    default stream profile\n\n\nLeRobot camera docs:\n\n  * LeRobot Cameras\n\n\n\nIf you are on Linux with USB/OpenCV cameras, I would also check the actual camera modes through V4L2:\n\n\n    v4l2-ctl --list-devices\n    v4l2-ctl --list-formats-ext -d /dev/video0\n    v4l2-ctl --list-formats-ext -d /dev/video2\n\n\nReplace `/dev/video0` and `/dev/video2` with your actual camera devices.\n\n## Why V4L2 / FFMPEG / MJPG may matter\n\nThere is a very relevant LeRobot issue:\n\n  * OpenCVCameraConfig settings fail under backend=ANY on Linux; the same settings work with backend=V4L2\n\n\n\nThe reported pattern is very close to a “resolution/FPS setting does not apply” situation:\n\n\n    backend=ANY\n    -> OpenCV selects FFMPEG\n    -> fourcc=\"MJPG\" does not apply\n    -> width/height/fps setting fails or is not reliably applied\n\n    backend=V4L2 + fourcc=\"MJPG\"\n    -> the same requested camera settings work\n\n\nThere is also a related Hugging Face forum post from an SO101 user:\n\n  * Lerobot Camera Backend Issues\n\n\n\nThat user saw errors like:\n\n\n    OpenCVCamera(/dev/video0) failed to set fourcc=MJPG\n    OpenCVCamera(/dev/video0) failed to set capture_width=640\n\n\nand temporarily fixed the issue by forcing OpenCV to use Video4Linux2:\n\n\n    cv2.VideoCapture(self.index_or_path, cv2.CAP_V4L2)\n\n\nI am not saying this is definitely your exact bug, but it is a strong candidate if:\n\n\n    - you are on Linux\n    - you use OpenCV cameras\n    - `lerobot-find-cameras opencv` reports FFMPEG backend\n    - width/height/fps settings do not apply\n    - 640x480 works in another camera tool but not through LeRobot\n\n\n## Camera config shape to verify\n\nFor real SO101 camera configuration, I would expect something like this shape, adjusted to your installed LeRobot version:\n\n\n    lerobot-rollout \\\n      --strategy.type=base \\\n      --policy.path=<your_policy_path_or_hub_repo> \\\n      --robot.type=so101_follower \\\n      --robot.port=/dev/ttyACM0 \\\n      --robot.cameras=\"{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}}\" \\\n      --task=\"<your_task>\" \\\n      --duration=60\n\n\nFor older versions, the equivalent may use `lerobot-record` instead of `lerobot-rollout`.\n\nThe important point is not this exact command. The important point is:\n\n\n    real robot camera settings should be passed through the robot/camera config path,\n    not through `--env.type=pusht`.\n\n\n## SmolVLA-specific thing to keep in mind\n\nIf the camera settings are eventually applied correctly but the policy still behaves badly, then I would check a different layer:\n\n\n    Do the camera names/count/order match what the SmolVLA policy expects?\n\n\nSmolVLA uses camera observations, robot state, and language instruction. So a mismatch in image keys can matter even when the camera itself works.\n\nReference:\n\n  * SmolVLA docs\n\n\n\nRelated issues/examples:\n\n  * Policy image features vs robot cameras mismatch #1620\n  * Best camera setup/order for SmolVLA + SO101 #1763\n  * SmolVLA / camera-count related debugging #2753\n  * lerobot-eval with SmolVLA and env mismatch #2246\n\n\n\nThis is probably not the first explanation for “640x480 is not applied,” but it may become relevant after the camera backend/config issue is fixed.\n\n## What I would post next for debugging\n\nIf you ask in the LeRobot Discord or continue here, I would include this exact information:\n\n\n    OS:\n    LeRobot version:\n    Install method: pip / source / uv / conda\n    Python version:\n    Policy path:\n    Robot type:\n    Camera type: OpenCV / RealSense / other\n    Camera model:\n    Full command used for eval:\n    Full command used for record:\n    Full command used for train:\n    Expected resolution/FPS:\n    Actual resolution/FPS:\n\n\nAnd these outputs:\n\n\n    lerobot-info\n    pip show lerobot\n    pip freeze | grep -E \"lerobot|opencv|av|ffmpeg|torch|gym|gymnasium|pusht\"\n    lerobot-eval --help\n    lerobot-record --help\n    lerobot-rollout --help\n    lerobot-find-cameras opencv\n    v4l2-ctl --list-devices\n    v4l2-ctl --list-formats-ext -d /dev/video0\n\n\nIf you have multiple cameras, also run:\n\n\n    v4l2-ctl --list-formats-ext -d /dev/video2\n    v4l2-ctl --list-formats-ext -d /dev/video4\n\n\nusing your actual devices.\n\n## Practical triage order\n\nI would debug in this order:\n\nStep | Question | Why\n---|---|---\n1 | Is `lerobot-eval` failing because optional dependencies are missing? | The original eval error was a dependency error.\n2 | Are you trying to evaluate PushT simulation or a real SO101 robot? | `--env.type=pusht` and real robot rollout are different paths.\n3 | Which LeRobot version are you using? | CLI names changed; current docs use `lerobot-rollout`, older examples may use `lerobot-record`.\n4 | Are camera settings explicitly passed through `--robot.cameras`? | Real camera settings need to be in the robot/camera config.\n5 | What does `lerobot-find-cameras opencv` report? | This shows actual camera backend and default profile.\n6 | Does V4L2 list 640x480 at the desired FPS? | The camera may not support the requested mode in the current format/backend.\n7 | Is OpenCV using FFMPEG instead of V4L2? | This is a known suspect on Linux/OpenCV webcam setups.\n8 | Do policy/dataset/robot camera keys match? | SmolVLA can fail or behave badly if observation keys/count/order differ.\n\n## My best current guess\n\nBased on the new details, I would not assume there is one root cause.\n\nMy best guess is:\n\n\n    1. `lerobot-eval` failed because the environment did not have the required optional dependencies for evaluation / PushT simulation.\n\n    2. The camera resolution/FPS problem in `record` / `train` is a separate real-camera backend/config issue, likely involving OpenCV backend selection, V4L2 vs FFMPEG, MJPG, or device-mode negotiation.\n\n    3. If the camera backend is fixed but the SmolVLA policy still behaves incorrectly, then check camera names/count/order and dataset-policy observation compatibility.\n\n\nSo I would first restore a clean LeRobot environment with the correct extras, then independently debug the camera mode with `lerobot-find-cameras opencv` and `v4l2-ctl`.",
  "title": "How to use the `lerobot_eval` command"
}