{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibcb5seanjv4zwrzlnnyegg5yltjbhx6t336or6l5tbnxfokfsqz4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnju3urqeg72"
  },
  "path": "/t/how-to-use-the-lerobot-eval-command/176555#post_2",
  "publishedAt": "2026-06-05T08:51:59.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "the LeRobot Discord",
    "LeRobot README — Inference & Evaluation",
    "lerobot_eval.py",
    "LeRobot Policy Deployment / lerobot-rollout",
    "LeRobot Cheat Sheet",
    "Inference lerobot/smolvla_base with SO-101 failed #2210",
    "OpenCVCameraConfig backend=ANY vs V4L2 issue #3198",
    "Lerobot Camera Backend Issues",
    "LeRobot Cameras docs",
    "SmolVLA docs",
    "Policy image features vs robot cameras mismatch #1620",
    "Best camera setup/order for SmolVLA + SO101 #1763",
    "Debugging poor eval with SmolVLA and two cameras #2753",
    "Problems with running pretrained lerobot/smolvla_base #2374"
  ],
  "textContent": "Since this is Physical AI, I would still recommend asking in the LeRobot Discord for a definitive answer. But at first glance, it looks like there may be a few common traps here:\n\n* * *\n\n## Short version\n\nI would not debug this as a camera-resolution bug first.\n\nThe first thing I would check is whether you are mixing two different LeRobot paths:\n\n  1. **Simulation/environment evaluation**\nExample shape: `lerobot-eval --env.type=pusht`\n\n  2. **Real SO101 + SmolVLA + physical cameras**\nExample shape: `lerobot-rollout` or older `lerobot-record`, with explicit `--robot.type`, `--robot.port`, `--robot.cameras`, and `--policy.path`\n\n\n\n\nYour description sounds like a real SO101 + SmolVLA + camera setup, but the command you showed uses:\n\n\n    --env.type=pusht\n\n\nThat is the first thing I would separate.\n\n## Decision tree\n\nQuestion | If yes | If no\n---|---|---\nAre you intentionally evaluating in the PushT simulation environment? | `lerobot-eval --env.type=pusht` may be the right direction. Debug the PushT env / policy / eval config. | Go to the next question.\nAre you trying to run a policy on a real SO101 with physical cameras? | Do not start from `--env.type=pusht`. Use the real-robot rollout/record path and pass camera settings through `--robot.cameras`. | The answer depends on the actual environment you want to evaluate.\nAre you already using the real-robot path and explicitly passing camera width/height/fps? | Then debug camera backend / device identity / FPS / USB bandwidth / V4L2 vs FFMPEG. | First fix the command path and camera config.\nDoes the policy expect 3 cameras, but the robot/dataset provides different camera names/count/order? | Check SmolVLA image features, dataset metadata, camera keys, and rename maps. | Then focus on the camera backend/device layer.\n\n## Why `--env.type=pusht` looks suspicious here\n\nLeRobot examples often use `lerobot-eval` with an environment such as `pusht` or `libero` when evaluating a policy in a benchmark/simulation-style environment.\n\nFor example, the LeRobot README shows evaluation with an environment argument like:\n\n\n    lerobot-eval \\\n      --policy.path=lerobot/pi0_libero_finetuned \\\n      --env.type=libero \\\n      --env.task=libero_object \\\n      --eval.n_episodes=10\n\n\nSee: LeRobot README — Inference & Evaluation\n\nThere are also PushT examples/tests using:\n\n\n    --env.type=pusht\n\n\nSee: lerobot_eval.py\n\nThat does not look like the same layer as configuring real SO101 cameras.\n\nFor a real robot, the important object is not only an `env`; it is the robot and its camera config.\n\n## Real SO101 / physical camera path\n\nFor real robot inference, current LeRobot docs point to `lerobot-rollout`.\n\nSee: LeRobot Policy Deployment / lerobot-rollout\n\nThe cheat sheet also says:\n\n> Inference means running the trained policy/model on a robot. For that we use `lerobot-rollout`.\n\nIt also notes that camera configuration needs to match what was used when collecting the dataset.\n\nSee: LeRobot Cheat Sheet\n\nSo for a real SO101 setup, I would expect the command to be closer to this shape:\n\n\n    lerobot-rollout \\\n      --robot.type=so101_follower \\\n      --robot.port=/dev/ttyACM0 \\\n      --robot.id=<your_robot_id> \\\n      --robot.cameras='{\n        front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30},\n        wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}\n      }' \\\n      --policy.path=<your_policy_path_or_hub_repo> \\\n      --task=\"<your task>\"\n\n\nExact arguments may differ by LeRobot version, so check your installed version first.\n\n## Version check\n\nBefore changing too much, I would collect:\n\n\n    lerobot-info\n    pip show lerobot\n    lerobot-eval --help\n    lerobot-rollout --help\n    lerobot-record --help\n\n\nThis matters because LeRobot CLI names and argument names have changed across versions. Some older examples use `record.py`, `control_robot.py`, or `lerobot-record` where newer docs use `lerobot-rollout`.\n\nFor example, there are real SO101 + SmolVLA issue examples using `lerobot-record` with explicit robot and camera config:\n\n\n    lerobot-record \\\n      --robot.type=so101_follower \\\n      --robot.port=/dev/ttyACM0 \\\n      --robot.cameras=\"{camera1: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, ...}\" \\\n      --policy.path=...\n\n\nSee: Inference lerobot/smolvla_base with SO-101 failed #2210\n\nSo I would not assume that a command from one LeRobot version maps cleanly to another version.\n\n## If the real-robot path is already correct, then check camera backend\n\nAfter confirming that you are using the real SO101 robot path, then the camera issue becomes very plausible.\n\nThere is a relevant LeRobot issue where Linux/OpenCV camera settings did not apply correctly with the default backend:\n\n  * `backend=ANY` auto-selected FFMPEG\n  * requested `fourcc=\"MJPG\"` did not apply\n  * width/height/fps setting failed or behaved incorrectly\n  * forcing `backend=V4L2` made the same settings work\n\n\n\nSee: OpenCVCameraConfig backend=ANY vs V4L2 issue #3198\n\nThere is also a Hugging Face forum thread from an SO101 user where forcing Video4Linux2 was used as a temporary fix for camera backend problems:\n\nSee: Lerobot Camera Backend Issues\n\nSo if you are on Linux with USB/OpenCV cameras, the camera backend is a strong second-level suspect.\n\n## Camera discovery\n\nI would also run:\n\n\n    lerobot-find-cameras opencv\n\n\nLeRobot’s camera docs say OpenCV and RealSense cameras support auto-discovery, and that identifiers may change after rebooting or re-plugging cameras.\n\nSee: LeRobot Cameras docs\n\nPlease check:\n\nItem | Why it matters\n---|---\n`/dev/video*` path or camera index | It may change after unplug/reboot.\nbackend API | FFMPEG vs V4L2 can matter on Linux.\ndefault stream profile | The camera may default to unexpected width/FPS.\nsupported modes | Some resolutions only work at lower FPS.\n`fourcc` / MJPG support | MJPEG is often needed for higher resolution/FPS USB camera modes.\n\n## SmolVLA-specific trap: camera keys/count/order\n\nSmolVLA uses multiple camera views plus robot state and language instruction.\n\nSee: SmolVLA docs\n\nSo after fixing the command path and camera backend, another possible issue is that the policy expects different image features than your robot/dataset provides.\n\nThings to check:\n\n\n    policy expected image keys\n    dataset image keys\n    robot camera names\n    camera order\n    number of cameras\n    image resolution\n    FPS\n    normalization/statistics\n\n\nThere are related issues where users hit camera feature/name/count problems with SmolVLA or SO101:\n\n  * Policy image features vs robot cameras mismatch #1620\n  * Best camera setup/order for SmolVLA + SO101 #1763\n  * Debugging poor eval with SmolVLA and two cameras #2753\n  * Problems with running pretrained lerobot/smolvla_base #2374\n\n\n\nThis may not be the first cause of your resolution/FPS symptom, but it is worth checking if the policy runs but behaves badly.\n\n## What I would ask you to post\n\nFor a useful LeRobot Discord/GitHub/forum question, I would include:\n\n\n    lerobot-info\n    pip show lerobot\n    lerobot-eval --help\n    lerobot-rollout --help\n    lerobot-record --help\n    lerobot-find-cameras opencv\n\n\nAlso include:\n\n\n    OS:\n    LeRobot version:\n    Install method:\n    Robot type:\n    Camera model(s):\n    Camera backend:\n    Full command:\n    Policy path:\n    Dataset repo_id:\n    Number of cameras used during training:\n    Number of cameras connected during evaluation:\n    Expected width/height/fps:\n    Actual width/height/fps:\n\n\n## Practical triage order\n\nI would debug in this order:\n\n  1. Decide whether this is **PushT simulation evaluation** or **real SO101 robot rollout**.\n  2. If it is real SO101, switch to the real-robot path rather than `--env.type=pusht`.\n  3. Pass cameras explicitly through `--robot.cameras`.\n  4. Run `lerobot-find-cameras opencv`.\n  5. If width/FPS still changes, check OpenCV backend: FFMPEG vs V4L2, MJPG, supported modes, and USB bandwidth.\n  6. If the policy runs but behaves badly, check SmolVLA camera keys/count/order and dataset metadata.\n\n\n\n## My best guess\n\nBased only on your command and description, the most likely first issue is not SmolVLA itself and not necessarily OpenCV yet.\n\nThe most likely first issue is:\n\n\n    The command is using a PushT environment-evaluation path,\n    while the problem description sounds like real SO101 robot-camera evaluation.\n\n\nAfter that is fixed, then the OpenCV/V4L2/FPS issue becomes the next thing to check.",
  "title": "How to use the `lerobot_eval` command"
}