{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifibidbijpgkwuzslru2j7nrevvmh7xyxl4zu5g5wrz73wpkojl3u",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mmz4lvio7rg2"
  },
  "path": "/t/physical-modelling-of-sim2real-so101-arm-project/176368#post_1",
  "publishedAt": "2026-05-29T17:04:07.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "SO101 Sim 2 Real Course",
    "NVIDIA’s Forums",
    "Train an SO-101 Robot From Sim-to-Real With NVIDIA Isaac",
    "Real Evaluation",
    "google drive link",
    "Given Dataset"
  ],
  "textContent": "Good afternoon everyone, this post has to relate with the moving of sim2real from NVIDIA’s recent SO101 Sim 2 Real Course. This is crossposted from NVIDIA’s Forums\n\n## Current Setup\n\n  * **Isaac Sim Version** : 5.1.0\n  * **OS** : Ubuntu 24.04 (arm64)\n  * **GPU Setup** : Dell Pro Max GB10 - Blackwell Architecture - CUDA v13.0, Driver 580.142\n  * **Lerobot Tag** : e670ac5daf9b76 (Just around v0.4.3)\n\n\n\n## Topic Description\n\nIs it possible to rollout example pretrained VLA policies in the real world if we copy the original workspace?\n\n### Detailed Description\n\nI am currently following the Train an SO-101 Robot From Sim-to-Real With NVIDIA Isaac Learning Tutorial. I’ve been able to setup the actual docker container just fine and have been able to rollout the various example policies that are provided in the tutorial in Isaac Lab simulation with high (70-90%) success rates. The issue lies with the Simulation to Real portion, particularly, the Real Evaluation section. I built the lightbox, mounted the light, external camera, and robot SO101 arm all as mentioned. These same conditions I would assume work between the provided example policies, the original real setup, and a new setup, yet I’ve not once been able to place a vial into the rack, probably over the span of 150-200 episodes, varying the environment plenty of times.\n\nWithin the “Real Evaluation” section there’s no mention of success rate, no video footage from the physical arm completing the task, and no dataset visualization in huggingface of an evaluation dataset. I took a look at one of the datasets in the Dataset Visualizer from LeRobot and noticed the position of the arm (in sim) wasn’t at the base of the lightbox, but instead the base of the mat.\n\nI understand VLAs like Gr00t N1.6 require finetuning for a variety of tasks, especially as it needs to get deployed in general scenarios and I don’t expect high success rates (aiming for around 30-40%), but I’m just recreating the original environment that was quite isolated from the rest of the world (hence the lighbox), so policies should be able to run on my setup as well.\n\n### Steps to Reproduce\n\n  1. Make sure the real docker container from the tutorial has been built\n\n  2. Start the real docker container, select a model, run the policy server\n2.1) `export MODEL=aravindhs-NV/grootn16-finetune_sreetz- so101_teleop_vials_rack_left/checkpoint-1000`\n2.2) `python Isaac-GR00T/gr00t/eval/run_gr00t_server.py –model-path /workspace/models/$MODEL`\n\n  3. Run the evaluation rollout for docker : attach the docker and run\n3.1) `docker exec -it real-robot /bin/bash`\n3.2) ````python Isaac-GR00T/gr00t/eval/real_robot/SO100/so101_eval.py `\n`–robot.type=so101_follower `\n`–robot.port=“$ROBOT_PORT” `\n`–robot.id=“$ROBOT_ID” `\n`–robot.cameras=“{`\n`wrist: {type: opencv, index_or_path: $CAMERA_GRIPPER, width: 640, height: 480, fps: 30},`\n`front: {type: opencv, index_or_path: $CAMERA_EXTERNAL, width: 640, height: 480, fps: 30}`\n`}” `\n`–policy_host=localhost `\n`–policy_port=5555 `\n`–lang_instruction=“Pick up the vial and place it in the yellow rack” `\n`–rerun True````\n\n  4. Let it run and watch it not succeed.\n\n\n\n\n### Error Messages\n\nNo errors, just poor success rate (0%)\n\n### Screenshots or Videos\n\nAdditional photos in the google drive link , sorry for the lack of “film production” but I hope this is a bit useful into insight of my environment. A bit unfortunate that I’m new and can only 2 photos/videos\n\n## Additional Information\n\n### What I’ve Tried\n\n  * I double checked with the setting up of the workspace section and all of my initial measurements are the same as the original authors.\n  * I also recalibrated both arms, verified they work in simulation, and checked the calibration, in which my standard deviations of joint position are on the same magnitude as the image\n  * Tried to recreate the environment based on a Given Dataset\n  * Tried Multiple Orientations of the External Camera to limit noise\n  * Tried Multiple Poses (position + orientation) of the rack relative to the arm\n  * Tried Multiple # of Vials along with vial poses\n  * Tried Multiple amounts of light intensity (25% - 100%) with the provided light bar\n\n\n\n### Related Issues\n\nN/A\n\n### Additional Context\n\nThe authors did a fantastic job on the tutorial, certainly more information provided than many other tutorials, I’m also hopeful to see the sim2real pipeline with VLAs. Just a bit unfortunate it’s lacking in the real evaluation portion.\n\nAlso, I’ve tried again recently have been getting the real arm to consistently, autonomously pick up a vial by positioning the vial central to the arm with random orientations, but below the top of the rack (imagine you drew a transversal line at the top of the rack close to the back wall, going across the width of the lightbox, the cap would be below that).\n\n### Edit:\n\nI began changing some things around the physical project setup to match the simulation environment (without domain randomization) based on relative position of objects and got it to function with 80% success in 10 episodes! It was very particular though, which is a bit concerning being applied to a generalist policy like Groot.\n**Conditions** : 50% light, 1 vial in top left of container, 1 vial on mat, top left of mat 4.5cm from back, top right of mat 2.5 cm from back. Rack: 18cm from left wall, 17 cm from back wall, 90 \\degree perfect. Vial 18.5 cm right from the rack, 2 cm from top of the rack transversal line, also 90 \\degree perfect.\n\n**Model** : so101_telop_vials_rack_left/checkpoint-100005",
  "title": "Physical Modelling of sim2real SO101 Arm Project"
}