{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreieqreq2hw4menakct3uwasmlhydg27r2p6e5pr5hhjgxpmstzkkbu",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mppys6ratws2"
  },
  "path": "/t/nanoi2v-building-an-image-to-video-model-from-scratch/177340#post_2",
  "publishedAt": "2026-07-03T04:46:07.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "self-contained, easy to tweak, beginner-friendly, and focused on one purpose",
    "(click for more details)",
    "Diffusers video generation guide",
    "Stable Video Diffusion in Diffusers"
  ],
  "textContent": "I tried a small Colab Free smoke run, and here is what I found:\n\n* * *\n\nI’m looking at this from a **first-time reader / small-contributor** perspective, not as a full training review or model-quality evaluation.\n\nMy main takeaway: the repo already has a useful educational component split — VAE, latent video, DiT, flow matching, conditioning, CFG — so most of my suggestions are about making the **contracts between those components** easier to test and document.\n\nI’m framing these as cheap checks that might help future readers and contributors, not as criticism of model quality.\n\nThis also seems aligned with the general style of small examples used in libraries like Diffusers, where examples are meant to be self-contained, easy to tweak, beginner-friendly, and focused on one purpose.\n\n## High-level summary\n\nIf I were adding a small onboarding/debug layer, I would probably add these:\n\nArea | Useful small addition\n---|---\ninstall | minimal dependency note or requirements file\ndata | tiny synthetic `.npy` video example\nVAE | valid `num_frames` / temporal shape-contract note\nVAE → DiT boundary | native VAE checkpoint interface smoke test\nDiT + flow | no-download fake-token smoke test\nconfig validation | patch-size divisibility check\nscheduler helper | `z_img` wrapper/lambda pattern note\nexternal models | keep SVD / other I2V pipelines as references, not drop-in checkpoints\n\nThe main idea is to give readers a cheap path like:\n\n\n    compile/import\n    → synthetic .npy preprocessing\n    → tiny VAE shape roundtrip\n    → native VAE checkpoint interface\n    → fake-token DiT/flow loss\n    → tiny overfit\n    → real conditioning / real training later\n\n\nThat way, readers can catch shape/interface issues before they spend time on expensive runs.\n\nScope of my smoke check (click for more details)\n\n* * *\n\n## 1. Install / dependency note\n\nIn my Colab/T4 run, the import probes passed. But I did not see a dependency manifest such as:\n\n  * `requirements.txt`\n  * `pyproject.toml`\n  * `environment.yml`\n\n\n\nA small install block or requirements file would help first-time readers reproduce the initial smoke path.\n\nIt may be useful to separate dependency layers:\n\nLayer | Example dependency category\n---|---\nCore VAE / DiT / flow smoke tests | PyTorch and basic Python libs\nConditioning | `transformers`, image/text processors\nTraining/logging | TensorBoard or equivalent logging deps\nVideo data utilities | video decoding / preprocessing deps\nOptional metrics | LPIPS or other evaluation helpers\n\nThis would let someone run the core VAE/DiT/flow smoke path before they need T5/CLIP downloads, video datasets, or metric dependencies.\n\nWhy I think this helps (click for more details)\n\n* * *\n\n## 2. A tiny synthetic `.npy` example would help onboarding\n\nThe dataset contract is easy to test with a tiny synthetic video.\n\nA minimal docs/example path could be:\n\n\n    raw .npy video:  [T, H, W, 3], uint8, [0, 255]\n    dataset output:  [3, T, H, W], float32, [-1, 1]\n\n\nA small moving-square `.npy` example would let readers test:\n\n  * axis order\n  * dtype\n  * normalization range\n  * short/exact/long sequence handling\n  * accidental double-normalization\n\n\n\nThis avoids gated datasets, large downloads, and unrelated video-decoding issues.\n\nIn my small check, float input was rejected with an assertion. I think that is useful, because it catches accidental double-normalization early.\n\nSuggested synthetic dataset smoke path (click for more details)\n\n* * *\n\n## 3. Document valid debug `num_frames` values for the keep-first temporal VAE path\n\nThis was one of the more useful shape-contract checks.\n\nIn a tiny VAE frame sweep, I saw the following pattern:\n\nInput `T` | Round-tripped through VAE?\n---|---\n5 | yes\n9 | yes\n13 | yes\n17 | yes\n21 | yes\n25 | yes\n33 | yes\n8 | no\n16 | no\n18 | no\n\nThis looked consistent with a keep-first temporal down/up path where debug frame counts like `4k + 1` are safer, at least for the tiny `temporal_ds=2` setup I tested.\n\nThe default `17` therefore looks well chosen.\n\nI would frame this as a **debug-config / shape-contract documentation item** , not as a quality issue.\n\nA short note like this would help:\n\n\n    For temporal_ds=2, use debug num_frames such as 5, 9, 13, 17, 21, ...\n    Avoid arbitrary even values such as 8 or 16 unless you intentionally handle temporal length mismatch.\n\n\nWhy this is useful (click for more details)\n\n* * *\n\n## 4. Native VAE checkpoint path: high-value interface smoke test\n\nThis is the check I would prioritize most.\n\nNanoI2V has both a native VAE path and a pretrained/wrapped VAE path. That means the DiT training/inference code needs a clear VAE inference contract.\n\nA useful test would be:\n\n\n    Can a native VAE3D checkpoint be loaded by the same train_dit.py / inference_dit.py path\n    and expose the same inference interface as the pretrained VAE wrapper?\n\n\nIn my tiny checkpoint simulation, I saw this interface shape:\n\nObject / path | Observed\n---|---\n`VAE3D` | had `encode`, `decode`, `forward`\n`VAE3D` | did not expose `encode_mean`\n`VAE3D` | did not expose `set_inference_mode`\nnative `train_dit.py` VAE path | expected `set_inference_mode`\nnative inference path | loaded `VAE3D`, but later expected `encode_mean`\nsmall adapter-style probe | worked\n\nI would not call this a model-quality issue. It is more of an **interface-contract** item between VAE training and DiT training/inference.\n\nA small bridge may be enough, depending on the intended semantics:\n\n\n    native VAE inference contract:\n    - encode_mean(video) -> latent mean or deterministic latent\n    - decode(latent) -> reconstructed video\n    - eval / inference mode behavior\n\n\nThe important part is that the native VAE checkpoint path and the pretrained VAE wrapper path expose the same small inference API.\n\nWhy this seems important (click for more details)\n\n* * *\n\n## 5. No-download DiT + flow smoke test\n\nA no-download DiT/flow test seems very useful for this project.\n\nBefore loading T5/CLIP, before loading a VAE checkpoint, and before using a real dataset, it is possible to test the core DiT/flow interface with fake tensors:\n\n\n    random z_t\n    randomflow test seems very useful for this project.\n\n    Before loading T5/CLIP, before loading a VAE checkpoint, and before using a real dataset, z_img\n    fake text tokens\n    fake image tokens\n    FlowMatchingScheduler.add_noise\n    velocity target\n    VideoDiT forward\n    MSE loss\n    backward\n\n\nIn my toy run, the following worked:\n\n  * `VideoDiT` output shape matched the velocity target shape.\n  * flow math was easy to verify independently.\n  * loss was finite.\n  * backward passed.\n  * a tiny fixed-batch DiT/flow overfit decreased loss over 25 steps.\n\n\n\nAgain, I would not treat that as quality evidence. It is only an **interface / optimizer sanity check**.\n\nThis is especially useful because video generation mixes time, space, conditioning, and latent shapes. The Diffusers video generation guide describes video generation as extending image generation into video-specific parameters and memory constraints, so isolating the DiT/flow contract before adding all expensive components seems valuable.\n\nSuggested no-download DiT/flow smoke test (click for more details)\n\n* * *\n\n## 6. `FlowMatchingScheduler.training_loss` and the `z_img` wrapper pattern\n\nOne small interface-clarity point:\n\nA direct reusable helper call like this:\n\n\n    FlowMatchingScheduler.training_loss(dit, ...)\n\n\ndoes not naturally pass the I2V-specific `z_img` argument expected by `VideoDiT`.\n\nIn my small check:\n\n  * direct helper call failed because `VideoDiT.forward()` expected `z_img`\n  * a small wrapper/lambda that closes over `z_img` worked\n  * Euler sampling with the same wrapper pattern also worked\n\n\n\nThis does not look like a blocker, because the current training/inference paths can handle `z_img` manually. But if `training_loss` is intended to be reused as a generic helper, documenting the wrapper pattern could avoid confusion.\n\nPossible wording in docs (click for more details)\n\n* * *\n\n## 7. Patch-size divisibility could be documented or asserted\n\nThe DiT patch-size grid is another small shape-contract item.\n\nIn a tiny patch-grid check:\n\nPatch size | Latent shape divisible? | Output shape matched input latent?\n---|---|---\n`(1, 1, 1)` | yes | yes\n`(1, 2, 2)` | yes | yes\n`(3, 2, 2)` | yes | yes\n`(2, 2, 2)` with latent `T=3` | no | no\n`(1, 3, 2)` with latent `H=4` | no | no\n`(1, 2, 3)` with latent `W=4` | no | no\n\nThe non-divisible cases could still run, but produced shorter output shapes. That can be confusing later.\n\nSo it may be useful to document or assert early:\n\n\n    patch_t must divide latent_t\n    patch_h must divide latent_h\n    patch_w must divide latent_w\n\n\nThis is a good candidate for a small config validation check.\n\nWhy this is worth an early assert (click for more details)\n\n* * *\n\n## 8. External I2V models as references, not drop-in checkpoints\n\nExternal I2V pipelines can still be useful, but I would keep them separate from NanoI2V-native smoke tests.\n\nFor example, Stable Video Diffusion in Diffusers is useful as a reference for image-to-video input/output conventions and memory tradeoffs. Its docs discuss settings such as CPU offload and `decode_chunk_size`, where lower memory can trade off with temporal consistency.\n\nBut external I2V models are not drop-in NanoI2V checkpoints. I would separate the docs into something like:\n\nPath | Purpose\n---|---\nNanoI2V no-checkpoint smoke tests | compile/import/shape/interface checks\nNanoI2V native checkpoints | exact VAE/DiT/config pairing\nExternal I2V references | compare pipeline conventions and memory tradeoffs\n\nThis avoids mixing up “reference pipeline” with “compatible weights”.\n\nSuggested separation (click for more details)\n\n* * *\n\n## Suggested small additions\n\nIf I were turning these observations into repo/docs tasks, I would suggest:\n\n### Documentation\n\n  * Minimal install block or dependency file.\n  * Tensor-shape table:\n    * raw video\n    * dataset output\n    * VAE input\n    * VAE latent\n    * image latent\n    * DiT input/output\n    * decoded video\n  * Valid debug `num_frames` note for the keep-first temporal VAE path.\n  * DiT patch-size divisibility note.\n  * Clear native VAE vs pretrained VAE wrapper interface contract.\n\n\n\n### Examples / smoke tests\n\n  * `examples/synthetic_moving_square.py`\n  * `examples/smoke_dataset_preprocessing.py`\n  * `examples/smoke_vae_frame_contract.py`\n  * `examples/smoke_native_vae_checkpoint_interface.py`\n  * `examples/smoke_dit_flow_fake_tokens.py`\n\n\n\n### Cheap test ladder\n\n\n    Level 0: py_compile + import probes\n    Level 1: synthetic .npy preprocessing\n    Level 2: tiny VAE frame roundtrip\n    Level 3: native VAE checkpoint interface\n    Level 4: fake-token DiT + flow loss/backward\n    Level 5: tiny fixed-batch overfit\n    Level 6: real conditioning encoders\n    Level 7: full training / inference\n\n\nThe main idea is: let readers validate component contracts before they spend time on expensive training.\n\nPossible first-reader path (click for more details)\n\n* * *\n\n## Closing note\n\nOverall, I think the component separation is already useful for an educational I2V repo. My main suggestion is to make the **contracts between components** more visible and cheaply testable.\n\nI would frame the smoke tests above as:\n\n  * not benchmarks\n  * not quality evaluation\n  * not a requirement that the full project taget Colab Free\n  * just cheap checks that help future readers and contributors avoid shape/interface confusion before expensive runs\n\n",
  "title": "NanoI2V: Building an Image-to-Video Model from Scratch"
}