{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreielnvv4b2igaykkbkjkoltjd2t42kqx64sh3qt74x5sdypnxzekb4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mpps2lnheqq2"
  },
  "path": "/t/what-are-the-best-practices-for-detecting-and-fetching-deltas-from-a-dataset/177360#post_2",
  "publishedAt": "2026-07-03T04:42:07.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "@lhoestq",
    "Hub Webhooks",
    "huggingface_hub",
    "(click for more details)",
    "editing datasets",
    "PyArrow integration",
    "/parquet endpoint",
    "Change Data Feed",
    "incremental and CDC query modes",
    "incremental reads"
  ],
  "textContent": "Hmm… my current read is that this does not exist as a first-class official feature yet; depending on the dataset, there may be workable workarounds:\njust in case, @lhoestq\n\n* * *\n\nI would split this into two separate problems:\n\n  1. **Detecting that the dataset repo changed**\n  2. **Deciding what the smallest safe fetch/reprocessing unit is**\n\n\n\nFor detection, I would look at Hub Webhooks first, or poll the dataset repo SHA with huggingface_hub if webhooks are not an option.\n\nFor deltas, I would not assume a generic “fetch only the new/changed rows since last time” API for arbitrary HF datasets. The practical workaround I would try first is **revision/file/shard-level tracking** :\n\n\n    last_processed_sha\n      -> detect new_sha\n      -> compare file manifests between the two revisions\n      -> find added/modified/deleted data files\n      -> download/reprocess only those changed files or shards\n      -> advance last_processed_sha only after success\n\n\nThis is not the same as true row-level CDC, but it is a low-risk pattern that is testable with current Hub APIs.\n\nSuggested minimal workflow (click for more details)\n\n## The key design question: what is the change boundary?\n\nThe workaround is much easier if the dataset is already organized into stable shards.\n\nDataset layout | Practical downstream strategy\n---|---\nAppend-only Parquet shards | Process only new/changed Parquet files. Best case.\nAppend-only WebDataset shards | Process only new/changed `.tar` shards. Also good.\nStable date/source partitions | Reprocess changed partitions or shards.\nExisting shards are rewritten | Reprocess the rewritten shards. Do not infer row deltas unless you have IDs/changelog.\nOne giant file/archive | Hard. Any small logical change may force whole-file reprocessing.\nCustom loading script hides file boundaries | Harder. Inspect actual repo files if possible.\nStable row IDs + changelog/tombstones | Row-level workflows become possible.\nNo stable IDs/changelog | Row-level delta is risky to infer.\n\nSo my first implementation would probably target **file/shard-level deltas** , not row-level deltas.\n\nWhy dataset layout matters (click for more details)\n\n## Things I would not over-assume\n\nThe main trap is mixing **transport/cache-level optimization** with **application-level delta semantics**.\n\nMechanism | Helps with | Does not automatically give\n---|---|---\nWebhook | Detecting repo updates | Changed rows/examples\nRepo SHA | Version boundary | Semantic delta by itself\nManifest comparison | Changed files/shards | Exact row-level insert/update/delete\n`snapshot_download` / `hf_hub_download` | Fetching selected files | Delta detection by itself\nHub cache | Avoiding redundant downloads | Pipeline processing state\nXet/chunk deduplication | Storage/transfer efficiency | Dataset-level changelog\nDataset Viewer Parquet files | Inspection/queryable derived artifacts | General delta API\n\nFor example, the Hub docs on editing datasets and PyArrow integration discuss optimized Parquet/Xet behavior that can reduce upload/download/storage costs. That is useful, but I would not treat chunk-level deduplication as a replacement for stable row IDs, manifests, or changelogs.\n\nSimilarly, the Dataset Viewer /parquet endpoint can list converted Parquet files for a dataset. That can be useful for inspection, but I would not treat viewer-converted Parquet files as a general-purpose dataset changelog unless your pipeline is explicitly designed around that derived representation.\n\nA few edge cases I would handle explicitly (click for more details)\n\n## If you really need row-level CDC\n\nIf the real requirement is “give me inserted/updated/deleted rows since version X”, I would treat that as a **changelog/table-format requirement** , not something to infer from arbitrary Hub files.\n\nFor comparison:\n\n  * Delta Lake has Change Data Feed for row-level changes between table versions.\n  * Apache Hudi has incremental and CDC query modes.\n  * Apache Iceberg has snapshot/manifest-based table metadata and incremental reads, though supported operations depend on the engine and mode.\n\n\n\nThose systems have explicit metadata, snapshots, operation logs, or change streams. A generic HF dataset repo may not have that contract unless the dataset producer publishes it.\n\nFor HF datasets, row-level CDC usually requires at least one of:\n\n\n    stable row/example IDs\n    producer-published changelog\n    producer-published manifest\n    tombstone/delete markers\n    table-format metadata\n    or accepting shard-level reprocessing instead\n\n\n## Minimal test I would run first\n\nBefore building a full system, I would try this small experiment:\n\n  1. Pick a dataset repo and two revisions: `old_sha` and `new_sha`.\n  2. Use `list_repo_tree(..., recursive=True, expand=True)` for both revisions.\n  3. Filter to likely data files.\n  4. Compare `path`, `size`, `blob_id`, `lfs.sha256`, and/or `xet_hash`.\n  5. Classify added / modified / deleted files.\n  6. Dry-run download only those changed paths.\n  7. Check whether the changed files are a useful reprocessing unit for your pipeline.\n  8. If yes, build the webhook/polling + retryable pipeline around that.\n  9. If no, the dataset layout or missing producer metadata is probably the limiting factor.\n\nSketch of the manifest comparison test (click for more details)\n\n## Short version\n\nI would start with this:\n\n\n    Use Webhooks or SHA polling for detection.\n    Store last_processed_sha.\n    Compare repo file manifests between old and new revisions.\n    Treat changed data files/shards as the delta.\n    Download and reprocess only those changed shards.\n    Handle deletions and schema changes explicitly.\n    Advance last_processed_sha only after downstream success.\n    Do not expect arbitrary row-level deltas unless the dataset provides stable IDs, changelog, or table-format metadata.\n\n\nThat is not a perfect built-in delta system, but it is a practical and testable path with current Hub APIs.",
  "title": "What are the best practices for detecting and fetching deltas from a dataset?"
}