External Publication
Visit Post

What are the best practices for detecting and fetching deltas from a dataset?

Hugging Face Forums [Unofficial] July 3, 2026
Source

Hmm… my current read is that this does not exist as a first-class official feature yet; depending on the dataset, there may be workable workarounds: just in case, @lhoestq

I would split this into two separate problems:

  1. Detecting that the dataset repo changed
  2. Deciding what the smallest safe fetch/reprocessing unit is

For detection, I would look at Hub Webhooks first, or poll the dataset repo SHA with huggingface_hub if webhooks are not an option.

For deltas, I would not assume a generic “fetch only the new/changed rows since last time” API for arbitrary HF datasets. The practical workaround I would try first is revision/file/shard-level tracking :

last_processed_sha
  -> detect new_sha
  -> compare file manifests between the two revisions
  -> find added/modified/deleted data files
  -> download/reprocess only those changed files or shards
  -> advance last_processed_sha only after success

This is not the same as true row-level CDC, but it is a low-risk pattern that is testable with current Hub APIs.

Suggested minimal workflow (click for more details)

The key design question: what is the change boundary?

The workaround is much easier if the dataset is already organized into stable shards.

Dataset layout Practical downstream strategy
Append-only Parquet shards Process only new/changed Parquet files. Best case.
Append-only WebDataset shards Process only new/changed .tar shards. Also good.
Stable date/source partitions Reprocess changed partitions or shards.
Existing shards are rewritten Reprocess the rewritten shards. Do not infer row deltas unless you have IDs/changelog.
One giant file/archive Hard. Any small logical change may force whole-file reprocessing.
Custom loading script hides file boundaries Harder. Inspect actual repo files if possible.
Stable row IDs + changelog/tombstones Row-level workflows become possible.
No stable IDs/changelog Row-level delta is risky to infer.

So my first implementation would probably target file/shard-level deltas , not row-level deltas.

Why dataset layout matters (click for more details)

Things I would not over-assume

The main trap is mixing transport/cache-level optimization with application-level delta semantics.

Mechanism Helps with Does not automatically give
Webhook Detecting repo updates Changed rows/examples
Repo SHA Version boundary Semantic delta by itself
Manifest comparison Changed files/shards Exact row-level insert/update/delete
snapshot_download / hf_hub_download Fetching selected files Delta detection by itself
Hub cache Avoiding redundant downloads Pipeline processing state
Xet/chunk deduplication Storage/transfer efficiency Dataset-level changelog
Dataset Viewer Parquet files Inspection/queryable derived artifacts General delta API

For example, the Hub docs on editing datasets and PyArrow integration discuss optimized Parquet/Xet behavior that can reduce upload/download/storage costs. That is useful, but I would not treat chunk-level deduplication as a replacement for stable row IDs, manifests, or changelogs.

Similarly, the Dataset Viewer /parquet endpoint can list converted Parquet files for a dataset. That can be useful for inspection, but I would not treat viewer-converted Parquet files as a general-purpose dataset changelog unless your pipeline is explicitly designed around that derived representation.

A few edge cases I would handle explicitly (click for more details)

If you really need row-level CDC

If the real requirement is “give me inserted/updated/deleted rows since version X”, I would treat that as a changelog/table-format requirement , not something to infer from arbitrary Hub files.

For comparison:

  • Delta Lake has Change Data Feed for row-level changes between table versions.
  • Apache Hudi has incremental and CDC query modes.
  • Apache Iceberg has snapshot/manifest-based table metadata and incremental reads, though supported operations depend on the engine and mode.

Those systems have explicit metadata, snapshots, operation logs, or change streams. A generic HF dataset repo may not have that contract unless the dataset producer publishes it.

For HF datasets, row-level CDC usually requires at least one of:

stable row/example IDs
producer-published changelog
producer-published manifest
tombstone/delete markers
table-format metadata
or accepting shard-level reprocessing instead

Minimal test I would run first

Before building a full system, I would try this small experiment:

  1. Pick a dataset repo and two revisions: old_sha and new_sha.
  2. Use list_repo_tree(..., recursive=True, expand=True) for both revisions.
  3. Filter to likely data files.
  4. Compare path, size, blob_id, lfs.sha256, and/or xet_hash.
  5. Classify added / modified / deleted files.
  6. Dry-run download only those changed paths.
  7. Check whether the changed files are a useful reprocessing unit for your pipeline.
  8. If yes, build the webhook/polling + retryable pipeline around that.
  9. If no, the dataset layout or missing producer metadata is probably the limiting factor.

Sketch of the manifest comparison test (click for more details)

Short version

I would start with this:

Use Webhooks or SHA polling for detection.
Store last_processed_sha.
Compare repo file manifests between old and new revisions.
Treat changed data files/shards as the delta.
Download and reprocess only those changed shards.
Handle deletions and schema changes explicitly.
Advance last_processed_sha only after downstream success.
Do not expect arbitrary row-level deltas unless the dataset provides stable IDs, changelog, or table-format metadata.

That is not a perfect built-in delta system, but it is a practical and testable path with current Hub APIs.

Discussion in the ATmosphere

Loading comments...