{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidw2mo2ipe3oqaazel67eefpwmjxq3samlkkk7e6ejhkfzajhcrue",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mp4eggqgyu52"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreidek4ikktsvfrb5ljs5udf2t4xp6kzi56ceamztqbj4xhavmfj4du"
    },
    "mimeType": "image/webp",
    "size": 65124
  },
  "path": "/ykpraveen/building-distributed-data-processing-with-spring-batch-6-spring-boot-4-5hf8",
  "publishedAt": "2026-06-25T11:37:27.000Z",
  "site": "https://dev.to",
  "tags": [
    "springboot",
    "batch",
    "java",
    "architecture",
    "github.com/ykpraveen/spring-batch-sample",
    "spring-batch-sample",
    "src/main/java"
  ],
  "textContent": "When people first use Spring Batch, they usually start with a simple single-threaded job. That works for small datasets, but once data volume grows, throughput becomes the bottleneck.\n\nIn this sample project, I implemented a **partitioned, multi-threaded Spring Batch pipeline** to process sales records in parallel using a master/worker step model.\n\n👉 Code repo: github.com/ykpraveen/spring-batch-sample\n\n##  Spring Batch\n\nAt its core, Spring Batch is built around a few key abstractions:\n\n  * **Job** : a complete batch workflow\n  * **Step** : one phase of a job\n  * **ItemReader / ItemProcessor / ItemWriter** : read-transform-write pipeline\n  * **Chunk processing** : process N items in one transaction (`chunkSize`) ### Why chunking matters\n\n\n\nIn chunk-oriented steps, Spring Batch reads and processes items until the chunk size is reached, then writes and commits in one transaction.\n\nSo with `chunk(500)`:\n\n  * 500 items are read/processed/written\n  * one commit happens per chunk\n  * failures can be retried at chunk boundaries\n\n\n\nThis gives a good balance between:\n\n  * too-small chunks (high transaction overhead)\n  * too-large chunks (long transactions, higher rollback cost)\n\n\n\n###  How Spring Batch scales\n\nSpring Batch offers multiple scaling patterns:\n\n  1. **Multi-threaded Step** : one step, concurrent chunk processing\n  2. **Partitioning** : split input domain into partitions, each handled by a worker step\n  3. **Remote Chunking / Remote Partitioning** : distribute work across processes/nodes\n\n\n\nThis project uses **partitioning + thread pool execution** (local distributed-style parallelism).\n\n##  How this project applies those concepts\n\nRepository: spring-batch-sample\n\nThe architecture is:\n\n  * A **master step** creates partitions (data ranges)\n  * A **worker step** executes each partition\n  * A `ThreadPoolTaskExecutor` runs workers concurrently\n\n\n\nKey classes (see `src/main/java` in repo):\n\n  * `BatchConfiguration` → job/step orchestration\n  * `SalesDataPartitioner` → partition boundary logic\n  * `SalesDataProcessor` → business transformation logic\n\n\n\nCode area: src/main/java\n\n##  Performance tuning used here\n\nThe sample uses:\n\n  * `gridSize: 8` (number of partitions)\n  * Thread pool: `corePoolSize=4`, `maxPoolSize=8`\n  * `chunk size: 500`\n  * Sample input: 5000 records\n\n\n\n###  Interpretation\n\n  * `gridSize` controls parallel work units.\n  * Thread pool size controls actual concurrent execution.\n  * Effective throughput depends on DB I/O, CPU, and item processing complexity.\n  * Increasing partitions beyond available threads can still help load balancing, but with diminishing returns.\n\n\n\n##  Database + metadata angle\n\nSpring Batch is not just a processing framework; it is also a **stateful execution framework**.\n\nIt tracks job/step execution state in metadata, enabling:\n\n  * restartability\n  * execution history\n  * failure diagnostics\n\n\n\nIn this sample, PostgreSQL stores both:\n\n  * domain tables (`sales_data`, `processed_data`, `processing_statistics`)\n  * batch execution context/metadata managed by Spring Batch\n\n\n\nThat combination is what makes batch jobs operationally reliable in real systems.\n\n##  Run locally\n\n###  1) Start PostgreSQL\n\n\n    docker compose up -d\n\n\n###  2) Build and run the app\n\n\n    mvn clean install\n    mvn spring-boot:run\n\n\n###  3) Trigger the batch job\n\n\n    curl -X POST http://localhost:8080/api/batch/start\n\n\n###  4) Stop PostgreSQL\n\n\n    docker compose down\n\n\n##  Why this pattern is useful in real projects\n\nThis design is a strong baseline for:\n\n  * ETL and data migration\n  * order/payment reconciliation\n  * large-volume reporting prep\n  * scheduled backend data shaping\n\n\n\nYou get:\n\n  * clear separation of orchestration vs business logic\n  * predictable transactional boundaries\n  * scalable parallel execution\n  * operational observability through batch metadata\n\n\n\n##  Next extensions\n\nIf you want to evolve this sample toward production-grade scale:\n\n  1. Add retry/skip policies for fault tolerance.\n  2. Export job metrics (Micrometer + Prometheus/Grafana).\n  3. Make partition strategy adaptive to dataset size.\n  4. Move to remote partitioning for multi-node execution.\n\n\n\nIf you’re learning Spring Batch or designing high-throughput processing pipelines, this pattern is a solid starting point: simple enough to understand, realistic enough to extend.",
  "title": "Building Distributed Data Processing with Spring Batch 6 + Spring Boot 4"
}