request batching pattern
Instead of one network call per item, accumulate items and send them in batches. Trades a small amount of latency for a lot of throughput. Pretty much every high-throughput system does this somewhere — Kafka producers, DB inserts, log shippers, GraphQL DataLoader.
Instead of one network call per item, accumulate items and send them in batches. Trades a small amount of latency for a lot of throughput. Pretty much every high-throughput system does this somewhere — Kafka producers, DB inserts, log shippers, GraphQL DataLoader.
symptoms
- CPU dominated by per-request overhead
- connection pool saturated by tiny requests
causes
- one-by-one calls in a loop
- no buffering layer
fixes
- accumulate up to N items or T ms, then flush
- DataLoader-style coalescing within a request
- tune batch size against latency budget
you might say
- batch the requests
- flush the buffer
- micro-batch