scatter-gather pattern
Send a query to every partition in parallel and merge the results. Necessary when the query doesn't include the partition key. Cost is N parallel calls and latency = slowest partition.
Send a query to every partition in parallel and merge the results. Necessary when the query doesn't include the partition key. Cost is N parallel calls and latency = slowest partition.
symptoms
- query slows linearly with partition count
- tail latency spikes from one slow shard
causes
- query lacks partition key
- secondary-index reads in a partitioned store
fixes
- denormalize / pre-compute by query key
- secondary index colocated with primary
- request hedging for tail latency
you might say
- scatter-gather across shards
- fan out the query