Webhooks power most serious Shopify integrations. They push order, product, and inventory events to your systems the moment they happen. But webhooks fail. Endpoints go down, deploys drop requests, and bugs corrupt data before anyone notices. When that happens, you need a way to replay the events you missed. That is where a Shopify webhook replay system earns its keep.
In this guide, we break down how replay systems work, why they matter, and how to build one that recovers lost events without creating duplicates or corrupting state.
What a Webhook Replay System Actually Does
A replay system stores incoming webhook events and lets you reprocess them on demand. Think of it as a time machine for your event stream. When something breaks, you rewind and run the events again.
Shopify sends webhooks for topics like orders/create, inventory_levels/update, and products/update. Each delivery is a single shot. Shopify retries failed deliveries for up to 48 hours, then gives up. If your system is down longer than that, those events are gone unless you have your own store.
A replay system solves three core problems:
- Recovering events lost during outages or deploys
- Reprocessing events after you fix a downstream bug
- Backfilling data when you add a new consumer or feature
Without replay, your only fallback is a full data sync through the Admin API. That is slow, rate-limited, and often incomplete for high-volume stores.
Why Shopify Events Get Lost in the First Place
Before you build recovery, understand the failure modes. Events disappear for predictable reasons.
| Failure cause | What happens | Replay helps? |
|---|---|---|
| Endpoint downtime | Shopify retries, then stops after 48h | Yes |
| Slow response timeout | Shopify marks delivery failed | Yes |
| Deploy drops in-flight requests | Request never reaches handler | Yes |
| Downstream bug corrupts data | Event processed wrong | Yes |
| Duplicate delivery | Same event arrives twice | Needs idempotency |
| Out-of-order delivery | Newer state overwritten by older | Needs ordering logic |
Shopify does not guarantee ordering or exactly-once delivery. That means replay alone is not enough. You pair it with idempotency strategies and logic for Shopify webhook ordering problems to keep state correct.
For deeper context on how these events flow, our primer on Shopify webhooks and event-driven architecture covers the fundamentals.
The Core Architecture of a Replay System
A good replay system splits ingestion from processing. You capture events fast, store them durably, then process them separately. This separation is what makes replay possible.
Here is the standard flow:
- Receive the webhook and verify its HMAC signature
- Store the raw payload immediately in a durable log
- Acknowledge Shopify with a fast 200 response
- Queue the event for async processing
- Process the event and mark its status
The key move is storing the raw payload before you do any work. If processing fails, the stored copy stays untouched and ready for replay.
This pattern aligns closely with queue-based Shopify webhook processing, where the queue acts as a buffer between Shopify and your business logic.
Fast Ingestion First
Never process heavy logic inside the webhook handler. Shopify expects a response within five seconds. If you run database joins, external API calls, or inventory math during the request, you risk timeouts.
Instead, write the event to storage and return immediately.
public function handle(Request $request)
{
$payload = $request->getContent();
$topic = $request->header('X-Shopify-Topic');
$eventId = $request->header('X-Shopify-Webhook-Id');
if (! $this->verifyHmac($request)) {
return response('Unauthorized', 401);
}
WebhookEvent::create([
'event_id' => $eventId,
'topic' => $topic,
'payload' => $payload,
'status' => 'received',
'received_at' => now(),
]);
ProcessWebhookJob::dispatch($eventId);
return response('OK', 200);
}
That event_id from the X-Shopify-Webhook-Id header is your anchor for deduplication and replay tracking.
Building the Event Store
The event store is the heart of your replay system. It must be durable, queryable, and able to hold events long enough to recover from any realistic outage.
Design your schema to capture everything you need for replay and debugging.
| Column | Purpose |
|---|---|
| event_id | Unique Shopify webhook ID |
| topic | Event type like orders/create |
| payload | Full raw JSON body |
| status | received, processing, done, failed |
| attempts | Retry count |
| received_at | Timestamp of arrival |
| processed_at | Timestamp of completion |
| error_message | Last failure detail |
Store the payload as raw text or JSONB. Raw storage means you replay the exact bytes Shopify sent, which matters for accurate HMAC verification and reprocessing.
For high-volume stores, consider a dedicated event log rather than a shared application table. Our write-up on Shopify event streaming architecture explains how to structure durable event logs at scale.
Retention Windows
Decide how long to keep events. A 30-day window covers most recovery scenarios and keeps storage manageable. For compliance or audit needs, archive older events to cold storage like S3.
Keep hot events in your primary database and move cold events to cheaper storage. This hybrid approach keeps queries fast while preserving history.
How Replay Actually Works
Replay reads events from the store and re-runs them through your processing pipeline. The trigger can be manual, automatic, or scheduled.
A simple replay command pulls failed events and re-queues them.
public function replayFailed(string $topic, Carbon $since)
{
$events = WebhookEvent::where('status', 'failed')
->where('topic', $topic)
->where('received_at', '>=', $since)
->orderBy('received_at')
->get();
foreach ($events as $event) {
$event->update(['status' => 'received', 'attempts' => 0]);
ProcessWebhookJob::dispatch($event->event_id);
}
return $events->count();
}
Notice the orderBy('received_at'). Replaying in the original order reduces the chance of stale data overwriting fresh state. This matters most for inventory and price updates.
Replay Triggers
You have several ways to kick off a replay. Choose based on how critical the data is.
| Trigger type | When to use |
|---|---|
| Manual | After fixing a known bug, controlled recovery |
| Automatic | Dead letter queue drains on a schedule |
| Scheduled | Nightly backfill or consistency checks |
| On-demand API | Ops team triggers via dashboard |
Automatic replay works well with a dead letter queue for Shopify webhooks. Events that exhaust retries land in the DLQ, and a scheduled worker replays them once the root cause is resolved.
The Duplicate Problem You Cannot Ignore
Replay creates duplicates by design. You re-run events that may have already succeeded partially. If your handlers are not idempotent, replay will corrupt data.
Say an orders/create event fired, created an order record, then failed before sending a confirmation email. On replay, you do not want a second order record. You want the system to recognize the order already exists and only complete the missing step.
This is why idempotency is non-negotiable in any replay system. Our guide on handling duplicate Shopify events covers the exact patterns.
A basic idempotency check keys off the event ID.
public function process(string $eventId)
{
$event = WebhookEvent::where('event_id', $eventId)->first();
if ($event->status === 'done') {
return;
}
$event->update(['status' => 'processing']);
try {
$this->handleBusinessLogic($event);
$event->update(['status' => 'done', 'processed_at' => now()]);
} catch (\Throwable $e) {
$event->increment('attempts');
$event->update(['status' => 'failed', 'error_message' => $e->getMessage()]);
throw $e;
}
}
For operations that touch external systems, add idempotency keys on the downstream side too. The deeper techniques in our idempotency strategies post help you avoid double-charging, double-shipping, and double-emailing.
Keeping Order During Replay
Replay can scramble event order if you are not careful. Imagine you replay a batch of inventory updates. If they run concurrently, an older stock level might overwrite a newer one.
Guard against this with versioning or timestamps. Compare the event timestamp against the current record before applying changes.
public function applyInventoryUpdate($event, $payload)
{
$record = InventoryLevel::find($payload['inventory_item_id']);
if ($record && $record->updated_at > Carbon::parse($payload['updated_at'])) {
return;
}
$record->update(['available' => $payload['available']]);
}
This stale-write guard is essential for inventory accuracy. Our articles on real-time Shopify inventory sync and preventing race conditions in order processing go deeper on this topic.
Verification and Security During Replay
Replay must preserve the same security guarantees as live ingestion. Since you store raw payloads, you can re-verify the HMAC signature on replay to confirm data integrity.
Always verify the signature before storing. A corrupted or spoofed payload should never enter your event store. Our guide on secure webhook verification systems details the full verification flow.
private function verifyHmac(Request $request): bool
{
$hmac = $request->header('X-Shopify-Hmac-Sha256');
$computed = base64_encode(
hash_hmac('sha256', $request->getContent(), config('shopify.webhook_secret'), true)
);
return hash_equals($hmac, $computed);
}
Store events only after they pass verification. This keeps your replay source clean and trustworthy.
Handling Volume and Backpressure
Replay can flood your system. If you re-queue ten thousand events at once, your workers and database may buckle. This is a classic backpressure scenario.
Throttle your replay. Process events in controlled batches rather than dumping them all into the queue.
WebhookEvent::where('status', 'failed')
->chunkById(500, function ($events) {
foreach ($events as $event) {
ProcessWebhookJob::dispatch($event->event_id)->delay(now()->addSeconds(rand(1, 30)));
}
});
The delay spreads load over time. For production systems, our coverage of backpressure handling in Shopify systems and scaling Shopify webhook infrastructure shows how to size queues and workers correctly.
During peak traffic like flash sales, replay can compete with live events for resources. Plan capacity ahead using the patterns in scaling Shopify systems during flash sales.
Observability: You Cannot Replay What You Cannot See
A replay system needs strong visibility. You must know which events failed, why, and whether replay fixed them. Blind replay is dangerous.
Track these metrics:
- Events received per topic per hour
- Failure rate by topic
- Replay success and failure counts
- Processing lag between received_at and processed_at
- Dead letter queue depth
Surface these in a dashboard so your ops team can spot problems early. Our guide on observability in Shopify ecosystems covers the tooling and metrics that matter.
When something breaks at scale, pair replay with a solid incident response playbook. Replay is a recovery tool, and recovery works best with clear runbooks.
A Practical Replay Workflow
Here is how a mature team handles a real incident.
First, an alert fires. The orders/create failure rate spiked after a deploy. Your observability dashboard confirms five hundred events failed in the last hour.
Next, you investigate. The deploy introduced a bug in the order handler that threw on orders with gift cards. You ship a fix.
Then you replay. You run a targeted replay for orders/create events in the affected window. Because your handlers are idempotent, events that partially succeeded complete cleanly, and fully failed events process fresh.
Finally, you verify. Your dashboard shows the failure count drop to zero and processing lag return to normal. The incident closes.
This workflow only works because you stored every raw event and built idempotent handlers. The investment pays off during exactly these moments.
Replay Versus Full Resync
Teams sometimes ask why they need replay when they can just resync from the Admin API. The answer is cost and completeness.
| Factor | Webhook replay | Full Admin API resync |
|---|---|---|
| Speed | Fast, targeted | Slow, full dataset |
| Rate limits | None on your store | Heavy API throttling |
| Granularity | Specific events and windows | All or nothing |
| State history | Exact event sequence | Current state only |
| Load impact | Controlled | Can be severe |
Full resync loses event history. It gives you the current state but not the sequence of changes. For audit trails, analytics, and debugging, that sequence matters. Replay preserves it.
A resync is still a useful fallback for total data loss. But for routine recovery, replay is faster, cheaper, and more precise. If you are weighing architecture choices, our post on fault-tolerant Shopify integration systems frames the tradeoffs.
Common Mistakes to Avoid
Teams building replay systems tend to hit the same walls. Learn from them.
Do not process inside the webhook handler. Store first, process async. This single rule prevents most timeout-driven event loss.
Do not skip idempotency. Replay without idempotency is a corruption machine. Every handler must be safe to run twice.
Do not ignore ordering. Inventory and pricing events need stale-write protection. Timestamps or version numbers solve this.
Do not replay without throttling. Uncontrolled replay can take down your own system. Batch and delay.
Do not forget cleanup. Unbounded event storage grows into a problem. Set retention policies early.
For a broader view of resilience, our articles on building reliable Shopify webhook consumers and preventing cascading failures in Shopify apps connect these ideas into a complete reliability strategy.
Scaling the Replay System
As your store grows, your replay system must grow with it. A single database table and one worker will not handle enterprise volume.
At scale, you partition the event store by topic or date. You run dedicated worker pools for replay separate from live processing. You distribute load across services using patterns from multi-service Shopify architecture and load balancing strategies.
For stores with complex inventory across locations, replay ties directly into enterprise inventory synchronization and multi-warehouse Shopify systems. Replay accuracy keeps stock levels correct across every channel.
When you need expert help designing these systems, the team at Kolachi Advanced Technologies builds resilient Shopify architectures for high-volume merchants.
Bringing It Together
A Shopify webhook replay system turns fragile event delivery into a recoverable, auditable pipeline. You capture every event, store it durably, and replay it whenever something breaks.
The three pillars are durable storage, idempotent processing, and controlled replay. Get those right and you can survive outages, fix bugs retroactively, and backfill new features without losing a single event.
Start simple. Store raw payloads, add an idempotency check, and build a manual replay command. Then layer in ordering guards, backpressure control, and observability as your volume grows. Each piece builds on the last, and together they give you a system you can trust when things go wrong.
Frequently Asked Questions
What is a Shopify webhook replay system?
It is a setup that stores incoming webhook events durably and lets you reprocess them on demand. When deliveries fail or a bug corrupts data, you replay the stored events instead of losing them.
How long does Shopify retry failed webhooks?
Shopify retries failed webhook deliveries for up to 48 hours. After that, it stops. A replay system protects you against outages longer than this window.
Why do I need idempotency for replaying Shopify events?
Replay re-runs events that may have already partially succeeded. Without idempotency, you create duplicate orders, double charges, or corrupted inventory. Idempotent handlers make replay safe.
What is the difference between webhook replay and full resync?
Replay reprocesses specific stored events quickly and preserves event order. Full resync pulls current state from the Admin API, which is slow, rate-limited, and loses the event history.
How do I prevent duplicates during webhook recovery?
Key your processing off the unique X-Shopify-Webhook-Id header and check event status before processing. Skip events already marked done, and add idempotency keys on downstream operations.
Can replay overwrite newer data with older events?
Yes, if you ignore ordering. Compare the event timestamp against the current record and skip stale updates. This stale-write guard is critical for inventory and pricing events.
Should I process webhooks inside the HTTP handler?
No. Store the raw payload, acknowledge Shopify fast, then process asynchronously through a queue. This prevents timeouts and keeps your replay source intact.
How long should I store webhook events for replay?
A 30-day hot window covers most recovery scenarios. Archive older events to cold storage like S3 for compliance or audit needs while keeping recent events fast to query.
