Integration Observability: Correlation IDs, Error Classes, and Idempotency

“The order didn’t go through. It failed somewhere.”

If you’ve done enterprise integration, you know that sentence. Not what failed. Not where. Just — somewhere. On my TechnoStore project, an order touched Salesforce, MuleSoft, Stripe, SAP, and a shipping provider before anyone got a tracking number. Ten external systems in total. When something broke, “somewhere” covered a lot of territory.

This article is about the layer I built so that “it failed somewhere” became “it failed here, for this reason, and here’s whether retrying is safe.” That layer has three pillars: correlation IDs, error classification, and idempotency keys. None of them are hard. All of them have to be there before the outage, not after.

The real enterprise problem isn’t failure — it’s finding it

One system failing is easy to debug. You read its logs, you find the error, you fix it.

Five systems failing is a different animal, because each system only sees its own slice. Salesforce says “I sent the request.” MuleSoft says “I forwarded it.” SAP says “I never got anything valid.” Everyone’s logs are individually true and collectively useless, because there’s no way to line them up. You end up matching timestamps by hand — “was the 14:03:07 error in MuleSoft the same request as the 14:03:09 error in Salesforce?” — which is guesswork wearing a lab coat.

Observability is the fix for exactly this. Not more logging — connected logging.

Correlation IDs: one thread through every system

A correlation ID is a unique identifier minted at the very start of a business transaction — when the order is placed, when the webhook arrives — and then passed along to every system that touches it, usually as an HTTP header.

X-Correlation-Id: ord-7f3a9c1e-2026

Every system logs it with everything it does. Salesforce logs it when it fires the request. MuleSoft logs it at every transformation step. The entry that goes toward SAP carries it. When something fails, you search all your logs for that one ID and the whole journey lines up in front of you like a filmstrip:

14:03:05  [salesforce]  ord-7f3a9c1e  order submitted, calling MuleSoft
14:03:06  [mulesoft]    ord-7f3a9c1e  received, transforming to OData payload
14:03:07  [mulesoft]    ord-7f3a9c1e  calling SAP → HTTP 403 CSRF token rejected
14:03:07  [mulesoft]    ord-7f3a9c1e  classified TRANSIENT, retry scheduled

There’s the whole story. No timestamp archaeology. The rules that make it work:

• Mint it at the edge. The first system to see the transaction creates the ID. Everyone downstream passes it through — never mints a new one mid-flow, or you’ve cut the thread.
• Log it everywhere, including errors. An error log line without the correlation ID is a clue with the name torn off.
• Store it on the record. In our build, the correlation ID lived on the integration log records in Salesforce, so a support person could start from an order and pull the full cross-system trace without touching a terminal.

This is genuinely 80% of the value of observability, and it costs you one header and one logging convention.

Error classification: not all failures deserve a retry

Once you can see failures, the next question is what to do with each one. The mistake I see constantly is treating every error the same — usually “retry everything,” which turns small problems into big ones. On TechnoStore, every error got classified into one of three classes:

Transient errors — retry

The request was fine; the moment was bad. Timeouts, HTTP 503, a rate limit, SAP rejecting a stale CSRF token. Retry with backoff and most of these heal themselves without a human ever knowing.

Permanent errors — fix, don’t retry

The request itself is broken. Malformed payload, HTTP 400, a field SAP’s OData service doesn’t recognize, an authentication failure. Retrying a permanent error is asking the same question louder and expecting a different answer. These need an alert and a developer, not a retry loop burning quota for hours.

Business errors — route to a human

Nothing is technically wrong. The systems all worked — the business said no. Credit limit exceeded, product blocked for that region, invoice already paid. No retry and no developer will fix these; they belong in a queue for the person who owns the business decision, phrased in business language.

The classification happened in the MuleSoft layer, at the moment of failure, and was stamped onto the log record. That one field drives everything downstream: transient → automatic retry, permanent → developer alert, business → user-facing queue. Without the classification, you either retry everything or nothing, and both are wrong.

Idempotency keys: making inbound webhooks safe

The third pillar guards your front door. External systems send you webhooks — Stripe says a payment succeeded, DocuSign says a contract was signed. And webhooks get delivered more than once. That’s not a bug; it’s the contract. Providers retry when they don’t get a timely 200, so at-least-once delivery is what you signed up for.

If your handler creates a payment record every time the “payment succeeded” webhook arrives, a single retry double-posts a payment. The fix is an idempotency key: a unique identifier per event (providers like Stripe give you one — the event ID) that you check before processing.

// Pseudo-code for an inbound webhook handler
key = webhook.eventId

if ProcessedEvent exists with Key == key:
    return 200          // already handled — acknowledge and do nothing
else:
    process the event
    store ProcessedEvent with Key == key
    return 200

Two details that matter in practice:

• Return 200 for duplicates. The sender doesn’t care that you’d seen it before; it cares that you received it. Any other status invites another retry.
• Verify before you trust. In our build, every inbound webhook was verified with an HMAC-SHA256 signature at the MuleSoft layer before Salesforce ever saw it — idempotency protects you from duplicates, the signature protects you from forgeries. You need both.

If this idempotency idea feels familiar, it’s the same principle from error handling and retries — there we attached keys to outbound requests so our retries were safe; here we honor them on inbound events so everyone else’s retries are safe too. Same principle, both directions.

What a useful integration-health dashboard answers

All of this feeds one place: a dashboard. But a dashboard is only useful if it answers questions people actually ask. Ours was built around four:

1. Is anything failing right now? Error counts per integration, per error class, over the last hours. Transient noise and permanent failures displayed differently — a spike of transient errors that all retried successfully is a non-event; three permanent errors is a fire.
2. Is anything getting slower? Latency trends per endpoint. Integrations rarely die suddenly; they degrade first. The dashboard that shows slower buys you days of warning.
3. Did retries actually recover? A retry that eventually succeeded is a system working as designed. A message that exhausted its retries needs a human — those surfaced in their own list, never silently dropped.
4. What happened to this specific order? The correlation ID search box. In practice this was the most-used feature by far — support could answer “where is my order?” in thirty seconds without escalating to a developer.

Notice what’s not on that list: vanity metrics. Total messages processed this month tells you nothing at 2 a.m. Build the dashboard around the questions asked during an incident, because that’s when anyone actually looks at it.

Build it before you need it

Here’s the honest part. None of this is exciting to build. There’s no demo moment where the room applauds a correlation ID. And that’s exactly why so many projects skip it — until the first cross-system failure, when a developer spends two days matching timestamps across five log files, and suddenly the observability layer gets approved.

Skip the two days. One header passed through every hop, one classification field on every error, one key check on every inbound webhook, and a dashboard that answers four questions. That’s the whole layer. It’s what turned “it failed somewhere” into a thirty-second lookup on a ten-system project — and it will do the same on yours.

Your next step

This article stands on the shoulders of Error Handling and Retries — read that first if retries and backoff are new to you. For a concrete flow where idempotency protects real money, see CAMT.053 bank reconciliation, and for keeping all these connections secure, authentication basics. The full journey lives in the Integration category.