Designing a Resilient MS Teams Contact Center: A Masterclass in Eventual Consistency and Call Hydration

The 0-to-1 Challenge

We were building a full-featured contact center natively on Microsoft Teams. Not a wrapper, not a bridge — a first-class application that routes PSTN and VoIP calls to skilled agents, manages queues, records interactions, and surfaces real-time analytics. The integration surface is the Graph Communications API: call-state change notifications are pushed directly to our bot’s registered callback endpoint, and REST methods on the call resource let us answer, transfer, redirect, and terminate calls programmatically.

In the lab this works beautifully. An inbound call hits the bot, a notification with state incoming arrives at the callback URL, the payload includes the call resource with source, direction, and callChainId, and your logic takes over. But somewhere between the demo and the first production deployment — now processing 1,000+ calls per day across a carrier-grade SIP trunk — a quiet problem surfaced that no amount of unit testing could have predicted.

The Mystery of the Empty Roster

During our first live pilot, the monitoring dashboards lit up with anomalies. Inbound calls were being received — routing logic fired correctly — but the participant roster was incomplete. Specifically, the source participant identity on the initial notification was either null or contained a temporary, un-resolved object ID that didn’t map to any known user or PSTN number. The originating participant was sometimes entirely absent from the roster in the initial notification.

Our first instinct was a bug in our deserialization layer. It wasn’t. The payload was faithfully representing exactly what the Graph Communications API sent us. We captured raw payloads and confirmed: the notification was delivering a delta update about a call that hadn’t fully materialized yet.

At first glance it looks like a platform bug — but it’s actually the system working exactly as designed. Understanding why requires thinking about what happens inside a globally distributed platform the moment a call arrives.

The Architectural Insight: Eventual Consistency in Real-Time Systems

Microsoft Teams is a globally distributed, hyper-scale communication platform. When a PSTN call enters the system, multiple subsystems activate in parallel: the Session Border Controller negotiates media, the identity service resolves the caller against Entra ID and the PSTN number mapping table, policy engines evaluate call routing rules, compliance recorders are notified, and the Bot Framework channel layer dispatches notifications to our bot’s callback URL.

These operations do not execute in a single transaction. They execute concurrently, across geographically distributed microservices, and they converge toward a complete call state over time. The notifications pushed to your callback endpoint are not snapshots of a finalized object — they are delta signals emitted as each subsystem completes its work. The notification with state: "establishing" fires the moment the call resource is created, which is before identity resolution is guaranteed to have completed.

This is textbook eventual consistency, applied to a real-time domain. The Graph Communications API is telling you the truth — the truth right now — but “right now” is a moving target in a distributed system.

Every one of these recommendations maps to a component we had already built through trial and error: the hydration loop (1 & 2), the PARTICIPANTS_PENDING state gate (3), and the differentiated timeout budget for PSTN vs. Entra ID participants (4).

The Solution: A Backend State Machine with Lazy Hydration

Once we framed the problem correctly, the architecture became clear. We needed two things: a state machine that models the call lifecycle independently of any single notification, and a hydration loop that lazily resolves incomplete state by polling the authoritative REST endpoint.

The State Machine

Every call in our system is modeled as a finite state machine. The Graph API itself defines the call states — incoming, establishing, ringing, established, hold, transferring, transferAccepted, redirecting, terminating, terminated — and we layer our own routing readiness on top:

INCOMING → ESTABLISHING → PARTICIPANTS_PENDING → ACTIVE → TERMINATING → TERMINATED

The PARTICIPANTS_PENDING state is our addition. The initial notification transitions the call from INCOMING through ESTABLISHING into PARTICIPANTS_PENDING. It does not advance to ACTIVE until the participant roster is fully resolved. This single design decision eliminated an entire class of routing defects — we stopped making decisions on incomplete data.

The Hydration Loop

When a call enters PARTICIPANTS_PENDING, a background task begins polling:

GET /communications/calls/{callId}/participants

The polling uses exponential backoff starting at 200ms, capping at 5 seconds, with a maximum of 8 attempts. In practice, identity resolution completes within 400–800ms for Entra ID users and 1–2 seconds for federated PSTN numbers. The moment the response contains a fully materialized participant with a resolved identity object, the state machine advances to ACTIVE and the call enters the routing engine.

If the hydration budget is exhausted — all 8 retries fail — the call is routed to a fallback queue with a synthetic “Unknown Caller” label, and an operational alert fires. In 18 months of production, this fallback path activates less than 0.02% of the time. But when it does, the system degrades gracefully instead of dropping the call.

Idempotency and Out-of-Order Delivery

Notifications are delivered at-least-once with no guaranteed ordering. An established notification can arrive before the establishing notification in rare race conditions. Our state machine handles this by accepting any valid forward transition and ignoring stale or duplicate events based on a monotonically increasing sequence derived from the notification’s changeType and resource version.

Every notification handler is idempotent. Processing the same notification twice produces exactly the same state. This is non-negotiable in any system consuming change notifications from an eventually consistent source.

Operational Patterns We Wish We Knew Earlier

The callback URL is your lifeline. Unlike subscription-based Graph change notifications (used for mailbox, calendar, or chat scenarios), the Communications API does not use /subscriptions. Instead, your bot registers a calling endpoint, and Microsoft Teams pushes call-state notifications directly to your callbackUri. If that endpoint goes down, notifications are lost — there is no replay mechanism and no subscription to renew. We front our callback with Azure Front Door and an active-passive failover to ensure zero dropped notifications.

Correlate everything end-to-end. Every call resource carries a callChainId that uniquely identifies the call across all participants. We use this as a correlation root and propagate it through Azure Service Bus, our state machine, SignalR hubs, and into the agent dashboard. When an agent reports “the caller ID didn’t show up,” we can trace the entire lifecycle from notification ingestion to UI render in Application Insights within seconds.

Design the front-end for convergence, not for snapshots. The agent dashboard renders immediately with whatever state is available — including an “Identifying caller…” placeholder — and updates in-place as hydration completes. This optimistic rendering pattern means agents see the call the instant it enters the system. The caller identity populates 400ms–2s later, which feels seamless. The alternative — blocking render until hydration finishes — adds perceptible latency to every single call and degrades the agent experience at scale.

The Leadership Takeaway

When you build enterprise SaaS on top of third-party platforms — Microsoft Graph, Salesforce, Twilio, AWS EventBridge — you are building on top of eventually consistent, distributed systems that you do not control. A notification is not a contract that says “here is the complete truth.” It is a signal that says “something changed; come get the truth when you’re ready.”

Architects who internalize this build systems that are fundamentally more resilient. They design state machines instead of request-response handlers. They implement hydration loops instead of trusting single payloads. They make every handler idempotent because they know the same event will arrive twice.

This is not a workaround. This is the architecture. The sooner a team embraces that, the sooner they ship systems that hold up under real-world load — and the fewer 2 AM pages they absorb after launch.