API Integration Services in India
API integration services in India, built for the CTO, founder or head of engineering in the US, UK, Canada, Australia or New Zealand who needs two systems to talk to each other and keep talking long after the demo. This page is about the engineering craft of connecting systems over APIs: delivery semantics, retries, webhook receipt, rate limits, auth and the failure handling that decides whether an integration survives its first bad week. Vendor-specific work has its own pages.
Why Do Integrations Break Months After Someone Signed Them Off?
Almost nobody calls us while an integration is being built. They call us eleven months later, when the same connector that passed acceptance testing is quietly dropping one record in four hundred and nobody can say which four hundred.
The reason is structural. An integration is written and tested during the only period of its life when both sides are healthy, the network is fine, the developer is watching, and the volume is a tenth of what it will be. Every hard part of this work is about the other 99% of the time: the seventeen seconds the provider was returning 502s, the deploy that restarted your workers mid-batch, the day marketing ran a campaign and your request rate tripled into a rate limit you had never hit, the token that expired at 03:00 because nobody tested the refresh path.
What that looks like from the business side is very specific and very familiar. A customer says they placed the order and your fulfilment system has never heard of it. Two invoices exist for one shipment because a retry created a second record. A support agent is looking at a status that was correct nine hours ago. Somebody has written a nightly script that compares two systems and emails the differences, and that script has become load-bearing. Worst of all, an engineer says the sentence that should worry any CTO: we are not sure how many are missing, because we only find out when a customer tells us.
The cost is rarely counted properly because it never appears as one line. It shows up as support hours, as manual reconciliation, as an engineer who now owns the connector nobody else understands, and as the growing reluctance to touch anything near it. Teams end up with an integration they are afraid of, which means the next integration gets built the same way by the same person, and the problem compounds. When a company finally decides to rebuild, the hard part is almost never the API call. It is working out what the correct state actually is after a year of quiet divergence.
Good integration engineering is mostly the discipline of assuming the unhappy path is the normal path. Every decision on this page comes from that assumption.
What an API Integration Engagement Actually Covers
The shape varies with how many systems are involved and who owns them, but this is what we build and what you get to keep. It all lands in your repository, in your pipeline, in a stack your engineers already run.
A read of the real interface, not the marketing page
We start by driving the actual API. What the documentation promises and what the endpoint returns diverge more often than anyone expects: undocumented fields, a status code the docs never mention, a pagination cursor that expires, a sandbox that behaves differently from production. We write that divergence down before designing anything on top of it, because a design built on the documentation alone is a design built on a claim.
The connector, with delivery semantics decided on purpose
The code that moves data, with an explicit answer to one question per operation: what happens if this runs twice. That answer drives idempotency keys, deduplication storage, and whether an operation can be safely retried at all. Deciding this deliberately, once, is the single largest difference between an integration that ages well and one that does not.
Failure handling as a designed feature
Timeouts on every outbound call, retry policy with backoff and jitter, a circuit breaker per dependency, bulkheads so one vendor cannot exhaust shared capacity, and a defined degraded mode. What the product does when the provider is down is a product decision, so we bring you the options rather than defaulting to a 500.
Inbound webhook receipt hardened properly
Signature verification against raw request bytes, replay protection, fast acknowledgement with processing moved to a queue, deduplication on the provider's event identifier, and a strategy for out-of-order arrival. Most webhook endpoints we are asked to review fail at least two of these.
Auth, tokens and secrets that survive a rotation
The right OAuth 2.0 grant for the situation, refresh handled in one place with clock skew accounted for, credentials in your secret manager rather than your environment file, and a documented rotation path that has actually been exercised in a lower environment.
Observability specific to integrations
Correlation identifiers propagated across the boundary, structured logs with the provider's request identifier captured, per-dependency success rate and latency, queue depth and dead letter alerts, and redaction of anything sensitive before it reaches your log store.
Reconciliation, because drift happens anyway
For anything that must stay in agreement over months, a scheduled comparison that reports divergence rather than waiting for a customer to find it. Detection first, then automated repair for the cases where repair is unambiguous.
Handover your team can pick up
A runbook that answers the questions a person will actually have at 02:00: how to replay a failed event, how to safely reprocess a dead letter queue, what to do when the provider posts an incident, and which failures are safe to ignore. One bar applies to every piece of work: your engineers have reviewed it, it runs on your infrastructure, something watches it, and the documentation is good enough that the next change does not need a call with us.
REST, GraphQL, gRPC or Webhooks: Which One Fits Which Job?
Half the time this is not your decision. The system you are integrating with has already chosen, and the skill is consuming what exists without letting its awkwardness spread through your codebase. Where you do own both ends, the trade-offs are real and the wrong pick is expensive to undo.
REST over JSON
The default for anything crossing an organisational boundary, and the default for good reasons. Every language has a mature client, every proxy and gateway understands it, caching works with HTTP semantics rather than against them, and any engineer can debug it with curl at three in the morning. Its weaknesses are equally real: over-fetching when you need three fields from a fat resource, under-fetching when a screen needs data from five endpoints, and no built-in schema unless somebody maintains an OpenAPI document. It is wrong when you have genuinely chatty internal traffic where the serialisation overhead and round trips actually show up in your latency numbers.
GraphQL
Earns its complexity in one situation: many different clients need different shapes of the same underlying data, and shipping a new backend endpoint for each of them has become the bottleneck. You get a typed schema, field-level deprecation and client-driven queries. You also inherit a set of problems that do not exist elsewhere. Errors arrive inside a 200 response, so naive client code treats a failure as a success. HTTP caching largely stops working, and you replace it with persisted queries and a cache at the resolver layer. A careless query can be quadratically expensive, so query depth and cost limits are not optional. And the N+1 problem moves into your resolvers, where DataLoader-style batching becomes mandatory rather than nice. It is the wrong choice for a simple service-to-service call between two teams who talk to each other daily.
gRPC
Strong where the traffic is internal, high volume and latency sensitive. Protobuf gives you a compact binary payload and a real schema that generates clients, HTTP/2 multiplexing removes head-of-line blocking at the connection level, and streaming in both directions is a first-class concept rather than a workaround. Deadlines propagate through the call chain, which is genuinely useful and something most REST estates never implement properly. The costs: browsers cannot speak it natively so you need grpc-web and a proxy, load balancing needs to be connection aware because long-lived HTTP/2 connections defeat naive layer 4 balancers, and debugging is harder because you cannot read the payload off the wire. It is the wrong choice for a public API consumed by partners whose stacks you do not control.
Webhooks and event delivery
Not a competing protocol so much as an inversion of who initiates. If you need to know when something changed and the alternative is polling every thirty seconds, webhooks are correct and the saving is large. The price is that you now run a public endpoint that a stranger posts to, which brings signature verification, replay protection, ordering that is not guaranteed, and duplicate delivery as the normal case rather than the exception. Webhooks are wrong when the event must be processed in a strict order that the provider does not guarantee, and they are wrong when you cannot expose an endpoint at all, where polling with a cursor or a provider-side queue is the honest answer.
The pragmatic combination
Most real estates end up with more than one, and that is fine as long as the choice per boundary is deliberate. A common and sensible arrangement is REST for anything external, gRPC between a handful of internal services that are genuinely chatty, and webhooks pushing change notifications that trigger a REST fetch for the authoritative state. What causes pain is protocol chosen by fashion, or a single style mandated across an estate where one boundary clearly needs something else.
Exactly-Once Delivery Is a Myth. Build for At-Least-Once Instead.
This is the idea that fixes the most bugs per sentence, so it is worth being precise. Across a network you do not control, exactly-once delivery cannot be achieved. A sender that sends a request and receives nothing back has no way to distinguish between two situations: the receiver processed it and the acknowledgement was lost, or the receiver never got it. Those are indistinguishable from where the sender stands, and no amount of protocol design removes the ambiguity. The sender either retries, risking a duplicate, or gives up, risking a loss.
So you choose which failure you prefer. For anything that matters, you choose at-least-once, and then you make duplicates harmless at the receiving end. At-least-once delivery plus deduplication gives you exactly-once effects, which is the property you actually wanted. Any vendor claiming exactly-once delivery is either describing deduplication inside their own boundary or is being loose with words.
Idempotency keys, and where teams get them wrong
The pattern is simple. The caller generates a unique key per logical operation, not per attempt, and sends it with every retry of that operation. The receiver stores the key with the result of the first successful execution and returns that stored result for any subsequent request carrying the same key. The header name Idempotency-Key has become the informal convention and is the subject of an IETF draft rather than a finished standard, so check what your provider actually expects.
Three mistakes are almost universal. Generating a new key on each retry, which defeats the entire mechanism. Accepting the same key with a different request body and silently returning the old result, which hides a genuine client bug and should be a 422 instead. And storing the key only after processing completes, which leaves a window where two concurrent requests both pass the check. That last one needs a write with a uniqueness constraint taken before the work starts, not an existence check followed by an insert.
When you cannot send a key
Plenty of providers do not support idempotency keys. Then you need a natural key: something in the payload that identifies the logical operation. An external order identifier, an invoice number, a hash of the meaningful fields plus a coarse time bucket. It is weaker, because the definition of "the same operation" now lives in your head rather than in the contract, but it is far better than nothing. Write down what you chose and why, in the code, next to the deduplication query.
Deduplication windows and storage
Deduplication state cannot grow forever. Pick a window from the provider's actual retry behaviour, which is often up to a few days for webhooks, then add margin. Redis with a TTL is the common choice and is fine when losing the set is survivable, which for a payment-adjacent flow it usually is not. A database table with a unique index and a scheduled purge is slower and much harder to lose. Choose per flow based on what a missed duplicate costs, not by habit.
The idempotent operations you get for free
Not everything needs machinery. Setting a field to a value is naturally idempotent. Adding to a set is. Incrementing a counter is not. Appending to a ledger is not. Where you have design freedom, expressing an operation as "make the state be X" rather than "change the state by X" removes the whole problem, and that reframing is available more often than people assume.
Retries, Backoff and the Retry That Makes Everything Worse
Retries are the most commonly added and most commonly misconfigured piece of integration code. Added carelessly, they convert a small provider wobble into your own outage, and they do it at exactly the moment the provider is least able to absorb it.
Decide what is even retryable
A 400 will be a 400 forever, so retrying it burns quota and hides a bug. A 401 needs a token refresh and then one attempt, not five. A 409 usually means your assumption about state was wrong. Timeouts, connection resets, 429s, 502s, 503s and 504s are the retryable set, with one large caveat: a timeout on a non-idempotent write is ambiguous, because the write may have succeeded. Retrying that without an idempotency key is how duplicate charges and double shipments happen. If you cannot make the call idempotent, the correct behaviour on timeout is often to stop and reconcile rather than to try again.
Exponential backoff, and why jitter is not optional
Fixed-interval retries synchronise. When a provider returns errors for twenty seconds, every one of your instances retries at the same moment, and the recovery attempt becomes a second load spike on a system that is already struggling. Exponential backoff spreads attempts out over time; jitter spreads them out across your fleet. AWS engineering has published a well-known analysis of the variants, and the practical summary holds up: full jitter, where the delay is a random value between zero and the current exponential ceiling, performs better under contention than adding a small random offset to a fixed schedule. Cap the ceiling so you do not end up with a retry scheduled forty minutes out on a customer-facing path.
Budgets beat attempt counts
Per-request attempt limits look safe and are not, because during a broad outage every request is retrying at its limit and your traffic to a failing dependency triples. The Google SRE book argues for a client-side retry budget instead: cap retries as a small proportion of overall request volume, so that when everything is failing the retries stop rather than multiply. It is a few lines of shared state. The arithmetic is the point: three attempts per request means a struggling dependency receives three times its normal traffic at the worst possible moment, while a budget caps the extra at whatever fraction you chose.
Timeouts and deadline propagation
A retry policy on top of an unbounded call is worse than no retry policy. Set connect and read timeouts separately, derive them from the provider's observed latency rather than a round number someone liked, and make sure the total of attempts plus backoff fits inside the request budget of whatever called you. In a chain of services, propagate a deadline rather than restarting the clock at each hop, or you build a system where the caller has already given up while three downstream services carry on doing work for nobody.
Retry storms and the recovery you did not plan
The failure mode to keep in mind: a dependency recovers, and the queued retries of every request from the last five minutes arrive at once and knock it over again. Jitter helps. Rate limiting your own outbound traffic helps more. A circuit breaker that reopens gradually rather than all at once is the piece that actually stops the oscillation, which brings us to the next section.
Circuit Breakers, Bulkheads and Containing a Bad Dependency
These two patterns come from Michael Nygard's Release It! and they remain the clearest thinking on the subject. Both exist to answer one question: when something you depend on fails, does the failure stay where it started?
What the breaker actually does
A circuit breaker watches the outcome of calls to one dependency. While failures stay under a threshold it stays closed and traffic flows. Once failures cross the threshold it opens, and calls fail immediately without touching the network. After a cooling period it moves to half-open and lets a small number of trial calls through. If they succeed it closes; if not, it opens again. The value is not that failing fast is nicer than failing slow, though it is. The value is that your threads and connections stop being consumed by calls that were going to fail anyway.
Thresholds people get wrong
A raw count of consecutive failures is a poor trigger because it ignores volume. A failure rate over a sliding window with a minimum call count is better, so a breaker does not trip on the first two requests after a quiet period. Slow calls deserve their own threshold: a dependency answering successfully in nine seconds is functionally down for a path with a two second budget, and a breaker that only counts errors will happily let it destroy your latency. Resilience4j on the JVM, Polly on .NET and gobreaker in Go all support these shapes, and a service mesh such as Envoy can do outlier detection at the proxy layer if you would rather keep it out of application code. Netflix stopped active development on Hystrix some years ago and pointed users toward Resilience4j, so new work should not start there.
Bulkheads, the part usually skipped
Named after ship compartments. If every outbound call shares one connection pool or one thread pool, a single slow dependency fills it and everything else queues behind traffic it has nothing to do with. Your database calls start timing out because a marketing enrichment API is slow. Giving each dependency its own bounded pool means the blast radius is the feature that uses it. This is cheap to implement and it is the difference between a degraded feature and a degraded product.
What happens when the breaker is open
This is a product question, not an engineering one, and it is worth asking the product owner explicitly. Sometimes the answer is a cached previous value with a staleness indicator in the UI. Sometimes it is queue it and process later, with the user told their request is accepted. Sometimes it is a clear error, which beats a spinner that never resolves. Occasionally the honest answer is that this path cannot degrade and must fail hard, which is fine as long as it is decided rather than discovered. We bring the options rather than picking silently, because the wrong fallback in a financial or clinical flow is worse than an error message.
Rate Limits and Living Inside Somebody Else's Quota
Rate limiting is where integrations that worked fine in testing fall over in production, because test volume never reaches the ceiling and the ceiling is often undocumented.
Read what the provider is telling you
A 429 with a Retry-After header is an instruction, and honouring it is both correct and the fastest route back to normal. Retry-After is defined in the core HTTP semantics specification and can carry either seconds or an HTTP date, so parse both. Many providers also return remaining-quota headers, and there is an IETF draft aiming to standardise a RateLimit family of fields, though it is a draft rather than a finished standard and provider names still vary. We capture whatever headers arrive on every response from day one, because that log is what tells you where the real ceiling sits when the documentation does not.
Limits are per fleet, not per process
The most common production surprise. A per-instance limiter that respects sixty requests a minute becomes six hundred a minute when autoscaling brings up ten pods. The quota belongs to your account, so the limiter has to be shared. A token bucket in Redis, refilled at the permitted rate, with every caller acquiring before it dials, is the standard shape and it is not much code. The subtlety is failure behaviour: decide explicitly whether losing the limiter means blocking all outbound calls or allowing them through, because both answers are defensible and the wrong one is discovered during an incident.
Budget the quota across features
Once a limiter is shared, a backfill job can consume the entire allowance and starve the checkout path. Split the budget by class of traffic: customer-facing calls get guaranteed headroom, batch and backfill jobs run on what is left and are expected to slow down. This is the one change that most reliably stops "we ran a data migration and the site stopped working."
Shape traffic before you hit the wall
Backing off after a 429 is reactive. Better to not send the request. Client-side pacing, batching where the provider offers a bulk endpoint, caching responses that do not change often, and collapsing duplicate in-flight requests for the same resource all reduce call volume without touching the provider. On one common pattern, a system polling a partner every minute for objects that change weekly, adding a conditional request with an ETag or a since-cursor cuts the effective load by orders of magnitude and costs an afternoon.
Distinguish 429 from 503
They need different responses. A 429 means slow down and you will be served; back off and continue. A 503 means the provider is unwell, and the right move is often to open the breaker rather than to keep politely retrying. Treating them identically is why some systems keep hammering a dependency that is asking to be left alone.
Receiving Webhooks Without Getting Burned
A webhook endpoint is a public URL that accepts instructions from outside your perimeter and usually mutates your data. It deserves more care than it typically gets. Here is the checklist we work through on every one.
Verify the signature against raw bytes
Most providers sign with an HMAC over the request body, commonly SHA-256, with the secret you were issued. Two details matter enormously. Compute the signature over the raw bytes exactly as received, before any JSON parsing and re-serialisation, because key ordering and whitespace changes will break verification in ways that look random. And compare using a constant-time function, not string equality, so you are not leaking information through timing. In many frameworks the raw body is consumed by middleware before your handler sees it, and retrieving it is the first practical problem to solve.
Replay protection
A valid signed payload captured once can be replayed forever unless something binds it to a moment. Providers that include a timestamp in the signed material let you reject anything outside a tolerance window, commonly around five minutes. Where the provider does not sign a timestamp, deduplication on the event identifier becomes your only defence, which makes that store security-relevant rather than merely useful.
Acknowledge fast, process later
Do the signature check, persist the raw event, return 2xx, and hand the work to a queue. Processing inline is the single most common design error we see. It means a slow database write causes the provider to time out and retry, generating duplicates; it means a deploy loses events in flight; and it means your endpoint's latency is coupled to your slowest downstream dependency. Providers typically retry with backoff for a period and then give up, and some disable endpoints that fail persistently, so slow processing eventually becomes silent data loss.
Duplicates are normal, so deduplicate
Webhook delivery is at-least-once everywhere. Store the provider's event identifier with a uniqueness constraint and drop repeats. Note that the same underlying change can also arrive as two different events, so business-level idempotency still matters. Deduplicating deliveries is not the same as making the handler safe to run twice, and you want both.
Ordering, which you do not get
Nothing guarantees that a created event arrives before the updated event that followed it, and retries make inversion likelier. If the payload carries a version or an updated timestamp, keep it and discard anything older than the state you hold. If it does not, treat the webhook as a notification and call the API for current state. The second approach costs a request and removes an entire category of bug, and for anything where correctness beats latency it is the one we recommend.
Hardening the endpoint itself
Reject oversized bodies before parsing. Rate limit the endpoint, because an attacker who cannot forge a signature can still exhaust your workers with junk. Allowlist provider source ranges where they publish them, as defence in depth rather than as your primary control. Keep signature secrets in the secret manager with a rotation path that accepts both old and new during a changeover, and make sure a failed verification produces an alert rather than a silent 401 nobody ever reads.
Prove it works before you need it
Local development with a tunnel, an automated test that posts a correctly signed fixture, a test that posts a wrong signature and asserts rejection, and a replay tool that can re-deliver a stored raw event to the handler. That replay tool is the thing you will be grateful for during the first incident, and it takes an hour to build in advance and a very stressful afternoon to build during.
Pagination, Cursors and the Sync That Never Finishes
Pulling a large collection through an API looks trivial and quietly produces some of the most persistent data quality bugs in an estate.
Offset pagination drifts while you read
Page through a list ordered by creation date while new records are being inserted, and rows shift between pages. You skip some and read others twice, and nothing errors. On a dataset that changes during the read, offset pagination is simply incorrect, and it also degrades badly on the server as the offset grows because the database still has to walk the rows it is discarding.
Keyset and cursor pagination
Ask for records after a specific position instead: after this identifier, after this timestamp plus this tie-breaker. Stable under insertion, and it stays fast at page ten thousand. When a provider hands you an opaque cursor, treat it as opaque, store it exactly, and check whether it expires, because a job that resumes after a long pause with a stale cursor may restart from the beginning without saying so. Always include a deterministic tie-breaker such as the primary key when sorting on a timestamp; without it, records sharing a timestamp can be skipped at a page boundary.
Incremental sync and the overlap window
Storing the highest updated timestamp you have seen and asking for everything newer is the standard incremental pattern and it has two traps. Clock differences and in-flight transactions mean a record can be committed with a timestamp slightly before your watermark, so re-request with a small overlap and rely on deduplication. And if the provider's timestamp has second resolution while hundreds of records share a second, an exclusive comparison drops the rest of that second permanently.
Deletes are the ones you lose
An incremental sync keyed on modification time sees creations and updates. It usually cannot see deletions at all, so records deleted upstream live on in your copy forever. If the provider exposes tombstones or a deletion event, consume it. If it does not, a periodic full reconciliation of identifiers is the only honest fix, and it should be scheduled from the start rather than added after somebody notices phantom rows.
Make long jobs resumable
Any backfill that runs for hours will be interrupted, by a deploy or a restart or a provider incident. Checkpoint the cursor after each page, make each page's processing idempotent, and the interruption costs a page rather than the job. Rate limit backfill separately from live traffic, and give it a way to be paused, because "we cannot stop the migration" during an incident is an unpleasant thing to say out loud.
Sync or Async, and When a Queue Stops Being Optional
Calling a third party inline inside a user's request is fine sometimes and catastrophic other times. The dividing line is clearer than most teams treat it.
When a direct call is right
The caller genuinely needs the answer to proceed, the call is fast relative to your latency budget, and abandoning the operation on failure is acceptable. A validation lookup, a search, a price check. Adding a queue here buys complexity and no reliability, because the user is waiting either way.
The four signals that make a queue mandatory
First, the work must survive the target being unavailable. Second, it takes longer than a person will wait. Third, the target rate limits you, so you need to control your own outbound pace independent of incoming traffic. Fourth, you need retries that outlive the request that started them, which an HTTP handler cannot provide because the connection is gone. Any one of these on its own is enough. Two of them and building it synchronously is a decision you will revisit within a quarter.
The outbox pattern, and the bug it prevents
Here is the bug: your handler writes a row to your database and then publishes a message. The write succeeds, the publish fails, and the rest of the world never learns about a record that exists. Or the publish succeeds and the transaction rolls back, and the world learns about a record that does not. You cannot make a database write and a network publish atomic. The transactional outbox fixes it by writing the message into a table in the same transaction as the business data, with a separate process reading that table and publishing. Delivery becomes at-least-once, which is exactly the property you have already designed for.
Ordering, partitions and what you give up
Queues trade ordering for throughput and the trade is worth understanding. Kafka gives ordering within a partition, so keying by entity identifier preserves order per entity while allowing parallelism across entities. SQS standard queues offer no ordering at all; FIFO queues do, with throughput constraints and a message group identifier that plays the same role as a partition key. RabbitMQ preserves order per queue until you add competing consumers, at which point you do not have it any more. Decide what actually needs ordering, which is usually per customer or per document rather than globally, and key accordingly.
What async costs you
Honesty matters here because queues are often sold as free. You take on infrastructure to run and monitor, eventual consistency that the UI has to represent, a harder debugging story across process boundaries, and a new failure mode in queue depth growing silently. The complexity is worth paying where one of the four signals applies. It is not worth paying because asynchronous sounds more scalable.
Error Taxonomy and Dead Letter Queues
"The integration failed" is not information. Most integration code treats every failure identically, which means the response is identical too, which means it is wrong for most of them.
Four classes, four responses
Transient infrastructure failures, meaning timeouts, resets and 5xx, get retried with backoff. Rate limiting gets backoff plus pacing, and it is not an error worth alerting on unless it is sustained. Client faults such as 400, 404 and 422 will not improve with time, so they go straight to a failure queue for a human or a repair job, and they should not consume retry budget. Semantic failures, where the call succeeded and the answer is unacceptable, such as a currency you do not support or a status you have no state machine branch for, need their own path entirely because retrying is meaningless. Every integration we build classifies before it reacts, and the classification is visible in logs and metrics.
Structured error responses
Where you own the API being called, RFC 9457 Problem Details gives a standard JSON shape for errors with a type, title, status and detail, plus room for your own fields. It replaced RFC 7807 and it is a small change that makes machine handling of errors possible instead of parsing prose. Where you are consuming someone else's API, capture their error body and their request identifier verbatim in your logs, because that identifier is what their support team will ask for and reconstructing it later is painful.
Dead letter queues used properly
A DLQ catches messages that failed after the maximum attempts. It is a diagnostic tool and a recovery buffer, and it only works if three things are true. Something must alert when depth goes above zero, because an unwatched DLQ is a data loss mechanism with extra steps. Each message must carry enough context to diagnose it: the original payload, the failure reason, the attempt count, the correlation identifier. And there must be a tested redrive path, so replaying is a command someone runs rather than a script written under pressure.
Poison messages and the loop that never ends
One malformed message that crashes the consumer before acknowledgement gets redelivered, crashes it again, and blocks everything behind it. Cap the attempts, move it aside, keep going. Also worth guarding: a redrive that immediately fails the same way and lands right back in the DLQ, which produces an alert loop nobody trusts within a day. Fix the cause, then redrive.
Alert on the right thing
Individual integration errors are usually noise, and paging on them trains people to ignore the channel. Alert on rate and on trend: error ratio over a window, DLQ depth above zero, queue age beyond a threshold, a breaker that has been open for more than a few minutes, and a scheduled sync that has not completed. Those five cover most of what genuinely needs a human at an odd hour.
Auth: OAuth 2.0 Flows, mTLS, API Keys and the Refresh Nobody Tests
Authentication is where integrations fail in the least convenient way, because a token problem tends to surface at an arbitrary hour with no code change to blame.
Choosing the grant
Server-to-server with no user involved is the client credentials grant, and it is the simplest thing that works. Acting on behalf of a user is the authorization code flow with PKCE, which is now the recommended default for confidential and public clients alike. Input-constrained devices use the device authorization grant. Two flows should not appear in new work: the implicit flow, which the OAuth working group moved away from, and the resource owner password credentials grant, which requires you to handle someone else's password and has been marked for removal in the OAuth 2.1 direction of travel. If a provider only offers password grant, treat that as a risk to record rather than a normal option.
Refresh tokens and rotation
Refresh token rotation, where each use issues a new refresh token and invalidates the old, is now common and it changes your code's obligations. Persist the new token atomically before you use the access token, or a crash between the two loses the ability to refresh at all. Rotation also enables reuse detection: if an old refresh token is presented again, the provider may invalidate the whole family on the assumption it was stolen, which is good security and means a race between two of your processes refreshing simultaneously can lock you out. Refresh in one place, with a lock, not in every worker that notices a 401.
The 03:00 expiry
Refresh proactively at some fraction of the token lifetime rather than reactively on the first 401, and account for clock skew when checking expiry, because a machine a few seconds fast will present tokens the provider considers not yet valid. Handle the case where refresh itself fails: distinguish a network failure, which should be retried, from an invalid grant, which means the connection needs a human to re-authorise and should raise a clear operational alert rather than a stack trace.
Validating tokens you receive
If you are the one receiving JWTs, verify the signature against the issuer's JWKS, match the key by its identifier, and cache the key set with a refresh path for rotation. Check issuer, audience and expiry every time. Never accept the algorithm the token itself declares; pin the algorithms you allow. These are old lessons and they are still being learned the hard way.
API keys and mutual TLS
Static API keys are still everywhere and can be handled responsibly: scope them to the narrowest permission that works, prefix them so they are recognisable in a leak scan, hash them at rest if you issue them, and build the rotation path before you need it. Mutual TLS gives you cryptographic client identity at the transport layer and is common in financial and healthcare integrations, sometimes as an OAuth client authentication method. Its operational cost is certificate lifecycle: expiry monitoring, renewal automation and a tested rollover, because an expired client certificate produces a total outage of that integration with an error message that rarely points at the real cause.
Where secrets live
In a secret manager your platform already runs, not in environment files committed to a repository, not in a shared password vault entry that four people copy from, and not in the integration's own database in plaintext. Credentials issued to us are scoped and revocable by you, sandbox where the provider offers one, and revoked by you at the end of the engagement. If your integration touches personal data, the lawful basis and cross-border transfer questions under GDPR or similar regimes are a matter for your counsel and privacy lead, and worth settling before the first byte moves rather than after.
Can You Answer "What Happened to Order 4471?"
That question is the real test of integration observability. A support agent asks it, and the answer should take two minutes with a search box, not two hours with an engineer and a database client.
Correlation across the boundary
Generate an identifier at the edge of your system and carry it everywhere: into log lines, onto queue messages, into outbound request headers, and back through webhook processing. The W3C Trace Context specification gives a standard traceparent header for this and most providers pass unknown headers through untouched, so send your own correlation header as well. Capture the provider's request identifier from their response and log it beside yours. That pairing is what turns a support ticket into a two-line query, and it is what a vendor's support team will ask for first.
Log what you can actually use
Structured logs with consistent field names beat prose. For every outbound call: dependency name, operation, status, duration, attempt number, correlation identifier and provider request identifier. For every inbound webhook: event identifier, event type, signature result, and processing outcome. Redact before writing, not after, and be specific about what counts as sensitive, because tokens and personal data in a log store are a compliance problem in a system that was never designed to be one.
Metrics that predict an incident
Per-dependency request rate, error rate and latency distribution, split by error class rather than lumped. Retry rate as its own series, because a rising retry rate with a flat error rate means a dependency degrading before it fails. Circuit breaker state changes as events. Queue depth and oldest message age. Time since the last successful sync per integration, which is the single metric most likely to catch a silently stopped job.
Tracing where it earns its keep
Distributed tracing with OpenTelemetry is worth it once a request crosses three or more services. Instrument the outbound HTTP client and the queue producer and consumer so a trace survives the asynchronous hop, which is the part naive instrumentation drops. Tail-based sampling keeps the traces you want, since a uniformly sampled trace set almost never contains the failure you are investigating.
The dashboard that answers the business question
One view per integration: is it flowing, how far behind is it, how many items are stuck, when did it last succeed. Engineers will happily read a latency histogram. The person who needs it at 09:00 on a Monday needs to know whether yesterday's orders all arrived, and building that view is usually a day of work that removes a recurring week of questions.
Versioning and Deprecation of an API You Do Not Control
Plenty has been written about versioning an API you publish. Consuming one is a different discipline, because the schedule belongs to somebody else and you will find out about the change at their convenience.
Pin, and know what you pinned
Where a provider offers explicit versions, pin them and put the version in configuration rather than scattered through the code. Keep a register of every external API you consume with the version in use, the account it runs under, who owns it internally, and where the credentials live. Most estates cannot produce that list on request, which is why deprecation emails go to an inbox nobody reads and the first sign of trouble is a 410.
Watch the signals the provider sends
The Sunset HTTP header, specified in RFC 8594, carries a machine-readable date on which a resource stops being available, and a companion Deprecation header field is in common use for the earlier warning. Log both when they appear on any response and alert on them, because a header is far more reliable than an email routed to whoever signed up three years ago. Subscribe to changelogs and status pages properly, with a real distribution list rather than one engineer's personal address.
An anti-corruption layer, and why it pays
Do not let a provider's payload shape spread through your codebase. Translate at the boundary into your own domain model, and keep the vendor's field names, enums and quirks confined to one adapter. When the provider changes, you rewrite an adapter instead of touching forty files. The same boundary is what makes swapping a vendor conceivable, and it is why teams with an adapter layer can change providers in weeks while teams without one treat it as a rewrite.
Detect their breaking changes before your customers do
A small scheduled suite hitting the provider's sandbox with real requests and asserting on response shape is cheap insurance, and it catches undocumented changes that no announcement mentioned. Where the provider publishes an OpenAPI document, diffing it on a schedule and alerting on changes to the endpoints you use costs almost nothing. This is one of the few places where a handful of tests genuinely prevents an outage, and it sits naturally alongside API testing services in India if the wider interface layer needs coverage too.
Plan the migration as work, not as an emergency
When a version is deprecated, the pattern that works is running both against a feature flag, comparing outputs on live traffic in shadow mode, then cutting over per segment with a fast rollback. The pattern that does not work is a big-bang switch on the deadline, which is what happens when the deprecation notice sat unread for five months.
Three Situations That Turn Up Again and Again
Patterns rather than named clients. If one of these reads like your system, the diagnosis usually follows the same path.
The duplicate that appears once a month
An operations team notices duplicate records at a low but persistent rate. The integration code has retries and looks correct on inspection. What is actually happening is a read timeout on a write that succeeded: the provider processed the request, the response never arrived, and the retry created a second record. It is invisible in testing because timeouts do not happen when latency is low. The fix is an idempotency key generated per logical operation with the result stored on first success, plus a decision that a timeout on a non-idempotent write is not automatically retryable. Then a one-off reconciliation to find and merge the duplicates already created, which is usually the longer half of the job.
The nightly sync that has been failing silently since March
A scheduled job pulls records from a partner. It succeeded for a year. At some point the partner's cursor started expiring after seven days, or a field became nullable, or the job began throwing after the first page and the wrapper swallowed the exception. Nothing alerted, because the monitoring checked that the job ran rather than that it completed and moved data. Meanwhile the local copy quietly drifted. The fix is layered: alert on time-since-last-successful-completion rather than on invocation, record counts per run with a variance check, a full reconciliation to find what is missing, and a resumable backfill to repair it. The engineering is straightforward. Working out what the correct state is after nine months of drift is the part that takes real time.
The vendor whose slowness became your outage
A checkout calls an enrichment or verification API inline. The vendor has a bad afternoon and starts answering in twenty seconds instead of two hundred milliseconds. The call has no timeout, so request threads accumulate, the shared connection pool fills, and requests with nothing to do with that vendor start failing. From the outside your entire site is down because a non-essential third party is slow. The fix is the containment stack: a timeout derived from observed latency, a dedicated pool for that dependency, a breaker with a slow-call threshold rather than only an error threshold, and a product decision about the fallback. Often the honest answer is that the enrichment step was never essential to completing the purchase and should have been asynchronous from the start.
How Does an Integration Team in India Work Across Your Timezone?
You would be giving engineers access to your repository, your queues and credentials for systems you do not own, from eight to thirteen hours away. That deserves specifics rather than reassurance.
The overlap window, stated plainly
We work 09:30 to 18:30 IST as standard. Set that against a British working day and you get around four hours of genuine live overlap in the UK afternoon, enough for a daily call and same-day iteration. Sydney and Auckland overlap through their afternoon. Put the same day next to a US Eastern nine to five and there is almost nothing left; against Pacific there is nothing. When a vendor promises round-the-clock coverage and does not mention shifts, the missing word is the expensive one. Shifts do work. They also mean fewer of your engineers and ours are awake together, more written handover between them, and steady effort to keep anyone on that rotation past the first few months. We fix the window with you before the first sprint, not in week three.
Why integration work tolerates a low overlap better than most
The artifacts here are pull requests, adapter code, queue configuration, dashboards and replayed events, all of which read identically at 09:00 in Manchester and 09:00 in Bengaluru. A failing contract check or a dead letter queue with a captured payload explains itself without a call. The exception is production incident response on a live integration, where a nine hour round trip is genuinely expensive, and that is precisely the case worth discussing before it happens rather than after.
How the day runs
Writing carries the day, talking supplements it. At the close of the Indian day you get a note: what shipped, what is stuck, what we need a decision on. It goes into the channel your team already uses, not into one manager's inbox, so nobody becomes a relay. There is one live call inside the overlap window and it is short unless something genuinely needs thinking through together. Tickets live in your tracker, never on a private board of ours. And every question we send you arrives with a proposed answer attached, so being blocked costs an hour of our time rather than a full day of it.
Code review and definition of done
Nothing lands without one of your engineers approving it. We open pull requests against your repository and your reviewers decide; merge rights on your main branch are yours to hand over only if you want us to have them. Finished means reviewed, deployed into your environment, carrying the metrics and alerts that change needs, and written into the runbook. An integration only the vendor understands is not finished, whatever the ticket says.
Access, ownership and data
Least privilege, granted by you and revoked by you. Provider sandbox credentials wherever a sandbox exists. Synthetic or scrubbed data in lower environments, and if a reconciliation genuinely needs production-like records, we agree in writing how they are handled before a single row is copied. Ownership of the code, IP assignment, confidentiality and data processing sit in the MSA and NDA, negotiated before anyone starts. Those belong in a contract you can hold us to rather than in marketing copy.
The talent pool for this particular skill
India has enormous depth in enterprise integration work, which is both the opportunity and the catch. A great deal of that experience is in vendor middleware suites and in point-to-point connectors built to a specification. Engineers who think naturally in delivery semantics, backpressure, idempotency and failure containment are a much smaller subset. We screen for that subset with a working exercise against a real flaky API rather than a tool quiz, because knowing a product's console is not the same as knowing what to do when a webhook arrives twice out of order at 40 requests per second.
What Actually Derails This Work
Every integration project has a way of going sideways. Here are the ways ours do, and what happens when they start to.
The provider's sandbox does not behave like production
Sandboxes commonly have different rate limits, cleaner data, no partner-specific configuration and sometimes an older API version. Code that passes there fails on the first real payload. We probe for the differences early, keep a list of behaviours we could not verify outside production, and plan a small controlled cutover with real traffic rather than assuming parity. Where a sandbox is missing entirely, we say so and we plan for a shadow-mode rollout instead of pretending the risk is not there.
Nobody can say what the correct data is
On a rebuild of an integration that has been drifting, the technical work is often easy and the reconciliation is not. Two systems disagree, both have a claim to being authoritative, and the business rule for resolving conflicts has never been written down. We surface that in the first fortnight as a decision for you rather than making it quietly in code, because a conflict rule chosen by a developer is a business rule chosen by accident.
The provider's support loop is the bottleneck
Sometimes the blocker is an undocumented behaviour that only the vendor can explain, and their response time is days. We work around it: capture and log everything, build against observed behaviour with the assumption recorded in code, and keep a running list of open vendor questions visible to you so the delay is transparent instead of appearing as unexplained slowness on our side.
Scope quietly turns into a data migration
Integration work has a habit of revealing that the two systems disagree about tens of thousands of records. That is a separate piece of work with its own risk, and we sequence it separately rather than letting it consume the connector build. Flagging it early gives you the choice of what to fix and what to accept.
Volume nobody mentioned
A design that is correct at fifty events per minute may need a different shape at five thousand: batching, partitioned consumers, a different queue, a different provider plan. We ask for the actual peak numbers and the growth expectation at the start, and where nobody knows, we build with the smallest thing that works and instrument it so the ceiling is visible before it is hit rather than after.
Continuity of the people doing the work
People change jobs everywhere, and integration knowledge is unusually easy to concentrate in one head. The mitigations are structural: more than one engineer familiar with each integration, everything in your repository under your review process, runbooks written as the work happens rather than at the end, and no undocumented tribal knowledge about which failures are safe to ignore. What notice applies, how handover runs and what happens if someone has to be replaced are all agreed with you and written down in the contract. Putting a number on any of that here, before we have spoken, would be inventing it.
Engagement Models
Three shapes, picked by what the work actually needs.
Integration review
A time-boxed assessment of what you already run. We trace the flows, read the failure handling, check webhook receipt and auth, look at what your monitoring would and would not catch, and reconcile a sample to find out whether the data actually agrees. The findings, the ranking and our reasoning are yours to keep, whatever you decide to do next.
Project engagement
One outcome, a start date and a finish. Building a named connector. Hardening one that keeps waking people up. Pulling a tangle of point-to-point calls into a queue-based flow. Getting off a provider version that is about to disappear. Scope is agreed before we begin, the work lands in your repository and pipeline, and the runbook ships as a deliverable rather than a promise.
Dedicated engineers
Integration engineers sitting inside your team, in your standups, pulling from your backlog. This fits estates where new connectors keep arriving and the old ones need somebody who genuinely knows them. How the team is composed, what notice applies and what it costs are things we agree with you and put in the contract.
Where This Sits Alongside Our Other Work
Everything above is the craft that applies whatever sits on the other end of the wire. Some categories bring constraints that deserve their own treatment, and we keep them separate rather than pretending one connector pattern covers them all. Enterprise suites have their own object models and their own release cadences, which is ERP integration services in India. Sales and marketing systems bring field mapping, ownership rules and sync loops, covered under CRM integration. Money brings a compliance perimeter that changes the design, which sits with payment gateway integration. Wiring in a particular vendor product is third-party integration, and moving records in bulk into a warehouse is a pipeline problem rather than an API one, so it belongs with data integration.
If the API being integrated needs to be designed or built rather than consumed, that sits with our API development services, and the delivery semantics are far cheaper to decide while the endpoints are still being written. Where several systems must be joined across an estate rather than two systems joined once, our enterprise integration services cover the architecture questions that come with that scale. Everything on this page assumes it runs automatically, which is why the work pairs closely with CI/CD pipeline services in India. If the shape you want is people sitting inside your own team instead of a scoped brief, the same skills come that way too: dedicated Node.js developers in India or Go developers in India, working your backlog under your own planning.
Frequently Asked Questions About API Integration in India
Is exactly-once delivery actually impossible, or just hard?
Impossible across a network boundary you do not control. A sender that gets no response cannot tell whether the receiver processed the message and the acknowledgement was lost, or whether nothing happened at all. Its only choices are to retry, which risks a duplicate, or to give up, which risks a loss. What you can build is at-least-once delivery plus deduplication at the receiver, which produces exactly-once effects. That is a different and achievable goal.
How do we stop one slow third-party API from taking down our whole checkout?
Three things, in order. Give every outbound call a timeout shorter than the request budget it sits inside, because an unbounded call is what actually exhausts your threads. Isolate each dependency in its own connection pool so a stalled vendor cannot consume the capacity your database calls need. Then put a circuit breaker in front of it so repeated failures stop being attempted and the feature degrades in a way you designed rather than in a way you discover.
Should we build this integration with REST, GraphQL, gRPC or webhooks?
Usually you do not choose. The system you are integrating with has already decided, and your job is to consume what exists well. Where you do own both ends, REST over JSON is the sane default for anything crossing an organisational boundary, gRPC pays off for chatty internal service calls where latency and payload size matter, GraphQL earns its complexity when many different clients need different shapes of the same data, and webhooks are the answer to being told about something rather than asking repeatedly.
Our vendor's webhooks arrive out of order. What do we do about it?
Stop trusting arrival order, because almost no provider guarantees it and retries make it worse. Two patterns work. If the payload carries a version, sequence number or updated timestamp, store it and discard any event whose version is older than the state you already hold. If it does not, treat the webhook as a notification only, then call the provider's API to fetch the current state of that object. The second pattern is slower and almost always more correct.
How do you handle a partner API whose rate limits are undocumented?
By measuring rather than guessing. We log every response header the provider returns, watch where 429s start appearing against concurrency and request rate, and build a shared limiter that enforces the observed ceiling across every process rather than per instance. Respecting Retry-After when it is sent, adding jitter so your fleet does not resume in lockstep, and separating bulk backfill traffic from customer-facing traffic on different quota budgets covers most of it.
Do we actually need a queue, or is a direct synchronous call fine?
A direct call is fine when the caller genuinely needs the answer before it can respond, the work finishes in well under your latency budget, and losing the operation on failure is acceptable. Change any one of those and a queue stops being optional. The specific signals are: the work must survive the target being down, it takes longer than a user will wait, the target rate limits you, or you need retries that outlive the request that started them.
Who owns the integration code and the API credentials you use?
The code is written in your repository, under your review process, in a language your team already maintains. Credentials stay yours: we work against your secret manager with scoped, revocable credentials that you issue and you revoke, and against sandbox keys wherever the provider offers them. Ownership, confidentiality and data handling are negotiated into the MSA and NDA before anyone starts, because those are commitments you should be able to hold us to rather than sentences on a marketing page.
What is the real timezone overlap with an integration team in India?
We run 09:30 to 18:30 IST. That buys about four live hours with a UK afternoon and most of an afternoon with Sydney or Auckland. For US Eastern, measured against an ordinary nine to five, it is nearly nothing, and for Pacific it is nothing at all. Shifting part of the team does create real American overlap, at a cost in coordination and in keeping people on that rotation. We settle the window with you upfront.