Site Reliability Engineering Services in India

Why Does Uptime Keep Slipping While Everyone Is Busy?

The symptoms arrive in a predictable order. First the pager gets noisy and nobody wants to be on it. Then two of your better engineers start spending their week on tickets that all look like the last ticket. Then a customer emails your CEO about an outage before your monitoring notices it. By the time somebody says the word reliability out loud in a board meeting, the team has usually been absorbing the damage quietly for six months.

What makes this hard to fix from inside is that no individual decision was wrong. The alert that fires forty times a week was added after a real outage. The manual database failover runbook exists because automating it looked risky. The nightly job that somebody restarts by hand every Tuesday has been failing the same way for a year, and restarting it takes four minutes, so it never reaches the top of a backlog. None of it is negligence. It is accumulated operational debt, and it compounds in exactly the way financial debt does.

The cost lands in three places, and only one of them shows up in a dashboard. There is the direct revenue you lose during an outage. There is the engineering capacity you are spending on repetitive operations instead of on product, which is usually the larger number and is invisible because it never gets logged as downtime. And there is attrition, because good infrastructure engineers leave jobs where the pager owns their evenings. Replacing one of them costs more than a quarter of reliability work.

Site reliability engineering services in India exist for a specific version of this problem: you know what needs doing, the work is real engineering rather than an afternoon of tidying, and hiring three experienced SREs in San Francisco or London is either too slow or too expensive to be the answer this year. India has a deep pool of Linux, cloud and Kubernetes operations engineers, and the working day sits directly on top of the hours your own team is asleep. Used properly, that is a structural advantage rather than a compromise.

SLIs, SLOs and Error Budgets: Picking Numbers That Mean Something

This is the part that gets performed rather than practised. A lot of organisations have SLO dashboards and no SLO practice, because the numbers were chosen in a workshop by picking a comfortable count of nines rather than by measuring anything.

Start with the indicator, not the target

A service level indicator is a ratio: good events over valid events. That framing does more work than it looks like it does, because it forces two decisions people usually skip. What counts as good, and what counts as valid. A health check endpoint returning 200 is not a good event if the checkout it fronts is failing. A request from a scraper hammering your search endpoint is arguably not a valid event, and if you include it your availability number becomes a measure of bot behaviour.

The indicators worth having describe something a user would notice. Request availability measured as successful responses over valid responses, at the edge rather than at the service, because the load balancer and the CDN are part of the path. Latency expressed as the proportion of requests served faster than a threshold. Freshness for anything pipeline shaped: the share of dashboard loads where the underlying data is less than fifteen minutes old. Correctness for batch work, sampled against a known answer. Coverage for a job that is supposed to process everything.

What does not belong in an SLI: CPU utilisation, memory, pod restart counts, queue depth. Those are useful signals and terrible objectives. A service can sit at ninety percent CPU and serve every request inside its latency target, and it can sit at ten percent CPU while returning errors to everyone.

Thresholds beat percentiles for budgeting

Teams instinctively write latency SLOs as p95 below 400ms. It reads well and it is awkward to budget against, because a percentile is not an event count and you cannot straightforwardly say how much of it you have spent. The threshold form, the proportion of requests served in under 400ms, gives you a good-events ratio that plugs directly into burn rate maths. Keep the percentile charts for debugging. Use the threshold for the objective.

One more thing about percentiles that catches people out. If you serve a million requests a day, your p99 is ten thousand requests. Those are real users having a bad time, and they are frequently your heaviest accounts, because heavy accounts have more data and slower queries. The tail is not an edge case, it is a customer segment.

What the nines actually cost you in minutes

Do the arithmetic out loud with your product lead in the room, because the difference between the numbers is not intuitive. Against a thirty day month, 99.9 percent leaves you 43 minutes and 12 seconds of budget. 99.95 percent leaves 21 minutes and 36 seconds. 99.99 percent leaves 4 minutes and 19 seconds, which is less time than most teams take to acknowledge a page, let alone diagnose one. Over a full year, 99.9 percent is 8 hours 45 minutes.

Look at 99.99 percent again. Four minutes a month means that if a human being has to be involved in recovery, you have already failed. It implies automated failover, tested regularly, on every component in the path, and it implies the same standard from every dependency you call. If a payment provider you cannot control gives you less than that, your ceiling is set by them regardless of how good your engineering is. This is usually the moment somebody quietly changes the target, and that is the correct outcome.

The policy is the product

An error budget only changes behaviour if something happens when it runs out. The policy needs to be written before the first SLO goes live and agreed by the person who owns the roadmap, not just by engineering. A workable version reads something like: while budget remains, ship normally; once it is spent, risky changes pause and the next sprint prioritises the reliability work that would have prevented the burn; if the budget is exhausted two quarters running, the target itself goes back on the table because either the architecture cannot deliver it or the target was wrong.

Notice what the policy does politically. It replaces the argument between product and engineering about whether to slow down with a number that both sides agreed to in advance. That is most of the value. The dashboard is a side effect.

Burn rate alerting, so the page arrives at the right time

Alerting on the SLO directly gives you a page after the damage is done. Alerting on burn rate gives you a page while it is happening, and lets you distinguish a fast catastrophic burn from a slow leak. The multi window, multi burn rate approach published in the Google SRE Workbook is the reference implementation: a burn rate of 14.4 times normal sustained over an hour consumes two percent of a monthly budget and pages immediately; six times over six hours consumes five percent and pages; three times over a day and one times over three days raise tickets rather than waking anyone.

The short window paired with each long window is what stops the alert firing on a spike that already recovered. Get this configuration right once and a large share of your out of hours pages disappear, because the slow burns that used to wake somebody now arrive as a ticket in the morning.

SLO, SLA and the gap between them

Your SLA is the contractual promise with a financial consequence. Your SLO is the internal target you manage to, and it should be stricter, because you want to be having the internal conversation well before you owe anyone a credit. Setting them equal is a common and expensive mistake: it means the first sign of a problem is a breach notice. If you carry an SLA you have never measured, measuring it is the first job, and occasionally the finding is uncomfortable enough to need a commercial conversation rather than an engineering one.

Observability: Metrics, Logs, Traces and the Cardinality Bill

Observability engagements go wrong in one of two directions. Either the team cannot answer basic questions during an incident, or the team can answer everything and the monthly bill has become a governance issue. Both are solvable, and both are usually caused by treating the three signal types as interchangeable.

Metrics, and the cardinality trap

Prometheus and its ecosystem, with Grafana on top, is still the default we build on for self hosted estates, and the managed options from Datadog, New Relic and Google Cloud Operations are reasonable where you would rather buy than run. Whichever you pick, the failure mode is the same: cardinality.

A time series is created for every unique combination of metric name and label values. Add a label for user identifier to a metric that already has five labels and you have multiplied your series count by your user count. Prometheus keeps every active series in memory, so a few million active series is where a single server starts needing sharding or a remote write backend such as Thanos, Mimir or VictoriaMetrics. On a usage priced vendor, that same mistake arrives as an invoice rather than an out of memory kill, which is arguably worse because nothing breaks until finance notices.

The rules we apply: never label with anything unbounded, which means no user identifiers, no request identifiers, no raw URL paths, no email addresses. Keep high cardinality dimensions in traces and logs where they belong. Decide histogram bucket boundaries deliberately at the start, because changing them later breaks the history you have been comparing against. And put a cardinality dashboard next to the cost dashboard so the growth is visible before it is expensive.

Logs, and why the bill is always a surprise

Logs are the signal people over produce, because adding one costs a single line of code and the cost lands somewhere else. Structured JSON logging with consistent field names is the baseline, since grepping unstructured strings at volume is how incidents get longer. After that it is economics: sample the high volume debug paths, keep errors unsampled, tier your retention so that seven days is hot and searchable while ninety days is cheap object storage, and review the top ten log producing services every quarter.

Loki is cheaper to run because it indexes labels rather than content, with the trade-off that arbitrary field searches over a wide time range are slow. Elasticsearch or OpenSearch gives you fast search over everything and charges you in storage and operational effort. Datadog and Splunk are excellent and priced by ingest, which makes a careless debug statement in a hot path expensive. There is no universally right answer, only a right answer for your query patterns and your tolerance for running stateful infrastructure.

Traces, and sampling you will not regret

OpenTelemetry is the instrumentation standard we build to now, for the practical reason that it decouples your code from your backend, so switching vendors becomes a collector configuration change rather than a re-instrumentation project. Jaeger, Tempo, Honeycomb and the commercial suites all consume it.

Sampling is where the design decision lives. Head based sampling decides at the start of a request, which is cheap and throws away the traces you most want, because a one percent sample of a rare failure is almost certainly zero. Tail based sampling waits for the trace to complete and keeps anything with an error or unusual latency, which is what you want during an incident and requires a collector holding spans in memory long enough to decide. On a mid sized system the tail based collector is real infrastructure with real memory requirements, and it earns its keep. Wire exemplars so a spike on a latency histogram links straight to an example trace, and you remove a whole category of incident guesswork.

The frameworks that stop dashboards multiplying

Two conventions keep dashboards useful. The four golden signals, latency, traffic, errors and saturation, for every user facing service. The USE method, utilisation, saturation and errors, for every resource. Tom Wilkie's RED variant, rate, errors and duration, is the same idea aimed at request driven services and maps neatly onto SLI definitions.

The practical rule is one dashboard per service that answers the question is it healthy, and separate diagnostic dashboards for when the answer is no. Teams that skip this end up with sixty dashboards, of which four get opened, and none of the four are the right one at 02:00.

Incident Command: Severity, Roles and the First Ten Minutes

Most organisations have an incident process on paper and a different one in practice, where the most senior person present starts debugging, everyone else watches, and nobody tells the customers anything for ninety minutes. The fix is structural, borrowed from emergency services, and it works because it separates coordinating from fixing.

Roles, and the one rule that matters

The incident commander owns the incident and does not debug. That single rule is the whole trick, and it is the one most teams break. The moment your commander opens a terminal, coordination stops: nobody is tracking who is doing what, the timeline goes unwritten, and the status page goes stale. On a small incident the commander may be your most junior person on the rota, and that is fine, because the job is running the room rather than knowing the system best.

The operations lead makes the changes and is the only person touching production, which prevents two engineers applying conflicting fixes. The communications lead handles the status page, the customer facing update and the executive who wants to know what is happening. A scribe keeps the timeline. On a small incident one person may hold several of these, but naming them out loud in the channel is what stops them silently going unowned.

Severity levels anchored to customers

Severity must be defined by impact rather than by whoever is shouting. A workable ladder: SEV1 for a total outage of a critical path or an active data loss or security event, paging immediately and pulling in leadership. SEV2 for significant degradation or a major feature down for a subset of customers, paging during any hours. SEV3 for a degraded non critical path or a single account affected, handled in business hours. Write the definitions down with examples from your own product, because abstract definitions get argued about at exactly the wrong moment.

Add a declaration threshold, which is the rule most teams lack. Ours: if two engineers have been investigating for fifteen minutes without a confident cause, declare an incident. Under declaration is far more expensive than over declaration. The worst incidents in most companies are the ones that stayed as a Slack thread for an hour because nobody wanted to seem dramatic.

The first ten minutes, in order

Declare and open a channel and a document. Name the commander in writing. Assess scope, which is who is affected and how badly, before touching anything. Mitigate before diagnosing, because rolling back the release from twenty minutes ago beats understanding it while customers are down. Post a first customer facing update inside the first fifteen minutes even if it says only that you are investigating, then keep a fixed cadence of every thirty minutes for a SEV1 regardless of whether there is news. Silence is read as incompetence, and updates that say nothing new are still updates.

Rollback beats root cause during an incident

The instinct to understand the failure before undoing it is the single most expensive habit in incident response. If a deploy correlates with the start, revert it. If a feature flag correlates, flip it. If a region is degraded, fail over. You can investigate at leisure once the bleeding has stopped, and you will investigate better because nobody is watching the clock. This requires that rollback is safe and rehearsed, which is why we test it during game days rather than discovering its edge cases mid outage.

Toil: Measure It Before You Try to Kill It

Toil has a precise definition, and using it precisely helps, rather than treating the word as a synonym for work you dislike. Toil is manual, repetitive, automatable, tactical rather than strategic, produces no lasting value, and scales linearly with the size of the service. Reviewing a design document is not toil even though it is not coding. Restarting the same stuck consumer every Tuesday is textbook toil.

Nearly every team we work with underestimates how much of it they carry, because it arrives in four minute increments that never get logged. So the first step is counting, not automating.

Counting it honestly

Two weeks of interrupt logging. Every page, every ticket, every ad hoc request in a chat channel gets recorded with rough minutes and a tag for toil or not toil. It is tedious and it is the only thing that produces an argument nobody can dismiss. The output is a ranked list by total hours, and the top item is frequently something absurd that everyone had normalised: an access request process that costs the team six hours a week, or a certificate renewal that three people each half remember.

Rank by total time, not by irritation. The thing everyone complains about is often twenty minutes a month. The thing nobody mentions is often four hours a week.

The half of a week that has to stay free

The convention we hold to is that operational work stays at or below half of an engineer's time. When it goes above that consistently, the team stops being able to reduce future work and the load becomes permanent. Passing the threshold is a signal to act on: hire, cut scope, or stop taking on new services until the automation catches up. It is not a target to admire on a slide.

What we automate first, and what we deliberately do not

First to go, in rough order of return: certificate renewal, which is also a common cause of self inflicted outages and belongs entirely to cert-manager or an equivalent. Access provisioning, moved into groups in your identity provider with a request flow. Manual scaling before known demand, replaced by scheduled scaling with a floor. Deploy babysitting, replaced by automated verification and rollback on SLI regression. Restarting known bad services, which should be fixed rather than automated, but a self healing controller buys time while the fix is written. Routine reporting, which is a scheduled query rather than a human with a spreadsheet.

Some things stay manual on purpose. A database failover that runs once a year is not worth full automation, but the runbook must be rehearsed, because an unrehearsed manual procedure at 03:00 is how a thirty minute incident becomes a four hour one. Anything genuinely destructive keeps a human in the approval path. Automation that runs rarely and is never tested is more dangerous than the manual process it replaced, because everyone assumes it works.

Chaos Engineering and Game Days

Chaos engineering has a reputation problem, mostly because it gets introduced as randomly killing things in production, which is a fine way to lose the argument permanently. The useful version is a controlled experiment with a hypothesis, a blast radius and an abort condition, and it usually starts nowhere near production.

Start with a game day, not a tool

A game day is a scheduled exercise where the team responds to a simulated or injected failure while somebody observes and takes notes. No automation required for the first several. The findings from a first game day are remarkably consistent across companies: the runbook references a dashboard that no longer exists, the alert routes to a rota that includes somebody who left in March, the escalation path is known by two people and neither is on call, and the rollback procedure has never been executed by anyone currently employed.

None of those are technical failures and none of them would have been found by a resilience tool. They are found by making people do the thing.

Experiments in a sensible order

Kill a single pod and confirm that traffic drains and the replacement joins without errors. Add 200ms of latency to a dependency and see whether your timeouts and circuit breakers behave or whether the delay propagates upward into a cascade. Fail a read replica. Return errors from a third party integration and check that the degraded path degrades rather than breaks. Expire a certificate in staging, which finds the renewal gap safely. Revoke an API credential. Fill a disk, which surfaces the logging path that fails closed. Simulate a zone loss if your architecture claims to survive one, because that claim is only as good as its last test.

Tooling when you get there: Chaos Mesh and LitmusChaos for Kubernetes, AWS Fault Injection Service where you are on AWS and want managed experiments, Gremlin where you want a commercial platform with guardrails included. Toxiproxy remains the simplest way to inject network conditions in staging.

Two rules we hold to

Do not run chaos experiments in production until you can run them in staging without surprises, and until you have an SLO with budget to spend, because an experiment consumes error budget and that is exactly how it should be accounted for. Second, always announce the first several. Unannounced experiments are for mature practices where the response process is already trusted. Running one before that point teaches people to distrust the exercise rather than to improve the system.

Four Situations This Work Is Usually Bought For

These are patterns rather than particular customers, and they cover most of the conversations that start with the phrase our uptime is fine but.

How We Run Offshore Delivery, and What Goes Wrong

You are handing production access to people you have never met, in a country whose working day may barely overlap with yours. It is reasonable to be sceptical. Here is how each of the standard worries is handled, including the ones without a comfortable answer.

Will the quality hold up?

The honest observation about the Indian market for this specific skill: the pool of strong Linux, AWS, Kubernetes and Terraform engineers is deep and genuinely competitive with anywhere. The pool of people who have personally owned an SLO, argued an error budget policy with a product manager and run an incident as commander is much thinner, because relatively few employers here have run the discipline properly. Anyone claiming otherwise has not hired for the role recently.

So screening has to separate operational fluency from reliability practice. Ours has four parts. A debugging exercise on a deliberately broken system with real metrics and logs available, which reveals method rather than memorised answers. A coding exercise in Go or Python, because automation is the job and an SRE who cannot write maintainable code becomes the toil. A simulated incident where they hold the commander role and are scored on coordination rather than on finding the bug. And a written postmortem of that incident, which is the strongest signal we have found for the thing clients actually worry about, since on-call handover across timezones is fundamentally a writing job. Poor written English shows up immediately in that exercise, and it is the reason we test writing rather than interviewing for accent.

Who do we actually talk to?

The engineers, directly, in your Slack and your standup. We do not put an account manager between you and the people doing the work, because relaying a technical question through a third party is how requirements get lost. There is a named delivery lead for the commercial relationship and for escalation, but they are not a communication checkpoint.

How do we manage people we never see?

The same way you manage anyone competent, plus written discipline. A daily standup inside the overlap window, written rather than spoken so the record survives the timezone. Your sprint cadence and your tracker, not ours. All work in your repositories with pull requests reviewed by your engineers, because code review is where quality is actually controlled. The monthly reliability report described above. And a rule we hold ourselves to: anything decided in a call gets written down within the hour, or it did not happen.

Who owns the code and the data?

You do, from the first commit. Intellectual property assignment, confidentiality and data processing terms are settled in the master agreement before work starts, and we would rather your counsel drafted them than accept a vendor template. Everything we build lives in your accounts and your repositories throughout, so ownership is a matter of fact rather than a matter of trust. Where personal data is in scope, the processing terms and any transfer mechanism your jurisdiction requires are a question for your data protection counsel, and we work to whatever they specify.

What about production access and security?

Named individual accounts in your identity provider, multi factor authentication mandatory, permissions scoped to the role and reviewed on a schedule. No shared logins, ever. Standing administrative rights removed in favour of a break glass path that raises privilege for a limited window and produces an audit record. Access to your environments through your bastion or your zero trust proxy, on managed devices with disk encryption and screen lock enforced. Offboarding runs the same day. Where your sector imposes specific control frameworks, we implement the controls you specify and your compliance advisers confirm what is sufficient for your obligations.

What are the costs nobody mentions?

Ramp up is real. An engineer joining a system they did not build is useful on tickets within a couple of weeks and trustworthy as primary on-call somewhere between week six and week ten, depending on how much is documented. Budget for that rather than expecting week one productivity. Your own team spends time on onboarding, review and answering questions, and that time is genuine cost. Written first working is slower per interaction and faster overall, but the slower part is felt immediately and the faster part accrues later. And any shifted or night working carries its own cost, which is agreed openly at the start rather than discovered in a renewal conversation.

What if it does not work out?

Knowledge transfer is designed in from the beginning rather than bolted on at the end, which is mostly a matter of where things live. Dashboards in your Grafana. Alert rules and infrastructure code in your repositories. Runbooks in your wiki. Incident history in your tracker. If the engagement ends, nothing of consequence leaves with us, because none of it was ever hosted somewhere we control. Handover and notice terms are agreed in the master agreement before the first sprint, while everyone is still cheerful and reasonable.

Where This Sits Alongside Our Other Work

Reliability work rarely arrives alone. If the underlying problem is that the platform itself needs building or rebuilding rather than stabilising, that is cloud engineering work and it should happen first, because putting an SLO on an architecture you are about to replace wastes both efforts. Where you need people rather than a programme, our DevOps engineers in India cover the pipeline and platform side, and AWS specialists handle account structure, networking and the managed services underneath.

If the systems keeping you awake are machine learning services, the reliability questions change shape: models degrade silently while every infrastructure metric stays green, which is why MLOps handles drift, retraining and model quality gates as separate machinery. And for the conceptual grounding behind everything on this page, our long form SRE guide covers the discipline, the metrics and the team structures in more depth than a service page should.