Hire Data Engineers in India
Hire data engineers in India to build the pipelines your reporting sits on, model the warehouse underneath it, and be the person who can say why a number changed. The title is used loosely enough that half the shortlists we see contain four different jobs. This page is mostly about telling them apart before you commit a quarter to the wrong one.
Data Engineer Is Five Job Titles Wearing One Label
Job ads in this field are written by people repeating a title they inherited. The result is a posting that asks for Spark, Tableau, causal inference and Kubernetes in the same bullet list, and a shortlist where no two candidates are comparable. Five distinct roles have been collapsed into one word. Separating them is the single highest-value thing you can do before you interview anybody.
The data engineer
Owns movement and reliability. Their unit of work is a job, not a request, and the question they are always answering is whether the thing can be run again safely. They think about extraction from systems that were never designed to be extracted from, delivery guarantees, schema drift, retries, partitioning, ordering, and what happens when a source is down for six hours. Most of their code is Python and SQL, and most of their difficulty is neither.
A useful test for whether this is who you need: is anybody currently unable to answer a question because the data is not in one place, or is in one place but wrong? That is an engineering problem.
The analytics engineer
Starts where the raw tables land and turns them into something a finance director can query without asking permission. This role emerged in the last several years, largely around dbt, and it is genuinely distinct: production SQL, modelling, tests, documentation and metric definitions, without the ingestion and infrastructure half. The best ones came from analysis and learned software practice, which shows up as a fierce interest in whether a number is right and a mild interest in how it got there.
If your complaint is that everyone calculates revenue differently, you may want this person rather than a data engineer. The data is arriving fine. The disagreement is about meaning.
The data scientist
Frames questions, finds signal, picks a metric that reflects a decision somebody will actually make, and explains the result to people who will act on it. Statistics and communication carry more weight than software craft. Hiring one before your data is trustworthy is the most common expensive mistake in this whole area, because the first three months become plumbing done reluctantly by someone who was hired for something else and priced accordingly.
The ML engineer
Takes something that worked offline and makes it a system: packaged, served, monitored, retrainable, rollback-able. Overlaps heavily with the data engineer on the feature side and with infrastructure on the serving side. If your model already works and cannot get into production, you are looking at a deployment and operations problem rather than a modelling one, which is the territory covered by our MLOps services in India rather than a data engineering hire.
The BI developer
Lives in Power BI, Tableau, Looker or Qlik. Builds the semantic layer, the dashboards, the row-level security and the refresh schedules, and knows why a DAX measure gives a different total than the SQL behind it. Frequently the most commercially fluent person in the data team and frequently the worst treated, because their output looks easy from the outside. A BI developer will not build you a pipeline that survives a schema change, and a data engineer will build you a dashboard nobody wants to look at.
Which one does your problem need?
Work backwards from the sentence somebody in your company said out loud last week.
"We cannot get the data out of the billing system" is a data engineer. "The dashboard and the invoice disagree" is usually an analytics engineer, occasionally a data engineer if the disagreement traces back to duplicate rows. "Nobody knows which customers are about to leave" is a data scientist, but only if the data to answer it already exists. "The churn model has been sitting in a notebook since March" is an ML engineer. "The board deck takes three days to assemble" is a BI developer with a decent semantic layer.
Where the answer is genuinely several of these at once, you are describing a platform build rather than a hire, and it is usually cheaper as a scoped engagement than as three simultaneous searches. Our data integration services in India page sets out what that looks like when it is bought as an outcome instead of as seats.
What buying the wrong one costs
Roughly a quarter, and it is not obvious for the first six weeks. The early work always looks fine, because the early work is exploratory and everybody is capable of that. The divergence shows up the first time something breaks at an awkward hour. The analyst-turned-engineer writes a pipeline that cannot be rerun without manual cleanup. The infrastructure engineer builds a beautiful platform on top of a source table with no reliable update timestamp. The data scientist quietly stops doing science and becomes an expensive SQL writer.
None of these people is bad. Each has been dropped into an adjacent job, and adjacency in this field is deceptive because the vocabulary is shared. Write down which of the five you need and what the first two tickets are. A shortlist built against that is comparable. A shortlist built against a title is not.
What Does a Data Engineer Do on an Ordinary Tuesday?
Not what the job ad says. Here is the honest breakdown of where the hours go, because it tells you what to screen for and what kind of person will still be happy in month eight.
Moving data that refuses to hold still
The stereotype is that this work is about building connectors. It is not. Connectors are the easy hour. The difficulty is that every source system was built for something other than being read by you, and it expresses that in a specific way. The CRM has an API with a rate limit and pagination that shifts if a record is updated mid-scan. The billing database has soft deletes in one table and hard deletes in another. The events from the mobile app arrive out of order because three of them were queued on a phone in a tunnel. The finance system exports a file whose column order changed when somebody upgraded it.
A good data engineer spends their morning making a system that tolerates all of that without a human in the loop, and their afternoon proving it tolerated it.
Turning landed data into tables people can actually use
Raw tables are not a product. Somebody has to decide what a customer is when three systems each hold a version of one, what happens to an order that was refunded in a later period, and whether a deleted row should vanish from history or be marked as ended. These are modelling decisions with business consequences, and in most companies the data engineer makes them by default because nobody else realises a decision is being made.
This is the part of the job that separates an engineer who will be valuable in year two from one who will have automated the current requirements and then stalled. Ask a candidate about a modelling decision they had to make where the business had not given them a rule. The good answers involve going and asking somebody, then writing the rule down where it can be found again.
Being the person who gets called when a number is wrong
This is the defining experience of the role and the thing that never appears in a job description. On a Thursday afternoon somebody in finance says last month's figure has changed since they looked at it on Monday. Nothing is down. No alert fired. The pipelines are green. And the number is different.
Working that back takes a particular temperament: the ability to hold a chain of six transformations in your head, to suspect your own code first, and to check the boring explanation before the interesting one. Nine times in ten the boring explanation wins. A late-arriving batch restated a closed period. A source system backdated a record. Somebody changed a filter in the BI tool and did not mention it. A dimension was overwritten in place, so the historical fact now joins to the current value rather than the one that was true at the time.
How a candidate describes one of these investigations is the most informative five minutes of any interview for this role. It exposes their debugging method, their honesty about their own bugs, and whether they went back and made the class of error impossible afterwards or just fixed the row.
The part nobody puts in the job ad
Talking to people. Finding out that the sales team stopped using the opportunity stage field in February and started encoding it in the deal name. Discovering that the warehouse system has a second database nobody mentioned. Sitting with an analyst to watch which of the sixteen columns they actually use. The engineers who are worth the most on a small team are the ones who do this without being asked, because most data problems are organisational problems that arrived in table form.
How Do You Screen a Data Engineer in One Conversation?
Tool inventories tell you almost nothing here. Everybody has Airflow on their CV. Depth in this job shows in how somebody reasons about failure, so every question below is a failure scenario with no clean answer, and the information is in the follow-up rather than the first response.
How would you move a table that changes 40 million rows a day, without hurting the source?
This is the question we open with. The weak answer is a query against the production database filtered on an updated timestamp, run every hour. It is not wrong, and a junior engineer giving it is fine, but the follow-ups tell you everything. Is that timestamp set by the application on every write path, including the batch job that finance runs? What happens to rows that were modified while the scan was in progress? How do hard deletes show up, given that a deleted row cannot appear in a query looking for recent changes? Is there an index on the column, and what does the plan do without one?
Stronger answers get to reading the database log instead of the tables, because a log reader sees deletes, sees every write path, and puts almost no query load on the primary. They also mention the operational cost of that choice honestly: replication slots that will fill your disk if a consumer stalls, an initial snapshot that still has to be taken somehow, and the fact that a schema change now arrives as an event you have to handle rather than an error you discover.
A producer changes a column type and tells nobody. What happens?
The answer we do not want is that the pipeline casts it to text and carries on. That is the worst outcome, because it succeeds. An integer identifier that quietly becomes a string still loads, still joins to nothing, and produces a report that is short by however many rows changed after the deploy. Somebody notices three weeks later.
Good candidates want the load to fail loudly and the bad batch to sit in quarantine rather than in the table. They talk about validating the incoming shape against a declared schema before writing, and if the organisation runs a schema registry with Avro or Protobuf, about which compatibility mode is set and who is allowed to change it. The best ones make a social point as well as a technical one: the fix is a conversation with the team that owns the producer, plus a contract that makes the next change visible before it ships rather than after.
Data for last Tuesday arrives on Thursday. What did you build?
Late arrival is normal, not exceptional, and a candidate who treats it as an edge case has only worked on systems where somebody else absorbed it. Listen for the distinction between the time an event happened and the time you received it, and for an opinion about which one the partitions use. Partition on arrival and loading is simple while every report is subtly wrong. Partition on event time and reports are right while your loads have to go back and rewrite periods you already published.
Then ask how wide the reprocessing window should be. The answer we like is that the width gets measured rather than picked: plot how long your own records take to show up, put the boundary beyond where that tail flattens out, and route anything slower through a deliberate restatement that somebody can explain to finance. An engineer who names a fixed number of days before asking anything about your sources has memorised a pattern rather than applied one.
Define idempotency, then apply it to a task that writes to three places
Most candidates can define it. Fewer can apply it once the task has side effects. The single-table version is well known: merge on a business key, take the ordering from a column the source controls rather than from your own ingestion clock, or append everything and read through a view that keeps the latest version per key. Any of those makes a rerun harmless.
The interesting version is a task that writes a table, updates a metadata row and posts to a downstream queue. Rerun that and the table is fine while the queue now has two messages. The answers worth hearing involve making the downstream consumer tolerant of duplicates by giving each message a deterministic key, or moving the publish to a separate step that reads committed state rather than doing it inline. A candidate who has actually been burned by this will bring up the queue before you do.
Why is a pipeline that reruns cleanly worth more than a fast one?
Ask it plainly. It sounds like a leading question and it is, but the reasoning behind the agreement is what you are measuring. The answer we are looking for is about who spends the time. A job that takes twenty minutes and can be rerun by anybody costs nothing when it fails at 03:00, because the retry is automatic and nobody wakes up. A job that takes four minutes and needs a human to work out which rows were half written costs an engineer their morning every time it breaks, plus the risk that the manual repair introduces a second error.
Speed matters when it is on the critical path of a decision somebody makes at nine. Most of the time it is not, and the engineer who optimises the runtime of a job that finishes at 02:00 has spent your money on the wrong axis. We would rather hire the person who made the eleven-minute job restartable than the one who made it a six-minute job that cannot be.
Tell me about a pipeline that broke and what you found
Open and unstructured on purpose. What you are listening for is whether the story has a real texture to it: the specific system, the specific wrong assumption, how they narrowed it down, and what they changed so it could not recur. Engineers who have owned something tell these stories easily and without defensiveness, and they usually include the part where the cause was their own code.
Two things should make you slow down. A candidate whose only incidents were caused by other teams has either been lucky or is editing. A candidate who describes the fix as fixing the affected rows, with no follow-up change, is telling you what your Tuesdays will look like.
Run all of this as a conversation about your own systems, not as a quiz. Should you also want something written, keep the task small and shaped like the job: a file containing duplicate rows, three timezones and one column whose type changed in July, to be loaded into a table that can be rebuilt from nothing, plus a note on what the candidate chose to do about the awkward rows and why. Two hours is plenty, and it discriminates between candidates far better than an algorithm puzzle does.
SQL Is the Non-Negotiable, and Most Screens Test It Badly
Almost every data engineering interview claims to test SQL. Most of them test whether the candidate remembers join syntax, which everybody does. The gap between a competent and an excellent data engineer sits in three places, and none of them is the join.
Window functions, because the whole job is built on them
Deduplication, change detection, sessionisation, running totals, comparing a row to the previous one, ranking within a group: all of it is ROW_NUMBER, LAG, LEAD and aggregates with an OVER clause. An engineer who reaches for a self join with a correlated subquery to find the latest row per key can produce a correct result and will produce a slow one, and more importantly is going to find the next six problems harder than they need to be.
Push one level further and ask about the frame clause. Most people who use window functions daily have never explicitly written ROWS BETWEEN and do not know that the default frame for an ordered aggregate is a range to the current row, which behaves differently from a row-based frame when there are ties. It is a small thing that quietly changes a running total. Candidates who know it have been reading the documentation rather than copying patterns, which is the trait you are really testing for.
Can they read an execution plan?
This is the question that separates the levels most reliably, and hardly anybody asks it. Put a plan in front of them, from Postgres or from your warehouse, and ask what the engine decided. You want to hear about join strategy and why the optimiser chose it, about the difference between the estimated and actual row counts and what a wild mismatch implies about statistics, about whether a sort or hash spilled to disk, and about whether a partition or cluster key was used to skip data rather than scan it.
An engineer who reads plans fixes performance problems by changing one thing on purpose. An engineer who does not fixes them by rewriting the query several ways until one is faster, which sometimes works and never teaches anybody anything. In a warehouse where scanning is the thing you pay for, this skill is directly a cost skill.
Why the ORM is not the answer here
Application engineers moving into data work often bring an ORM with them, and it is the wrong instrument. An ORM exists to map rows to objects for transactional code that touches a handful of records at a time. Analytical work touches millions of rows in one statement, needs window functions and set operations the ORM either cannot express or expresses awkwardly, and lives or dies on a query plan the ORM is designed to hide from you.
There is a second, subtler problem. ORM code encourages a per-row mental model, and a per-row mental model in a columnar warehouse produces work that is orders of magnitude slower than the set-based equivalent while looking perfectly reasonable in review. The right tool for the transformation layer is SQL in version control with tests around it, which is what dbt exists to organise. Keep the ORM for the application it belongs to.
A short exercise that actually discriminates
Give the candidate a table of events with a customer identifier, a timestamp and a status, containing duplicates, some out of order rows and one customer whose events span a daylight saving change in their local zone. Ask for the current status per customer, then for a monthly count of status changes, then ask what they would do about the duplicates in the pipeline rather than in the query. Fifteen minutes, three follow-ups, and you will know exactly what you are dealing with.
Dimensional Model, Wide Tables or One Big Table?
Modelling arguments in this field are older than most of the people having them, and they have been reshaped by the fact that storage got cheap and columnar engines made joins less frightening than they were. Here is what each approach still buys, and what a candidate's answer tells you.
The star schema, and what it still buys you
Facts in the middle, dimensions around the edges, keys joining them. The Kimball approach is thirty years old and it has aged better than its critics expected, because the thing it optimises for was never really storage. It was agreement. A conformed customer dimension means the marketing team and the finance team are counting the same customers, and that consistency is worth more than any query performance argument.
Where it costs you is speed of change and the skill required. Building and maintaining dimensions with surrogate keys is real work, and a small team without a dedicated modeller will do it badly and end up with the overhead and none of the benefit. If your organisation has two consumers of the data and both are the same person, a star schema is ceremony.
Wide tables, the pragmatic middle
Take the fact table and denormalise the attributes people always want onto it. Order lines carrying the customer segment, the product category and the region, so the common query needs no joins at all. This is where a great deal of real analytics work lands, and it is a sensible default for a small team on a columnar warehouse, because unused columns cost you almost nothing at read time.
The cost arrives when an attribute changes. A customer moves from one segment to another, and now you have to decide whether the historical orders show the segment as it was or as it is. If you have not thought about that, the rebuild silently rewrites history and last quarter's report stops matching the copy somebody saved.
One big table
Everything flattened into a single wide table per subject area, no joins at all, straight into the BI tool. It is fashionable, it makes self-service genuinely work for non-technical users, and on some engines it is the fastest thing you can do. It is also the least flexible: every new attribute means a rebuild, storage grows quickly, and restating anything means restating everything.
Where it fits is a stable, well-understood subject area with many casual consumers, particularly when the BI tool has a weak semantic layer and the alternative is fifty people writing their own joins. Where it fails is anywhere the definition of things is still moving.
Slowly changing dimensions, the question underneath all of it
Every one of these choices eventually reduces to a single question: when an attribute changes, does history change with it? Overwriting in place is simple and loses the past. Keeping versioned rows with validity ranges, the pattern usually called type 2, lets you ask what a customer's segment was on the day they ordered, which is the only way to make historical reporting reproducible.
Ask a candidate which they would use for your specific case and listen for whether they ask what the data is used for before answering. Type 2 everywhere is over-engineering, and it inflates every join. Overwriting everywhere means somebody in finance will one day discover that a report they printed in March cannot be reproduced in June, and nobody will be able to explain why.
Why it depends is the honest answer
A candidate who says one of these three approaches is correct in all cases has worked in one shop. The determinants are specific and worth writing down before you interview: how many people consume the data and how technical they are, how often the dimension attributes change, whether anybody needs to reproduce a historical report exactly, what your BI tool does well, and whether you have anyone who will maintain a dimensional model properly.
What we would push back on is doing nothing while the discussion runs. Most teams are better served by a modest star for the two or three subjects that matter and wide tables everywhere else, built in a way that can be changed, than by a perfect design that is still being debated in month four.
A Scheduler and a Dependency Graph Are Not the Same Thing
This distinction gets skipped in interviews and it is the one that predicts how your platform behaves at three in the morning. Somebody who has only ever used a scheduler will build you a system that is fine until the day something is late.
What cron cannot express
Cron fires a command at a time. That is the whole feature, and it is a good feature. What it cannot say is that this task should run because the previous one succeeded, that it should not run at all if the upstream data is missing, that a failure should retry twice with a delay and then stop rather than continue, or that if today's run fails then tomorrow's should not quietly proceed on stale inputs.
So teams encode dependencies in the schedule instead. The extract runs at 02:00, the transform at 03:00, the export at 04:00, with the gaps sized by hope. It works until the extract takes ninety minutes one night, at which point the transform runs against half-loaded data and produces a plausible, wrong report that nobody questions. Everything was green. That is the failure mode, and it is why a dependency graph is not a nicer cron.
Airflow
The default in most organisations, and the one most candidates have. Tasks and dependencies declared in Python, a scheduler, an executor, and a web interface where you can see what failed and clear a task to rerun it. It has real operational weight to it and it repays somebody who knows its habits.
Screen for the two things people get wrong. The first is putting datetime.now() inside task logic, which makes every run depend on when it happened rather than which interval it covers, and destroys the ability to backfill correctly. Airflow gives you the interval for a reason. The second is a graph of a few enormous tasks rather than many small ones, so a failure eight steps in means rerunning everything before it. Ask a candidate what their largest task did, and whether they could rerun one day of one table without touching anything else.
Dagster
Built around assets rather than tasks: you declare the table you are producing and what it depends on, and the orchestrator works out the graph. That inversion sounds cosmetic and is not. When somebody asks which tables are affected by a broken source, the answer comes from the system rather than from a person's memory, and lineage stops being a documentation exercise.
The trade is a stronger set of opinions to learn and a smaller pool of people who have run it in production. On a new platform with nobody's legacy to carry, it is a genuinely good choice and we say so more often than the market share would suggest.
Prefect
Closer to ordinary Python, with flows that can change shape at runtime. Where the work is not a fixed graph, such as fanning out over however many client accounts exist this morning, expressing it in Prefect is direct where in Airflow it takes effort. The looser structure is the cost: with less enforced shape, the discipline has to come from the team.
The orchestration question worth asking
Describe this and let them answer: a file usually arrives at 02:00, occasionally at 05:00, and once a month not at all. Your downstream job is scheduled for 03:00. What do you build?
Weak answers move the schedule later. Better answers wait for the file with a sensor and a timeout rather than a fixed time. The answers we want include what happens on the month it never arrives: the run does not proceed on yesterday's data, somebody is told before the business day starts, and the downstream tables are visibly stale rather than silently wrong. That last distinction is most of the job.
Data Quality Belongs in the Path, Not on a Dashboard
Nearly every company we talk to has a data quality dashboard. Almost none of them has one that has changed anybody's behaviour, because a screen full of amber squares that publishes the numbers anyway is a record of problems rather than a defence against them.
A test that cannot stop anything is decoration
The question to ask a candidate is not whether they write tests. Everybody says yes. The question is what happens when one fails at 04:00 with nobody awake. If the answer is that a warning is logged and the table publishes anyway, then the test has no authority and the first person to see the problem will be a customer or a board member.
The version that works is a gate. Transformations write to a staging location, the assertions run there, and only a clean result gets promoted to the tables people query. A failure leaves yesterday's good data in place and raises an alert that names the failing assertion. Stale and correct beats fresh and wrong in almost every business context, and the exceptions are worth naming explicitly rather than assuming.
The four tests worth having before anything clever
Uniqueness on the primary key of every model, because duplicates are the single most common cause of an inflated total. Not-null on the columns that joins and filters depend on. Referential checks that every foreign key in a fact resolves to a row in its dimension, which catches late-arriving dimension records that would otherwise silently drop rows from a report. And an accepted-values check on the low-cardinality columns that drive segmentation, which is how you find out that a producer added a new status code.
All four are one-line declarations in dbt, and if you are not on dbt then Great Expectations or Soda cover the same ground with more configuration. The tooling is not the interesting part. Whether the test blocks publication is.
Freshness is the test people forget
A pipeline that fails is visible. A pipeline that succeeds while reading a source that stopped updating on Tuesday is not, and it is the more common outage. The check is trivial: assert that the newest record in a table is younger than some threshold, set from how often the source genuinely changes, and alert when it is not. Ask a candidate whether they monitor freshness separately from job success. A surprising number have never separated the two, and it is a quick way to find out whether they have run a platform or only built one.
Warehouse Compute Is the Line Item That Surprises People
Nobody warns finance teams about this one. The bill for a modern data platform is not driven by how much data you store, which is usually cheap, but by how much of it gets read and how often. Almost every one of those decisions is made by an engineer writing SQL.
The engineer decides most of the bill
The pricing models make it explicit. BigQuery's on-demand mode charges for bytes scanned, so a query that filters on a partitioned column and one that does not can differ by orders of magnitude while returning the same result. Snowflake bills for warehouse time, so an idle warehouse with a slack auto-suspend setting is money spent on nothing, and an oversized one finishes a small job faster than you needed and charges accordingly. Neither of those is a procurement decision. Both are made by whoever wrote the model and configured the job.
This is why cost belongs in a technical interview rather than only in a contract negotiation. An engineer who has never seen the bill for their own pipelines will build something that works and costs whatever it costs.
The habits that move the number
Incremental models rather than full refreshes, which is usually the single biggest lever and usually the one skipped because a full rebuild is simpler to reason about. Partitioning and clustering chosen to match the filters people actually use, rather than the ones that seemed natural when the table was designed. No SELECT * in a chain of views, because a columnar engine reads only the columns you ask for until somebody asks for all of them. Dashboards that refresh on a schedule matched to how often the underlying data changes, instead of every five minutes because the default said so. And an honest look at anything streaming: real-time processing is right when a decision is made in real time, and expensive theatre when the report is read at nine the next morning.
What to ask about cost in an interview
Ask whether they have ever had to reduce a warehouse bill, and what they found. The specific answers are all interesting. A dbt model rebuilding an entire history nightly because incremental logic was never added. A dashboard set to auto-refresh sitting on a full table scan. A single analyst's exploratory query pattern running against the raw layer instead of a modelled table. An engineer who can tell you where the money went in their last platform is a different proposition from one who has never been shown the invoice.
Can Anyone Answer Where This Number Came From?
This is the question a chief financial officer asks the first time a figure moves, and the answer time is a fair proxy for the maturity of the whole platform. Ten minutes is healthy. Two days means the knowledge lives in one person's head.
Table-level, column-level, and the gap between them
Table-level lineage says this table was built from those three. It comes almost free once transformations are declared rather than scripted, and dbt produces it from the graph it already has to build. It answers what breaks if a source goes away.
Column-level lineage says this figure in the board deck traces to that column in that source, through these two transformations, one of which applies a currency conversion. That is what you actually need during an investigation, and it is meaningfully harder to produce. Several tools parse SQL to derive it, and OpenLineage exists as an open standard for emitting the events across different systems rather than being locked to one vendor's catalogue.
What good looks like in practice
Honestly, most teams do not need a catalogue product to start. What they need is transformations expressed as code in version control rather than as queries saved in a BI tool, a graph the orchestrator can show, and a written definition of each metric that names the owner. Those three cover the majority of investigations.
The test for a candidate is direct. Ask how they would trace a wrong figure in a report back to its source, and whether they have ever had to do it under pressure. Someone who has will describe a method rather than a tool, and will usually mention checking whether the definition changed before assuming the data did.
The Data Engineering Seniority Ladder, Without the Title Inflation
Years count for less here than in most engineering disciplines, because a lot of people have spent three years running pipelines somebody else designed. What separates the bands is the size of the failure they can be trusted to be alone with.
One to two years: builds inside a pattern somebody else set
Writes solid SQL, adds a model to an existing dbt project, adds a task to a graph somebody else drew, follows the conventions in front of them. Productive where a platform already exists and somebody has time to review. Put them in as the only data person in a company and it goes wrong quietly, because the damaging decisions at this level are the ones nobody realises are decisions.
Three to five years: owns pipelines and their failures
Has been the person called about a wrong number, and found it. Designs for reruns without being told to, reads a query plan, argues about partition keys with evidence rather than preference. Most teams are hiring in this band whether their job ad says so or not, and it resists interviewing, since nothing separates it from the band below until an inconvenient hour arrives.
Six years and up: talks you out of things
Judgement is the product at this level, not output. Argues down the streaming architecture nobody needs, works out that the whole problem is one source system's soft deletes, can say when a paid connector is cheaper than the months it would take to replace it. Expect an opinion about your architecture within a fortnight, ideally phrased as a question.
Lead: makes the platform legible to other people
Sets the modelling standard, owns the metric definitions, breaks work up so three engineers do not collide in the same models, and translates between what the business asked for and what the data can honestly support. Worth adding at three or more engineers, or where analytics and ML are pulling on the same tables. Not worth adding to supervise one person.
One caveat specific to this field. Seniority in data engineering does not transfer cleanly from the roles it borders. A senior backend engineer is a mid-level data engineer the first time they own a warehouse, because their instincts are transactional and the failure modes here are not. A senior analyst who writes better SQL than anyone on your team may never have made a job restartable. Both will get there, and neither should be introduced as senior in a role they have not held. We will tell you which of the five jobs from the top of this page a candidate is actually senior in.
Four Hiring Situations and What We Would Staff
Each of these is a composite, drawn from the shape of briefs that reach us rather than from any single company. They are here because the reasoning transfers better than a logo would.
The analyst who became the pipeline
One capable analyst has forty scheduled queries, six spreadsheets with connections nobody else understands, and a laptop that has to stay on. Everything works. The company is one resignation away from not knowing its own revenue by segment.
The instinct is to hire a senior data engineer to replace the setup, and it is the wrong first move, because the knowledge is in the analyst's head and a replacement will spend three months rediscovering it. What works better is a mid-to-senior data engineer whose first job is explicitly not to rebuild anything: get every query into version control, make each one reproducible, and write down what each is for and who reads it. Only then is the rebuild ordered by what would hurt most if it stopped. The analyst should stay an analyst, which is usually what they wanted.
The data scientist hired a year too early
An experienced data scientist joined nine months ago to build a churn model. They are still writing extraction scripts against three APIs, the model has been demonstrated twice and deployed never, and both they and the founder are frustrated. This happens constantly and it is not the scientist's fault.
The fix is to hire the engineer that should have come first and to be honest about the sequence. The engineer's first quarter is landing the sources properly and producing a modelled, tested table of the events the model needs, with history. The scientist's job becomes modelling again, which is what they are expensive for. If the model then needs to run on a schedule and be monitored, that is a distinct piece of work again, closer to what our MLOps services in India page describes than to either of these two hires.
The nightly database dump that outgrew itself
Reporting runs off a full dump of the production Postgres database, restored into a second instance overnight. It has worked for years. Now the dump takes six hours, it is starting to collide with the morning, and the finance team wants intraday figures that this design fundamentally cannot provide.
The temptation is a platform rebuild. The proportionate move is a data engineer who first works out which tables actually need to be current, because it is rarely all of them, and moves only those to incremental loading with a proper ordering column. Full copies of the slow-moving tables can stay nightly for now. The reason to do it in that order is that it delivers the intraday figure in weeks rather than after a migration, and it produces the information you need to decide whether the full platform is worth building at all.
The pipeline that cannot be rerun
A team of four has pipelines that work most of the time and, when they fail, require somebody to work out which rows were half written and repair them by hand. There is a document describing the repair. Everyone quietly avoids the person's week when it happens.
This is a design problem, not a staffing one, and it is a good example of work that does not need a permanent hire. The job is to make loads idempotent, add the ordering and keying that lets a rerun be harmless, split monolithic jobs into steps that can be retried individually, and then delete the repair document. It has a clear end state, which makes it a poor fit for a dedicated seat and a good fit for a scoped engagement with a senior engineer, and we would say so rather than sell you a headcount.
Hiring Data Engineers in India: The Working Day, Honestly
Most offshore engagements that disappoint people fail on coordination rather than on ability, and the cause is usually arithmetic nobody did before signing. None of these numbers move afterwards, so do them now.
The timezone arithmetic, done properly
One zone covers the whole of India, at UTC+5:30, and no part of the year moves it. The second half of that sentence does more work than the first. Your own office hours slide by an hour twice a year while the Indian block stays exactly where it is, so a 09:30 start and an 18:30 finish in Pune reads as 04:00-13:00 UTC in February and reads the same in August.
| Your office | Local 09:00-17:00, converted | Shared hours against 04:00-13:00 UTC |
|---|---|---|
| Dubai (UTC+4) | 05:00-13:00 UTC | Eight. Effectively the same day. |
| Berlin, summer (UTC+2) | 07:00-15:00 UTC | Six. Best of any Western market. |
| London, summer (UTC+1) | 08:00-16:00 UTC | Five. |
| London, winter (UTC+0) | 09:00-17:00 UTC | Four, sitting entirely in your morning. |
| Sydney, winter (UTC+10) | 23:00-07:00 UTC | Three, right as the Indian desk fills up. |
| Auckland, winter (UTC+12) | 21:00-05:00 UTC | One. Nothing at all after the clocks move. |
| New York and Toronto, standard (UTC-5) | 14:00-22:00 UTC | None. |
| San Francisco, standard (UTC-8) | 17:00-01:00 UTC | None. |
Look hard at the bottom two rows, since they are the ones this industry hurries past. Keep both sides on ordinary office hours and an American team shares nothing with an Indian one. Not a thin sliver. Nothing. Any vendor claiming otherwise has silently moved the Indian day and forgotten to mention it.
American overlap therefore costs somebody their evening. Push the Indian day to 13:30-22:30 and the block sits at 08:00-17:00 UTC, handing New York roughly three shared hours. Getting the same for the Pacific coast means running later again, which is a night shift, and calling it anything else does nobody any favours. Arrangements like this hold up when both sides discussed them and staffed for them deliberately. They collapse when a client quietly assumes the hours and an engineer quietly absorbs them until they burn out.
The advantage that is specific to this kind of work
Here is the part of the timezone conversation that runs in your favour, and it applies to data engineering more than to almost any other role. Batch pipelines run overnight in your zone, which is the middle of the Indian working day.
Work it through for a New York business. A nightly job kicking off at 02:00 Eastern starts at 07:00 UTC, which is 12:30 in the afternoon in India. If it fails, an engineer is at a desk, awake, with hours of working day left, rather than being paged out of bed. The same job for a San Francisco business at 01:00 Pacific starts at 09:00 UTC, or 14:30 in India. Your overnight window is somebody's ordinary afternoon, which means a failed load can be diagnosed and rerun before your team logs in, and a backfill can hammer a source database at the quietest possible moment.
Be aware of where this argument stops. For a UK or European business the benefit largely disappears, because a 02:00 London job is 07:30 in India and lands before the working day starts. The overnight advantage is real for the Americas and thin for Europe, and Europe gets the overlap hours instead. It is worth choosing which of the two matters more to you before you decide how the day should be arranged.
Continuous coverage is a staffing decision, not a bonus
Cover across every hour of the day needs several people, a rota written down somewhere both sides can see, a handover note that still makes sense to whoever reads it cold at 06:00, and a shared rule about which failures justify phoning somebody at home. All of that is extra people and extra coordination effort. Asking for it is entirely legitimate. Being told it comes bundled at no cost with an offshore team is not, because it never has. When a platform genuinely warrants that kind of rota, we work the shape out with you first and you can see what it takes before committing.
How the day runs when it is working
Writing comes before talking. There is a note waiting in your channel when you sit down: what landed overnight, what is being picked up, where something is stuck. You start the day informed instead of asking. Pull requests go up early and stay small, so your review happens during the shared hours rather than as one enormous branch dropped on a Friday afternoon. Anything decided on a call gets typed back into the ticket the same day, because a conversation three people half remember is not documentation.
Two things matter disproportionately for data work in particular. One is a development setup with realistic sample data in it, so a model can be built and checked without an engineer waiting on production access or on somebody awake in your office to explain a column. The other is metric definitions written down and agreed. The expensive delays in this discipline are almost never technical. They are an engineer in Mumbai losing eighteen hours to a question about whether a refund lands in the month it was issued or the month of the original sale.
Code, access and your data
Everything is built inside your repository, on infrastructure you control, behind access you grant, and the commit log is the audit trail. Terms covering confidentiality and ownership of the work are put in writing ahead of any engineer starting, and what those terms contain belongs in the agreement between us rather than in a paragraph on a website. This discipline pushes the question harder than most, because production data is the raw material rather than an occasional debugging convenience. Tell us at the brief stage if your data carries regulatory constraints or restrictions on where processing may happen. It genuinely changes how the engagement is built, and one common answer is that engineers develop against masked or synthetic data while your own team triggers the runs that touch the real thing. Where your legal advisers have residency or processing requirements, designing around them at the start is far cheaper than discovering them in week six.
What actually goes wrong
Three things account for most of the disappointment, and none is technical ability. The first is an underspecified request: a ticket saying build a pipeline for orders will get an interpretation, and distance makes a wrong interpretation expensive because it survives longer before anyone notices. The second is review latency. A pull request waiting two days for a reviewer means the engineer either stalls or starts something else, and unmerged work stacks up. The third is a missing decision-maker on the business side, because so many data questions are business questions in disguise. Nobody can define active customer except you. All three are fixable, and all three sit on the client side of the line at least as often as ours.
Why This Role Takes Longer to Fill Than a Backend Developer
If you have hired backend engineers before, calibrate for a slower search here, and know why.
The pool is smaller and it is diluted. Data engineering is a destination role rather than a starting one, so the people who are good at it arrived from backend work, from analysis or from database administration, which means the label sits on a wider spread of actual ability than most titles do. On top of that, the two biggest employers of this skill are the ones you are competing with: platform teams inside large product companies, and the consultancies who staff them. Both move quickly and neither is short of budget.
The result is a search that runs for months and then meets whatever notice the candidate's current employer holds them to. An offer accepted in March can become a start date in June, and the pipeline that prompted the search is still breaking every Tuesday in the meantime. That gap is the cost nobody puts in the business case.
Our answer to it is unremarkable but it is the honest one. We keep data engineers between engagements rather than starting a search when your brief arrives, so the people already exist by the time you ask. That is what makes a shortlist inside 48 hours realistic, and it is why the engineer you choose is working on your codebase within 7 days rather than after a resignation cycle. Availability is doing the work in that sentence, not recruiting speed, and it only holds up if your brief is specific enough to match somebody against. A request for a data engineer produces a worse shortlist than a request for someone who can take a Postgres source with soft deletes onto incremental loading and model the result for a finance team.
If the wider question is whether to hire in this market at all, or which discipline to start with, our hire developers in India page covers the options across stacks. Where the work is mostly Python engineering with a data flavour rather than warehouse ownership, the distinction is worth reading about on the hire Python developers in India page before you write the brief.
The Tooling Our Data Engineers Work Across
Listed for completeness, not as a filter. We would rather send you someone who reasons well about ordering and failure and has to learn your orchestrator than someone whose only qualification is having clicked through the same console you use.
What a Data Engineer Should Do in Weeks One and Two
The order matters more here than in application work, because a change made before the data is understood produces a result nobody can review with confidence. Use this as a checklist whoever you hire.
Reads the actual rows before reading the code
Counts, distinct values, null rates, the distribution of dates, the customers with forty addresses. Every codebase encodes assumptions about the data, and half of them are already false. Finding out which half is week one.
Draws the map that does not exist
Which sources feed which tables, which tables feed which dashboards, and which of them nobody has opened in six months. It is usually the first time anyone has seen the whole picture, and the unused branches are the cheapest thing you will ever delete.
Finds out what breaks most often
Job history, the alert channel, and whoever answers the messages about wrong numbers. Failures cluster around two or three causes in almost every platform, and naming them turns a vague reliability problem into a scoped piece of work.
Checks whether anything can be rerun
Picks one non-critical job and runs it twice against the same input. If the totals move, that is the finding, and it usually applies to more than the one job tested.
Writes down the metric definitions currently in force
Not the ones people say. The ones the SQL implements. Where the two disagree, that gap is worth surfacing early, because it is the source of most arguments about whether the data is correct.
Puts a uniqueness and a freshness test on the busiest table
Two assertions on the model most people read is a small change with a fast payback. It also establishes the pattern before there is a backlog of models to retrofit.
Looks at the bill
Which queries and which jobs account for the spend. There is almost always one full refresh or one auto-refreshing dashboard doing a disproportionate share, and finding it early funds a lot of goodwill.
Ships something reviewable by day three
A change compact enough to get through review during the shared hours, with a description explaining the reasoning rather than restating the diff. Getting a few merges done early sets the rhythm before anything hard depends on it.
Engagement Models
Three shapes. Which one fits depends on two questions: does the work have an ending, and who is holding the technical direction. Working hours, contractual terms and the shared window all get settled with you up front, which is exactly why no figures appear anywhere on this page. The exception to that silence: a data engineer who is not working out is replaced within 48 hours, and the person who follows is one of several candidates you get to choose between, not a single name.
A Defined Piece of Work
Suits anything with a visible finish line: making a set of loads restartable, moving reporting off an overnight dump, building the modelled layer your data science team never got. What you receive is a working result and a handover, and the arrangement stops there.
A Dedicated Data Engineer
Usually the right answer once a platform exists and the tickets keep coming. One engineer, inside your repo, on your board, in your daily call, taking direction from you. It is what most teams have in mind when they ask to hire dedicated data engineers.
Engineer Plus Analytics Engineer
Where ingestion and modelling both need attention at once and a single hire would be strong at one and learning the other. Somebody holds the modelling standard, which spares you refereeing arguments between two specialists in a discipline you do not practise.
About money, the honest contribution is a refusal. There is no rate card here, because a price quoted against no scope, no seniority and no working-hours requirement tells you nothing worth knowing. Budget instead for the things people leave out: the weeks before a new engineer is productive, the review time this costs your own staff, the coordination overhead of any shifted day, and the warehouse compute that new pipelines will start consuming. That last item belongs to this discipline specifically, and it is the one most often missing from the business case altogether.
Frequently Asked Questions About Hiring Data Engineers
What is the difference between a data engineer and an analytics engineer?
A data engineer owns everything up to and including the raw tables landing in your warehouse: extraction, delivery guarantees, schema changes, retries, orchestration. An analytics engineer starts where that ends and turns raw tables into modelled ones your business people query. The first is closer to a backend engineer, the second closer to an analyst who writes production SQL. Small teams often want one person covering both, and that person is rarer and more senior than either alone.
Do we need a data engineer or a data scientist first?
If nobody can currently produce a clean table of the last two years of orders without a manual export, hire the engineer. A data scientist with no reliable data spends the first quarter doing plumbing badly and resenting it, and you pay a research salary for it. The order flips only when the tables already exist, people trust them, and the open question is genuinely a modelling question.
How much of our working day will we actually share with an engineer in India?
A single zone covers India, at UTC+5:30, unmoved by daylight saving. So a 09:30 start and 18:30 finish converts to 04:00-13:00 UTC in every week of the year. Against that block, London gets four hours in winter, Berlin six in summer, Dubai eight and Sydney three. Both American coasts get nothing on ordinary office hours. Buying US overlap means moving the Indian day later, and that shape is settled with you before an engineer starts.
How do you test whether someone can really write SQL?
We give them a small messy dataset and ask for the latest row per customer, then ask for the running total by month, then ask them to explain the query plan the warehouse produced. The first two need window functions. The third separates people who write SQL from people who read what the engine did with it. Candidates who reach for a self join with a MAX subquery on the first question are not wrong, but they are usually a level below where they claimed.
Our data all lives in one Postgres database. Is it too early to hire a data engineer?
Probably, and we will tell you so. While one database answers every question, an analyst with good SQL and a read replica goes a long way. The moment to hire is when reporting queries start competing with production traffic, when a second system holds data the first one needs, or when somebody is exporting a spreadsheet on a schedule. Any of those three is the signal, not headcount or funding stage.
What should a data engineer already know, and what can they pick up on the job?
Non-negotiable on arrival: SQL to window function depth, Python for the glue, and the reasoning about idempotency, ordering and failure that this page is largely about. Picked up in weeks: your specific warehouse dialect, your orchestrator, your BI tool, whichever ingestion product you already pay for. Hiring on tool names filters out good engineers and lets through people who have only ever clicked through one vendor console.
Can we hire a dedicated data engineer rather than buying a project?
Yes, and it is the shape most teams want once the platform exists and the work is continuous. A dedicated engineer sits in your repository and your ticket queue under your direction. Where the work has an obvious end state, such as one migration or one set of pipelines to build and hand over, a scoped engagement finishes sooner because nobody is paying for a seat after the work runs out.