Questions to Ask an Engineering Partner About Production Reliability

I noticed this shift by comparing notes across a handful of recent partner searches, and it's worth naming clearly: hiring an engineering partner used to start with a budget conversation, and now it starts with a durability conversation instead. The 2024 Deloitte Global Outsourcing Survey backs this up: cost dropped from the top driver of outsourcing decisions to a secondary concern, replaced by access to specialized talent. Organizations aren't shopping for the cheapest hands on a keyboard anymore. They're shopping for teams that can hold a system together once it's live.

Here's the part I had to sit with before it made sense. The same survey found that vendor misalignment, not poor technical skill, is the primary delivery risk, and it isn't a lack of talent so much as a coordination failure. That's a coordination failure, and it usually shows up as engineering leads turned into full-time babysitters for a partner they can't quite trust to make the right call without oversight.

What does misalignment actually look like day to day? Late defect discovery is a big one, where bugs surface weeks after they were introduced because nobody was watching closely. Thin knowledge transfer is another, where documentation exists but nobody who wrote it is around to explain the parts that matter. And then there's the one that keeps founders up at night: no clear owner when something breaks in production at 11pm on a Friday.

So the evaluation conversation has to change. It's no longer "can they build it?" That question got answered a decade ago; plenty of teams can build. The real question is whether they'll still be answering the phone when it breaks.

What production readiness actually means — and how rarely it is formalized

Production readiness is a specific thing, not a vibe. It's the set of criteria that confirm a service can run reliably once actual customers, not test accounts, are pushing traffic through it. That sounds obvious when I say it out loud, but in practice, most teams skip the formalizing part entirely.

What I see instead, over and over: a Confluence page with some operating procedures on it, and a spreadsheet somewhere with a list of criteria nobody updates after the third sprint. Both are static, and both require someone to remember they exist. Both get skipped the moment a deadline gets tight, which is exactly the moment you need them most.

Cortex surveyed 50 engineering leaders and found that 32% admitted they have no formal process for maintaining production standards outside of responding to incidents. Read that again: the incidents are the process, and something breaks, and that's how the team finds out what standard they failed to meet. Ninety-eight percent of those same leaders had seen at least one significant consequence of failing production readiness, most often downstream revenue loss or a delayed launch. Specific numbers from that survey: roughly 62% saw their change failure rate climb, 56% saw mean time to resolve get worse, and 54% watched developer productivity drop.

Here's the gap worth surfacing in a partner conversation. Do they run a pre-launch production readiness review, a real one with named owners and specific criteria? Or do they find out what's missing from production the same way most teams do, which is when it's already on fire?

The reliability vocabulary a partner must be able to use precisely

There are three terms a partner should be able to define without pausing to think: SLA, SLO, and SLI. If someone hesitates here, that's data.

An SLA is the contractual promise made to customers. An SLO is the internal target the team holds itself to, which should be stricter than the SLA. An SLI is the actual number you measured. Say a product promises a high-uptime guarantee in its SLA. A serious partner sets an internal SLO that is stricter than the external promise, tighter than what's promised externally. Then the SLI comes in below the SLO but above the SLA threshold, meaning the customer promise got kept even though the internal target got missed. A partner who understands the difference will flag that miss and dig into why, even though nobody outside the team noticed.

Why build in that gap between SLO and SLA at all? Because it's a buffer. Without it, every small operational hiccup risks tripping a contractual penalty. With it, the team has room to catch problems before they become customer-facing ones.

Then there's the error budget, which is where reliability math turns into an actual decision-making tool. When the budget is spent, healthy teams stop shipping new features and shift to stability work. That's not a suggestion, it's a rule the good teams actually follow, even when a product manager is pushing hard for the next release.

One more thing worth understanding: reliability degrades as systems compose. Six services, each individually reliable at a high percentage, don't stay that reliable once they're chained together, because the math compounds against you. Practically, this means removing a dependency from your critical path often matters more than squeezing a better SLA number out of a vendor contract. So ask a partner: do they define SLOs before launch, or do they back into them after the first outage teaches them what they should have measured?

Questions that reveal how a partner handles the MVP-to-production transition

Technical debt from an MVP isn't a failure of discipline. It's a predictable stage of building anything fast, and the real question is whether a partner tracks that debt on purpose, or lets it pile up quietly until it's load-bearing and nobody remembers why.

CAST's 2025 analysis looked at over 10 billion lines of code across 47,000 applications and found that 45% of the world's code is fragile, 32% is bloated, and 31% is too rigid to change without breaking something. That's the baseline condition most partners inherit the moment they take over an existing codebase, whether they built it or not.

A good CTO will name the tension directly: moving fast during the MVP phase creates shortcuts, and those shortcuts are rational early and destructive later. The partner's actual job is knowing exactly where that line sits, and saying so out loud before it's a problem.

A few questions worth asking directly:

• "What's your process for flagging MVP shortcuts as tracked debt before the first production deployment?"
• "How do you decide when a product is genuinely production-ready versus just deployed?"
• "Walk me through a pre-launch production readiness review. What's on the checklist, and who owns each item?"

A strong answer names an artifact you could actually look at: a debt register, a PRR checklist, a risk-rated backlog that a non-technical stakeholder could read and understand. A weak answer sounds like "we do a final QA pass before go-live." QA and production readiness are not the same thing, and a partner who conflates them hasn't thought hard about the difference.

Questions that expose how a partner manages technical debt over time

The scale here is bigger than most founders assume. Accenture's 2025 Digital Core Report puts the annual cost of global tech debt at an estimated $2.41 trillion in the US alone, with $1.52 trillion needed to remediate it in the US. That's not a backlog line item; it's an operating cost, every year, whether or not anyone's looking at it.

Deloitte's Tech Trends 2024 report found developers spend roughly a third of their time on technical debt maintenance, and a large majority of technology leaders rank debt as the top cause of lost productivity. One-third of engineering time, gone to upkeep instead of new work, and that's the tax nobody budgets for upfront.

And AI tooling is making this worse, not better, in a lot of shops. GitClear's 2025 analysis of hundreds of millions of lines of code found an eightfold increase in duplicated code blocks in 2024, while refactored lines dropped from roughly a quarter of all changes in 2021 down to a much smaller fraction in 2024. Code is getting written faster and cleaned up less, which is the opposite of what the AI pitch usually promises.

The benchmark worth knowing: companies need to put roughly 15% of IT budget toward debt remediation just to avoid compounding, and firms that do this consistently outperform peers on revenue growth. So ask a partner directly what percentage of each sprint goes to debt work, and ask them to make that visible to you even if you're not technical. Ask if they can show a sample debt register from a past engagement, an actual artifact, not a description of one. And ask specifically how they keep AI tooling from accelerating the exact problem it's supposed to help with. A partner who can't produce a debt artifact usually doesn't have a systematic process, and debt's getting handled reactively, after it causes the incident, not before.

Questions that test whether a partner measures delivery performance at the system level

DORA's four metrics, deployment frequency, lead time for changes, change failure rate, and mean time to recover, have been the industry standard for measuring delivery performance since 2013. If a partner doesn't track these, or tracks something adjacent but can't speak to these specifically, that tells you something.

The 2024 DORA State of DevOps report found elite performers deploy on demand with a change failure rate near 5%. It also found the high-performance cluster of teams shrank while the low-performance cluster grew, meaning delivery capability, as an industry, moved backward last year. That's worth sitting with for a second; it's not a one-way ratchet where things only get better with time.

Here's the part I find most important for founders evaluating partners right now: for the second year running, DORA's research shows AI tooling correlates with worse software delivery performance at the system level, even while it improves how individuals feel about their own productivity. A partner who points to how fast their team ships with AI as proof of reliability is answering a question you didn't ask.

Worth asking directly: which DORA metrics does the partner track, and can they share their typical range from recent engagements? How do they separate individual developer output from system-level reliability in how they report progress to you? And when a deployment makes reliability worse, what's the actual decision process for what ships next? GitLab's 2024 DevSecOps survey found roughly half of security professionals say vulnerabilities are most often caught only after code merges into a test environment. That gap shows up directly in change failure rate, and it's exactly the kind of thing these questions are designed to surface.

Questions specific to nonprofits and mission-driven organizations

Nonprofits carry a version of this risk that's easy to underestimate. Smaller organizations spend a disproportionately large share of total budget on IT relative to bigger ones, which means a reliability failure costs more, proportionally, not less, and there's less room to absorb a bad week.

Nonprofit Tech for Good's 2024 Tech Report found a substantial share of nonprofits cite technology limitations as a barrier to hitting their organizational goals. That's not an IT department problem tucked away somewhere; that's mission delivery, directly tied to whether the engineering partner takes reliability seriously.

Here's a concrete example worth sitting with. Mobile drives the majority of nonprofit web traffic, but desktop drives 70% of donation revenue, at nearly double the average gift size. A checkout failure on desktop has a real, measurable cost per transaction, not a vague one. And recurring giving, which makes up a significant portion of all online revenue for nonprofits, depends on payment flows working correctly month after month after month. A broken donation flow isn't a one-time hiccup; it's a subscription quietly failing in the background until someone notices the numbers are off.

There's also a security piece that doesn't get enough attention. Cyberattacks on civil-society nonprofits jumped sharply between 2024 and 2025, and 70% of nonprofits still have no formal cybersecurity policy in place. A partner who doesn't raise this during scoping, unprompted, isn't asking the right questions of themselves.

Nonprofit leaders should ask: "Have you built for organizations with no internal technical staff to catch and escalate production issues? What does monitoring and alerting handoff look like in that world?" Also worth asking: how does the partner scope reliability work inside a tight budget, what's non-negotiable, and what gets deferred with explicit tracking rather than just quietly dropped? And specifically: is there a separate reliability process for payment and donation flows, distinct from general uptime? A partner who genuinely understands this space should be able to describe operating as the sole technical point of contact during a real incident, at a real organization, not just handing over a folder of docs and wishing you luck.

Questions about what happens after the engagement ends — or doesn't

The single most important reliability question isn't about tools or pipelines; it's about ownership. Who's responsible for production at 2am, six months after launch, when the person who built the original feature has moved on to three other projects?

Partners who hand off and disappear create a predictable failure mode. The founder becomes the accidental production owner, with none of the context needed to actually fix anything, and every incident pulls them out of go-to-market work and client conversations they should be having instead. That's a bad trade, and it's an avoidable one.

Documentation matters, but it's not enough on its own. Runbooks, architecture docs, incident playbooks: these tell you what happened after something breaks. They don't tell you who's actively working to prevent the next one.

Ask directly: after launch, who owns monitoring and incident response, your team or theirs, and is there a hard handoff date on the calendar or just a vague promise? What does ongoing maintenance actually look like in practice, what's covered, what are the response time commitments, and how is that different from the build phase? Can they describe a time they caught and fixed a production issue during a live engagement, not during a build sprint, and how they communicated that to the client at the time? And if the relationship ends, what does knowledge transfer look like, and how long before another team can take over without triggering an incident in the process?

What you're listening for is specificity about ongoing accountability, not reassurance about how thorough the documentation is going to be. Worth noting as one data point: this is how we've structured things at QUWA Labs, where a rebuilt product stays on an ongoing maintenance plan afterward, with our team owning production so the founder can stay focused on growth instead of getting pulled into pager duty. Whatever the arrangement, the model that tends to hold up is a long-term technical partnership, not a build-it-and-walk-away transaction.

How to use these questions as a filter, not a scorecard

No partner is going to answer every single one of these perfectly, and that's fine, honestly expecting perfection is its own mistake. What matters is how they respond to the hard ones: do they get specific, or do they reach for reassurance instead?

Specificity is the tell. A partner who can name their actual DORA range, pull up a real debt register, and walk through a real incident from start to finish is treating production as a first-class concern. A partner who answers with process language, lots of confident phrases and no artifacts to back them up, probably isn't.

Values alignment shows up here before any contract gets signed. A partner who seems uncomfortable with these questions now is going to be uncomfortable owning production later, when something actually breaks and someone needs to be reachable.

My practical suggestion, after running through this exercise myself more than once: bring three to five questions from each section into that first conversation. The answers will tell you more about what the partnership actually looks like than the proposal ever will. For nonprofit leaders specifically, weight the constrained-budget and sole-technical-contact questions heavily; a partner who hasn't operated in that world will show it fast, usually within the first two or three answers.

Choosing on capability instead of cost means knowing exactly which questions expose a partner's real standards, not just their pitch, and it means being willing to walk away from one who can't answer them.