What should a company look for in an AI DevOps partner?
Short answer: Five things to look for. Three things that should immediately disqualify. The rest is taste. We'll tell you which five and which three — including the questions we'd ask about us.
The five things to look for
1. Senior engineers on the keyboard
Ask: "Will the people in this room be writing the code?" If the answer is "we have a delivery team for that," walk. The gap between sales-engineering and delivery is where 80% of bad consulting outcomes happen. The right answer is "yes, the same humans you talked to in the assessment build the system."
2. Fixed-scope, not time-and-materials
Time-and-materials engagements have no incentive to finish efficiently. You want fixed-scope after a paid assessment. The assessment may be T&M because nobody knows the scope yet — that's fair. The build phase should not be.
3. Written assessment before code
You should receive a document — architecture, ADRs, risk register, scope plan — before anyone writes production code. If the proposed flow is "we'll figure it out as we go," you're paying to be the discovery phase.
4. Evals and observability as default deliverables
Ask: "What evals will be in CI when you hand this off?" If the answer is "we'll discuss in scoping," that's a no. Evals and observability should be table stakes, not upsells. The cost of adding them after launch is roughly 3× adding them during the build.
5. Explicit handoff with runbooks
Ask: "Who owns this in month four? What does the handoff document include?" Good answers: written runbooks, dashboard tour, oncall trial-run, eval suite expansion plan. Bad answer: "we'll be available."
The three disqualifiers
1. Vague case studies
"We helped a Fortune 500 client save millions" is nothing. You want: which company, what stack, what specific result, what was the team size, what did it cost. If they can't share specifics under NDA, they likely don't have the specifics.
2. Refusal to commit scope in writing
If they won't write down what's in and what's out before you sign, expect a moving target.
3. A custom framework instead of standard tools
"Our proprietary AI orchestration platform" is usually a bad sign. The standard tools (Kubernetes, Terraform, ArgoCD, LangGraph, OpenTelemetry, MCP) are standard for a reason. Vendor lock-in to a consultancy's framework is worse than lock-in to a major cloud — at least the cloud has open documentation.
The questions we'd want you to ask us
In the spirit of transparency:
- "Show me an architecture you delivered last quarter." Yes. Under NDA, with permission.
- "Show me the runbooks you handed off." Same.
- "What do you do when an engagement goes off the rails?" We've had two tough engagements out of 30+. In both, we paused, did a written reset, refunded one milestone, and finished. We'd tell you their names if they consented; one did, one didn't.
- "Who's actually on the team?" Founder-led. Senior engineers (10+ years). No outsourced delivery. We tell you exactly who's on your engagement before you sign.
- "What can't you do well?" ML training and model fine-tuning at frontier scale. We integrate, we operate, we evaluate — we don't pretrain.
The check we'd run on any partner (including ourselves)
- Ask for an assessment writeup from a recent engagement (redacted).
- Ask for the eval suite they shipped.
- Ask the partner's last client for a 15-minute reference call.
- Ask: "What would you change about how that engagement went?" Anyone who says "nothing" is lying.
If you want to put us through this checklist, book an intro. We'll bring the receipts.