The two questions that decide annotation vendor risk
Most annotation vendor evaluations spend their energy on price per label and a portfolio of sample work. Both matter, but neither predicts whether the engagement survives contact with production. The two areas that do are the performance SLAs a vendor is willing to commit to in writing, and the security posture they can prove with artefacts rather than assurances.
These two areas are also where vendor pitches are weakest under questioning. Any vendor will claim "98% accuracy" and "enterprise-grade security" on a sales call. The job of your evaluation is to turn those claims into specific, measurable commitments before a contract exists - because once work has started, renegotiating an SLA or discovering a security gap costs far more than asking the right question up front.
This is a buyer-side interview script. It covers the SLA questions that reveal whether a vendor can actually deliver at the quality and cadence they promise, and the security questions that reveal whether your training data is safe in their hands. For each, we include what a strong answer sounds like and the red flag that should make you pause. Use it before the contract; the answers become the terms you negotiate.
How to run the evaluation: get every answer in writing
Two rules make this checklist work. First, ask for evidence, not policy. A vendor describing a "rigorous QA process" tells you nothing; a vendor showing you a real inter-annotator agreement report from a comparable project tells you everything. Every question below has an artefact the vendor can produce if the capability is real.
Second, put the answers in writing during evaluation, not after. Verbal commitments on a sales call do not survive the handoff to the delivery team. The questions that produce a specific number, threshold, or named certificate are the ones worth building your SLA around. Anything the vendor will only commit to verbally is a signal in itself.
The checklist splits into two parts: performance SLAs (questions 1 to 5) and security posture (questions 6 to 11). Both parts matter for every engagement, but weight them to your data. A public-domain image project leans on the SLA questions; a project involving PII, medical, financial, or unreleased-product data makes the security questions load-bearing.
1. What accuracy do you guarantee, and how exactly do you measure it?
This is the SLA question every other quality term depends on, and the one where vague answers hide the most risk. A number without a measurement method is meaningless: "98% accuracy" measured as raw percent-agreement on an easy class distribution is a weaker guarantee than "92% F1" measured per class against an adjudicated gold set. Push past the headline number to the method.
Ask the vendor to specify, in writing, all of the following:
What a strong SLA answer sounds like versus a red flag, across the five performance questions
| Question | Strong answer | Red flag |
|---|---|---|
| Accuracy metric | Per-class F1 or kappa against a refreshed gold set, per-batch floor | A single "98%" with no metric or measurement method named |
| Turnaround | Tiered TAT with a named late penalty and escalation path | "As fast as possible" or best-effort with no penalty clause |
| Rework | Free rework below the SLA floor plus a root-cause report | Rework billed as new work, or unlimited-scope disputes |
| Scale | Named ramp time and gold-panel gate for new annotators | Headline headcount only, no onboarding-to-production time |
| Reporting | Per-batch quality dashboard with disagreement clusters | A summary email at project end, no in-flight visibility |
- The metric itself: inter-annotator agreement (Cohen's kappa, Krippendorff's alpha), per-class F1, or pixel-level accuracy for segmentation - not just an undefined "accuracy" percentage.
- The ground truth: who builds the gold-standard set, how large it is (200 to 1,000 adjudicated items is typical), and how often it is refreshed as the schema evolves.
- The audit sampling rate: what percentage of every production batch is checked against the gold set (5 to 10% per batch is the industry baseline).
- Floor versus average: whether the target is a per-batch floor or a project-wide average. A per-batch floor is far more protective - an average lets a vendor hide bad batches behind good ones.
- Defect severity: whether critical errors (wrong class) are counted the same as cosmetic ones (a bounding box 3% loose). Only critical errors should trigger remediation.
2. What turnaround do you commit to, and what happens when you miss it?
Turnaround time (TAT) is where a vendor either respects your model-development cadence or quietly sets their own. The commitment should be tiered by urgency and, critically, should specify the consequence of a miss. A TAT with no penalty is a preference, not a commitment.
A defensible turnaround SLA structures commitments in tiers and attaches a real cost to lateness:
- Standard tier: delivery within 3 to 5 business days for batches up to roughly 10,000 items, as the default cadence.
- Priority tier: 24 to 48 hours for smaller surge batches, at a stated rate premium so both sides know the cost of urgency in advance.
- Bulk tier: milestone-based schedule for very large volumes, with weekly progress checkpoints rather than a single deadline.
- Late-delivery penalty: a specific credit (a common structure is 5% of batch value per business day late, capped at 25%), not a vague promise to "make it right".
- Escalation path: who you call when a batch is at risk, and how far in advance the vendor commits to flagging a slip.
3. How do you handle rework when a batch fails QA, and who pays?
Every annotation engagement produces a batch that misses the bar eventually. What separates a professional vendor is whether the rework path is defined before it is needed. The rework question has three parts: what triggers it, who bears the cost, and how fast it is fixed.
The answers you want to hear, and hold the vendor to:
- Trigger: any batch that falls below the contracted accuracy floor is reworked, measured by the same gold-set method from question 1.
- Cost: rework to reach the agreed SLA is at the vendor's expense, not billed as new work. This is the single most important word in the answer.
- Speed: rework completes within roughly 50% of the original TAT for that tier, so a failed batch does not stall the pipeline for a full new cycle.
- Root cause: any reworked batch comes with a written root-cause analysis within 48 hours, so the same failure does not recur silently.
- Cap and escalation: whether free rework is unlimited or capped at two passes, and what happens (secondary vendor, termination right) if two consecutive batches fail.
4. Can you hold quality as volume scales, and how fast can you ramp?
Total annotator headcount is the number vendors lead with and the least useful one. Two vendors with 500 annotators can have completely different effective capacity for your task depending on domain coverage, language coverage, security tiering, and current project load. The scaling questions that actually predict delivery are operational, not headline.
Ask specifically:
- Ramp time: how quickly the vendor can go from 10 to 50 annotators on your specific task, and whether the ramp is gated by hiring, training, or guideline calibration - each has a different timeline.
- Onboarding-to-production: the typical time for a new annotator to reach production, and the gold-panel score they must hit before their labels ship.
- Quality-at-scale evidence: a per-class quality trend from a past project as it scaled, showing accuracy held (or where it dipped and how they recovered).
- Retention: a 12-month annotator-retention statistic for the pool that would staff your work - churn resets calibration and is a hidden quality tax.
- Bottleneck handling: how reviewer load is redistributed when a single senior reviewer becomes the constraint on throughput.
5. How will I see quality while work is in flight, not after?
A vendor who reports quality only at project end has removed your ability to intervene when it matters. Real-time visibility is both an SLA term and a trust signal: vendors confident in their process expose the numbers; vendors who are not, summarise them after the fact.
The reporting artefacts to require:
- A per-batch quality dashboard showing accuracy against the gold set, refreshed as batches land - not a single end-of-project summary.
- Disagreement-cluster reports: the classes and edge cases where reviewers disagreed most. This is the highest-value QA signal and the one buyers most often forget to ask for.
- An audit trail linking every label to the annotator and reviewer who produced it, available on request rather than only after an incident.
- A named cadence for review calls (weekly is standard for production runs) where quality trends and schema questions are worked through together.
6. What security certifications can you prove, not just claim?
This begins the security half of the evaluation, and the framing matters: a certification is only worth the audit behind it. Ask for the certificate and its scope, not a logo on a slide. A vendor "aligned with" or "working toward" a standard is telling you they do not hold it.
The baseline security credentials for enterprise annotation work:
- ISO/IEC 27001 certification (common and well-accepted among APAC vendors) or a SOC 2 Type II report - with the certificate, the scope of certification, and the most recent audit date, not just a claim.
- A signed NDA and a Data Processing Agreement (DPA) in place before any sample data changes hands, not after the contract is signed.
- Named certification scope: confirm the certificate covers the delivery site and workforce that will actually handle your data, not a different corporate entity or office.
- For regulated data, the specific instrument: a HIPAA Business Associate Agreement for healthcare data, or documented controls mapped to GDPR, PDPA, or the relevant regime for your jurisdiction.
7. Where does my data live and move - and can you keep it in-region?
Data residency is where security meets regulatory exposure. For many APAC and EU engagements, the location where data is stored and processed is not a preference but a compliance requirement. A vendor without a clear answer here is a vendor that has not thought about your regulatory position.
The residency and sovereignty questions to pin down:
- The specific country or region where your data is stored and where annotation is physically performed - both, since they can differ.
- A written commitment that data does not leave the agreed jurisdiction without your explicit consent, including for backup, tooling, or overflow capacity.
- Sub-processor disclosure: every third-party tool or platform that touches your data (annotation platform, cloud storage, any offshore overflow team), named in writing.
- For EU or APAC personal data, confirmation that the residency arrangement satisfies the applicable regime (GDPR, Vietnam's PDPD, Singapore's PDPA, and equivalents).
8. Who can access my data, and how is that access controlled?
Certifications describe intent; access controls describe daily reality. The most common gap between a vendor with the right certificate and real security maturity is in the day-to-day controls on who can see your data and from where. These operational questions surface that gap.
What to ask about workforce and access security:
- Named-individual logins with no shared accounts, so every action traces to a person - a precondition for a meaningful audit trail.
- Role-based access control and multi-factor authentication on the annotation platform and any data storage.
- A secure-room policy for sensitive work: no personal devices, no mobile phones, no remote-from-home access on the most sensitive subset - and whether that policy is enforced or aspirational.
- Individual annotator NDAs (not just an entity-level NDA), so confidentiality obligations reach the people actually handling the data.
- Client access to annotator-access logs on request, not only after an incident.
9. How is data protected at rest, in transit, and in the annotation environment?
Encryption and environment isolation are the technical floor of data security. They are also easy to claim and easy to verify, so a vendor should answer these crisply. Vagueness here is disqualifying for any sensitive engagement.
The technical controls to confirm:
- Encryption at rest (AES-256 or equivalent) and in transit (TLS) across storage, transfer, and the annotation platform.
- Environment isolation for sensitive projects: VPC-only or on-premise deployment offered as a first-class engagement model, not a bolt-on.
- Segregation between clients, so your data is not co-mingled with other engagements on shared infrastructure without isolation.
- A documented, time-bound data-deletion protocol on project completion (30 days is a common maximum), with a written certificate of deletion provided.
- Whether any data is used to train the vendor's own models or improve their tooling - which should be explicitly prohibited in writing.
10. What is your incident response and breach-notification commitment?
No security posture is perfect, so the real test is what happens when something goes wrong. A vendor with a rehearsed incident-response answer has thought about failure; a vendor who improvises the answer on the call has not. This is one of the most revealing questions in the entire evaluation.
The incident-response commitments to get in writing:
- A breach-notification window: notification within 24 to 72 hours of discovering any suspected breach, regardless of whether impact is yet confirmed.
- A defined escalation contact and process, so notification reaches your security team directly rather than through account management.
- A documented incident-response plan the vendor can describe: containment, investigation, client notification, and remediation steps.
- A right-to-audit provision, so you can verify controls during the engagement rather than trusting the annual certificate.
- Post-incident obligations: root-cause analysis and remediation evidence, not just an apology.
11. For regulated data, can you meet my industry's specific obligations?
General security maturity is necessary but not sufficient when your data is regulated. Healthcare, financial, defence, and personal-data projects carry obligations that a generically secure vendor may still fail. This final question tailors the evaluation to your specific exposure.
Match the question to your data type:
- Healthcare and medical imaging: a HIPAA Business Associate Agreement, and where relevant, local clinical-data rules and clinician-reviewer access controls.
- Financial and KYC documents: SOC 2 Type II, and familiarity with the financial-data regime in your jurisdiction (MAS in Singapore, and equivalents).
- Personal data (PII): a DPA mapped to the governing regime (GDPR, PDPD, PDPA), lawful-basis handling, and data-subject-rights support.
- Defence, proprietary, or unreleased-product imagery: secure-room execution, on-premise or VPC-only deployment, and the tightest access tier the vendor offers.
Turn the answers into contract terms
The point of this interview is not the conversation - it is the contract that follows it. Every specific answer you extract (the accuracy metric, the TAT penalty, the rework cost allocation, the deletion window, the breach-notification clock) becomes a clause you negotiate before work begins. Answers you could only get verbally are the clauses a vendor will resist putting in writing, which tells you where the risk sits.
A useful final filter: a vendor confident in their SLAs and security will welcome these questions and answer with artefacts, because the questions let them differentiate from weaker competitors. A vendor who treats the questions as friction is showing you how the engagement will feel when something goes wrong. That reaction is itself a data point worth as much as any single answer.
Once you have the answers, the next step is codifying them. Our companion guide on annotation SLAs walks through the exact contract clauses - accuracy definitions, penalty structures, IP ownership, and the pilot clause that de-risks the whole engagement - so the commitments you gathered here survive into an enforceable agreement.
DataX Power answers every question in this checklist with documented artefacts - ISO/IEC 27001 aligned operations, per-batch quality reporting, defined SLA and remediation terms, and data-residency options across APAC. Our engagement terms are structured for enterprise procurement and security review.
See DataX Power's annotation engagement terms- What are the most important SLA questions to ask an AI annotation vendor?
- The five that matter most are: (1) what accuracy metric you guarantee and how it is measured against a gold set, (2) what tiered turnaround you commit to and the penalty for missing it, (3) how rework is triggered and who pays when a batch fails QA, (4) how quickly you can scale annotators while holding quality, and (5) how quality is reported in real time rather than only at project end. Each answer should come with an artefact - a real IAA report, a sample dashboard, a penalty clause - not just a verbal assurance.
- What security requirements should an AI data annotation vendor meet?
- At a baseline: current ISO/IEC 27001 certification or a SOC 2 Type II report with named scope, a signed NDA and DPA before any data is shared, data-residency commitments that keep data in the agreed jurisdiction, named-individual access with MFA and role-based controls, encryption at rest and in transit, environment isolation (VPC or on-premise) for sensitive work, a 24-to-72-hour breach-notification window, and a time-bound deletion certificate on completion. For regulated data, add the specific instrument - a HIPAA BAA for healthcare, or a DPA mapped to GDPR, PDPD, or PDPA for personal data.
- How do I verify an annotation vendor can actually meet their SLA and security claims?
- Ask for evidence rather than policy. For every SLA claim, request an artefact from a comparable past project: an inter-annotator agreement report, a per-batch quality dashboard, a versioned guideline, a sample penalty clause. For every security claim, request the certificate and its scope, the DPA, sub-processor disclosures, and access-log samples. Capabilities that are real can be shown pre-contract; capabilities that only exist on the sales slide cannot. Then put every specific answer into the contract so the commitment is enforceable.
