Why most annotation contracts are dangerously thin
Enterprise AI teams routinely spend months selecting a data annotation vendor – evaluating quality scores, reviewing case studies, negotiating per-label pricing – and then sign a contract that fits on two pages. That contract rarely defines what "98% accuracy" actually means, who owns the data during the engagement, or what happens when the vendor misses a deadline.
The result: disputes surface mid-project, when renegotiating is expensive and switching is nearly impossible. Fixing quality issues in production costs ten times more than getting the annotation right the first time, and fixing contract terms once work has begun costs even more in goodwill and project momentum.
This guide covers every clause your data annotation SLA should include, written from the buyer side, with specific language and benchmarks to request.
Quality accuracy clauses: define the metric, not just the number
The most common SLA clause is an accuracy target – typically "95% accuracy" or "98% accuracy". These numbers are meaningless without defining the measurement method. Ask your vendor to specify all of the following in the contract.
- Inter-annotator agreement (IAA) method: Cohen's Kappa, Krippendorff's Alpha, or simple percent agreement – each gives different results for the same dataset.
- Gold standard benchmark: who creates the ground truth set, how large it is (minimum 200–500 items), and how often it is refreshed.
- Sampling rate: what percentage of production batches are audited against the gold set (industry standard: 5–10% per batch).
- Accuracy floor vs. average: is the 98% target a per-batch floor or a project average? A per-batch floor is far more protective.
- Defect classification: distinguish between critical errors (wrong label class), minor errors (bounding box 5% off), and cosmetic errors (annotation note formatting). Only critical errors should trigger remediation SLAs.
- Measurement frequency: daily, per batch, or per milestone – weekly is the minimum acceptable for production runs.
Turnaround time commitments: tiers and escalation paths
Turnaround SLAs should reflect your actual business cadence, not the vendor's preferred batch size. Structure turnaround commitments in tiers based on urgency, and require the contract to specify what happens when those tiers are missed.
A standard tier structure for data annotation services:
- Standard tier (default): delivery within 3–5 business days for batches up to 10,000 items.
- Priority tier (surge): delivery within 24–48 hours for up to 2,000 items, at a 20–40% rate premium.
- Bulk tier (>50,000 items): milestone-based delivery schedule agreed per project with weekly progress checkpoints.
- Late delivery penalty: require a specific penalty clause – typically a 5% credit per business day late, capped at 25% of the batch value.
- Force majeure carve-out: clearly define what events excuse lateness (power outage, natural disaster) vs. what does not (internal staffing shortages).
Revision and rework policy: who pays when quality fails
Every vendor contract needs a rework clause that answers three questions: at what accuracy level does rework trigger, who bears the cost, and within what timeframe must rework be completed.
Industry-standard rework terms to negotiate:
- Rework trigger threshold: if batch accuracy falls below the contracted SLA floor, the vendor reworks the affected batch at no additional charge.
- Rework turnaround: rework should be completed within 50% of the original turnaround time for that batch tier.
- Root cause report: for any batch requiring rework, require a written root cause analysis within 48 hours of identification.
- Escalation to replacement: if two consecutive batches fall below the accuracy floor, the contract should grant the right to engage a secondary vendor at the primary vendor's cost for the duration of remediation.
- Cap on free rework cycles: clarify whether unlimited rework is covered or whether a maximum of 2 rework passes is included, with additional passes billed at cost.
Data security and compliance clauses
Data security SLAs are non-negotiable for any project involving personally identifiable information (PII), medical records, financial data, or proprietary product imagery. These are the minimum certifications and obligations to specify in writing.
- ISO 27001 certification: require proof of current certification (not just a claim), including scope of certification and most recent audit date.
- GDPR / PDPA / HIPAA applicability: explicitly state which regulation governs the engagement based on data type and origin geography.
- Data residency: specify the country or region where data may be stored and processed. Vietnam-based vendors should be able to confirm data does not leave agreed jurisdictions without written consent.
- Annotator NDA: require that all annotators working on your data sign individual NDAs, not just the vendor entity.
- Data deletion protocol: define the timeline and method for secure data deletion upon project completion (maximum 30 days after final delivery, certificate of deletion provided).
- Breach notification: require notification within 24–72 hours of discovery of any suspected breach, regardless of confirmed impact.
- Sub-processor disclosure: vendor must disclose any third-party tools or platforms used to process your data (annotation platforms, cloud storage providers).
Pricing, payment terms, and scope change clauses
Pricing disputes are the second most common source of annotation project failures after quality issues. Protect yourself with explicit terms on what is included in the quoted rate and how scope changes are handled.
- All-inclusive vs. base rate: confirm whether the quoted rate includes QA review, project management, tooling fees, and data transfer, or whether those are billed separately.
- Volume commitment and minimum: many vendors require a minimum volume commitment (e.g., 5,000 items/month). Understand the penalty for falling below it.
- Rate lock period: negotiate a rate lock for the duration of the initial project (typically 6–12 months). Annual CPI-linked adjustments are reasonable; ad hoc increases mid-project are not.
- Change order process: any scope change (new annotation type, additional attributes, format change) must be formalized in a written change order before work begins, with a revised rate and timeline.
- Milestone payment schedule: for large projects, link payments to delivery milestones and quality acceptance, not just calendar dates.
Data ownership, IP, and model training rights
One of the most frequently overlooked contract sections is IP ownership. The default in many vendor contracts is ambiguous – intentionally so. Make these terms explicit.
- Raw data ownership: the client retains full ownership of all input data at all times. The vendor receives a limited, revocable license to process the data solely for annotation purposes.
- Annotation output ownership: all labeled datasets produced under the engagement are owned by the client upon delivery and payment.
- Prohibition on training use: explicitly prohibit the vendor from using your data – raw or annotated – to train any internal models, improve their own tooling, or share with third parties.
- Work product assignment: in jurisdictions where annotators may have creator rights, the contract should include a work-for-hire clause assigning all rights to the client.
- Survivor clause: data ownership and prohibition on use clauses must survive termination of the contract.
Pilot project clause: the single best contract protection
The most effective risk mitigation in any annotation vendor contract is a structured pilot clause. Before committing to a full production volume, require a paid pilot run of 200–500 items at standard production conditions – same annotators, same tooling, same QA process that will apply to the main engagement.
The pilot results should trigger a binary decision: if accuracy meets the SLA floor, the main engagement proceeds automatically. If it does not, the client may terminate without penalty, or negotiate an extended pilot with a revised quality plan.
A well-structured pilot clause costs the vendor nothing if they are confident in their process and saves the client from a much larger mistake. Any vendor that resists a pilot clause is telling you something important about their confidence in their own quality.
