Step 1: define your annotation requirements before talking to vendors
Before reaching out to vendors, document exactly what you need. Vendors who receive a clear brief respond faster, quote more accurately, and produce better results. Your brief should include: data type and volume, annotation task definition (with examples), accuracy requirements, file format for delivery, timeline, and any domain-specific knowledge requirements.
- Data type: images, text, video, audio, 3D point cloud, or mixed.
- Task definition: bounding boxes, NER, sentiment, segmentation – with annotated examples.
- Accuracy target: minimum inter-annotator agreement percentage.
- Delivery format: JSON, CSV, COCO, Pascal VOC, custom schema.
- Volume and timeline: total assets and weekly delivery cadence.
- Domain expertise needed: general, medical, legal, automotive, etc.
Step 2: shortlist 3–5 vendors
Request quotes from at least three vendors. Evaluate them on: experience with your specific data type, QA process transparency, data security practices, communication responsiveness, and references from comparable projects. Beware of vendors who promise the lowest price with no explanation of how they maintain quality at that cost.
Step 3: run a paid pilot
Never commit to a large annotation project without a pilot. A pilot of 200–500 items gives you real accuracy data, reveals workflow gaps, tests communication, and confirms the vendor understands your task. Pay for the pilot – free pilots often receive less attention and do not reflect production quality.
- Define a clear acceptance criterion before the pilot starts (e.g., 95%+ accuracy on your gold standard).
- Annotate 10–20% of the pilot samples yourself as a gold standard for comparison.
- Measure inter-annotator agreement, not just final accuracy.
- Review edge cases – these reveal whether annotators understood the task deeply.
Step 4: structure the contract properly
Key contract elements to negotiate: data confidentiality and NDA, IP ownership (your data, your labels), SLA for accuracy and delivery timelines, rework policy (who pays when quality falls short), data deletion post-project, and pricing model (per-item, per-hour, or fixed project).
Step 5: build a feedback loop
The best annotation partnerships improve over time. Share model performance feedback with your vendor – when your model struggles on specific data types or edge cases, that signals annotation gaps. Regular calibration sessions between your ML team and the annotation team create a continuous quality improvement cycle.
Red flags when evaluating annotation vendors
These are the patterns that consistently predict an engagement that will end badly. Walk away if you see them:
- No pilot process – reputable vendors welcome pilots.
- Vague QA descriptions – "we have quality checks" without specifics.
- Unwilling to share sample work or client references.
- No NDA or data security documentation available.
- Quotes without seeing your data.
- No dedicated project manager – you will be managing the annotation team yourself.
