Why vendor choice on image annotation matters more than buyers expect
Computer-vision models hit a quality ceiling set by their labelled training data. MIT's 2021 study of label errors in canonical benchmarks found measurable noise in every one of ten widely-used datasets – including ImageNet, which carried roughly a 6% label-error rate in its test set. If even the public benchmarks have hidden a 1-in-20 error rate for years, the realistic prior on a vendor-produced dataset is that quality varies dramatically across the market.
The cost of a quality mismatch shows up downstream as longer training runs, lower mAP on validation, and silently degraded production performance. The cost of choosing the right vendor up front is a one-time evaluation cycle. The ratio is in favour of doing the evaluation properly. Below is the framework we see successful computer-vision teams use when scoping an image annotation services partnership.
Core annotation types your vendor must support
A credible image annotation vendor should support the full range of computer-vision annotation types. If a vendor only does bounding boxes, they are not equipped for production-grade projects that evolve over time. The schema almost always expands as the model matures – starting at bounding boxes, then adding semantic segmentation for road surface or background, then keypoints for pose estimation, then 3D for depth-aware perception.
- Bounding boxes (2D, axis-aligned and rotated). The baseline for object detection. The complexity driver is class count and per-box attribute density, not the box itself.
- Polygon annotation. For irregular shapes – plants, soft tissue in medical imaging, manufacturing defects – where bounding boxes lose meaningful precision.
- Semantic segmentation. Pixel-level class labelling for scene understanding. The expensive modality used in medical imaging, autonomous driving, satellite imagery, and dense urban perception.
- Instance segmentation. Separating individual objects of the same class – critical for crowd counting, multi-object tracking, and retail-shelf analysis.
- Keypoint annotation. Marking specific points for pose estimation (joints on a human body), facial landmarks (AR, biometrics), hand keypoints (gesture recognition), or industrial-inspection landmarks.
- Image classification and multi-label tagging. The simpler end of the spectrum – assigning one or more labels per image. Volume-heavy, accuracy-sensitive on the rare classes.
- Lane marking, drivable-area, and road-feature annotation for ADAS and autonomous-driving perception. Often combined with 3D point cloud work in a unified scene-understanding programme.
- Cuboid annotation for 3D bounding boxes – the bridge between 2D image work and full 3D point-cloud annotation, particularly for AV programmes that fuse camera and LiDAR data.
- OCR and document image annotation for structured extraction. Heavily used in financial, legal, healthcare, and government workflows.
Quality assurance – the single most important differentiator
Quality is where most annotation vendors differentiate or fail. Ask every shortlisted vendor to describe their QA process in detail. Vague answers ("we have quality checks") are a strong red flag – they almost always indicate a single-pass workflow with no formal inter-annotator agreement protocol.
A rigorous QA programme has six observable artefacts. Ask for evidence of each one, not just the promise:
- Versioned annotation guidelines, including worked examples for hard cases and a documented schema in source control. A vendor that cannot show you the guideline for their last project will not produce a defensible dataset on yours.
- Annotator training and calibration against a gold panel before any production batches start. New annotators score against the panel; the vendor publishes the calibration scores.
- Inter-annotator agreement (IAA) measurement on a stratified sample of every batch. The metric should be Cohen's kappa, Krippendorff's alpha, or per-class F1 against a gold panel. Reporting should be per-class, not just headline-averaged.
- Multi-pass review: annotator self-check, peer review, senior-reviewer adjudication on the decision boundary. Disagreements are logged, not silently overwritten.
- Disagreement-cluster reports per batch – the classes and cases where reviewers disagreed most often. This is the highest-leverage QA signal in any annotation programme, and the artefact buyers most often forget to ask for.
- A versioned gold panel of 200–1,000 adjudicated examples that travels with the project. Used to score new annotators at onboarding, detect drift over time, and document the dataset for audit.
Tooling and format compatibility
Your annotation vendor should be tool-agnostic or support the formats your ML pipeline requires. Common delivery formats include COCO JSON, Pascal VOC XML, YOLO TXT, BIO tags, and custom schemas. Ask whether the vendor can work within your existing tooling (Labelbox, SuperAnnotate, V7, Encord, CVAT, Scale Nucleus, Roboflow, Label Studio) or deliver in your required output format.
For multimodal programmes – image plus document, image plus 3D point cloud, image plus video – ask explicitly which combinations they have shipped. A vendor experienced with single-modality work may struggle to maintain cross-modal label consistency on a unified schema.
Two operational signals are easy to miss: how the vendor handles schema migration mid-project (every long-running programme has at least one), and how they version the gold panel across schema changes. The vendor who has a documented playbook for both is meaningfully more reliable than the one who improvises.
Scale and throughput
Annotation needs spike. A new training run, a dataset expansion, a new product vertical, a regression batch after the model fails on new data – production AI is a series of these spikes, not a flat baseline. Your vendor needs to scale with you.
Ask three specific questions. First, what is your current annotator headcount and how many of them have shipped work in my specific data type (medical imaging, AV perception, satellite, retail, etc.). Second, how quickly can you ramp a 10× volume increase – two weeks, four weeks, eight weeks? Third, how do you staff specialist tasks like clinician-reviewed medical imaging or AV perception with safety-driver context.
A mature mid-market vendor like a Vietnam-based data annotation services pod typically scales from a 5–10 annotator pilot to a 100–200 annotator production programme in two to four weeks, assuming the schema is stable. Specialist headcount (medical reviewers, AV perception leads) takes longer and is the bottleneck on regulated programmes.
Data security and confidentiality
Image datasets for unreleased products, proprietary medical scans, defence imagery, customer-generated content, or financial KYC documents require strict confidentiality controls. The vendor's security posture is not a checklist – it is an audit-worthy artefact. Ask for these specifically:
- NDA and DPA signed before any sample data is shared. A vendor that wants data first and paperwork later is the wrong vendor.
- ISO 27001 alignment, with current certification or a written statement of alignment from a senior leader. SOC 2 readiness for regulated US-facing work.
- Annotator access controls: named-individual logins, no shared accounts, full audit trail of who labelled what.
- Work-from-secure-room policy for sensitive projects: no personal devices, no mobile phones, no remote-from-home access. For medical and defence work this is increasingly standard.
- Data storage and transmission security: encrypted at rest and in transit, signed-URL or VPN-bound delivery, no S3-public buckets ever.
- On-premise or VPC-only deployment option. For PII, medical, financial, and regulated documents this should be available as a first-class engagement model, not a special case.
- Post-project data deletion with a written deletion certificate – default within 30 days of project close unless audit requires longer retention.
Pricing model and contract structure
A clean image annotation contract priced by task type aligns incentives between buyer and vendor. Three things to look for in the proposal:
- Per-task pricing with no blended rates. A separate line for bounding boxes, polygons, segmentation, keypoints, and specialist work. Blended rates almost always mean the simple tasks subsidise the complex ones – fine for some buyers, expensive for others.
- Rework policy in writing. The vendor reworks any batch that misses the SLA at their cost; the buyer pays only for batches that pass. Vague rework clauses end in invoice disputes.
- No minimum monthly commit for pilot or first-quarter engagements. A vendor confident in their quality lets the buyer ramp at their own pace; a vendor that needs a minimum commit is hedging.
Questions to ask before signing
Take this list into the vendor call rather than reading it back from a deck after the fact. The conversation that comes from these questions reveals more than any written proposal:
- Can you share sanitised samples of previous image annotation work in my specific domain (or, for regulated work, can you arrange a reference call with a similar client)?
- What is your inter-annotator agreement protocol, and can I see a sample batch report?
- What is your average turnaround time for a 10,000-image bounding-box pilot, end to end, from NDA signature?
- Do you support annotation of proprietary or sensitive images under NDA, and can you deploy inside our VPC or on-premise environment if needed?
- What annotation tools do your teams use, and can you integrate with our pipeline (Labelbox, V7, CVAT, Label Studio, custom)?
- How do you handle disagreement between annotators on ambiguous cases? Who adjudicates, and is the decision logged?
- How do you scale from pilot to production – specifically, what is your published timeline for going from 10 annotators to 100?
- What happens at end of engagement – data deletion certificate, knowledge transfer of guidelines and gold panel, formal handover audit?
Red flags – patterns that consistently predict bad engagements
The patterns that come up before the worst image-annotation engagements we have observed. Any one of these is a strong negative signal:
- Vendor declines a paid pilot or pushes for a multi-month commit before any sample work has been done.
- QA description in the proposal does not name a specific IAA metric, gold-panel size, or review-pass count.
- No willingness to share sanitised samples or arrange a reference call with a similar client.
- Per-asset rate dramatically below the market floor with no explanation of how quality is sustained at that price.
- No dedicated project manager named on the engagement. The buyer ends up coordinating with annotators directly.
- Vendor cannot describe how they handle the migration from one labelling platform to another – a near-universal need in long-running programmes.
Frequently asked questions
A short reference for the questions enterprise computer-vision teams ask most often when scoping an image annotation services engagement:
- What is the right way to pilot an image annotation vendor? Send a 200–500 image representative sample with the schema you intend to use in production. Score the returned batch against your own 50–100 image gold panel. Compare two or three vendors apples-to-apples on the same sample.
- How long does a pilot take? Five to ten business days from NDA signature in mature vendors. Anything materially longer than that on a 500-image batch suggests operational immaturity.
- Can a Vietnam-based pod deliver on regulated medical or AV-perception work? Yes – with the right specialist-reviewer panel on the QA tier and an on-premise or VPC deployment option. The pattern is well-established for both modalities.
- How do I avoid paying for rework? Negotiate a rework clause that obliges the vendor to redo any batch that misses the SLA at their cost. Define the SLA metric explicitly (per-class IAA or per-class F1 against the gold panel) before signing.
- What is the typical cost difference between a per-asset rate and a per-hour rate for the same image annotation work? Per-asset rates work out cheaper for well-defined high-volume work; per-hour rates work out cheaper for complex schemas with significant guideline iteration. The right test is to scope a paid pilot under each model and compare the true cost per labelled asset at the end.
