Core annotation types your vendor must support
A credible image annotation vendor should support the full range of computer vision annotation types. If a vendor only does bounding boxes, they are not equipped for production-grade projects that evolve over time.
- Bounding boxes (2D and rotated) – the baseline for object detection.
- Polygon annotation – for irregular shapes where bounding boxes lose precision.
- Semantic segmentation – pixel-level class labeling for scene understanding.
- Instance segmentation – separating individual objects of the same class.
- Keypoint annotation – for pose estimation and facial landmark detection.
- Image classification and multi-label tagging.
- Lane marking and road feature annotation for ADAS.
Quality assurance: the most important factor
Quality is where most annotation vendors differentiate – or fail. Ask every vendor to describe their QA process in detail. Vague answers ("we have quality checks") are a red flag. A rigorous process should include: annotator training and calibration, gold standard benchmarking, inter-annotator agreement measurement, and multi-pass review.
- What is your minimum inter-annotator agreement threshold?
- How do you handle edge cases or ambiguous labels?
- What percentage of annotations receive senior review?
- How do you measure and report accuracy to clients?
- What happens when annotation quality falls below the agreed SLA?
Tooling and format compatibility
Your annotation vendor should be tool-agnostic or support the formats your ML pipeline requires. Common delivery formats include COCO JSON, Pascal VOC XML, YOLO TXT, and custom schemas. Ask whether the vendor can work within your existing tooling (Label Studio, Labelbox, CVAT, Roboflow) or deliver in your required output format.
Scale and throughput
Annotation needs often spike – a new training run, a dataset expansion, or a new product vertical. Your vendor needs to scale with you. Ask about their annotator headcount, how quickly they can ramp for a 10x volume increase, and how they staff specialist tasks like medical imaging or autonomous vehicle annotation.
Data security and confidentiality
Your training data is a competitive asset. Image datasets for unreleased products, proprietary medical scans, or customer-generated content require strict confidentiality controls. Ask vendors about NDA policies, annotator access controls, data storage and transmission security, and post-project data deletion.
Questions to ask before signing
Take this list into the vendor call rather than reading it back from a deck after the fact:
- Can you share samples of previous image annotation work in a similar domain?
- What is your average turnaround time for a 10,000-image bounding box project?
- Do you support annotation of proprietary or sensitive images under NDA?
- What annotation tools do your teams use, and can you integrate with our pipeline?
- How do you handle disagreements between annotators on ambiguous cases?
