Why NLP annotation is a linguistics problem first
Image annotation is fundamentally a perception problem: what is in this picture, and where? NLP annotation is fundamentally a linguistics problem: what does this text mean, and according to whose interpretation? The difference is what makes NLP annotation harder than it looks, and what makes the schema design phase load-bearing for the entire project.
A 5% label-error rate on a bounding-box dataset typically degrades model performance by a measurable but bounded amount. A 5% label-error rate on an NER or sentiment dataset can collapse the model in ways that are harder to detect: the model learns the labelling noise as a feature rather than treating it as random error, and the production failures cluster on the exact text patterns the annotation team disagreed about. Operationally, the cost of this is much higher than it looks on the per-batch quality dashboard.
The framework that follows describes the eight primary NLP annotation techniques used in enterprise programmes in 2026, the language-coverage decisions that determine whether the dataset transfers across markets, and the quality discipline that turns a noisy text dataset into one a model can actually generalise from.
Text classification
The simplest NLP annotation pattern: assign one or more class labels to a document, paragraph, or sentence. Common applications include spam detection, topic categorisation, content moderation, support-ticket routing, document-type classification for downstream extraction pipelines, and the broad class of pre-filtering classifiers that route text to more expensive downstream models.
The principal quality concern is ambiguous-case handling. A message that is both a complaint and a product inquiry needs an explicit escalation rule in the guideline: which class wins, or is the schema multi-label? A document that is partially in English and partially in Vietnamese needs an explicit language-tag rule. The guideline either decides up front or the annotators decide on the fly – and on-the-fly decisions are how a class boundary silently fragments across batches.
For taxonomy-based classification (the 200-class product-category classifier, the 50-class intent classifier for a regional chatbot), the failure mode is class proliferation. Taxonomies with overlapping or near-overlapping classes produce per-class IAA that varies wildly, with the overlapping classes producing the most disagreement. The defensible fix is taxonomy curation at the schema stage – fewer, cleaner class definitions consistently outperform larger, noisier taxonomies in production.
Named Entity Recognition (NER)
NER involves labelling specific spans of text as entity types: PERSON, ORGANISATION, LOCATION, DATE, PRODUCT, MONEY, EVENT, and the broader class of custom domain-specific entities (PROTEIN and GENE in biomedical NER, DIAGNOSIS and MEDICATION in clinical NER, CASE_NAME and STATUTE in legal NER).
NER annotation requires annotators who understand context, not just surface patterns. "Apple" is a company in a tech article and a fruit in a recipe; "Washington" is a person, a state, or a city depending on the surrounding text. The guideline has to specify the context-resolution rule for every ambiguous entity in the domain, with worked examples for the cases the annotators are most likely to encounter.
The principal operational concerns are span boundary precision and overlapping-entity handling. Span boundary precision (whether "President Joe Biden" is one entity or two) varies across schemas; the defensible decision is to specify the convention in the guideline (typically: include titles in the entity for PERSON, exclude possessives, include co-references when they appear in apposition) and reinforce it with worked examples for every entity type. Overlapping entities – a phrase that is both a LOCATION and a PRODUCT – require an explicit schema decision about whether entities can nest or must be flat.
Sentiment and emotion analysis
Beyond the standard positive/negative/neutral classification, modern sentiment annotation typically expands across three orthogonal dimensions: aspect-based sentiment (what specific feature is the user reviewing – battery life, camera quality, customer service), intensity scoring (5-star scale rather than binary), and emotion categorisation (anger, joy, fear, surprise, disgust, contempt, sadness from the Ekman taxonomy).
This level of granularity requires annotators who are native or near-native speakers of the target language. Sentiment is heavily culturally and linguistically context-dependent: a Vietnamese product review uses different sarcasm and understatement conventions than a US review, and a Bahasa Indonesia comment thread may signal disagreement through politeness conventions that look neutral to a non-native annotator. The defensible model is native-speaker annotation with bilingual reviewer adjudication on the hard cases.
For multi-aspect sentiment (the standard for product-review NLP work), the schema decision is whether aspects are pre-defined or open. Pre-defined aspect lists (typically 8–20 per product category) produce cleaner per-aspect models but require taxonomy curation before annotation starts. Open aspect extraction is more flexible but materially harder to evaluate and slower to annotate; in practice, most production programmes use a hybrid where the most common aspects are pre-defined and the long tail is captured under an "other" category.
Intent classification and slot filling for conversational AI
Building chatbots, virtual assistants, voice-commerce systems, and the broader class of conversational AI requires two parallel annotation streams. Intent classification labels each user utterance with what they want to do ("BookFlight", "CheckBalance", "TransferFunds", "EscalateToAgent"). Slot filling extracts the structured arguments inside the utterance (departure city, date, account number, amount).
The principal challenge is utterance coverage. The same intent ("CheckBalance") can be phrased thousands of different ways across users, typos, slang, abbreviations, and code-switched bilingual inputs. A defensible intent dataset has to cover the full diversity of phrasings the production system will encounter, which is materially more annotation effort than the headline intent count suggests.
For multilingual production systems, the schema choice is whether intents are language-agnostic (the same intent ID for "Check my balance" in English and "Kiểm tra số dư của tôi" in Vietnamese) or language-specific. The cleaner pattern is language-agnostic intents with language-specific utterance examples – this lets the model learn one intent space across languages while still capturing language-specific phrasing patterns.
Coreference and span-level relation annotation
Coreference resolution identifies when different words or phrases refer to the same entity ("John said he was tired" – "John" and "he" are coreferent). This is technically demanding annotation that requires annotators to understand discourse structure across sentences, paragraphs, and entire documents, not just individual utterances.
Relation extraction goes further: labelling structured relations between entities (PERSON works_at ORGANISATION, COMPANY acquired COMPANY, DRUG treats DISEASE). The annotation produces triples (subject, relation, object) that train knowledge-extraction and graph-construction models. The schema decision is whether relations are restricted to a closed predicate set (the 50 predicates in a biomedical knowledge graph) or open (any predicate that appears in the text).
Both tasks require senior annotators with linguistic training. Open-relation extraction in particular is materially harder than NER because the relation surface is unbounded, and IAA tends to be lower than on classification or NER work. Production programmes typically target κ > 0.65 on open-relation extraction and κ > 0.75 on closed-predicate relation extraction.
Language and dialect coverage for APAC programmes
NLP models are only as multilingual as their training data. For APAC-facing production systems, the annotation team must include native speakers of the target languages – machine-translation-of-English approximations consistently produce datasets that fail in production because the translation does not preserve the linguistic phenomena the model needs to learn.
- Vietnamese: tonal language with no inflection but rich diacritics. Native annotators handle tone-mark recovery, regional accent transcription, and the high rate of borrowed-from-French and borrowed-from-Chinese vocabulary. VinAI's PhoBERT and related Vietnamese NLP work document the difficulty empirically.
- Thai: no word spacing in the written language, which requires tokenisation expertise the native annotator base handles routinely. Tone marks (mai ek, mai tho) and complex consonant clusters are routine annotation considerations.
- Bahasa Indonesia and Bahasa Malay: shared linguistic root with meaningful regional variation. Sentiment, intent, and entity annotation has to handle code-switching with English, Mandarin, and regional languages.
- Tagalog and Filipino: strong English code-switching patterns ("Taglish") that are central to conversational AI work for the Philippines' fintech and e-commerce sectors.
- Mandarin and regional Chinese variants: distinct conventions across Simplified Chinese (mainland China, Singapore, Malaysia) and Traditional Chinese (Hong Kong, Taiwan), with different vocabulary and stylistic norms in each.
- Low-resource APAC languages (Khmer, Lao, Burmese, Tetum): the IIT Madras AI4Bharat initiative has documented best practices on low-resource APAC language annotation that the broader regional ecosystem now applies.
LLM-era NLP annotation: RLHF, evaluation panels, and structured outputs
The rise of large language models has not removed NLP annotation – it has changed what annotation is for. The dominant LLM-era annotation patterns in 2026 fall into three categories.
- RLHF (Reinforcement Learning from Human Feedback): pairwise comparison annotation where annotators rank model outputs against each other, used to train preference models that align downstream LLMs. The annotation quality determines what the model is aligned to – noisy preferences silently bias the model toward whatever the annotation team consistently agreed on.
- Evaluation-set annotation: structured benchmarks that test specific model capabilities (mathematical reasoning, multi-hop question answering, safety boundaries, instruction following). Evaluation sets are smaller than training sets but materially more important to label correctly because every model on the leaderboard is scored against them.
- Structured output and tool-use annotation: labelling the structured outputs (JSON, function calls, tool invocations) that production LLMs need to produce. The annotation work specifies what the correct structured output looks like for each input, and the model is trained against that specification.
Quality control for NLP datasets
The QA practices that consistently separate usable NLP datasets from noisy ones:
- Multi-annotator labelling for subjective tasks: sentiment, emotion, intent, and any task where reasonable people can disagree should have at least two annotators per example with adjudication on the disagreement.
- Chance-corrected IAA per class: Cohen's kappa or Fleiss' kappa for categorical tasks, Krippendorff's alpha for ordinal/interval tasks. Target κ > 0.80 on the headline metric, κ > 0.75 on the hardest class. Single-headline IAA hides per-class failure.
- Held-out test sets that annotators never see, used only for final model evaluation. Without this, train-test contamination is the standard silent failure.
- Calibration sessions every 4–6 weeks where annotators re-annotate the same samples to detect drift over time. The drift metric is its own quality signal, independent of headline IAA.
- Confusion matrix analysis: which entity types, intent classes, or sentiment categories are most often mislabelled. Drives targeted guideline improvement rather than blanket retraining.
- Per-language quality reporting on multilingual datasets. A single global IAA number on a multilingual dataset hides the fact that the Tagalog subset is at κ = 0.60 and the English subset is at κ = 0.90.
Common mistakes in NLP annotation programmes
Recurring patterns we see across NLP annotation engagements that consistently produce noisy datasets:
- Underspecified guidelines. "Label the main topic" without defining what "main" means produces high disagreement and noisy labels. The guideline has to specify the decision rule for hard cases, with worked examples.
- Ignoring edge cases. Not providing rules for ambiguous, multi-label, or partially-applicable cases forces annotators to guess, and the guesses diverge across annotators in ways that look like noise but are actually schema ambiguity.
- Single-annotator labelling for subjective tasks. One person's interpretation of sentiment, intent, or relevance is not ground truth. Subjective tasks require multi-annotator review with adjudication.
- Neglecting class balance. Overrepresentation of frequent intents or entity types and underrepresentation of rare-but-important ones biases the model toward the head of the distribution and away from the rare cases that often matter most in production.
- Skipping linguistic review on the source text. Grammar errors, OCR artefacts, typos, and code-switching in the source corpus all need explicit annotator guidance. Without it, two annotators handle the same artefact differently and the model learns the noise.
- Using non-native or machine-translation-assisted annotators for the target language. The dataset looks fine on a spot check and fails systematically in production on the language phenomena the non-native team did not catch.
Frequently asked questions
Common questions raised by NLP and conversational AI teams scoping an annotation programme:
- How many annotators do I need per example? Single annotation is acceptable for objective tasks (NER on well-defined entities, structured extraction with closed schemas). Subjective tasks (sentiment, intent on conversational data, relation extraction) require at least two annotators with adjudication on disagreements.
- Should LLM pre-labelling reduce my annotation cost? Yes, on tasks where the LLM is competent (entity extraction on well-known domains, classification on standard taxonomies). Human review still catches the errors the LLM systematically makes, which can be the most important signal for the eventual production model. The cost reduction is typically 30–50% when done well.
- How do I evaluate a multilingual NLP annotation vendor? Run paid pilots in each target language with native-speaker reviewer adjudication. The per-language kappa and the per-language audit pass rate are the comparable artefacts. A vendor that quotes the same accuracy across all languages without per-language reporting is either inexperienced or rounding the numbers.
- How should I handle PII and sensitive content in NLP datasets? Treat the annotation pipeline as a regulated data flow: signed NDA and DPA before any data is shared, named-user annotator access, encrypted-at-rest storage, and post-project deletion with a written certificate. For regulated content (financial KYC, healthcare records, legal correspondence) add work-from-secure-room policies and on-premise / VPC-only deployment as appropriate.
- What is the right schema for an open-domain NER project? Start with the standard PERSON / ORG / LOCATION / DATE / MONEY / EVENT schema, add 5–10 domain-specific entity types, and iterate the schema based on disagreement-cluster reports from the first 2–3 batches. The schema that emerges from the first batch is rarely the right schema; the one that emerges from the first three is usually defensible.
