Labeling (3) - Data Management - Full Stack Deep Learning

Data labeling hinges less on the annotation software’s feature list and more on the human decisions inside the labeling workflow—especially when “reasonable” people would draw different boundaries. Vision labeling tools typically offer the same core primitives—bounding boxes, segmentation polygons, keypoints, and cuboids for 3D or video—but quality depends on clear instructions, consistent interpretation, and active quality control. A concrete example is an object partially hidden behind another item: annotators must be told whether to label the entire imagined object or only the visible portion. Without that guidance, label noise rises from subjective judgment calls, and even well-intentioned annotators can drift.

That quality challenge shapes the three main ways teams source labeling labor: hire in-house annotators, crowdsource through marketplaces, or outsource to full-service labeling companies. Hiring directly offers tight control and faster iteration once trusted annotators are identified, including the ability to promote top performers into quality control roles and secure the work via contracts. The tradeoffs are cost, slower scaling (finding and managing real people takes time), and administrative overhead. Crowdsourcing—via services such as Amazon Mechanical Turk—can reduce cost and speed up throughput, but it adds complexity because annotators are transient and often poorly paid, which typically requires an additional quality-control layer.

Full-service data labeling companies aim to remove that operational burden by providing both the annotation interface and the quality assurance machinery. The workflow often looks like: spend a few days evaluating vendors, create a gold-standard labeled dataset internally to understand the true labeling complexity, send that same dataset to candidate providers, and compare their outputs against the internal standard. Pricing matters because vendors charge for recruiting labor, enforcing quality, and managing the process. Examples mentioned include Figure Eight (formerly CrowdFlower), which has labeled over 100 million images and accumulated over 1 billion human judgments by 2018, and Scale AI, which is positioned as an API-like service for labeling new image content. Other providers include Labelbox and Supervisely, while Prodigy is highlighted as an annotation tool that supports active learning and can reduce the amount of labeling needed by prioritizing uncertain samples.

When teams can’t afford heavy quality control, the transcript’s practical stance is blunt: robustness to bad samples can’t be assumed without seeing the relevant distribution. Models may generalize to unseen conditions, but there’s no reliable way to guarantee resilience to specific outliers unless those cases are labeled, tested, and then incorporated into training if needed.

Finally, the discussion turns to aggregation and “subjective” labels. If multiple annotators label the same task, teams can use majority vote or averaging for objective outputs like bounding boxes. For ranking or search relevance—where people disagree—there are multiple strategies: bin items into clearly high/low categories and treat the ambiguous region as noisy, or frame ranking as a regression problem by predicting average rank. Related questions about leveraging negative examples (e.g., images a user didn’t choose, or search links a user didn’t click) remain an active research area because “not chosen” doesn’t always mean “incorrect”—it may reflect randomness or user behavior rather than ground truth.