Lab 06: Data Annotation (FSDL 2022)

Data annotation is treated as a make-or-break step in the full machine-learning pipeline: rich, carefully structured labels—often at finer granularity than the final task—are what turn raw handwritten inputs into training data that neural networks can actually learn from. The lab’s first half focuses on how handwritten-text datasets are represented and persisted for PyTorch/PyTorch Lightning, emphasizing that the “flavor” of the raw data matters less than the principle of capturing detailed annotations. Even when the end goal is a whole paragraph, the workflow benefits from labeling at the line, word, and character levels. That detail enables synthetic data generation, such as recombining lines from different paragraphs into new synthetic paragraphs to stretch limited datasets. The lab frames data synthesis as an underrated early-stage strategy for bootstrapping ML systems, especially when data is scarce, and notes that modern image/text synthesis advances (including approaches associated with Stable Diffusion–style image generation and GPT-3–style text generation) make this kind of augmentation increasingly important.

The second half shifts from dataset structure to the practical mechanics of producing annotations from the real world. Raw handwritten pages are easy to collect by scanning and digitizing, but the labels must be created manually. For that, the lab uses Label Studio, a secure web-based annotation tool. Because Label Studio runs as a local web service during the exercise, the setup includes creating a username/password, installing Label Studio, and using ngrok to expose the local service to the public internet without wrestling with firewalls or port forwarding. The lab also uses a publicly accessible FSDL handwriting dataset stored on S3, but in an unannotated form; instead of uploading images directly, Label Studio ingests a manifest (a CSV of URLs). In local development, the manifest can be uploaded directly; in Colab-style workflows, the manifest must be downloaded to the machine running the browser.

Once Label Studio is running, each row in the CSV becomes a “task” to annotate. The lab then walks through building an annotation interface using Label Studio’s domain-specific language, starting from an OCR template and customizing it to match the desired output: annotators mark each line of handwritten text. The interface supports zooming, rotating, and precise region selection via draggable controls. A key part of the workflow is UI debugging and ambiguity resolution—spending time annotating a few forms end-to-end to understand edge cases such as whether annotators should tightly bound individual letters (even if that overlaps neighboring lines), how much rotation to apply to follow the text baseline, and whether to correct misspellings. The lab’s guidance is explicit: annotators should make a best effort to capture the letters present in the handwriting, not “correct” spelling or replace uncertain handwriting with the printed prompt.

Finally, the lab highlights a practical constraint: if downstream data handling expects rectangular regions, polygon selectors should be removed from the UI even if polygon precision seems useful. It also recommends writing clear instructions inside Label Studio and testing them with real annotation passes. The exercise ends with a teardown step that shuts down the Label Studio service and returns the environment to the model-development setup for the next lab, where trained models are deployed into production.