Lab 07: Web Deployment (FSDL 2022)

A practical deployment pipeline turns a trained PyTorch text recognizer into a portable, shareable model service—first by compiling it to TorchScript, then by wrapping it in a Gradio interface, and finally by splitting the UI from model execution so it can run behind a stable public URL. The payoff is a workflow that moves beyond “just training” into an application people can actually use: upload an image, submit it, and receive handwritten text output through a web link.

The process starts with model weights saved during training. Checkpoints produced for restarting training are repurposed for inference by loading them into a Lightning module and converting the resulting model to TorchScript via the model’s TorchScript conversion method. That conversion yields a lighter artifact that can run without a Python runtime and without the heavy development dependencies typically required for training stacks. Once compiled, the TorchScript binary is stored in Weights & Biases (W&B) cloud storage, and a small “stage model” script automates the handoff from training to production.

A key operational advantage of using W&B for both training artifacts and production artifacts is traceability. The compiled TorchScript model is linked back to the specific training run that produced the checkpoint, along with logged metrics and experiment metadata. Early on, manual tracking is manageable, but as teams and model versions grow, programmatic access to that lineage becomes essential for debugging, auditing, and iterating.

With the model packaged, the lab introduces a dedicated Python module—ParagraphTextRecognizer—designed to load the TorchScript artifact with a single call (torch.jit.load). Inputs are formatted to match what the model expects, and outputs are converted back into strings, enabling an image-to-text workflow. To keep iteration safe, an end-to-end test script runs the full chain: data ingestion, gradient updates, checkpoint saving, TorchScript conversion, uploading to W&B, and pulling the compiled model back down. Any failure in that pipeline causes the test to fail.

For deployment performance, the lab emphasizes that GPUs are not always necessary for inference. In the provided notebook environment, inference runs without a GPU, and the discussion highlights why batching matters when GPUs are used: training can batch efficiently because it controls data flow, while production requests arrive independently. Profiling comparisons show that batch size 16 maintains high GPU utilization, while batch size 1 drops utilization and shifts the bottleneck to CPU-side work.

Next comes the user experience. A Gradio app wraps the recognizer’s predict function, automatically generating an interface with image input widgets and text output. Gradio also provides an API endpoint and a REST-style workflow for programmatic access, including base64 encoding for image payloads. A minimal web test verifies that the UI and API respond without errors.

Finally, the lab moves toward a model-as-a-service architecture. The Gradio frontend can call an AWS Lambda function that loads the TorchScript model and returns predictions via JSON over HTTP. This separation allows the UI to run locally while model inference runs on AWS infrastructure. To make the app shareable beyond Gradio’s temporary URLs, the lab uses ngrok to expose the locally running Gradio service over HTTPS, then outlines production deployment on an EC2 instance and an optional Docker-based approach to automate environment setup and execution.

Overall, the workflow—compile to TorchScript, package and trace with W&B, wrap with Gradio, serve via serverless or separated backends, and expose with stable public tunneling—creates a realistic path from research artifacts to an application that users can interact with immediately.