Lecture 7: Machine Learning Teams - Full Stack Deep Learning - March 2019

Machine learning teams face a widening talent gap that makes hiring—and building effective teams—far harder than most companies expect. Estimates cited for the number of people who can truly build AI systems range from 5,000 to 200,000–300,000 globally, while the broader pool of software developers is orders of magnitude larger (3.6 million in the US; 18.2 million worldwide). That imbalance fuels intense competition for candidates, pushing recruiting toward “frenzied” tactics and driving up salaries, with hiring described as unusually time-consuming and difficult even for startups.

Against that backdrop, the lecture breaks down the main machine-learning-adjacent roles companies use and why their skill mixes differ. DevOps engineers focus on deploying and monitoring production systems, producing a working deployed product. Data engineers build and maintain data pipelines—often using tools like Hadoop and Airflow—so training data can be accessed reliably and quickly. Machine learning engineers train and deploy prediction models, bridging the gap between experimentation and production code and workflows. Machine learning researchers concentrate more on training and iteration, then hand models off for deployment. Data scientist is treated as a catch-all term with no single definition: in some organizations it means reporting and business analytics (including heavy use of Excel), while in others it overlaps with research or engineering.

A key practical takeaway is that these roles sit on a 2x2 space: software engineering skill versus machine learning knowledge. The lecture also emphasizes that success depends not only on technical depth but on communication and technical writing. Among the roles, machine learning engineer is portrayed as a “rare unicorn” hire because it requires both state-of-the-art ML capability and enough engineering fluency to integrate models into real systems. Backgrounds vary widely: some candidates come from software engineering with self-taught ML; others are CS or stats PhDs; and increasingly some transition from research fellowships such as the Google Brain Fellowship or the Facebook Fellowship.

Team structure lacks a single best answer, but common consensus points emerge. Most organizations aim for a blend of engineering and ML skills on the same team, with the expectation that everyone can write production-ready code. There’s also disagreement about how to staff ML researchers: some prefer more engineering-heavy teams because collaboration can be difficult, while others argue that deep ML expertise is essential for moving fast. Data engineering placement varies too—some embed data engineers with ML teams as the primary customer, while others keep data labeling in-house to build tooling that speeds annotation.

Running these teams is described as uniquely challenging because ML work is hard to estimate, progress is nonlinear, and projects can stall for weeks. Even early performance signals can mislead, as improvements may happen quickly at first and then flatten. Cultural gaps can also appear when research-oriented norms (publishing, exploration) clash with engineering priorities (shipping, reliability). Finally, leaders sometimes impose standard software-engineering planning frameworks that don’t translate well to ML.

Hiring and interviewing add another layer of uncertainty. Job seekers are encouraged to apply directly to target companies despite the usual advice for software roles, since the ML talent shortage can make direct applications more effective. Interview loops are less standardized than software engineering, with common elements including pair debugging of ML code, take-home ML projects, and ML theory or linear algebra questions. Preparation focuses on ML fundamentals (e.g., bias-variance tradeoffs, diagnosing why loss might rise) rather than memorizing recently released architectures. The lecture ends by highlighting an exam designed to mirror ML engineering interview questions, ranging from residual networks’ purpose to diagnosing noisy learning curves and selecting appropriate loss functions for tasks.