Orgs (3) - ML Teams - Full Stack Deep Learning

Machine-learning organizations don’t have a single “correct” structure yet, but companies tend to evolve through a recognizable ladder: from ad hoc experimentation, to ML embedded in product teams, to a centralized ML function, and finally to an “ML-first” model where AI expertise and infrastructure permeate the whole company. The practical payoff of moving up the ladder is better access to data and talent, faster iteration, and smoother deployment—but each step also introduces new bottlenecks around resources, ownership, and operational handoffs.

At the base of the “ML organization mountain” sit companies where machine learning is mostly informal—sometimes a few enthusiasts, but no dedicated function. The upside is that teams can often find “low-hanging fruit” by spotting business problems that simple predictive models can improve quickly. The downside is structural: limited infrastructure, limited compute, and weak leadership buy-in. Those gaps matter because ML projects often run on different timelines than typical software work, and it can be hard to attract ML talent without a real support system.

A common next move is to place ML inside R&D. In this setup, researchers—often PhD-heavy—work with business data and produce models and sometimes papers, while enjoying more freedom from near-term product deadlines. This can appeal to researchers who prefer building models over worrying about what happens after deployment. But the model often stalls: R&D groups struggle to obtain data from business units, and without visible business wins, investment stays small.

Many organizations then embed ML engineers directly into product or business teams. This arrangement creates a clear line from ML work to customer-facing improvements and enables rapid feedback loops: prototype, test, iterate. It also tends to unlock more funding because results can be tied to product metrics. Yet dispersing ML across the organization makes it harder to build “ML as a function”—including hiring and developing top ML talent who want to collaborate with other ML specialists. It also strains resources like data and compute, and engineering leaders may push back when ML delivery doesn’t match engineering’s expectations.

To address those issues, some companies build an independent ML function. Centralization increases talent density, supports tooling and deployment practices, and—because the group often reports to senior leadership—can help break down data-access barriers. The tradeoff is handoff friction: centralized teams must deliver models to business users who may not have the expertise to know when models apply, how to monitor them, or how to operate them responsibly.

The end state is an ML-first organization, where leadership is fully committed, a centralized ML division tackles the hardest problems, and every business unit has ML capability built in. Google and Facebook are cited as examples, along with ML-focused startups. This structure aims to combine the best of both worlds—data access plus deployment and talent development—but it’s difficult to implement, expensive to staff, and culturally hard to shift.

The transcript also highlights key design choices that determine how teams work: whether ML teams prioritize software engineering versus research, how much control they have over data ownership and pipelines, and whether they deploy and maintain models or hand them off. Embedded teams often prioritize production code and work closely with data engineers; R&D teams often have less data control and focus on research; ML-first organizations can support shared understanding between research and engineering and may take responsibility for company-wide data infrastructure and model operation. The discussion closes by weighing career fit: people motivated by improving products tend to thrive embedded in business teams, while those driven by state-of-the-art modeling and large-scale tooling often prefer centralized ML roles. Bias and fairness are also flagged as an increasingly important responsibility as ML systems become more mature.