Build Anything with CrewAI, Here’s How

CrewAI is presented as a fast way to build multi-step AI “agents” that can generate lead-targeting search queries, search the web, scrape relevant pages, and extract contact details—without requiring the viewer to write everything from scratch. The practical payoff is a lead-generation pipeline that can be adapted to any niche, location, and lead volume, turning basic inputs (industry + city + number of leads) into structured contact outputs like email and social profiles.

The build starts in Google Colab, where Python runs in notebook cells. After installing CrewAI and connecting to Anthropic’s Claude models via an API key, the workflow uses Claude 3.5 Sonnet (with notes about model/token settings and cheaper alternatives). Two Claude client instances are created: one “consistent” (temperature set low) for predictable outputs, and one “creative” (higher temperature) for generating varied phrasing. This split matters because the pipeline needs both reliable formatting and diverse search-query ideas.

The first CrewAI agent, dubbed a “variation agent,” takes a niche and location and produces 10 distinct, concise search queries intended to surface business leads. A second step wraps that agent into a Crew, then “kickoffs” execution so the generated queries appear as a clean list. The transcript emphasizes output control—no extra text, no quotation marks, and one query per line—so downstream steps can parse results reliably.

Next, the pipeline adds a web-search agent using Serper (via CrewAI tools). The search agent receives the generated queries and returns a list of websites. The build then highlights a key limitation: asking the model to hit a large lead count in one go can cause early stopping or counting errors, since LLMs struggle with strict numeric targets. The suggested fix is structural: use loops outside the agent (and deduplicate) so the system keeps searching until the desired number of qualifying leads is reached.

For scraping, the workflow installs Firecrawl and uses it to convert websites into LLM-ready markdown. After selecting a target site from the search results, Firecrawl scrapes the homepage content (truncated for readability), then a new CrewAI agent analyzes that homepage to find URLs for “about” and “contact” pages. Those pages are scraped as well, and the combined scraped text is fed into a final Anthropic call with a tight prompt instructing the model to extract email, Twitter, and LinkedIn—outputting “none” when fields aren’t found.

The result is a working end-to-end lead extractor: search → scrape → page discovery → contact extraction. The transcript closes by outlining concrete upgrades for production quality: save results to CSV, add personalized unique fields per business, run search and scraping in loops to reach exact lead counts, avoid counting leads with missing contact info, deduplicate websites, and refactor into functions so the whole process can run in a single cell. The overall message is that CrewAI plus purpose-built tools (Serper, Firecrawl, Anthropic) can assemble a usable lead engine quickly, while code-based loops and validation handle the parts where LLMs are least reliable.