Open Source AI Inference API w/ Together

Together’s inference API is positioned as a fast, reliable way to run open-source text, chat, image, and code models without building and hosting your own stack—while keeping an escape hatch to run the same models locally later. The core pitch is practical: for any given model, pricing and speed can shift across providers, so having an API that can swap models quickly matters. Based on the creator’s testing, Together delivers consistently quick token generation and dependable responses, often outpacing other hosted options they tried.

A major reason to use an API at all is convenience and iteration speed. Even though the models behind Together are open source (and can be downloaded and run on your own hardware), standing up local inference—especially for larger models—takes time and compute. The workflow starts with creating an account, setting up billing, and retrieving an API key from settings. Developers then install the Together Python package (via pip) and can begin with simple calls—like listing available models and hitting a “playground” first to validate prompt behavior before writing production code.

Model selection is framed as both broad and fast-moving. Together adds new models regularly, and the transcript highlights “Mixol,” described as a Mistral AI mixol 8x7B mixture-of-experts model. For text generation models, the key operational detail is prompt structure: each model may require a specific formatting style, so checking the Together Playground (or the referenced Hugging Face page) becomes part of the setup. The transcript also emphasizes streaming: instead of waiting for a full response, the API can stream tokens as they’re generated, improving user experience for chat-like applications where output may be long.

Beyond plain completion, Together offers instruct and chat variants. The transcript stresses that these are still text-generation models under the hood; they’re trained to follow particular prompt conventions that encourage structured dialogue. It also notes that Together models can be queried through an OpenAI-compatible package, which can ease migration for codebases built around OpenAI-style endpoints.

For instruct-style prompting, the transcript walks through a smaller “RedPajama” instruct model (tied to a GPTJ lineage) and discusses zero-shot vs one-shot examples using question/answer patterns. It also highlights the importance of stop sequences to prevent the model from drifting into extra text. When moving to code, the transcript spotlights “Code Llama” (find code llama 34b python v1), describing it as fast and accurate for Python generation and noting that larger coding models can be tens of gigabytes and slow to run locally.

Finally, the transcript demonstrates building a “terminal-capable” assistant: using a system prompt plus example formatting to elicit bash commands, extracting those commands with regex, and executing them via os.system. It suggests a multi-model approach—using one model to generate, another to evaluate, and a third to moderate—while also proposing conversation-history management and summarization once context grows beyond practical limits. The overall takeaway is an engineering workflow: prototype in the Playground, integrate via the API with streaming and stop controls, and only then decide whether to host locally for privacy or compliance needs.