Unlock Open Multimodality with Phi-4

Microsoft’s Phi-4 family just got more practical for local, multimodal work: the Phi-4 3.8B “mini instruct” lineup now includes function calling and a dedicated Phi-4 multimodal model that can process images and audio. The release matters because it brings “agent-ready” capabilities—choosing tools via function calling—plus on-device multimodal understanding, without requiring the largest Phi-4 models or cloud inference.

Weights for the Phi-4 14B had been discussed since December, but official downloads lagged until January. That same pattern of delayed weight availability is now resolved for the smaller, more deployable 3.8B “mini instruct” models. Hugging Face hosts multiple variants, including instruct-tuned models, “Onyx” versions, and a GG UF option. Notably, the base (non-instruct) model weights aren’t provided for the Phi-4 3.8B line, limiting full fine-tuning workflows that were easier with earlier releases.

For the Phi-4 mini instruct (3.8B), one standout addition is function calling. That enables local tool-use patterns—small agents that don’t need heavy reasoning but do need decision points about which tools to invoke. The transcript also emphasizes deployment reality: many users run models on devices through stacks like LM Studio and Ollama, often using quantized formats such as GG UF and Onyx. Microsoft’s release of an Onyx runtime is framed as a direct response to that trend, with support aimed at serving models across environments ranging from Raspberry Pi to mobile devices.

On multimodality, the Phi-4 multimodal instruct model uses the same 3.8B backbone but extends it with both vision and audio encoders. The vision side converts images into tokens via a SigLIP encoder (described as likely SigLIP-1 rather than the newer SigLIP-2). The model can handle image inputs up to 1344×1344 and is trained to generate interleaved image-and-text token sequences. For audio, the audio encoder is larger than the vision encoder, and the system uses audio-specific LoRA adapters to route audio tokens into the Phi-4 backbone. Training details cited include pre-training on 2 million hours of speech pairs, followed by 100 million curated speech/audio supervised fine-tuning samples.

In practical tests using the Transformers library, the model runs with an Auto processor that handles both images and audio. Image results are described as strong for recognition and OCR: it identifies a bee on a flower, answers color and content questions, and transcribes text from a screenshot of a blog post with high apparent accuracy. Some limitations show up in counting (planes in an image are miscounted), and bounding boxes appear inconsistent in placement even when the model returns many boxes.

Audio performance is presented as a key differentiator. The model transcribes an MP3 interview clip into text and is described as accurate enough to feel competitive with or surpass Whisper on certain cases—while also enabling translation of the transcribed text into French. The transcript closes by positioning the Phi-4 multimodal model as a strong open-weights option for local multimodal tasks and agent workflows, especially when paired with function calling and tool-use patterns.