ChatGPT Now Supports Native Multimodal Capabilities: Vision, Speech Input, and Speech Output

Multimodal Capabilities Fully Deployed

OpenAI has deployed the full multimodal capabilities of the GPT-4o model natively in ChatGPT, covering vision (image understanding), speech-to-text (STT), and text-to-speech (TTS) — all handled by a single unified model without plugin dependencies or pipeline orchestration. End-to-end latency is significantly reduced (median < 320ms), enabling conversational-turn timing comparable to human dialogue.

Technical Implementation and Availability

• Currently available only to ChatGPT Plus subscribers and ChatGPT Enterprise customers; free-tier users do not have access;
• Image understanding supports JPG, PNG, HEIC, and WEBP uploads, with robust parsing of charts, handwritten notes, screenshots, and whiteboard content;
• Speech functionality is built directly into GPT-4o’s architecture—no separate ASR or TTS models—delivering high accuracy under noisy conditions and multilingual speech (including code-switched Mandarin-English);
• iOS and Android apps have been updated with a dedicated voice conversation button; desktop web interface now displays the voice input icon.

Limitations and Boundaries

• No support for real-time video stream analysis (static images only);
• TTS output does not yet offer customizable voice profiles or adjustable speaking rate;
• Image understanding is not exposed via public API; currently limited to in-app use within ChatGPT;
• All audio and image data undergo on-device preprocessing by default; OpenAI states raw audio/image files are not persistently stored.