# Multimodal input: text, image, and audio in one model

[Skip to content](#lm-inhoud)Network/[NL](/en/multimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model&text=Multimodal%20input%3A%20text%2C%20image%2C%20and%20audio%20in%20one%20model)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model&title=Multimodal%20input%3A%20text%2C%20image%2C%20and%20audio%20in%20one%20model)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model&text=Multimodal%20input%3A%20text%2C%20image%2C%20and%20audio%20in%20one%20model)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-input-uitgelegd-tekst-beeld-en-audio-in-n-model&title=Multimodal%20input%3A%20text%2C%20image%2C%20and%20audio%20in%20one%20model)[](#)

 
# Multimodal input explained: text, image, and audio in one model

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 
 What you need to know beforehand: This article falls under Module 5: Under the hood. To fully grasp the mathematical and architectural choices below, it helps if you're already familiar with the basics of vectors and weights. For an overview of all technical terms around weights, tensors, and vectors, you can consult the [central AI glossary](https://leren.llmnet.nl/en/ai-begrippenlijst) . In addition, we build directly on the fundamental concept of attention mechanisms, which is covered in detail in the article on [the attention mechanism in AI](https://leren.llmnet.nl/en/attention-uitgelegd).

 

 Traditional large language models were strictly textual prediction machines: they accepted sequences of integers (tokens) corresponding to subwords, and predicted the next token. In a multimodal architecture, however, completely different physical signals — continuous sound waves, two-dimensional pixel grids, and discrete characters — are processed simultaneously by one central transformer backbone. This is not achieved by running a separate neural network alongside the language model for each modality that passes along textual summaries, but by converting all signals into continuous embedding vectors with exactly the same dimension.

 In this article, we break down the internal mechanisms behind multimodal architectures. We look at how raw audio recordings and pixel grids are transformed into vectors, how projection layers ensure that different sensory inputs end up on one shared conceptual line, and what the computational consequences are for the context window and the KV cache.

 
## 1. The shift from pipeline coupling to native multimodality

 Early systems that combined text and image used a composite chain (often referred to as late fusion or a pipeline architecture). Here, an external speech model like Whisper would first transcribe an audio fragment into text, or a Vision Transformer (such as a classic CLIP image recognizer) would describe a photo in textual labels. Only after that did the central language model receive that extracted text as a regular prompt. Although this modular approach is easy to build, fundamental context is lost: intonation, background noise, breathing, spatial relationships between objects, and subtle visual nuances cannot be translated into a flat text string without loss of quality.

 Modern multimodal networks, by contrast, use early fusion or deep cross-modal integration. In a native multimodal model, raw sensory data is embedded directly into the transformer's active hidden state (the hidden dimension $d_{\text{model}}$) via specialized linear projections. This allows the network to model cross-modal relationships using the same self-attention layers that analyze word relationships. A model thereby learns directly that a visual image of a red apple, the Dutch word "appel," and the specific audio pattern of someone pronouncing the word all refer to the same underlying concept in the latent space.

 
## 2. Image input: from pixels to vision patches and linear projection

 A two-dimensional image has no inherent order of discrete symbols the way written text does. To feed an image of, for example, 448 by 448 pixels with three color channels (RGB) through a transformer, the image is first divided into a grid of smaller squares, so-called patches. A common patch size is 14 by 14 or 16 by 16 pixels.

 For an image with resolution $H \times W$ and patch size $P \times P$, this produces $N = (H \cdot W) / P^2$ individual patches. At a resolution of $448 \times 448$ and a patch size of $14 \times 14$, this yields exactly $(448/14) \times (448/14) = 32 \times 32 = 1024$ visual patches. Each individual patch consists of $14 \times 14 \times 3 = 588$ raw pixel values.

 A linear projection layer or a pretrained Vision Transformer (ViT encoder) then transforms each flattened 588-dimensional patch vector into the hidden model dimension $d_{\text{model}}$ (for example, 4096). A two-dimensional position embedding (2D positional encoding) is also added, so the model knows where a patch was located in the original grid. For a thorough comparison of how different data formats are divided up, the explanation of [multimodal tokenization of images and audio](https://leren.llmnet.nl/en/multimodale-tokenisatie-uitgelegd) offers valuable depth.

 
 
 
 
 Resolution | 
 Patch size | 
 Number of visual tokens | 
 Memory pressure (FP16 per token) | 
 Typical use | 
 

 
 
 
 224 × 224 | 
 14 × 14 | 
 256 tokens | 
 Low (~0.5 KB activation per layer) | 
 Fast mobile classification, thumbnails | 
 

 
 448 × 448 | 
 14 × 14 | 
 1,024 tokens | 
 Medium (~2 KB activation per layer) | 
 Standard document OCR and object detection | 
 

 
 896 × 896 | 
 14 × 14 | 
 4,096 tokens | 
 High (~8 KB activation per layer) | 
 Complex technical diagrams, dense PDFs | 
 

 
 1344 × 1344 | 
 14 × 14 | 
 9,216 tokens | 
 Very high (requires dynamic cropping) | 
 High-resolution satellite and medical images | 
 

 
 
 

 
## 3. Sound and speech: spectrograms and continuous audio tokens

 Sound waves are one-dimensional continuous signals with a high sample rate, typically 16,000 to 48,000 measurements per second. If we treated every audio sample directly as a token, a ten-second audio fragment would already result in 160,000 input elements, which immediately leads to memory shortages in the quadratic attention mechanism.

 Therefore, audio data is first converted via a Short-Time Fourier Transform (STFT) into a so-called log-mel spectrogram. This transforms the time signal into a two-dimensional representation with frequency bands on the vertical axis and time segments (frames) on the horizontal axis. An audio encoder (often based on a Conformer or 1D convolutional architecture) then compresses these frames in time: typically, every 40 to 80 milliseconds of audio is summarized into one audio token.

 As a result, one second of spoken audio produces roughly 12.5 to 25 tokens. A ten-second speech recording thus translates into 125 to 250 vector embeddings. These audio tokens carry not only semantic information about the spoken words, but also preserve acoustic properties such as intonation, speaking rate, emotional charge, and ambient noise.

 
## 4. The mathematics of the shared vector space

 The core of a modern multimodal model is that after the input and projection phase, all modalities take on the same mathematical form: an ordered matrix of tensors with dimension $[B, S, D]$, where $B$ is the batch size, $S$ is the total sequence length (the sum of text tokens, image tokens, and audio tokens), and $D$ is the hidden model dimension ($d_{\text{model}}$).

 Let $X_{\text{text}} \in \mathbb{R}^{N_t \times d_t}$ be the classic subword embeddings, $X_{\text{image}} \in \mathbb{R}^{N_v \times d_v}$ the extracted feature vectors from the Vision Transformer, and $X_{\text{audio}} \in \mathbb{R}^{N_a \times d_a}$ the output of the audio encoder. Because $d_t$, $d_v$, and $d_a$ can initially differ from each other, trained projection matrices $W_v \in \mathbb{R}^{d_v \times D}$ and $W_a \in \mathbb{R}^{d_a \times D}$ are applied:

 # Conceptuele samenstelling van een gecombineerde multimodale sequentie
import torch
import torch.nn as nn

class MultimodalProjector(nn.Module):
 def __init__(self, d_vision, d_audio, d_model):
 super().__init__()
 # Meerlaagse perceptron (MLP) projectielagen met niet-lineariteit
 self.vision_proj = nn.Sequential(
 nn.Linear(d_vision, d_model),
 nn.GELU(),
 nn.Linear(d_model, d_model)
 )
 self.audio_proj = nn.Sequential(
 nn.Linear(d_audio, d_model),
 nn.GELU(),
 nn.Linear(d_model, d_model)
 )

 def forward(self, text_emb, vision_feat, audio_feat):
 # text_emb: [batch, len_text, d_model]
 # vision_feat: [batch, len_vision, d_vision]
 # audio_feat: [batch, len_audio, d_audio]
 
 proj_vision = self.vision_proj(vision_feat) # -> [batch, len_vision, d_model]
 proj_audio = self.audio_proj(audio_feat) # -> [batch, len_audio, d_model]
 
 # Samengevoegde invoer voor de transformer backbone
 interleaved_seq = torch.cat([text_emb, proj_vision, proj_audio], dim=1)
 return interleaved_seq

 Once these vectors have been merged, the backbone transformer no longer sees any functional difference based on a token's origin. In every self-attention layer, the query, key, and value vectors compute the relationships across modality boundaries via the well-known formula:

 Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

 This allows a textual query vector ("Where is the bike?") to directly assign high attention scores to specific visual image tokens in which the pattern of spokes and wheels is encoded.

 
## 5. Attention distribution and cross-modal alignment

 Although all vectors reside in the same matrix, multimodal models exhibit specific attention behavior during training and inference. Mechanistic analyses show that lower layers of the transformer often process modality-specific features: visual tokens communicate mainly with neighboring visual tokens to refine edges and textures, while text tokens analyze syntactic structures.

 In the middle and higher layers, cross-modal integration emerges (late alignment). Here, the information converges into abstract semantic representations. When the model, for example, answers a Dutch question about a spoken complaint or an uploaded photo of a receipt, the higher attention heads combine the auditory intonation, the visual layout of numbers, and the grammatical structure of the desired output.

 To evaluate whether a prompt with image or audio responds more reliably than a purely text-based instruction, it's advisable to validate model performance empirically; for this, see the methodology of [A/B testing for prompts](https://benchmark.llmnet.nl/en/ab-testen-prompts). For structurally validating quality differences between multimodal models, the dossier on [evaluating multimodal AI and measuring performance](https://benchmark.llmnet.nl/en/multimodale-evaluatie) offers deeper guidance.

 
## 6. Computational complexity, KV cache, and context management

 The integration of images and sound has direct consequences for hardware load and inference costs. While an average written paragraph spans roughly 60 to 100 subword tokens, a single photo at standard resolution already consumes 576 to 1,024 tokens. A thirty-second audio fragment adds another 400 to 750 tokens on top of that.

 Because the memory usage of the Key-Value cache (KV cache) grows linearly with the total sequential context length, and attention processing scales quadratically with respect to the input sequence, multimodal inputs place heavy pressure on graphics memory (VRAM). An interactive session with five consecutive screenshots can quickly require more than 5,000 tokens of active context, even before the user has typed a single question.

 To reduce this memory pressure, modern multimodal architectures use specific optimization techniques:

 
 
- Dynamic patch reduction (token pruning): Algorithms that identify and remove non-informative patches (such as uniform white backgrounds in documents) before they reach the transformer.
 
- Spatial Perceiver Resamplers: Cross-attention modules with a fixed number of learnable queries (for example, 64 or 128) that compress an arbitrarily large number of visual tokens into a compact, fixed-length representation.
 
- Quantization of the KV cache: Storing key and value tensors at 4-bit or 8-bit precision to free up memory space during long multimodal dialogues.
 

 
## 7. Pitfalls: cross-modal hallucinations and temporal desynchronization

 Despite the mathematical elegance of a shared embedding space, multimodal models struggle with specific weaknesses that don't occur in pure text models:

 1. Visual hallucination due to language prior (Language Prior Dominance): When the language model knows a strong statistical correlation from its textual training data (for example: "a zebra always has stripes"), it can ignore visual details. Shown a photo of a rare unpatterned zebra, the model will often incorrectly claim the animal has normal stripes, because the textual probability overrides the visual patch representation.

 2. Temporal desynchronization in streaming audio: In real-time speech interaction, the model must listen and speak in sync. Because audio tokens arrive sequentially, delay in the cross-attention layer can cause the model to react to an intonation shift that occurred several hundred milliseconds earlier, leading to stuttering interruptions.

 3. Resolution loss with small fonts: In document analysis and reading scans (OCR), the fixed patch grid is a structural limitation. Letters smaller than a few pixels fall within a single patch and blur into an unrecognizable average of RGB values, making numbers in financial tables prone to hallucination.

 
## 8. Application in software architectures and agent systems

 In modern software environments, native multimodality allows developers to build robust pipelines in which visual inspection, speech recognition, and structured JSON output come together in a single call. Instead of managing separate OCR and ASR (Automatic Speech Recognition) servers, one model endpoint that accepts heterogeneous payloads suffices.

 Those who want to design autonomous systems that combine live camera feeds and spoken commands can follow the practical route to [becoming an AI agent engineer in 2026](https://leren.llmnet.nl/en/ai-agent-engineer-worden-2026). Here, the multimodal model acts as the central sensory processing system that directly converts visual and acoustic observations into targeted function calls (tool calls).

 
 Continue with: Now that we've seen how multimodal inputs are brought together under the hood into a single token sequence, the logical next step is to study model optimizations that keep these large contexts efficient. Read on about how specific attention mechanisms limit memory consumption in the articles on [grouped-query attention and memory usage](https://leren.llmnet.nl/en/grouped-query-attention-uitgelegd) and managing long sequences via [the construction of KV caching in transformer architectures](https://leren.llmnet.nl/en/kv-caching-opbouw).