Multimodal embedding combines different types of data models into a shared embedding space. It is a powerful approach in machine learning that aims to combine and represent information from different modalities in a shared latent space. By embedding multiple modalities together machines can better understand complex concepts that are difficult to capture from a single modality alone.
These models embed different types of data into a shared vector space which makes it possible to directly compare them.
Use separate encoders for each modality and train the model so that related pairs are closer in the embedding space than unrelated ones.
For example: CLIP (by OpenAI)
2. Cross modal transformers
Cross modal transformers is a transformer based model that uses cross attention to learn relationships between modalities.
Each modality has its own encoder which allows one modality to attend to another learning rich interactions.
For example: ViLBERT, LXMERT, VisualBERT.
3. Multimodal Auto encoders
Auto encoders that take inputs from multiple modalities and learn a shared latent representation that can reconstruct one or both modalities.
A shared encoder processes inputs from different modalities into a single embedding and reconstruct the inputs and teaching the model to capture shared meaning.
For example: Multimodal Variational Auto encoders (MVAE)
4. Graph Neural Networks (GNN)
It uses a graph structure to model multimodal relationships where each node might represent text, image region and edges model how they relate.
Creates a graph where each node is a modality element and GNN layers update node embeddings based on neighbors and helping model complex interactions across modalities.
For example: Used in social media analysis, medical diagnosis (text + scan).
For example
CLIP (Contrastive Language-Image Pretraining) is trained to connect images and their textual descriptions by mapping both into a shared embedding space where semantically similar text and images lie close together.
2. Two Encoders: An image encoder (like a CNN or Vision Transformer) turns the image into a vector and a text encoder (like a Transformer) turns the sentence into a vector.
3. Joint Embedding Space: Both vectors are projected into the same high dimensional space. During training the model is optimized so that:
Matching image text pairs have high similarity.
Non matching pairs have low similarity.
Advantages
Better understanding and Richer representations: Combining multiple modalities help to capture more complete information than a single modality.
Improved performance on tasks: Multimodal embeddings improve results in tasks like: Visual question answering and image captioning because the model leverages complementary signals from each modality.
Robustness and flexibility: Models can handle missing or noisy data better by relying on other modalities. For example: if audio is unclear text can still provide clues.
Enables complex AI applications: Supports models that can understand, generate or relate multiple data types like Conversational AI with images and speech and Multimodal recommendation systems.
Disadvantages
Complexity of model design: Combining multiple modalities requires designing different encoders and fusion mechanisms. The architecture can become quite complex making training and tuning harder.
Data Requirements: Multimodal models often need large datasets with aligned data and such paired multimodal datasets are scarce and expensive to collect.
Modality Imbalance: Sometimes one modality dominates or is more informative than others. If one modality is missing or noisy it can hurt performance if not handled properly.
Computational cost: Processing multiple modalities increases memory and computation requirements. Training and inference become slower and require more resources.
