In the previous chapter, we built applications that work with text and documents using RAG. Now we move beyond text entirely.
Multimodal AI allows an AI system to understand and work with multiple types of information such as text, images, PDFs, audio, and other media.
Text + Image + PDF + Audio → AI ApplicationOpenAI APIs support applications involving text generation, computer vision, image/file inputs, audio transcription, text-to-speech, and realtime multimodal interactions.
8.1 What Is Multimodal AI?
Traditional AI: Text → AI → Text
Multimodal AI: Text / Image / PDF / Audio / Video → AI → Text / Audio / Action
Example:
User uploads: Invoice Image
User asks: What is the total amount?
AI: The total amount is ₹25,400.Another:
User: [Uploads a screenshot]
Question: Why is this application showing an error?
AI: The screenshot shows a database connection error.8.2 Why Multimodal AI Matters
Many real-world problems aren’t text-only:
Customer Support: Customer → Uploads Product Image → AI Identifies Problem → Support Response
Healthcare Administration: Medical Document → AI → Extract Structured Information
Manufacturing: Machine Image → AI Vision Analysis → Detect Visible Defect
Finance: Invoice PDF → AI → Extract: Vendor, Amount, Date, Tax
Meetings: Audio Recording → Transcription → Summary → Action Items
8.3 The Multimodal Architecture
USER
│
┌─────┼─────┐
▼ ▼ ▼
Text Image Audio
│ │ │
└─────┼──────┘
▼
Multimodal AI
│
┌────┼────┐
▼ ▼ ▼
Text JSON ActionFor a production system:
User Input → Input Validation → Media Processing →
OpenAI API → Structured Output → Business Logic → Response8.4 The Project We Will Build
We’ll build an AI Document & Image Analyzer supporting images, PDFs, text, and audio, producing summaries, extracted information, classifications, and structured JSON.
We’ll build it in stages: Stage 1 — image analysis. Stage 2 — PDF analysis. Stage 3 — audio transcription. Stage 4 — a combined multimodal business workflow.
8.5 Industry Project Structure
multimodal-ai-analyzer/
│
├── src/
│ └── multimodal_analyzer/
│ │
│ ├── main.py
│ │
│ ├── config/
│ │ ├── __init__.py
│ │ └── settings.py
│ │
│ ├── clients/
│ │ ├── __init__.py
│ │ └── openai_client.py
│ │
│ ├── models/
│ │ ├── __init__.py
│ │ └── analysis_models.py
│ │
│ ├── services/
│ │ ├── __init__.py
│ │ ├── image_analysis_service.py
│ │ ├── document_analysis_service.py
│ │ └── audio_analysis_service.py
│ │
│ ├── validators/
│ │ ├── __init__.py
│ │ └── media_validator.py
│ │
│ └── utils/
│ ├── __init__.py
│ └── logging_config.py
│
├── input/
│ ├── images/
│ ├── documents/
│ └── audio/
│
├── tests/
│ ├── test_image_analysis.py
│ ├── test_document_analysis.py
│ └── test_audio_analysis.py
│
├── .env
├── .env.example
├── .gitignore
├── requirements.txt
└── README.mdThe design separates Input Validation → Media Processing → OpenAI Client → Analysis Service → Structured Result.
8.6 Configuration
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
openai_api_key: str
class Config:
env_file = ".env"
settings = Settings().env.example:
OPENAI_API_KEY=your_api_key_here.gitignore:
.env
__pycache__/
.pytest_cache/8.7 OpenAI Client
from openai import OpenAI
from multimodal_analyzer.config.settings import settings
def create_openai_client() -> OpenAI:
return OpenAI(
api_key=settings.openai_api_key
)This keeps client creation centralized, as in previous chapters.
8.8 Image Understanding
A vision-capable model can receive an image alongside text instructions. The Responses API supports image inputs using image URLs, uploaded file IDs, or base64 data URLs.
Text Question + Image → AI Model → AnswerExample:
Image: [Invoice]
Question: What is the invoice total?
Response: The invoice total is ₹25,400.8.9 Basic Image Analysis
response = client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Describe this image.",
},
{
"type": "input_image",
"image_url": image_url,
},
],
}
],
)The image and text question are sent to the model together in a single request.
8.10 Image Analysis Service
class ImageAnalysisService:
def __init__(self, client) -> None:
self.client = client
def analyze(
self,
image_url: str,
question: str,
) -> str:
response = self.client.responses.create(
model="gpt-5",
input=[
{
"role": "user",
"content": [
{
"type": "input_text",
"text": question,
},
{
"type": "input_image",
"image_url": image_url,
},
],
}
],
)
return response.output_textThis service has exactly one job: analyze an image based on a question.
8.11 Example: Product Damage Analyzer
Input: damaged_product.jpg
Question:
Analyze this product image.
Identify:
1. Whether visible damage exists.
2. The type of damage.
3. The approximate severity.
4. Any visible uncertainty.Possible response:
Visible damage: Yes
Damage type: Crack on the outer casing.
Severity: Moderate.
Uncertainty: The internal condition cannot be determined from the image.Notice the important principle here: the AI should distinguish visible evidence from assumptions — it explicitly flags what it can’t determine rather than guessing confidently.
8.12 Structured Output
Instead of "The product appears damaged and the damage looks moderate," a production application needs:
{
"damage_detected": true,
"damage_type": "crack",
"severity": "moderate",
"confidence": "medium",
"uncertainty": [
"Internal damage cannot be determined from the image."
]
}Architecture: Image → AI → Structured JSON → Application Logic. For example, damage_detected = true can directly trigger Create Support Ticket.
8.13 Image Validation
Before sending an image to an AI API, validate that it exists, is a supported format, is within size limits, and is readable:
from pathlib import Path
ALLOWED_IMAGE_TYPES = {
".jpg",
".jpeg",
".png",
".webp",
}
def validate_image(path: str) -> None:
file_path = Path(path)
if not file_path.exists():
raise FileNotFoundError(
f"Image not found: {path}"
)
if file_path.suffix.lower() not in ALLOWED_IMAGE_TYPES:
raise ValueError(
"Unsupported image format."
)8.14 Why Validation Matters
Without validation: User → Invalid File → API Request → Application Failure (a raw, unhelpful crash).
With validation: User → Validate → Invalid? → Yes → Clear Error / No → Process. A message like Invalid image format. Supported formats: .jpg, .jpeg, .png, .webp is far better than a generic Internal Server Error.
8.15 Image Use Cases
Receipt Analyzer — extract merchant, date, items, total, tax from a receipt photo.
Screenshot Debugger — identify errors, UI problems, and possible causes from a screenshot.
Visual Quality Inspection — detect visible defects in product images.
Accessibility Assistant — generate descriptions of images for visually impaired users.
8.16 PDF and Document Understanding
A PDF may contain text, tables, charts, images, and diagrams. Simple text extraction can grab paragraphs but often loses layout, tables, and visual relationships. A multimodal model can reason about the document as a much richer input. OpenAI’s API supports sending files and PDFs as inputs for analysis.
8.17 PDF Analysis Workflow
PDF → Validate → Upload → Send to AI → Analyze → Structured ResultExample:
Invoice.pdf → AI → {
"vendor": "...",
"invoice_number": "...",
"total": "...",
"due_date": "..."
}8.18 Invoice Extraction Project
Input: invoice.pdf
Question:
Extract the following:
- Vendor name
- Invoice number
- Invoice date
- Total amount
- Tax amount
- Due dateExpected output:
{
"vendor_name": "ABC Technologies",
"invoice_number": "INV-2026-001",
"invoice_date": "2026-07-20",
"total_amount": 25400,
"tax_amount": 3879,
"due_date": "2026-08-20"
}This output can flow directly into an accounting system, invoice database, and approval workflow.
8.19 Production Invoice Pipeline
Invoice Uploaded → File Validation → AI Document Analysis →
Structured JSON → Schema Validation → Business Rules → DatabaseExample:
Extracted Total: ₹25,400
Is amount > ₹50,000?
├── Yes → Manager Approval
└── No → Normal ProcessingThis combines multimodal AI + structured output + business logic — the recurring pattern across this entire course.
8.20 Audio AI
Audio applications usually have multiple steps:
Audio → Speech-to-Text → Transcript → AI Analysis → SummaryOpenAI provides audio transcription capabilities through the audio transcription API, with supported audio formats including common formats such as MP3, MP4, M4A, OGG, WAV, and WebM.
8.21 Meeting Summarizer
Input: meeting.mp3
Audio → Transcription → Transcript → AI → SummaryOutput:
Meeting Summary
Key Decisions:
- Launch moved to August 15.
Action Items:
- Ravi: Update deployment plan.
- Priya: Complete testing.
Risks:
- API integration remains incomplete.8.22 Transcription Service
class AudioAnalysisService:
def __init__(self, client) -> None:
self.client = client
def transcribe(
self,
audio_file,
) -> str:
transcription = (
self.client.audio.transcriptions.create(
model="gpt-4o-transcribe",
file=audio_file,
)
)
return transcription.textThe current transcription API supports multiple transcription models and options such as language hints, timestamps for supported formats, and diarization capabilities for supported workflows.
8.23 Audio → Transcript → Summary
Production workflow:
Audio File → Validate → Transcribe → Transcript Validation →
Summarize → Extract Action Items → Store ResultMeeting Audio → Transcript →
┌─────────────────────┐
│ Summary │
│ Decisions │
│ Action Items │
│ Risks │
└─────────────────────┘8.24 Speaker Diarization
A meeting may contain multiple speakers. A diarized transcript identifies speaker segments:
[Speaker A] We should launch next week.
[Speaker B] Testing is not complete.
[Speaker A] Let's move the launch date.The AI can then generate: Decision: Launch date moved due to incomplete testing. This is useful for meetings, interviews, customer calls, and call centers.
8.25 Text-to-Speech
The reverse workflow: Text → Text-to-Speech Model → Audio.
AI: Your order has been shipped.
↓
Audio: Spoken responseOpenAI provides speech generation through the audio speech API, with configurable models and voices.
8.26 Voice Assistant Architecture
User Speaks → Audio Input → Speech Recognition → Text →
AI Agent → Text Response → Text-to-Speech → Audio ResponseComplete pipeline: Microphone → Speech-to-Text → LLM → Tools / RAG → Text-to-Speech → Speaker. This connects everything from previous chapters: audio + LLM + tools + RAG + agents, all in one system.
8.27 Multimodal AI + RAG
This combination is very powerful. Suppose a user uploads a product image:
Image → AI Vision → Identify Product → RAG Search →
Product Documentation → AnswerExample:
User: [Uploads image of machine]
Question: How do I fix this error?System: analyze image → identify error code → search technical documentation → generate grounded instructions.
Image → Vision → Error Code → RAG → Technical Documents → AI → Solution8.28 Multimodal AI + Function Calling
User uploads invoice → AI extracts: Total = ₹75,000 →
Function Calling → create_approval_request()Invoice → AI → Structured Data → Business Rule → Tool Callif invoice.total_amount > 50000:
create_approval_request()8.29 Complete Enterprise Workflow
User Uploads Invoice → File Validation → AI Vision →
Structured Extraction → Schema Validation → Business Rules
│
┌───────┴───────┐
▼ ▼
Approval Database
ToolThis is the kind of architecture worth internalizing as a GenAI developer.
8.30 Strong Multimodal Prompt
Weak:
Analyze this image.Better:
You are an invoice extraction system.
Analyze the uploaded invoice.
Extract:
- Vendor name
- Invoice number
- Invoice date
- Due date
- Subtotal
- Tax
- Total amount
Rules:
1. Do not guess missing values.
2. Use null for unavailable fields.
3. Preserve the original currency.
4. Return only the requested structured data.The strong version defines role, task, fields, rules, and output behavior explicitly.
8.31 Handling Uncertainty
Suppose an invoice image is blurry. Bad: confidently stating Total: ₹25,400 anyway. If the image is unclear, the system should instead return:
{
"total_amount": null,
"uncertainty": "The total amount is not clearly readable."
}This is far better than inventing a plausible-sounding value.
8.32 Confidence and Human Review
AI Result → Confidence / Validation →
High → Automatic Processing
Medium → Review Queue
Low → Human ReviewExample: Invoice Extraction → Total clearly visible? → Yes → Process Automatically / No → Human Review. AI should not always be the final decision-maker, especially for financially consequential decisions.
8.33 Multimodal Security
Media inputs can carry sensitive information: identity documents, invoices, private conversations, customer photos, voice recordings. Consider access control, data retention, encryption, sensitive data handling, file deletion, and audit logging. OpenAI’s platform documentation also describes data controls for image and file inputs, including handling and retention considerations that should be reviewed for your deployment requirements.
8.34 File Lifecycle
A secure file pipeline:
Upload → Validate → Process → Extract Result →
Store Required Data → Delete Temporary FileExample: invoice.pdf → AI Extraction → JSON Result Saved → Temporary File Deleted. Don’t keep files forever without a specific reason.
8.35 Testing Multimodal Applications
Test valid images (expected analysis), invalid images (validation error), blurry images (uncertainty response), missing fields (e.g., invoice without due date → due_date = null), and audio with multiple speakers (speaker segments correctly identified).
8.36 Example Test
def test_invalid_image_extension():
with pytest.raises(ValueError):
validate_image(
"document.exe"
)def test_missing_invoice_field():
result = extract_invoice(
invoice_without_due_date
)
assert result.due_date is None8.37 Production Observability
Log request ID, user ID, input type, file size, processing time, model used, error, and output status:
request_id=abc123
input_type=image
processing_time=2.4s
status=successAvoid logging sensitive media contents directly — full passport images, full private audio, or complete sensitive invoices — unless your data governance explicitly allows it.
8.38 Cost and Performance
Multimodal inputs can be more expensive than plain text. For images: resize when appropriate and remove unnecessary content before sending. For audio: segment long recordings before transcribing. For documents: extract only relevant pages rather than analyzing an entire huge document. The guiding principle: send the minimum necessary input required to solve the task.
8.39 Capstone Project: Intelligent Expense Processing System
User uploads invoice.pdf.
System: validate PDF → analyze document → extract invoice fields → validate schema → apply business rules → store invoice → request approval if needed.
Invoice PDF → File Validation → Multimodal AI → Structured JSON →
Schema Validation → Business Rules
│
┌──────┴──────┐
▼ ▼
Database Approval Tool8.40 Example Capstone Result
Input: invoice.pdf
Output:
{
"vendor_name": "ABC Technologies",
"invoice_number": "INV-2026-001",
"invoice_date": "2026-07-20",
"currency": "INR",
"subtotal": 21521,
"tax": 3879,
"total": 25400,
"requires_approval": false
}Business logic:
if invoice.total > 50000:
requires_approval = True
else:
requires_approval = False8.41 Advanced Capstone: Multimodal Support Agent
Now combine Chapters 5–8:
Customer
├── Text
├── Image
└── Audio
│
▼
AI Support Agent
│
┌────┼─────────┐
▼ ▼ ▼
RAG Order Tool Ticket Tool
│
▼
Final AnswerExample:
Customer: [Uploads image of damaged product]
"This arrived damaged. Can I get a replacement?"The agent: analyze image → identify visible damage → search return policy using RAG → check order using a tool → create replacement request → respond to customer. This is a complete production-style AI workflow, pulling together everything from the last four chapters.
8.42 Practical Exercises
Exercise 1: Image Description Assistant — Describe the image for a visually impaired user.
Exercise 2: Receipt Extractor — extract merchant, date, items, subtotal, tax, total from receipt.jpg.
Exercise 3: Screenshot Debugger — Identify the visible error and suggest possible debugging steps.
Exercise 4: Meeting Summarizer — process meeting.mp3 into summary, decisions, action items, risks.
Exercise 5: Multimodal RAG — from a machine image: identify error code → search documentation → generate solution.
Chapter 8 Checklist
You should now understand: what multimodal AI is, image understanding, image validation, PDF analysis, invoice extraction, audio transcription, text-to-speech, speaker diarization, structured multimodal output, uncertainty handling, human review, multimodal + RAG, multimodal + function calling, multimodal + agents, security, file lifecycle, testing, observability, and cost optimization.
Central architecture:
Media Input → Validation → AI Understanding → Structured Output →
Validation → Business Logic → ActionThe most important principle: multimodal AI should not simply “look at files.” It should convert real-world media into reliable, validated information that your application can use.
Next chapter: Chapter 9 — AI Safety, Security & Responsible AI, where we’ll cover prompt injection, sensitive data, access control, hallucination mitigation, content safety, and production security.