Multimodal AI interview questions
Multimodal AI questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
What is the difference between running OCR on an image and giving the image to a vision-capable model?
The strongest answer: OCR transcribes characters, often with positions and confidence, cheaply and literally. A vision model interprets the image and answers questions about it, but costs more and can paraphrase or invent text. Exact transcription or understanding decides
Why it works: Right — literal transcription versus interpretation. Many pipelines use both: OCR for the exact characters, a model for what they mean.
- Transcription versus interpretation.
- OCR: literal text, positions, confidence.
- Vision model: meaning, at higher cost.
- Verify anything that must be exact.
Marketing wants the image generator to produce banners with the exact product name and price, and the same mascot in all ten. What do you tell them?
The strongest answer: It samples a fresh image from the prompt, so mood and composition are its strength; exact text, precise layout and an identical character each time are not guaranteed. Generate the art, overlay the text in code
Why it works: Exactly — a generator is a sampler, not a layout engine. Let it draw, and render names and prices with ordinary, testable code.
- A generator samples; it does not lay out.
- Strength: style, mood, composition.
- Exact text and identity are not guaranteed.
- Overlay text in code; review before publishing.
Your voice assistant chains speech-to-text, an LLM and text-to-speech. Each stage looks fast alone, but users say it is slow to answer. Where does the time go?
The strongest answer: The stages add up because each waits for the last to finish. Stream end to end: partial transcripts, streamed tokens, speech starting on the first sentence. Tune end-of-speech detection and measure time to first audio
Why it works: Right — perceived latency is time to first audio, and sequential hand-offs inflate it. Streaming overlaps the stages; end-of-speech detection decides when the clock even starts.
- Optimise time to first audio.
- Stream between every stage.
- Tune end-of-speech detection.
- Trace latency per stage.
A chat feature sends the user’s full-resolution photo to a vision model, and re-sends it with every follow-up message. What do you look at?
The strongest answer: An image costs tokens and latency, usually more the larger it is. Downscale or crop to what the task needs while text stays legible, keep an extracted description instead of re-sending, and check quality on test images
Why it works: Right — resolution drives cost and history re-sends multiply it. Resize to the task, send once, and prove on a test set that quality held.
- Image size drives cost and latency.
- Resize or crop to the task.
- Do not re-send the image each turn.
- Re-test quality after resizing.
Transcripts of your support calls read well, but customer names, product codes and order numbers are often wrong — and those are what the downstream summary depends on.
How do you approach it?
The strongest answer: Those are speech-to-text’s known weak spots. Measure error on our own audio for those entities, not overall; add vocabulary hints where supported, correct against the catalogue and order system, improve the audio, and flag low-confidence segments
Why it works: Exactly — a good overall word error rate hides failures on the rare words that matter. Measure per entity, then constrain with the lists you already own.
- Average accuracy hides entity errors.
- Measure on your own audio, per entity.
- Constrain with known lists and hints.
- Flag low-confidence segments for review.
Users upload dashboard screenshots and ask questions about them. The answers are fluent, but some of the quoted figures do not appear anywhere in the screenshot.
What do you check first, and what do you change?
The strongest answer: Resolution first: if we downscale, small digits become unreadable and the model guesses. Send legible crops or tiles, have it quote what it reads and say “unreadable”, cross-check digits with OCR, and keep real screenshots as evals
Why it works: Right — most “hallucinated” digits begin as pixels the model could not resolve. Fix the input, give it a way to say so, and verify exact values independently.
- Check what the model can actually see.
- Keep small text legible: crop or tile.
- Let it say “unreadable”; quote what it read.
- Verify exact values; eval on real screenshots.
A field technician sends a photo of a faulty device and a voice note describing the symptom. The repair manuals are PDFs. The assistant should suggest the next step.
How do you combine the inputs?
The strongest answer: Give each input its own step: transcribe the voice note and show it for correction, read the photo for model and visible fault, retrieve manual sections with that text, then assemble one source-labelled prompt with citations
Why it works: Right — per-modality steps give you checkpoints you can inspect, the photo’s findings sharpen retrieval, and source labels make the answer traceable.
- Each modality gets its own checkpoint.
- Transcribe, read the photo, then retrieve.
- Label every source in the prompt.
- Confirm identifiers; cite the manual.
For a voice product, how do you choose between a speech-to-speech audio model and a pipeline of speech-to-text, LLM and text-to-speech?
The strongest answer: A native audio model buys latency, tone and natural interruption; the pipeline keeps a text checkpoint for guardrails, tools, captions, logging and swapping parts. The use case decides, and I would prove it on our own calls
Why it works: Exactly — it is a trade of naturalness against control. Regulated or tool-heavy flows tend to want the text in the middle; a conversational companion may not.
- Naturalness versus control.
- Native audio: latency, tone, interruptions.
- Pipeline: text checkpoint, guardrails, captions.
- Decide per use case; test on real calls.
The product team wants the support bot to speak in a cloned voice of a popular real agent, and to use recorded customer calls to tune how it sounds.
What is your call as the engineering lead?
The strongest answer: Not until the basics exist: the agent’s explicit, written, revocable consent with a defined scope, callers told the voice is synthetic, and a lawful basis plus retention limits for the recordings. Meanwhile, a licensed stock voice
Why it works: Right — a voice identifies a person. Documented, scoped, revocable consent, disclosure to callers and a proper basis for the recordings come before any model work.
- A voice belongs to a person.
- Written, scoped, revocable consent first.
- Tell callers the voice is synthetic.
- Recordings need a basis and a limit.