AI Engineering and LLMOps Interview Questions: Scenario-Based Answers

1. How Would You Debug a Wrong RAG Answer?

Separate retrieval from generation. Verify source ingestion and metadata, then inspect top-k IDs, scores, filters, and text for the exact query. Confirm ingest and query use the same embedding model and normalization. If evidence is missing, fix retrieval. If good evidence is present but the answer contradicts it, inspect prompt assembly and faithfulness. Report source IDs and test a no-answer case.

2. Why Is Embedding Dimension a Contract?

The vector store collection expects a fixed dimension, but model identity matters beyond shape. Two models can emit the same dimension with incompatible semantic spaces. Record model digest, dimension, normalization, and chunker version with the index and rebuild rather than mixing versions.

3. What Do MLflow and DVC Each Track?

DVC versions datasets and large artifacts through pointer files committed with code. MLflow records experiment runs, parameters, metrics, tags, and model artifacts. Connect them by logging Git and DVC revisions into every MLflow run. A model is reproducible only when code, data, environment, parameters, and evaluation split can all be resolved.

4. Which Model Metrics Would You Use?

Choose metrics from the cost of errors. For imbalanced classification, accuracy alone is weak; inspect precision, recall, F1, class distribution, and the confusion matrix. For RAG, score retrieval relevance and answer faithfulness separately. Define release thresholds against a stable holdout set and compare with the deployed baseline.

5. How Do You Measure LLM Latency?

Record queue time, time to first token, inter-token latency or tokens per second, and total response time. Tag by model and workload class. A slow first token suggests queueing, loading, or prompt processing; slow generation suggests compute or cache pressure. Use percentiles, not averages alone.

6. How Would You Secure a Terminal Agent?

Replace arbitrary shell access with typed, allowlisted tools. Run least-privileged identities in isolated workspaces. Treat retrieved content and tool output as untrusted data. Require explicit approval for destructive or production actions. Bound iterations and time, independently verify results, redact traces, and log every authorization decision.

7. What Is Prompt Injection?

It is untrusted input attempting to alter the model's instructions or tool behavior. It can arrive through user text, web pages, tickets, logs, or RAG documents. Prompt wording alone is not a security boundary. Enforce tool and data access in deterministic code, isolate secrets, label trust boundaries, and test poisoned inputs.

8. Liveness Versus Readiness for a Model Server?

Liveness proves the process responds. Readiness proves the required model, tokenizer, storage, and dependencies are loaded and able to serve. A process can be alive while loading a model or after losing its model store. Only ready instances should receive traffic.

9. When Should a Model Proxy Fall Back?

Fallback on defined transient failures with bounded retries and a compatible alternative. Validate context length, response schema, tool support, policy, and data residency. Trace the route and actual served model. Never hide a fallback that materially changes quality or capability.

10. How Do You Prevent PII Leakage?

Minimize collection, avoid logging raw prompts by default, classify sensitive fields, redact both complete and streamed output, and use allowlists for structured responses. Regex catches known formats but needs contextual or model-assisted detection for broader entities. Test chunk boundaries and preserve an auditable redaction decision without retaining the original value unnecessarily.

11. What Does Continuous Batching Solve?

It schedules tokens across active sequences and admits new work as others finish, improving accelerator utilization compared with waiting for fixed batches. It introduces queue and KV-cache capacity tradeoffs. Monitor TTFT, throughput, active sequences, cache use, and rejection rate together.

12. Describe a Strong AI Release Gate

Pin code, data, model, embedding, prompt, and evaluator versions. Run classical or RAG evaluation against a stable, representative set. Require operational checks for latency, error rate, readiness, injection resistance, guardrail behavior, and trace completeness. Preserve the evidence and provide rollback to the prior versioned bundle.

Build concrete examples through the eight-phase AI Engineering & LLMOps course and its 17 incident-based labs.