🌑

Stephen's Blog

Top Interview Questions for AI Engineers

Stephen Cheng

 

Intro

As an AI engineer, we build and deploy AI systems in production environments. Then they turn models like LLMs into reliable applications that deliver business value. In this post, we’ll cover interview questions on LLMs, core AI concepts, system design, deployment and MLOps, coding and implementation, and real-world scenarios for AI engineer interviews.

Basic

What is the difference between an AI model and a machine learning model?

An AI model is any system that performs tasks that predominantly require human intelligence, including some rule-based or symbolic approaches. A machine learning model, on the other hand, is a subset that learns patterns directly from the fed data. In production terms, AI engineers most often work with ML models, especially LLMs, but must also integrate some non-ML components, such as rule engines or knowledge graphs, when needed for a highly reliable output.

What is an LLM?

A large language model is, as the name suggests, a large, transformer-based neural network that is trained on massive text corpora to generate coherent text. In production systems, LLMs serve as the reasoning engine behind the chatbots, summarizers, and agents.

What is tokenization?

Tokenization converts raw text into numerical tokens that the model can process. It directly affects the cost, the context length, and how the model sees text fundamentally. In other words, text segments are assigned numerical identifiers, called tokens, so the text is ready to be fed to the model. Production systems use subword tokenizers like BPE and monitor the token counts to control the API expenses and latency.

What is a prompt?

A prompt is the input text that is fed to the LLM to produce some output. It is generally an engineered artifact that contains instructions, sometimes examples, and a full context to maximize the adaptability of the LLM.

What is the difference between fine-tuning and prompting?

Prompting changes the behavior at inference time. However, fine-tuning updates the model weights, which is slower but generally leads to more customized large language models. This makes prompting faster and cheaper for most production use-cases.

What is retrieval-augmented generation RAG?

RAG retrieves the relevant documents from a knowledge base that you assign and adds them to the prompt. The LLM therefore uses that extra verified data to predict more reliable information and reduce hallucinations.

Core Concepts

What are embeddings, and how are they used?

Embeddings are dense vector representations that understand semantic meanings. They help with semantic search and retrieval in vector databases. You can think of an embedding as a table that takes the numerical input and assigns it a high-dimensional vector in a different, more representative space.

How do we evaluate a large language model system?

Combine automated metrics, like faithfulness and relevance, with human review and business KPIs. In production, we basically require a continuous evaluation pipeline.

What causes hallucinations in LLMs? How do we reduce hallucinations?

Gaps in the training data and the overconfident nature of next-token prediction can force the model to predict tokens without sufficient grounding, which is known as hallucination.

RAG is one of the most reliable and efficient methods, since it delivers grounding against trusted sources you provide. Combining it with structured output formats, self-consistency checks, and fact-checking further reduces hallucinations.

LLM and Generative AI

How do transformers work at a high level?

They process in parallel using self-attention instead of a recurrence. Modern large language models are decoder-only and predict the tokens autoregressively.

What is attention?

Attention computes weighted relevance between the tokens to allow for long-range dependencies regardless of their position.

What is a context window?

The maximum number of tokens the model can process in one inference call. It limits prompt design and conversation history management.

What is the temperature in generation?

It controls the randomness in sampling: lower values produce more deterministic output, while higher values increase creativity. In production systems, it’s tuned per use case to balance reliability and variety.

What is prompt engineering?

The systematic design, testing, verification, tuning, and iteration of prompts to achieve the best reliable behavior. Prompts are versioned with code in production and depend heavily on the model itself rather than a generic template.

AI System Design and Deployment

How to design a chatbot using an LLM?

Frontend, conversation memory, prompt orchestration, LLM API call, output validation, logging, rate limiting, and cost tracking.

How to reduce latency in an AI system?

Apply prompt compression, caching, smaller or distilled models, speculative decoding, and hardware acceleration.

How tp evaluate and monitor outputs?

Log requests/responses, run automated quality scores, sample for human review, track business metrics, and set degradation alerts.

How do you deploy an LLM-backed application?

Containerize with FastAPI, integrate via SDKs or self-hosted servers like vLLM, orchestrate with Kubernetes or serverless, and use CI/CD for prompts and code.

What tools are used for AI deployment?

Docker/Kubernetes, vLLM or Text Generation Inference, vector databases like Pinecone or Weaviate, LangSmith, and cloud platforms like AWS Bedrock, Azure OpenAI, or GCP Vertex AI.

What is model drift in AI systems?

The degradation is caused by some changes in input distribution, the user behavior, or the data sources. It appears as a declining relevance or rising hallucinations.

How do you scale inference?

Apply continuous batching, quantization like 4/8-bit, model parallelism, and engines such as vLLM or TGI.

, , — Jul 28, 2026

Search

    Made with ❤️ and ☀️ on Earth.