Model Builders live on the front lines of capability and accountability in the fast-moving AI space. Observability is how Model Builders learn quickly, ship safely, and defend margins. It is key to how they translate model ambition into user trust and business results. The formula is simple, even if the systems aren’t: get the signals right, bind them to SLOs that reflect real experience, tag them to the business, and iterate relentlessly. If Part 1 of this 5-part blog series on AI observability set the foundation, Part 2 shows how observability becomes the operating system for building and scaling LLMs.

This blog is part two of our five-part series, explaining how different segments of the AI market approach observability. In part two here, we’ll dive into LLM Model Builders and explain how observability drives success for these companies. Here’s the rest of our series:

Part 1: Overview
Part 2: LLM Model Builders (You are here)
Part 3: GPU Providers (Coming on 10/28)
Part 4: AI-Natives (Coming on 11/4)
Part 5: Feature Builders (Coming on 11/11)

When we say “Model Builder,” we mean the teams behind the hottest thing in AI today, the large language models or LLMs (think OpenAI, Anthropic, xAI, and the many builders inside enterprises standing up specialized LLMs). LLMs are a type of artificial intelligence trained on vast amounts of text that can understand context and generate human-like responses. LLMs power everything from ChatGPT to customer service bots to code assistants.

Click here for the full glossary of AI terminology.

The work of the Model Builders includes shaping model architectures and data pipelines, running massive model training jobs, and serving low-latency inference. Inference is the process of using a trained AI model to make predictions, decisions, or generate new content based on new, unseen data.

• For Model Builders, the work never really ends; models evolve, datasets shift, and user expectations climb. Day to day, LLM Model Builders move through a build-operate-learn loop.
• They evaluate model quality.
• They ship new versions behind safe rollout gates.
• They’re also on the hook for margins: GPUs aren’t cheap, and training runs cost real money and trust.

In other words, they are chasing state-of-the-art results, but they’re also running a business.

There are four key steps used by Model Builders in the process of building, launching, and serving large language models.

Step 1: Training

Training is the initial phase where a large language model is built from scratch, or a pre-existing base model is further developed. This involves feeding the model vast amounts of data to learn patterns, grammar, facts, and general knowledge, hardening its core capabilities.

Step 2: Fine-tuning

Fine-tuning takes a pre-trained base model and adapts it to a specific domain or task. This involves training the model on a smaller, more targeted dataset relevant to the desired application, allowing it to specialize and perform better in that particular context.

During both training and fine-tuning, Model Builders monitor:

• Step time: The total time taken to complete a single training step.
• Tokens/sec: The rate at which tokens are processed per second, indicating model throughput.
• GPU utilization by stage: The percentage of time the GPU is actively working during different stages of the training process.
• Data-loader wait: The time spent waiting for data to be loaded from storage, which can indicate I/O bottlenecks.
• Host-to-device throughput: The speed at which data is transferred from the host CPU to the GPU.
• Collective-comms stalls: Delays caused by communication operations between multiple GPUs in a distributed training setup.

Step 3: Retrieval-augmented generation

Retrieval-augmented generation (RAG) enhances the model’s ability to generate accurate and up-to-date responses by incorporating a retrieval step. Before generating a response, the model fetches relevant information from a knowledge base or external data source, and then uses this fresh context to inform its output.

During RAG, these key metrics tell us if answers will be relevant and fast:

• Retrieval Latency: The time it takes for a system to retrieve requested information or data.
• Recall@K: A metric used to evaluate the performance of retrieval systems, representing the proportion of relevant items found within the top K results.
• Index Build/Compact Times: The time required to create a data index or to optimize and reduce the size of an existing index.
• Source Freshness: How up-to-date or recent the information from a particular data source is. tell us if answers will be relevant and fast.

Step 4: Inference

Inference is the final stage where the trained, fine-tuned, and potentially RAG-augmented model is put into action to generate a response or make a prediction based on a given input from a user. Inference is when the model gets put into “Production” and is put to use. This is where all the previous work culminates in delivering user outcomes.

In inference, the following define user experience and token economics:

• Request rates: The number of requests a system receives per unit of time.
• P50/P95/P99 latency: Percentiles indicating the time taken for a certain percentage of requests to be completed. P50 (median) means 50% of requests are faster than this time, P95 means 95% are faster, and P99 means 99% are faster.
• Timeouts: A preset time limit for a process or request to complete. If the limit is exceeded, the process is aborted.
• Queue depth: The number of items or requests waiting to be processed in a queue.
• Tokens/sec per GPU: The rate at which a Graphics Processing Unit (GPU) can process tokens (units of text or data) per second.
• KV-cache behavior: How the Key-Value cache, often used in large language models to store attention keys and values for efficiency, is being utilized and managed.
• Autoscaling events: Occurrences where a system automatically adjusts its resources (e.g., adding or removing servers) based on current demand.

The key is to stitch these signals with metrics, events, logs, and traces so you can move from “it feels slow” to “this node pool’s batcher is thrashing the cache.”

The OpenTelemetry open source observability framework is a great way to gather this telemetry while ensuring flexibility and utility going forward. Open-source SDKs like OpenInference and OpenLLMetry gather much of the above key telemetry directly from the LLMs and make it available in OpenTelemetry format for integration into observability platforms.

Likewise, GPU vendors like NVIDIA provide powerful tooling like their Datacenter GPU Management (DCGM) system for exporting GPU performance and utilization metrics in Prometheus format for observability ingestion.

Model Builders win on three axes: model performance, training efficiency, and inference optimization.

• Model performance is quality under real conditions: factuality, safety, coherence, and responsiveness. Miss here and adoption stalls.
• Training efficiency is time-to-fitness per dollar. Idle GPUs and I/O bottlenecks aren’t just technical nits; they’re roadmap drag.
• Inference optimization is the margin lever. Every millisecond off P99 and every uptick in tokens/sec/GPU compounds across traffic.

If they get these success drivers right, the macro tailwind for Model Builders is massive: A McKinsey report sizes generative AI’s potential at $2.6T–$4.4T in annual value, but short-term returns depend on disciplined execution which is what observability enables.

Model performance

For model performance, it’s important to treat evaluations as first-class workloads.

• In AI, evaluations are the process of assessing a model’s performance and effectiveness on various tasks, typically using metrics and test datasets.
• In addition, you can track model drift by measuring how far apart the current performance of the model is from its expected performance, using techniques like embedding distances, which measure the similarity between data points in a high-dimensional space.
• Monitor the model’s success rate compared to a baseline (a “champion” model). If any of the built-in safeguards (guardrails) are triggered, identify the specific inputs that caused them to activate, allowing you to understand and address potential issues.

The payoff is confidence in shipping: when online quality dips, you can pinpoint if it’s the model, the retrieval layer, or a silent schema change upstream.

Training efficiency

For optimal training efficiency and to gain comprehensive insights into model performance, it is crucial to expose stage-level telemetry throughout the entire end-to-end process. This means instrumenting each distinct stage of the training pipeline – from data ingestion and preprocessing to model training, validation, and deployment – with robust telemetry collection mechanisms.

By doing so, key metrics can be meticulously monitored, bottlenecks identified, and areas for optimization pinpointed at every step. This granular visibility allows for quicker debugging, more informed decision-making regarding hyperparameter tuning, and ultimately, a more streamlined and efficient training workflow.

Inference optimization

For inference optimization, Service Level Objectives (SLOs) are centered on user-visible experience like 99.9% of requests under 500 ms with <0.5% timeouts.

• • The batcher (a component that groups multiple requests together for more efficient processing), queues (data structures that hold requests waiting to be processed.)
  • KV-cache (a memory cache used to store key-value pairs, often for attention mechanisms in large language models) are also closely monitored.
  • Spiky traffic can hide inside rolling averages, so time-slice views and per-tenant breakdowns are used to catch the five-minute windows that matter.

Google’s SRE playbook remains the north star: make reliability measurable, then manage it.

Even world-class teams get tripped up. Anthropic recently published a postmortem on three infrastructure bugs that intermittently degraded Claude’s responses; the takeaway was: instrument deeply, detect faster, and design rollouts to reduce blast radius. That’s observability doing its job.

Start where risk and cost are highest. Instrument step time and data-loader wait before chasing exotic optimizations.

• Make P99 a first-class citizen; averages will lie to you.
• Standardize tags (model_name, model_version, dataset_id, run_id, tenant_id, region) so costs and SLOs roll up to the things you sell.
• Treat your vector DB like a core service with its own SLOs.
• Integrate evaluations into your deploy pipeline so you’re correlating quality shifts with infra changes, not guessing.

Observability is mission critical for AI Model Builders. In plain terms: better visibility helps them ship improvements faster, catch issues before customers feel them, and control GPU spend—so AI stays reliable, useful, and cost-effective.

(If you missed it, circle back to 5 keys to AI observability, Part 1: Foundations for a refresher on telemetry strategy, and later in this series we’ll dive into GPU Providers, AI-Natives, and Feature Builders for the wider ecosystem view.)

What do we actually mean by “Observability for AI Model Builders”?

We’re talking about end-to-end visibility for teams that build, fine-tune, and serve LLMs. It connects training, RAG, and inference telemetry so we can link user outcomes (latency, accuracy, reliability) to causes (data quality, GPU bottlenecks, batcher behavior) and make decisions fast.

How do I start with observability without boiling the ocean?

Begin where risk is highest. We start with three boards:

• Training efficiency: step time, tokens/sec, GPU util by stage, data loader wait, NCCL/all-reduce time.
• Inference health: request rate, P95/P99 latency, errors/timeouts, queue depth, tokens/sec/GPU, KV-cache evictions.
• RAG quality: retrieval latency, hit rate, recall@K, index build time, feature freshness.

Then we bind a few SLOs to user impact and iterate.

Which SLOs make sense for LLM systems?

Use user-visible SLOs (e.g., “99.9% of requests < 500 ms,” “<0.5% timeouts”) plus component SLOs (retrieval < 50 ms, index freshness < 5 min, batcher queue depth < N). For spiky traffic, time-slice SLOs catch windows where rolling averages hide pain. See our internal [Timeslice SLOs] primer.

How do I detect model drift in production?

We pair offline evals with online signals: embedding distance vs. a baseline, win-rate against a champion, guardrail triggers, and user feedback. We also track upstream feature freshness and schema changes. Shadow traffic and progressive rollouts (canary/A-B) reduce blast radius.

What tags should I standardize across metrics, logs, and traces?

The minimum viable schema: model_name, model_version, dataset_id, run_id, tenant_id, region, endpoint, batch_size, lr_schedule. These tags unlock per-model cost and SLO roll-ups, help with noisy-neighbor isolation, and make incident timelines coherent.

How do I keep observability costs in check while expanding coverage?

Understand the cost vs. value delivered for every piece of telemetry in your system. With this information you can make smart optimization decisions. Drop unused/low value telemetry. Adapt collection strategies dynamically based on circumstances without redeploying services or collectors. Set quotas to hold teams accountable and ensure your observability spend is predictable.

What does “good” look like for training efficiency?

Short answer: GPUs spend time pushing tokens, not waiting. We look for:

• High sustained tokens/sec and GPU util per stage.
• Low data loader wait and stable step time.
• Predictable checkpoint duration and minimal OOM/retry churn.
  If those trend the right way, time-to-fitness and cloud spend follow.

How do I monitor RAG and the vector database like a first-class service?

Treat it as prod: SLOs for recall@K, index build/compact times, connector timeouts, cache hit rate, and staleness of embeddings and sources. Correlate retrieval latency with answer quality and tail latency—those two tend to move together during incidents.

What’s the playbook for multi-tenant inference and “noisy neighbors”?

Track per-tenant: request rate, P99, errors/timeouts, tokens/sec, cost, and limits. Add circuit breakers and fair rate-limits. Watch batcher efficiency and KV-cache hit rate; tune batch size and cold-start paths. Per-tenant SLOs make it obvious when one customer is hurting others.

How do I integrate with existing tooling and OpenTelemetry?

Standardize on OpenTelemetry for traces/metrics/logs, export through a pipeline that can enrich with tags, and consolidate views so model, infra, and business telemetry live together.