How to Use AI Powered Observability Platform: A Practical Walkthrough

Why Your Dashboards Miss What AI Actually Breaks

Your status dashboard glows green, every request returns a 200, and latency sits comfortably under 100ms — and the model is still confidently telling a customer something false. That is the core problem with traditional observability: it watches infrastructure, not meaning.

AI systems fail in ways metrics, logs, traces, uptime checks, and status codes were never designed to catch. As Braintrust's framing of AI observability makes clear, a model can return a fast, successful HTTP response and still hallucinate, misstate facts, or produce harmful output. IBM research cited in that same analysis measured a mean coefficient of variation of 63% in execution paths for identical inputs to agentic systems — the same prompt does not reliably produce the same journey.

This is why the market is moving so fast. Gartner predicts that 40% of organizations deploying AI will use dedicated AI observability tools by 2028, and the sector is expanding at a 38.8% CAGR toward roughly $16.1 billion by 2034. The tooling is catching up to a gap that has existed since the first model hit production.

The cost of that gap is not abstract. New Relic's 2025 observability forecast puts the median outage at $2M per hour, halving to $1M per hour with full-stack observability — yet 73% of organizations still lack it. Engineers spend 33% of their time firefighting and another 33% on maintenance, while roughly 70% of collected observability data is unnecessary.

Tool sprawl makes it worse rather than better:

• New Relic reports the average organization now runs 4.4 observability tools, down 27% from 6 in two years.
• ECI Research, cited by The Cube Research, finds 75% of organizations still juggle six to 15 tools.
• 52% plan to consolidate onto unified platforms within 12–24 months.

Both figures are real, dated, and contradictory — a sign of how unsettled the stack still is. When you are stitching together half a dozen tools, nobody owns the question of whether the AI was actually right. Agent skills can automate parts of that monitoring workflow; SkillzBench is building a free directory and MCP connector to help teams discover and test them.

Gartner's Padraig Byrne names the gap plainly: AI is everywhere, but most organizations are still figuring out how to monitor and trust these systems — and that visibility gap makes scaling risky.

The Layered Approach: What to Instrument, In What Order

The hardest part of AI observability isn't choosing a tool — it's knowing what to instrument first. An AI system can return a healthy 200 status code in under 100 milliseconds and still produce factually wrong output that no infrastructure metric will ever catch, which is why the instrumentation plan must cover every layer of the stack, not just servers.

The clearest framework comes from Dynatrace's AI observability guidance, which lays out six layers to instrument in sequence:

• Application layer — user interactions and feedback events, like a thumbs-down click
• Orchestration layer — prompt/response pairs, retries, and tool execution timing
• Agentic layer — reasoning steps, memory state, and tool invocations
• Model/LLM layer — prompts, completions, token usage, cost, and failure patterns
• Vector database layer — embedding quality, retrieval relevance, and latency
• Infrastructure layer — GPU, memory, and network

The order matters because every step must be traceable — from a user's thumbs-down click in the UI, through the agent's decision graph and prompt processing, down to the underlying workloads. Skip a layer and you get gaps where quality problems hide.

Once instrumentation is in place, the implementation sequence is just as deliberate. Practitioner guidance breaks it into four steps: start with traces, define quality criteria early (accuracy, relevance, tone, compliance), build small production datasets — 10 to 200 examples is enough to start catching regressions — and close the loop connecting traces to evaluations to iteration. That last step matters because a single agent interaction can generate dozens of spans and megabytes of trace data; without evals, all that telemetry goes unread.

The non-determinism problem explains why. IBM research measured a 63% mean coefficient of variation in execution paths for identical inputs to agentic systems — the same prompt can take a completely different route through your agent. That's why prompt-tweaking alone fails. As Sarav Bhatia, Senior Director of Engineering at Navan, put it, he's "amazed at how many teams still think about continuing to build AI by just refining prompts all the time... rather than actually institutionalizing the process of evals." Dropbox VP of Engineering Josh Clemm goes further: "Evals have become the new PRD."

OpenTelemetry is emerging as the vendor-neutral default here, integrating with Datadog, Prometheus, and AWS CloudWatch. Its GenAI semantic conventions formalize standard attributes for model calls, token usage, and tool invocations, with a separate proposal extending coverage to agentic systems — so the traces you build today stay portable tomorrow.

The same traceability discipline applies to the building blocks of your agents themselves. Skills are code too, and roughly a quarter of published agent skills carry at least one vulnerability. SkillzBench is building free benchmarking, security scanning, and version-pinned verification so you can see exactly how a skill behaves before it enters your stack — join the waitlist to follow the launch.

Step-by-Step: Standing Up Your First AI Observability Loop

Standing up AI observability is less about installing a product and more about building a loop: instrument, evaluate, iterate. The good news is that a capability-by-capability sequence works across platforms, so you can start today without committing to a single vendor.

Step 1: Build on a vendor-neutral foundation. OpenTelemetry has become the emerging default standard for AI telemetry, and it integrates cleanly with Datadog, Prometheus, and AWS CloudWatch, according to observability trend analysis from CNCF. Starting here avoids lock-in and keeps your traces portable as tooling consolidates.

Step 2: Instrument one layer at a time. Begin with application and orchestration traces so prompt/response pairs, retries, and tool execution timing are captured before you chase model-level detail. Dynatrace's six-layer instrumentation model sequences this well — application, orchestration, agentic, model/LLM, vector DB, infrastructure — and the principle is simple: every step must be traceable and measurable, from a user's thumbs-down click through the agent's decision graph to the underlying workloads.

Step 3: Define quality criteria before scaling. As Arize AI's Aparna Dhinakaran puts it, evaluating non-deterministic products "no longer just becomes about is it right or wrong, but it becomes about actually measuring the quality of the responses," per The Cube Research. Lock in accuracy, relevance, tone, and compliance criteria early.

Step 4: Build your first evaluation dataset from real traffic. Ten to 200 production examples is enough to start catching regressions, and a single agent interaction can generate dozens of spans and megabytes of trace data. Treat this set as a living regression suite rather than a one-time artifact.

Step 5: Control cost from day one. Roughly 70% of collected observability data is unnecessary, so:

• Sample key traces instead of capturing everything
• Store only the logs that matter for debugging
• Tier less critical data to cheaper storage
• Attribute token usage to requests, model versions, components, or users

This discipline makes 60–80% cost reduction achievable — an unoptimized inference endpoint can burn thousands of dollars in minutes.

Step 6: Close the loop — carefully. Traces feed evals, evals feed iteration. Autonomous remediation is the direction of travel, but it requires guardrails, policy controls, validation mechanisms, and clearly defined boundaries before anything acts on its own. As Dynatrace's Steve Tack notes, automation confidence depends on the precision and accuracy of the analytics underneath.

Gartner's guidance for I&O leaders maps directly onto this: mandate AI model monitoring policies for production, standardize frameworks across teams, prioritize distributed tracing of AI inference calls, and fold AI monitoring and cost management into IT strategy. Agent skills can automate parts of this workflow — SkillzBench is building a free directory and MCP connector so you can browse, test, and equip skills per agent before trusting them in production.

Making It Stick: Standards, Consolidation, and What Comes Next

Getting AI observability to stick is less about the initial setup and more about the habits, standards, and governance you build around it. The market is already voting with its tooling: the average number of observability tools per organization dropped 27% over two years, from six down to 4.4, according to a 2025 observability forecast, and 52% of organizations plan to consolidate onto unified platforms within the next 12 to 24 months.

That said, the picture is not entirely settled. A separate analysis citing ECI Research found that 75% of organizations still run between six and 15 different observability tools. Both figures can be true depending on when and how they were measured — consolidation is a trend in motion, not a finished story.

Standards are what make consolidation work at scale. OpenTelemetry is emerging as the vendor-neutral default, integrating with Datadog, Prometheus, and AWS CloudWatch, while its GenAI semantic conventions formalize standard attributes for model calls, token usage, and tool invocations. Gartner's guidance for I&O leaders is explicit: standardize monitoring frameworks across teams rather than letting each one improvise. This matters because the endgame isn't just engineers — observability needs to become "usable for everyone," including non-IT domain experts who understand what a correct answer looks like.

Regulation is accelerating the timeline. EU AI Act enforcement deadlines, SEC and OCC model risk guidance, and EBA guidance are turning AI monitoring from a best practice into a compliance necessity. Market analysts expect vendors offering integrated ML/LLM monitoring with built-in EU AI Act documentation to become the standard governance layer for enterprise AI programs.

The convergence is already visible in M&A. Dynatrace's acquisition of Arize AI — a company whose co-founder argues that "observability is no longer about humans looking at dashboards... It's about action" — signals that AI observability and traditional observability are merging into a single discipline.

There's one more layer worth watching: the agentic one. Agent skills can automate much of this workflow — instrumentation, eval harnesses, even alert triage. The catch is quality control: published research found roughly 26% of skills carry at least one vulnerability. That's the problem SkillzBench is being built to solve. As a free, independent directory, it will verify every skill (commit-SHA pinning, security scans, license checks, quality judging) and benchmark it in a sandbox to publish the Lift — the measurable performance gain — before you ever deploy it. Readers will be able to browse, test, and equip agents with only the skills they choose via a free MCP connector. The waitlist is live now for teams that want early access.

Where does it all land? Sam Suthar of Middleware puts it plainly: predictive alerting, powered by AI, is set to become an industry standard, enhancing reliability and reducing unplanned downtime. With outages costing a median $2M per hour — and $1M with full-stack observability — the organizations that standardize early won't just comply faster. They'll fail less.