Golang The Series EP.167: Monitoring AI Performance & Latency with Prometheus

Welcome to EP.167! It's been 166 episodes. Our Go-based AI Backend is fully equipped with Multi-LLM support, Redis Caching, Rate Limiting, Load Balancing, and even Circuit Breakers—ready to scale at an enterprise level.

But in the production world, there's a classic engineering adage: "If you can't measure it, you can't improve it."

I still remember deploying an AI system to production for the first time. During peak traffic, the system inexplicably slowed down. It took hours to hunt down which AI Provider was bottlenecking our app.

AI systems present latency challenges vastly different from standard CRUD APIs. A typical API might respond in 50ms, whereas an LLM can take anywhere from 500ms to 10 seconds, depending on prompt size, context length, and output tokens. Without proper Observability & Monitoring, we have no way of knowing:

• Which LLM provider is degrading during peak loads?

• Is the latency distribution (Percentiles: p50, p90, p99) meeting our SLA requirements?

• How often are cache hits/misses or circuit breaker trips occurring?

Today, we’re going to attach a "heart rate monitor" to our Go application using Prometheus Metrics!

Understanding Prometheus Metric Types

Prometheus offers several metric types tailored for different use cases. For tracking AI performance, we'll focus on three main ones:

• Counter: A cumulative metric that only goes up (unless the service restarts). Perfect for tracking the total number of requests (ai_requests_total) or total token usage.

• Gauge: A metric that can arbitrarily go up and down. Ideal for measuring the number of concurrent requests currently being processed (ai_concurrent_requests).

• Histogram: Samples observations and counts them in configurable "buckets" to calculate statistical distributions. This is the absolute best choice for measuring Latency or Processing Duration (ai_request_duration_seconds).

Installing Dependencies

We'll be using the official Prometheus Go client:

Bash

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

Structuring and Embedding Prometheus Metrics in Go

We will create custom metrics to clock the response times of various AI providers (e.g., OpenAI, Anthropic, Ollama Local) and categorize them by response status (Success / Error).

Part 1: Declaring Prometheus Variables and Mocking the AI Call

Go

package main

import (
	"fmt"
	"log"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	// 1. Counter: Total requests categorized by provider and status
	aiRequestsTotal = promauto.NewCounterVec(
		prometheus.CounterOpts{
			Name: "ai_requests_total",
			Help: "Total number of AI API requests",
		},
		[]string{"provider", "status"},
	)

	// 2. Histogram: Processing time (Latency) categorized by provider
	aiRequestDuration = promauto.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "ai_request_duration_seconds",
			Help:    "AI API processing duration in seconds",
			// Define statistical buckets ranging from 0.1s to 10s
			Buckets: []float64{0.1, 0.5, 1.0, 2.0, 5.0, 10.0},
		},
		[]string{"provider"},
	)

	// 3. Gauge: Number of concurrent requests currently running
	aiActiveRequests = promauto.NewGaugeVec(
		prometheus.GaugeOpts{
			Name: "ai_active_requests",
			Help: "Number of concurrent AI requests currently processing",
		},
		[]string{"provider"},
	)
)

// simulateAICall mocks an AI provider request and measures processing time
func simulateAICall(provider string) (string, error) {
	// Increment active requests in Gauge
	aiActiveRequests.WithLabelValues(provider).Inc()
	defer aiActiveRequests.WithLabelValues(provider).Dec()

	// Start timer
	startTime := time.Now()

	// Simulate AI processing time (0.2s to 3.0s)
	processingTime := time.Duration(200+rand.Intn(2800)) * time.Millisecond
	time.Sleep(processingTime)

	duration := time.Since(startTime).Seconds()

	// Record Latency in Histogram Bucket
	aiRequestDuration.WithLabelValues(provider).Observe(duration)

	// Simulate a 10% chance of error
	if rand.Float32() < 0.1 {
		aiRequestsTotal.WithLabelValues(provider, "error").Inc()
		return "", fmt.Errorf("AI Provider %s Timeout / Service Unavailable", provider)
	}

	aiRequestsTotal.WithLabelValues(provider, "success").Inc()
	return fmt.Sprintf("Response from %s (took %.2fs)", provider, duration), nil
}

Part 2: Setting up HTTP Endpoints and Prometheus Scraping: main.go

Go

func main() {
	// 1. Endpoint for Prometheus Server to scrape metrics
	http.Handle("/metrics", promhttp.Handler())

	// 2. Mock API Endpoint for AI processing
	http.HandleFunc("/api/v1/generate", func(w http.ResponseWriter, r *http.Request) {
		provider := r.URL.Query().Get("provider")
		if provider == "" {
			provider = "openai" // Default Provider
		}

		resp, err := simulateAICall(provider)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		_, _ = w.Write([]byte(resp))
	})

	log.Println("📊 Prometheus Metrics Endpoint running at http://localhost:2112/metrics")
	log.Println("🚀 AI Service API ready to receive requests at http://localhost:2112/api/v1/generate")
	log.Fatal(http.ListenAndServe(":2112", nil))
}

Building a Grafana Dashboard with PromQL

Once Prometheus starts scraping metrics from our /metrics endpoint into its time-series database, we can use PromQL (Prometheus Query Language) in Grafana to build sleek, insightful dashboards:

1. Calculating the 95th Percentile Latency (How fast 95% of users get a response)

histogram_quantile(0.95, sum(rate(ai_request_duration_seconds_bucket[5m])) by (le, provider))

2. Calculating the Success Rate (%)

(sum(rate(ai_requests_total{status="success"}[5m])) / sum(rate(ai_requests_total[5m]))) * 100

🎯 Daily Mission

Try running the code above and use cURL to send multiple requests, swapping between providers:

Bash

curl "http://localhost:2112/api/v1/generate?provider=openai"
curl "http://localhost:2112/api/v1/generate?provider=gemini"

Then, open your browser and navigate to http://localhost:2112/metrics.

Food for Thought: Try searching for the ai_request_duration_seconds_bucket text on the /metrics page. Notice how the counts in each bucket correlate with the actual processing time? Furthermore, if you wanted to track "Token Usage" in the future to calculate daily costs, which metric type would you choose: Counter, Gauge, or Histogram? Try designing the structure yourself!

🙋‍♂️ FAQ

Why not just use standard logging to track execution time? Why go through the hassle of using Prometheus?

Using standard logs (like log.Printf("Took %v", duration)) is easy when starting out. However, when your system handles thousands of requests per second, the log volume becomes massive. Computing real-time averages or P95 from plain text logs is incredibly resource-intensive. Prometheus is purposely built to store Time-Series Data, which is far more space-efficient and drastically faster when querying for dashboard visualizations.

How should I configure Histogram Buckets for typical AI workloads?

AI models inherently take much longer to respond than standard APIs. Setting bucket boundaries at the millisecond level (e.g., 0.01s, 0.05s) isn't very useful here. It's best practice to configure buckets that cover the realistic processing span of LLMs, such as [0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0] seconds. This ensures you can accommodate even extremely large prompts.

Will frequent scraping by Prometheus slow down my server?

Not at all! The /metrics endpoint is designed to be extremely lightweight and fast. Configuring Prometheus to scrape every 10 or 15 seconds is a standard best practice and will not negatively impact your server's core performance.

📝 Conclusion

In this article, we learned how to arm our Go backend with robust observability to monitor AI performance using Prometheus Metrics. We explored the three primary metric types (Counter, Gauge, Histogram), wrote clean Go code to accurately measure latency, and applied advanced PromQL queries for Grafana to calculate enterprise-grade statistics like P95 and Success Rates. Your system is now production-ready and fully equipped to confidently answer the business team when they ask about performance!

Coming up next (EP.168): We have an excellent performance monitoring system in place, but one of the biggest headaches for AI developers is that "AI does not always return errors as standard HTTP Status Codes." Sometimes the API returns a 200 OK, but the actual message inside reads "Sorry, I cannot fulfill this request," or it spits out a malformed JSON that our Go system fails to parse! Next time, we're diving deep into "Error Handling in AI — Managing and mitigating unexpected AI responses." Don't miss it, Gophers!

Follow Superdev Academy on all platforms:

• 🔵 Facebook: Superdev Academy Thailand

• 🎬 YouTube: Superdev Academy Channel

• 📸 Instagram: @superdevacademy

• 🎬 TikTok: @superdevacademy

• 🌐 Website: superdevacademy.com