Golang The Series EP.163: Caching AI Responses - Reducing Costs and Boosting Speed for AI Apps with Redis Caching

Hello Gophers! After exploring how to call multiple AI models in parallel in EP.162, the next inevitable challenge when scaling AI applications for a large user base comes down to two things: Cost Optimization and Latency Reduction.

If you observe user behavior in production, a clear pattern emerges: people frequently ask the same popular questions. Examples include, "What are the company holidays this year?" or "How do I submit a travel reimbursement?" If your architecture continuously hits the LLM (Large Language Model) API to process identical questions, you are wasting API tokens and forcing users to wait several seconds for a response they could have received instantly.

The simplest and most effective solution to this problem is implementing the Cache-Aside Pattern using Redis. By placing Redis in front of your AI engine, you cut the token cost of repetitive questions to zero and return responses to your users in milliseconds.

System Architecture (Cache-Aside Pattern for AI)

The Cache-Aside (or Lazy Loading) strategy follows a clean, straightforward workflow as shown below:

Plaintext

  [User Question]
         │
  1. Generate Cache Key (SHA-256)
         │
  ┌──────┴────────────────────────┐
  ▼                               ▼
[Check in Redis]           [Check in Redis]
  (Cache HIT)                (Cache MISS)
      │                           │
Return instantly (ms)      2. Query the actual LLM (3s)
                                  │
                           3. Save to Redis + Set TTL
                                  │
                             Return to User

Installing the Redis Client

We will use the highly popular go-redis library (latest version) to manage the connection pool and handle data replication with Redis:

Bash

go get github.com/redis/go-redis/v9

Data Structure and Key Generation

To keep our codebase clean, let's separate the Redis client initialization and the function that hashes long questions into short, optimized Redis keys.

package main

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"fmt"
	"log"
	"time"

	"github.com/redis/go-redis/v9"
)

var (
	rdb *redis.Client
	ctx = context.Background()
)

// generateCacheKey hashes long user prompts into fixed-size keys to optimize Redis memory usage.
func generateCacheKey(question string) string {
	hash := sha256.Sum256([]byte(question))
	return fmt.Sprintf("ai:cache:%s", hex.EncodeToString(hash[:]))
}

Core Cache-Aside Logic

Next, we implement an LLM mock function (using time.Sleep to simulate network latency) and the core function handling the Cache Hit and Cache Miss flow.

// askLLMSimulator mocks an LLM API call with natural latency and token consumption costs.
func askLLMSimulator(question string) string {
	fmt.Println("🤖 [LLM] Cache Miss! Querying the actual model (takes 3 seconds)...")
	time.Sleep(3 * time.Second) 
	return fmt.Sprintf("Answer for '%s': According to company policy...", question)
}

// getAIResponseWithCache handles the lookup and population logic for the cache.
func getAIResponseWithCache(question string) (string, string) {
	cacheKey := generateCacheKey(question)

	// 1. Inspect Redis first (Cache Hit Check)
	val, err := rdb.Get(ctx, cacheKey).Result()
	if err == nil {
		return val, "HIT" // Found in cache, return immediately
	}

	// Log unexpected connection errors (ignoring key-not-found errors)
	if !errors.Is(err, redis.Nil) {
		log.Printf("⚠️ Redis connection error: %v", err)
	}

	// 2. Cache Miss -> Proceed to query the actual LLM
	answer := askLLMSimulator(question)

	// 3. Cache the response in Redis with a 1-hour Time-To-Live (TTL)
	err = rdb.Set(ctx, cacheKey, answer, 1*time.Hour).Err()
	if err != nil {
		log.Printf("❌ Failed to save response to cache: %v", err)
	}

	return answer, "MISS"
}

Execution Control: main.go

Let's tie everything together in the main function to test querying the exact same question twice and observe the performance execution gap.

func main() {
	// Establish connection to local Redis server
	rdb = redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})
	defer rdb.Close()

	question := "How many vacation days do new hires get?"

	fmt.Println("⚡ Running AI Response Caching Test...")
	fmt.Println("==================================================")

	// --- Turn 1: Initial state (Should result in a Cache Miss) ---
	start1 := time.Now()
	ans1, status1 := getAIResponseWithCache(question)
	fmt.Printf("[%s] Response: %s\n⏱️ Execution Time: %v\n", status1, ans1, time.Since(start1))
	fmt.Println("--------------------------------------------------")

	// --- Turn 2: Exact same question (Should result in a Cache Hit) ---
	start2 := time.Now()
	ans2, status2 := getAIResponseWithCache(question)
	fmt.Printf("[%s] Response: %s\n⚡ Execution Time: %v\n", status2, ans2, time.Since(start2))
	fmt.Println("==================================================")
}

Taking It Further with Semantic Caching

The code provided above demonstrates Exact Match Caching, which evaluates incoming prompts character by character. If a user alters a word, modifies spacing, or makes a minor typo, the system treats it as a Cache Miss even if the core meaning is identical, such as:

"How many vacation days do I get?"
"Can you tell me the number of annual leave days for new staff?"

To solve this limitation, modern production-grade AI backends typically transition to Semantic Caching using Redis Vector Similarity Search (VSS). In this setup, we convert the incoming question into a Vector Embedding first. We then execute a similarity query inside Redis to check for close matches (e.g., calculating a Cosine Similarity score of >= 0.95). If an existing question with a matching semantic threshold is found, we serve its cached response instantly without ever touching the LLM.

FAQ: Frequently Asked Questions

Why hash user prompts with SHA-256 before saving them to Redis?

User prompts submitted to an LLM can sometimes span entire pages. Using exceptionally long strings as direct Redis keys consumes an immense amount of memory and slows down indexing lookups. Hashing ensures every key remains a consistent, predictable, and memory-efficient size.

What is the recommended Time-To-Live (TTL) for caching AI responses?

This depends on your underlying data. Static internal data like company policies can be safely cached for 1 day to 1 week. For highly dynamic content, shorter windows like 1 to 2 hours are safer, or you can implement manual cache invalidation hooks triggered by source data changes.

Is there a risk that Semantic Caching serves an unrelated answer?

Yes. If your similarity threshold is configured too low, the system might misinterpret distinct questions as having identical intent. Implementing semantic caching requires tuning your threshold carefully (typically between 0.95 and 0.98) and testing it thoroughly against real user query logs before rollout.

Conclusion & Next Episode (EP.164)

Adopting the Cache-Aside Pattern with Redis is a staple architecture standard that yields massive savings on API token bills while optimizing frontend responsiveness for recurring workloads.

In the next episode (EP.164): While caching shields your backend from repetitive queries, what happens when malicious actors or rogue bots flood your system with an immense volume of unique questions? A cache won't stop them, and your corporate API quota could vanish within minutes. Next time, we'll build a secure perimeter by implementing Rate Limiting AI Requests with the Token Bucket Algorithm in Go. Stay tuned, Gophers!

Follow Superdev Academy on all platforms:

🔵 Facebook: Superdev Academy Thailand
🎬 YouTube: Superdev Academy Channel
📸 Instagram: @superdevacademy
🎬 TikTok: @superdevacademy
🌐 Website: superdevacademy.com