View : 126

21/07/2026 01:53am

Multi-LLM System Diagram using Goroutines and Channels in Go

Golang The Series EP.162: Goroutines for Multi-LLM - Querying Multiple AIs in Parallel for Result Comparison

#Goroutines

#Multi-LLM

#Go

#Golang

#AI Comparison

#Parallel Processing

Hello Gophers! Welcome back to EP.162. Lately, anyone building backends for AI systems probably faces a similar issue: we don't want to tie our system to just one LLM provider. In some use cases, we want to "ask all of them at once"—whether it's OpenAI, Gemini, Claude, or even a local model like Ollama—to use the answers for a fact-checking voting system or to let users compare them side-by-side.

The problem is, if we write code to call each API one by one (Sequential), users will be waiting forever before everyone finishes (assume 3 seconds per provider, 3 providers would take almost 10 seconds). Today, we're going to use Go's specialty, Goroutines and Channels, to perform Concurrent Querying. We'll fire off the question to all providers simultaneously. Whoever is the slowest, that becomes our maximum wait time!

Multi-LLM Architecture via Concurrent Querying

The idea is to spawn Goroutines as independent Workers to call each AI's API. We'll use sync.WaitGroup to orchestrate and wait for all of them to finish. Then, we use Go Channels to collect the results centrally, preventing any Data Race issues.

Plaintext

┌──> [Goroutine 1] ──> Call OpenAI API (1.5s) ──┐
                  │                                             │
[User Question]   ├──> [Goroutine 2] ──> Call Gemini API (2.5s) ──┼──> [Go Channel] ──> Aggregated Result (max wait 3.5s)
                  │                                             │
                  └──> [Goroutine 3] ──> Call Ollama Local (3.5s) ┘

Data Structure and Worker (Types & Function)

To keep our code clean, we separate the Data Structure for the results and create a specific Worker function for querying each provider.

Go

package main

import (
	"context"
	"fmt"
	"strings"
	"sync"
	"time"
)

// LLMResponse is a struct to store results and stats from each AI provider
type LLMResponse struct {
	Provider string
	Answer   string
	Duration time.Duration
	Error    error
}

This function acts as a Worker that calls the API. (In this example, we'll simulate the latency using time.After).

Go

// callLLM acts as a Worker to independently call each Provider's API
func callLLM(ctx context.Context, provider string, delay time.Duration, question string, ch chan<- LLMResponse, wg *sync.WaitGroup) {
	defer wg.Done()
	startTime := time.Now()

	// Simulate Network Latency
	select {
	case <-time.After(delay):
		// Simulate the model's response
		answer := fmt.Sprintf("[%s]: After analyzing the question '%s', I found that...", provider, question)
		
		ch <- LLMResponse{
			Provider: provider,
			Answer:   answer,
			Duration: time.Since(startTime),
			Error:    nil,
		}
	case <-ctx.Done():
		// Case where Context times out or gets canceled
		ch <- LLMResponse{
			Provider: provider,
			Error:    ctx.Err(),
		}
	}
}

Main Orchestrator

In main.go, we run 3 Workers in parallel and set an overall Timeout of 4 seconds to prevent the system from hanging endlessly.

Go

func main() {
	question := "What should be the focus of SEO trends this year?"
	
	// Set an overall timeout of 4 seconds
	ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
	defer cancel()

	var wg sync.WaitGroup
	// Create a Buffered Channel with a size equal to the number of providers
	responseChan := make(chan LLMResponse, 3)

	// 1. Distribute parallel AI queries
	wg.Add(1)
	go callLLM(ctx, "OpenAI (GPT-4o)", 1500*time.Millisecond, question, responseChan, &wg)

	wg.Add(1)
	go callLLM(ctx, "Google Gemini", 2500*time.Millisecond, question, responseChan, &wg)

	wg.Add(1)
	go callLLM(ctx, "Ollama (Llama 3)", 3500*time.Millisecond, question, responseChan, &wg)

	// 2. Run a Background Worker to wait and close the Channel once everyone is done
	go func() {
		wg.Wait()
		close(responseChan)
	}()

	fmt.Printf("🚀 Sending the question: \"%s\" to all AI providers simultaneously...\n\n", question)

	// 3. Loop to fetch incoming results from the Channel
	for resp := range responseChan {
		if resp.Error != nil {
			fmt.Printf("❌ [%s] Failed or Timed out: %v\n", resp.Provider, resp.Error)
			fmt.Println(strings.Repeat("-", 50))
			continue
		}
		fmt.Printf("🤖 %s (Took: %v)\n", resp.Provider, resp.Duration)
		fmt.Printf("📝 Answer: %s\n", resp.Answer)
		fmt.Println(strings.Repeat("-", 50))
	}
}

Bonus Technique: First Past the Post

For some projects where we don't need to compare results but just want "maximum speed" (whoever answers first wins), we can easily adapt this code. Just change the loop condition: once we successfully pull the first data from responseChan, call cancel() immediately. This signal will tell other Goroutines waiting for APIs to abort their requests, saving API tokens and significantly reducing server load.

🎯 Daily Mission

Try running this code and push the delay of some providers up to 5000*time.Millisecond. You'll notice that those specific providers get cut off with a Timeout Error instead.

Challenge: If the requirement changes to "Must wait for all 3 answers, then send all of them to a Judge LLM to summarize the best one," how would you design a Slice to receive values from the Channel and pass them on? Try picturing the architecture in your head!

FAQ: Frequently Asked Questions

Why do we need to create a Buffered Channel make(chan LLMResponse, 3) with a specific size?

To prevent Goroutine Leaks. Suppose a Worker finishes and wants to send data into the Channel. If we use an Unbuffered Channel (without a specified size), the Worker gets Blocked and waits until the main code comes to receive the value. Adding a buffer equal to the number of Workers allows them to drop the result into the box and close their job immediately.

Is sync.WaitGroup necessary since we're already looping through the Channel?

Yes, in this case. Because we set up close(responseChan). If we don't use a WaitGroup to wait before closing the Channel, a Worker might not be finished yet but will try to send data into an already-closed Channel, which causes the program to Panic and crash instantly.

If we have 100 AI providers to call, would firing 100 Goroutines at once consume too many machine resources?

Goroutines are extremely lightweight. Running 100 of them is not a problem for Go. The real bottleneck to watch out for is the Network or API Rate Limits. If you scale to that level, it's recommended to use the Worker Pool Pattern or implement a Semaphore to control the number of concurrent requests.

Conclusion & Next Episode (EP.163)

Goroutines and Channels are the true heroes that make Go handle I/O-Bound tasks—like concurrently calling multiple AI APIs—efficiently and safely.

In the next episode (EP.163): Even though this parallel querying is faster, it still incurs redundant API costs for the "same old questions" users frequently ask. Next time, we'll reduce costs by implementing Caching AI Responses with Redis, allowing our system to answer popular questions in milliseconds without bothering the AI. Stay tuned, Gophers!

Follow Superdev Academy on all platforms: