[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"academy-blogs-en-1-1-all-golang-the-series-ep165-load-balancing-ai-servers-all--*":3,"academy-blog-translations-uhi7wotg3d6qsjq":87},{"data":4,"page":72,"perPage":72,"totalItems":72,"totalPages":72},[5],{"alt":6,"collectionId":7,"collectionName":8,"content":9,"cover_image":10,"cover_image_path":11,"created":12,"created_by":13,"expand":14,"id":80,"keywords":81,"locale":52,"published_at":82,"scheduled_at":68,"school_blog":76,"short_description":83,"status":74,"title":84,"updated":85,"updated_by":86,"slug":77,"views":79},"Building an AI Load Balancer in Go with Reverse Proxy and Background Health Check","sclblg987654321","school_blog_translations","\u003Cp>Welcome to EP.165! In our previous episode, we implemented Rate Limiting to prevent spam and control API request traffic. However, as your enterprise AI system gains popularity and legitimate user requests flood in, a single instance can no longer handle the heavy workload.\u003C\u002Fp>\u003Cp>Especially if your organization runs its own \u003Cem>Local LLM Infrastructure\u003C\u002Fem> (such as hosting models with Ollama, vLLM, or TGI across multiple GPU nodes), each request involving long prompts or document ingestion consumes massive amounts of GPU and VRAM resources. Routing all requests to a single machine will quickly lead to soaring latency, freezing, or system crashes.\u003C\u002Fp>\u003Cp>The solution is \u003Cstrong>Load Balancing\u003C\u002Fstrong>—using Go to build a proxy intermediary that distributes requests to AI server workers efficiently and reliably!\u003C\u002Fp>\u003Ch2>Load Balancing Algorithms for AI\u003C\u002Fh2>\u003Cp>When distributing workloads across AI inference nodes, there are three popular strategies:\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cp>\u003Cstrong>Round Robin:\u003C\u002Fstrong> Distributes requests sequentially (1, 2, 3, then loops back to 1). Ideal for nodes with identical hardware specs and similar task sizes.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>Least Connections:\u003C\u002Fstrong> Directs new requests to the node with the fewest active processing connections. This is best suited for AI workloads since prompt token processing times vary.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>Weighted Round Robin:\u003C\u002Fstrong> Assigns higher request proportions to nodes with superior GPU specifications (e.g., higher weights for more powerful hardware).\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>Architecture of an AI Load Balancer\u003C\u002Fh2>\u003Cp>We will build a lightweight Reverse Proxy in Go that performs periodic \u003Cem>Health Checks\u003C\u002Fem> on each AI node, routing traffic using a \u003Cem>Round Robin with Health Check\u003C\u002Fem> strategy to ensure requests never hit dead or down nodes.\u003C\u002Fp>\u003Cp>Plaintext\u003C\u002Fp>\u003Cpre>\u003Ccode>[Client Requests]\n                                      │\n                                      ▼\n                        ┌──────────────────────────┐\n                        │    Go Load Balancer      │\n                        │  (Round Robin + Proxy)   │\n                        └─────────────┬────────────┘\n                                      │\n           ┌──────────────────────────┼──────────────────────────┐\n           │ (Active)                 │ (Active)                 │ (Down ❌)\n           ▼                          ▼                          ▼\n   ┌──────────────┐           ┌──────────────┐           ┌──────────────┐\n   │  AI Node 1   │           │  AI Node 2   │           │  AI Node 3   │\n   │ (GPU Node A) │           │ (GPU Node B) │           │ (GPU Node C) │\n   └──────────────┘           └──────────────┘           └──────────────┘\n\u003C\u002Fcode>\u003C\u002Fpre>\u003Ch2>Component & Proxy Logic\u003C\u002Fh2>\u003Cp>To maintain Clean Architecture principles, we separate our node data structures and reverse proxy logic from the core execution flow.\u003C\u002Fp>\u003Ch3>Part 1: Node Structures and Health Checker\u003C\u002Fh3>\u003Cp>Go\u003C\u002Fp>\u003Cpre>\u003Ccode>package main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"net\u002Fhttp\"\n\t\"net\u002Fhttp\u002Fhttputil\"\n\t\"net\u002Furl\"\n\t\"sync\"\n\t\"sync\u002Fatomic\"\n\t\"time\"\n)\n\n\u002F\u002F AINode stores details and status for each AI server\ntype AINode struct {\n\tURL          *url.URL\n\tAlive        bool\n\tReverseProxy *httputil.ReverseProxy\n\tmu           sync.RWMutex\n}\n\n\u002F\u002F SetAlive updates the availability status of the node\nfunc (node *AINode) SetAlive(alive bool) {\n\tnode.mu.Lock()\n\tnode.Alive = alive\n\tnode.mu.Unlock()\n}\n\n\u002F\u002F IsAlive checks if the node is operational\nfunc (node *AINode) IsAlive() bool {\n\tnode.mu.RLock()\n\tdefer node.mu.RUnlock()\n\treturn node.Alive\n}\n\n\u002F\u002F AILoadBalancer manages a group of AI nodes\ntype AILoadBalancer struct {\n\tnodes   []*AINode\n\tcurrent uint64\n}\n\n\u002F\u002F GetNextNode selects the next available node using Round Robin\nfunc (lb *AILoadBalancer) GetNextNode() *AINode {\n\tnodeCount := len(lb.nodes)\n\tif nodeCount == 0 {\n\t\treturn nil\n\t}\n\n\tnext := atomic.AddUint64(&lb.current, 1)\n\t\n\tfor i := 0; i < nodeCount; i++ {\n\t\tidx := int((next + uint64(i)) % uint64(nodeCount))\n\t\tif lb.nodes[idx].IsAlive() {\n\t\t\treturn lb.nodes[idx]\n\t\t}\n\t}\n\treturn nil\n}\n\n\u002F\u002F HealthCheck pings the \u002Fhealth endpoint of each node at set intervals\nfunc (lb *AILoadBalancer) HealthCheck() {\n\tclient := http.Client{\n\t\tTimeout: 2 * time.Second,\n\t}\n\n\tfor _, node := range lb.nodes {\n\t\tgo func(n *AINode) {\n\t\t\tresp, err := client.Get(n.URL.String() + \"\u002Fhealth\")\n\t\t\tif err != nil || resp.StatusCode != http.StatusOK {\n\t\t\t\tif n.IsAlive() {\n\t\t\t\t\tlog.Printf(\"⚠️ [Health Check] AI Node %s is unresponsive -> Marked as DOWN\", n.URL.String())\n\t\t\t\t\tn.SetAlive(false)\n\t\t\t\t}\n\t\t\t\treturn\n\t\t\t}\n\t\t\t_ = resp.Body.Close()\n\n\t\t\tif !n.IsAlive() {\n\t\t\t\tlog.Printf(\"✅ [Health Check] AI Node %s is back online -> Marked as UP\", n.URL.String())\n\t\t\t\tn.SetAlive(true)\n\t\t\t}\n\t\t}(node)\n\t}\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\u003Ch3>Part 2: Core Server Logic and Proxy Routing: main.go\u003C\u002Fh3>\u003Cp>Go\u003C\u002Fp>\u003Cpre>\u003Ccode>func main() {\n\trawURLs := []string{\n\t\t\"http:\u002F\u002Flocalhost:11434\", \u002F\u002F AI Node 1 (Ollama Instance A)\n\t\t\"http:\u002F\u002Flocalhost:11435\", \u002F\u002F AI Node 2 (Ollama Instance B)\n\t\t\"http:\u002F\u002Flocalhost:11436\", \u002F\u002F AI Node 3 (Ollama Instance C)\n\t}\n\n\tvar nodes []*AINode\n\tfor _, rawURL := range rawURLs {\n\t\ttargetURL, err := url.Parse(rawURL)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"Invalid URL: %v\", err)\n\t\t}\n\n\t\tproxy := httputil.NewSingleHostReverseProxy(targetURL)\n\t\tnodes = append(nodes, &AINode{\n\t\t\tURL:          targetURL,\n\t\t\tAlive:        true,\n\t\t\tReverseProxy: proxy,\n\t\t})\n\t}\n\n\tlb := &AILoadBalancer{nodes: nodes}\n\n\t\u002F\u002F Background Health Checker running every 10 seconds\n\tgo func() {\n\t\tticker := time.NewTicker(10 * time.Second)\n\t\tfor range ticker.C {\n\t\t\tlb.HealthCheck()\n\t\t}\n\t}()\n\n\t\u002F\u002F HTTP Handler acting as a Reverse Proxy\n\thttp.HandleFunc(\"\u002Fapi\u002Fai\u002Fgenerate\", func(w http.ResponseWriter, r *http.Request) {\n\t\ttargetNode := lb.GetNextNode()\n\t\tif targetNode == nil {\n\t\t\thttp.Error(w, \"❌ No available AI servers at the moment\", http.StatusServiceUnavailable)\n\t\t\treturn\n\t\t}\n\n\t\tfmt.Printf(\"🔀 [Load Balancer] Proxying request to -> %s\\n\", targetNode.URL.String())\n\t\ttargetNode.ReverseProxy.ServeHTTP(w, r)\n\t})\n\n\tlog.Println(\"🚀 AI Load Balancer running on port :8080...\")\n\tlog.Fatal(http.ListenAndServe(\":8080\", nil))\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\u003Ch2>Important Considerations for AI Load Balancing\u003C\u002Fh2>\u003Col>\u003Cli>\u003Cp>\u003Cstrong>Server-Sent Events (SSE) \u002F Streaming Responses:\u003C\u002Fstrong> Modern AI models often stream tokens back incrementally via SSE. Go's reverse proxy flush intervals must be carefully handled so chunks stream to the client in real-time without buffering in memory.\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>Stateless Node Design:\u003C\u002Fstrong> Design AI workers to be as stateless as possible. For chat history or conversation memory, offload context storage to a centralized database (such as Redis or PostgreSQL) so subsequent requests from the same user can hit any AI node without losing context.\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Fol>\u003Ch2>🎯 Daily Mission\u003C\u002Fh2>\u003Cp>Try running the code above and test real-world production scenarios:\u003C\u002Fp>\u003Cp>\u003Cstrong>Challenge:\u003C\u002Fstrong> If your system includes one ultra-fast GPU node (e.g., NVIDIA H100) and two standard GPU nodes (e.g., RTX 4090), traditional Round Robin may overload the slower nodes while the powerful node sits idle.\u003C\u002Fp>\u003Cp>As a systems engineer, how would you modify the \u003Ccode>GetNextNode()\u003C\u002Fcode> function and \u003Ccode>AINode\u003C\u002Fcode> fields to implement \u003Cstrong>Weighted Round Robin\u003C\u002Fstrong> based on hardware capabilities? Give it a try!\u003C\u002Fp>\u003Ch2>❓ Frequently Asked Questions (FAQ)\u003C\u002Fh2>\u003Ch3>Why build a load balancer in Go instead of using Nginx or HAProxy?\u003C\u002Fh3>\u003Cp>For standard AI infrastructure, Nginx and HAProxy are exceptional reverse proxies and load balancers. However, writing a load balancer in Go lets you tailor custom logic specifically for AI workloads—such as handling LLM token streaming, implementing specialized health checks, or integrating authentication and rate limiting directly without complex modules.\u003C\u002Fp>\u003Ch3>If an AI node hangs rather than completely crashing, will health checks catch it?\u003C\u002Fh3>\u003Cp>In our basic example, we use a short timeout (2 seconds) to ping the \u003Ccode>\u002Fhealth\u003C\u002Fcode> endpoint. If a node responds too slowly due to processing a long queue, it is temporarily marked as down. In production environments, you may also want to monitor GPU VRAM utilization or active connection counts.\u003C\u002Fp>\u003Ch3>What are the disadvantages of Round Robin load balancing for AI?\u003C\u002Fh3>\u003Cp>The main drawback is \u003Cstrong>processing time inequality\u003C\u002Fstrong>. Since prompts vary in length and token generation time, long requests piling up on Node A while Node B sits idle can cause resource imbalances. This is why Least Connections or Weighted Round Robin strategies are recommended for advanced setups.\u003C\u002Fp>\u003Ch3>Does maintaining chat history affect the load balancer?\u003C\u002Fh3>\u003Cp>Yes. If AI nodes store conversation memory internally, a follow-up request routed to a different node by the load balancer will lose context. The solution is to design stateless AI workers and persist chat history in a centralized cache like Redis.\u003C\u002Fp>\u003Cdiv data-type=\"horizontalRule\">\u003Chr>\u003C\u002Fdiv>\u003Ch2>Summary\u003C\u002Fh2>\u003Cp>In this article, we explored how to resolve bottlenecks when enterprise AI systems face high concurrent traffic by building a Go-based load balancer with a reverse proxy. We also implemented a background health check system to monitor node availability in real-time, preventing requests from hitting dead servers and improving overall enterprise AI infrastructure stability.\u003C\u002Fp>\u003Cp>\u003Cem>In the next episode (EP.166):\u003C\u002Fem> Even with a great load balancer, if your primary AI API or backend destination completely goes down, slow timeouts can trigger \u003Cstrong>Cascading Failures\u003C\u002Fstrong> across your entire backend architecture. Next time, we will look at installing automated circuit breakers with \u003Cem>\"Circuit Breaker Patterns: Handling Failures When AI APIs Collapse\"\u003C\u002Fem>. See you there, Gophers!\u003C\u002Fp>\u003Cp>\u003Cstrong>Follow Superdev Academy on all platforms:\u003C\u002Fstrong>\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cp>\u003Cstrong>🔵 Facebook: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.facebook.com\u002Fsuperdev.academy.th\">\u003Cstrong>Superdev Academy Thailand\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🎬 YouTube: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.youtube.com\u002F@SuperdevAcademy\">\u003Cstrong>Superdev Academy Channel\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>📸 Instagram: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.instagram.com\u002Fsuperdevacademy\u002F\">\u003Cstrong>@superdevacademy\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🎬 TikTok: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.tiktok.com\u002F@superdevacademy?lang=th-TH\">\u003Cstrong>@superdevacademy\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🌐 Website: \u003C\u002Fstrong>\u003Ca rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fsuperdevacademy.com\">\u003Cstrong>superdevacademy.com\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Ful>\u003Cp>\u003C\u002Fp>","503phgms7w8j_sbtzh8gp13.png","https:\u002F\u002Ftwsme-r2.tumwebsme.com\u002Fsclblg987654321\u002Fii2obwvs49bbfkx\u002F503phgms7w8j_sbtzh8gp13.png","2026-07-29 04:37:34.765Z","76qprkevbgfdps8",{"keywords":15,"locale":46,"school_blog":56},[16,22,26,30,34,38,42],{"collectionId":17,"collectionName":18,"created":19,"created_by":13,"id":20,"name":21,"updated":19,"updated_by":13},"sclkey987654321","school_keywords","2026-07-29 04:34:06.787Z","p6osctqon25pao8","Golang Load Balancer",{"collectionId":17,"collectionName":18,"created":23,"created_by":13,"id":24,"name":25,"updated":23,"updated_by":13},"2026-07-29 04:34:09.278Z","z7eed2k1l9rdllx","AI Load Balancing",{"collectionId":17,"collectionName":18,"created":27,"created_by":13,"id":28,"name":29,"updated":27,"updated_by":13},"2026-07-29 04:34:11.928Z","t8naihqgaknjfz4","AI Infrastructure",{"collectionId":17,"collectionName":18,"created":31,"created_by":13,"id":32,"name":33,"updated":31,"updated_by":13},"2026-07-29 04:34:14.332Z","ip3nk1hcdyn1mua","Reverse Proxy Go",{"collectionId":17,"collectionName":18,"created":35,"created_by":13,"id":36,"name":37,"updated":35,"updated_by":13},"2026-07-29 04:34:17.842Z","n8qbf7t2vmpf6pu","Ollama Scaling",{"collectionId":17,"collectionName":18,"created":39,"created_by":13,"id":40,"name":41,"updated":39,"updated_by":13},"2026-07-29 04:34:20.763Z","wp2x2p9g14g8igb","Health Check Go",{"collectionId":17,"collectionName":18,"created":43,"created_by":13,"id":44,"name":45,"updated":43,"updated_by":13},"2026-07-29 04:34:26.738Z","psau1viorbkrbtp","Local LLM Infrastructure",{"code":47,"collectionId":48,"collectionName":49,"created":50,"flag":51,"id":52,"is_default":53,"label":54,"updated":55},"en","pbc_1989393366","locales","2026-01-22 11:00:02.726Z","twemoji:flag-united-states","qv9c1llfov2d88z",false,"English","2026-04-10 15:42:46.825Z",{"category":57,"collectionId":58,"collectionName":59,"created":60,"expand":61,"id":76,"slug":77,"updated":78,"views":79},"wqxt7ag2gn7xcmk","pbc_2105096300","school_blogs","2026-07-28 14:20:27.450Z",{"category":62},{"blogIds":63,"collectionId":64,"collectionName":65,"created":66,"created_by":13,"id":57,"image":67,"image_alt":68,"image_path":69,"label":70,"name":71,"priority":72,"publish_at":73,"scheduled_at":68,"status":74,"updated":75,"updated_by":13},[],"sclcatblg987654321","school_category_blogs","2026-03-04 08:33:53.210Z","59ty92ns80w_15oc1implw.png","","https:\u002F\u002Ftwsme-r2.tumwebsme.com\u002Fsclcatblg987654321\u002Fwqxt7ag2gn7xcmk\u002F59ty92ns80w_15oc1implw.png",{"en":71,"th":71},"Golang The Series",1,"2026-03-16 04:39:38.440Z","published","2026-06-07 06:45:03.856Z","uhi7wotg3d6qsjq","golang-the-series-ep165-load-balancing-ai-servers","2026-08-03 17:03:09.401Z",122,"ii2obwvs49bbfkx",[20,24,28,32,36,40,44],"2026-08-03 05:00:54.824Z","Learn how to scale AI infrastructure for high traffic by building a custom Reverse Proxy Load Balancer in Go, implementing the Round Robin algorithm and Background Health Check to ensure enterprise-grade stability.","Golang The Series EP.165: Load Balancing AI Servers - Scaling AI Workloads for High Concurrency","2026-08-03 05:00:54.825Z","423vhnv3ckczcyn",{"th":77,"en":77}]