Golang The Series EP.159: Hybrid Search - Combining SQL and Vector Search in Go

Hello Gophers! Welcome to EP.159.

Following our previous episode on Context Injection, where we successfully built a RAG system capable of smoothly retrieving answers from documents. However, when deploying this to a real production environment, I believe many of you might encounter the same "pain point."

The issue is that most enterprise data doesn't just consist of raw, unstructured document files that we convert into vectors. There's also a massive amount of structured data resting in traditional Relational Databases like PostgreSQL or MySQL—whether it's employee departments, access permission levels, system update dates, or document statuses (Active/Draft).

Imagine a requirement from the business side: "We want employees to be able to search manuals using AI, but restrict it so that it only retrieves the latest version of documents, and the user asking must have the correct permissions (Role = Admin)."

If we rely solely on a Vector DB without considering other conditions, there's a high chance that confidential cross-department data could leak. This is exactly where Hybrid Search, or the technique of blending the SQL world with Vector Search, comes into play in our Go applications!

Hybrid Search Design Concepts in Go Projects

When we need to filter data across these two architectures, we generally have two main approaches:

1. Two-Stage Query (The Classic Approach: SQL First, then Vector)

   This approach involves using the Go Backend to query the SQL Database first to retrieve a list of Document IDs that meet the conditions (e.g., correct permissions, active status). Then, we send this list of IDs as a filter to the Vector DB for the actual search. This method clearly separates responsibilities, but if the ID list is enormous, it can introduce significant latency.

2. Integrated Filtering or Metadata Pre-filtering (Recommended Approach: One-Shot)

   This method shifts the necessary conditional data (Metadata) to be saved into the Payload section of the Vector Database (like Qdrant) right from the data extraction phase. From there, we use Qdrant's Payload Indexing feature to perform a vector search alongside the conditions in a single query.

   And the Qdrant Go SDK provides highly efficient functions for this, drastically reducing network latency!

Go Example for Hybrid Search (Vector Search + Metadata Filter)

Instead of writing all our code inside main(), let's divide the structure into two main parts to keep the code clean and easy to extend:

1. Text Conversion (Embedding)

2. Data Search and Filtering (Hybrid Search)

1. Function to Convert Queries to Vectors (Embedding)

We will extract the OpenAI API interaction into a separate function that takes text input and returns vector coordinates ([]float32).

Go

// generateEmbedding takes a query and converts it into vector coordinates via OpenAI
func generateEmbedding(ctx context.Context, client *openai.Client, text string) ([]float32, error) {
	embReq := openai.EmbeddingRequest{
		Input: []string{text},
		Model: openai.SmallEmbedding3Small, // Using standard 1,536 dimensions
	}
	resp, err := client.CreateEmbeddings(ctx, embReq)
	if err != nil {
		return nil, fmt.Errorf("embedding failed: %w", err)
	}
	return resp.Data[0].Embedding, nil
}

2. Function to Execute Hybrid Search Query on Qdrant

This function takes the vector coordinates from step 1 along with the "conditions" (Metadata) we want to filter by, such as department and document status, to construct the search query.

Go

// searchDocuments performs a semantic search with Metadata filtering (Department, Status)
func searchDocuments(ctx context.Context, client *qdrant.Client, queryVector []float32, dept, status string) ([]*qdrant.ScoredPoint, error) {
	searchLimit := uint64(3)
	
	// Create filter specifications by matching keywords in the Payload
	searchFilters := &qdrant.Filter{
		Must: []*qdrant.Condition{
			qdrant.NewMatchKeyword("department", dept),
			qdrant.NewMatchKeyword("status", status),
		},
	}

	// Execute the query for both semantic coordinates and Metadata filtering in one go
	return client.Query(ctx, &qdrant.QueryPoints{
		CollectionName: "ai_knowledge_base",
		Query:          qdrant.NewQuery(queryVector...),
		Filter:         searchFilters,
		Limit:          &searchLimit,
	})
}

3. Assembling in the main() Function

By separating the functions, our main() code becomes as readable as a book. It clearly simulates the workflow.

Go

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"github.com/qdrant/go-client/qdrant"
	"github.com/sashabaranov/go-openai"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	// --- [Setup Clients] ---
	apiKey := os.Getenv("OPENAI_API_KEY")
	if apiKey == "" {
		log.Fatal("Please set your OPENAI_API_KEY environment variable")
	}
	openaiClient := openai.NewClient(apiKey)

	qdrantClient, err := qdrant.NewClient(&qdrant.Config{
		Host: "localhost",
		Port: 6334,
	})
	if err != nil {
		log.Fatalf("Qdrant connection failed: %v", err)
	}
	defer qdrantClient.Close()

	// --- [Define Query & Filters] ---
	userQuery := "What are the medical reimbursement regulations?"
	
	// Simulated values retrieved from the core system's SQL Database
	targetDepartment := "Human Resources"
	documentStatus := "active"

	// --- [Step 1: Embedding] ---
	queryVector, err := generateEmbedding(ctx, openaiClient, userQuery)
	if err != nil {
		log.Fatalf("Error generating embedding: %v", err)
	}

	// --- [Step 2: Hybrid Search] ---
	searchResp, err := searchDocuments(ctx, qdrantClient, queryVector, targetDepartment, documentStatus)
	if err != nil {
		log.Fatalf("Hybrid Search query failed: %v", err)
	}

	// --- [Step 3: Display Results] ---
	fmt.Printf("🔍 Hybrid Search Results for: '%s'\n", userQuery)
	fmt.Printf("📋 [System Conditions] Department: %s | Status: %s\n\n", targetDepartment, documentStatus)
	
	for i, point := range searchResp {
		payloadMap := point.Payload
		
		// Prevent Panics by checking keys before retrieving values
		var content, dept, status string
		if val, ok := payloadMap["content"]; ok { content = val.GetStringValue() }
		if val, ok := payloadMap["department"]; ok { dept = val.GetStringValue() }
		if val, ok := payloadMap["status"]; ok { status = val.GetStringValue() }
		
		fmt.Printf("[%d] Score: %.4f | Department: %s | Status: %s\n", i+1, point.Score, dept, status)
		fmt.Printf("   Reference Content: %s\n\n", content)
	}
}

Why is Go Perfect for Hybrid Search Architecture?

Having the backend talk to two databases (both SQL and Vector) inevitably introduces latency concerns if the workflow isn't managed well. Fortunately, we're writing in Go! Go has great features that make handling this a breeze:

• Goroutines for Concurrent Querying: If your requirements force you into the Two-Stage Query approach (fetching data from separate silos), you can simply run them in parallel using Goroutines and synchronize them with sync.WaitGroup. Once you have results from both sides, perform data intersection in Go. This significantly cuts down processing time compared to waiting for sequential queries.

• Type-Safety with Go Structs: When we receive payload data back from the Vector DB (often as JSON or Maps) alongside SQL records, we can unmarshal them into strictly typed Go Structs right in memory. This allows for confident data transformation, completely eliminating the minor runtime errors typically found in dynamically typed languages.

🎯 Challenge (Daily Mission)

Try applying the filter-attached query structure from this article to your own project. The challenge is: "Retrieve only documents that are no older than a specified duration" (using Range Filters).

Food for thought: Try modifying the searchFilters variable to support numeric or Unix Timestamp comparisons (e.g., finding documents where the created_at field is greater than 30 days ago). Hint: Check out the Qdrant Go Client documentation to see how qdrant.NewRange(...) handles numeric parameter passing, run the query, and observe the results in your terminal!

💡 FAQ (Frequently Asked Questions)

Is Qdrant strictly required for Hybrid Search?

Not necessarily. Other leading Vector Databases like Milvus, Pinecone, Weaviate, or even pgvector in PostgreSQL also support Metadata Filtering. However, in this series, we chose Qdrant because of its easy-to-use, high-performance Go SDK and excellent support for Payload Indexing.

If Metadata (like access permissions) changes very frequently, which approach is better?

If access permissions change on a per-second basis (Highly Dynamic), using the Two-Stage Query approach (checking SQL for permissions first, then passing IDs to Vector) might be better for Data Consistency. But if changes aren't that frequent, the Metadata Pre-filtering approach discussed here offers much better speed performance (you just need to build a mechanism to keep the Qdrant Payload synced with SQL).

Does Payload Indexing consume more system resources?

Creating indexes will consume slightly more RAM and Storage. But the tradeoff is a massive leap in query speed. For production environments, sacrificing a little storage for lower latency is a highly worthwhile investment.

Summary

In this article, we've explored how to integrate Relational Databases (SQL) and Vector Databases via Hybrid Search. This is crucial for building enterprise AI search systems that are secure, permission-aware, and retrieve only ready-to-use data using the Metadata Pre-filtering technique efficiently completed in a single query via the Go SDK.

In the next episode (EP.160): Congratulations! The theoretical foundation and advanced components for our AI Integration development are now 100% ready. Next time, we'll conclude the season with a grand finale: "Workshop 2: Internal Tool Document Q&A System." We will combine the Gin Web Framework, PDF Extraction, the RAG Pipeline, and Hybrid Search to build a real enterprise application. Don't miss it, Gophers!

Follow Superdev Academy on all platforms:

• 🔵 Facebook: Superdev Academy Thailand

• 🎬 YouTube: Superdev Academy Channel

• 📸 Instagram: @superdevacademy

• 🎬 TikTok: @superdevacademy

• 🌐 Website: superdevacademy.com