Golang The Series EP.157: Building a Document Ingestion Pipeline for PDF/Word to Power Your RAG System

Welcome to EP.157! In our previous episode, we explored the power of Semantic Search and how it allows us to find information based on meaning. However, in the real world, an organization's knowledge base rarely exists as clean, short snippets of text. Instead, it is usually buried inside complex documents like PDFs or Microsoft Word (.docx) files that span dozens of pages.

Therefore, our mission as Go backend developers today is to build a Document Ingestion Pipeline. This automated data pipeline will seamlessly handle the entire workflow: "Extract Raw Text ➡️ Chunk Text ➡️ Generate Vectors ➡️ Upsert to Qdrant Database."

Fire up your favorite code editor, and let's dive in!

Ingestion Pipeline Architecture

To ensure our system is flexible enough to support various file types, we will design a Modular Architecture. This separates the responsibilities into 4 main components, making it highly maintainable and scalable:

Plaintext

[PDF / Word File] ──> 1. Text Extractor (Extracts raw text content)
                               │
                               ▼
                      2. Chunking Engine (Splits text with a defined overlap)
                               │
                               ▼
                      3. Embedding Client (Sends API requests to generate []float32)
                               │
                               ▼
                      4. Vector DB Client (Stores Vector embeddings and Metadata in Qdrant)

Choosing Go Libraries for PDF and Word Processing

• For PDF Files: We recommend github.com/dslipak/pdf. It is lightweight, fast, and handles complex scripts (including Thai language tone marks and vowels) remarkably well compared to other alternatives. (Note: Extraction accuracy can still depend heavily on the font and how the PDF was originally exported).

• For Word Files (.docx): We recommend github.com/nguyenthenguyen/docx. This library parses the XML structure of Word files directly and rapidly without requiring any external software or Microsoft Office dependencies.

Implementation: Building the Pipeline in Go

Here is a complete example of a Go pipeline that extracts text from a PDF file and processes it for a RAG (Retrieval-Augmented Generation) system:

Go

package main

import (
	"bytes"
	"context"
	"fmt"
	"log"
	"os"

	"github.com/dslipak/pdf"
	"github.com/google/uuid"
	"github.com/qdrant/go-client/qdrant"
	"github.com/sashabaranov/go-openai"
)

func main() {
	ctx := context.Background()
	pdfPath := "company_policy.pdf"

	// 1. Text Extraction: Extract text from PDF
	fmt.Println("⏳ Extracting text from PDF file...")
	text, err := extractTextFromPDF(pdfPath)
	if err != nil {
		log.Fatalf("Failed to extract text: %v", err)
	}

	// 2. Chunking: Split text (using the chunking function from EP.155)
	// Chunk size: 500 characters, Overlap: 100 characters
	chunks := ChunkText(text, 500, 100)
	fmt.Printf("🧩 Successfully split text into %d chunks\n", len(chunks))

	// Initialize API Clients (Using Environment Variables for security)
	apiKey := os.Getenv("OPENAI_API_KEY")
	if apiKey == "" {
		log.Fatal("Please set your OPENAI_API_KEY environment variable")
	}
	openaiClient := openai.NewClient(apiKey)

	// Connect to Qdrant Vector Database
	qdrantClient, err := qdrant.NewClient(&qdrant.Config{
		Host: "localhost",
		Port: 6334, // Port 6334 is used for gRPC communication with the Go Client
	})
	if err != nil {
		log.Fatalf("Qdrant connection failed: %v", err)
	}
	defer qdrantClient.Close()

	// 3 & 4. Embedding Generation & Upserting to Qdrant
	fmt.Println("🚀 Generating embeddings and saving to Qdrant...")
	for i, chunk := range chunks {
		// Generate Vector embedding for the current chunk
		embReq := openai.EmbeddingRequest{
			Input: []string{chunk},
			Model: openai.SmallEmbedding3Small, // Standard 1,536-dimension model
		}
		embResp, err := openaiClient.CreateEmbeddings(ctx, embReq)
		if err != nil {
			log.Printf("Error embedding chunk %d: %v", i, err)
			continue
		}
		vector := embResp.Data[0].Embedding

		// Prepare Qdrant Point and attach Metadata Payload
		pointID := uuid.New().String()
		payload := map[string]interface{}{
			"content":   chunk,
			"source":    pdfPath,
			"chunk_idx": i,
		}

		// Upsert data into the Collection
		_, err = qdrantClient.Upsert(ctx, &qdrant.UpsertPoints{
			CollectionName: "ai_knowledge_base",
			Points: []*qdrant.PointStruct{
				{
					Id:      qdrant.NewIDUUID(pointID),
					Vectors: qdrant.NewVectorsDense(vector),
					Payload: qdrant.NewValueMap(payload),
				},
			},
		})
		if err != nil {
			log.Printf("Error upserting chunk %d to Qdrant: %v", i, err)
		}
	}

	fmt.Println("✅ Pipeline process completed! Your AI knowledge base is ready.")
}

// Helper function to extract raw text from a PDF file
func extractTextFromPDF(path string) (string, error) {
	r, err := pdf.Open(path)
	if err != nil {
		return "", err
	}
	
	var buf bytes.Buffer
	b, err := r.GetPlainText()
	if err != nil {
		return "", err
	}
	
	_, err = buf.ReadFrom(b)
	if err != nil {
		return "", err
	}
	
	return buf.String(), nil
}

// ChunkText (from EP.155) - Splits text based on runes to fully support multi-byte characters
func ChunkText(text string, chunkSize int, overlap int) []string {
	runes := []rune(text)
	var chunks []string
	
	if len(runes) == 0 {
		return chunks
	}
	
	for i := 0; i < len(runes); {
		end := i + chunkSize
		if end > len(runes) {
			end = len(runes)
		}
		
		chunks = append(chunks, string(runes[i:end]))
		if end == len(runes) {
			break
		}
		
		i += (chunkSize - overlap)
	}
	return chunks
}

Scaling Up with Go's Concurrency

When your application grows and users start uploading hundreds or thousands of files simultaneously (such as massive product manuals or financial statements), a sequential for loop will rapidly become your bottleneck. Your system would waste time waiting for OpenAI and Qdrant network responses one by one.

This is where Go truly shines! We can optimize the code above using a Worker Pool Pattern powered by Goroutines and Channels. By running text extraction, embedding generation, and vector database upserts concurrently, you can slash your data pipeline processing time from minutes down to a few seconds.

🎯 Daily Mission

Try creating a sample PDF containing some text (like a short company policy or a standard operating procedure). Drop it into your project directory and run the code provided above.

Bonus Homework: Refactor the loop using Go's concurrency patterns (e.g., using a sync.WaitGroup) to generate embeddings and upsert chunks concurrently. Once executed, don't forget to visit your Qdrant Web UI dashboard at http://localhost:6333/dashboard to verify that your points and payloads have successfully landed!

FAQ

What happens if my PDF contains complex tables or images? Can standard text extractors read them?

Standard text extractors only look at raw character strings. If they encounter a complex table layout, the text output might become misaligned or interleaved incorrectly. Furthermore, images will be completely ignored. If your production environment requires high-accuracy parsing of complex tables and images, consider migrating to a Document Intelligence API (such as Azure Document Intelligence or Google Document AI) or utilizing Vision LLMs to read the documents.

Why do we need to store metadata like source and chunk_idx inside Qdrant's payload?

This is crucial for Citations. When the RAG system retrieves information to formulate an answer, you typically want to show users exactly where that information came from. Storing the source filename and the chunk_idx allows your frontend application to effortlessly point back to the source document, helping users verify facts and boosting trust in your AI system.

Summary & Next Up: EP.158

Outstanding job! You've successfully built an automated pipeline to ingest external documents and structure them into an AI's "long-term memory."

In the next episode, we will complete the entire RAG loop by looking into Context Injection. How do we take the relevant knowledge chunks retrieved from our Vector DB and strategically feed them into an LLM's prompt to force accurate, hallucination-free answers?

Stay tuned, and happy coding, Gophers!

Follow Superdev Academy on all platforms:

• 🔵 Facebook: Superdev Academy Thailand

• 🎬 YouTube: Superdev Academy Channel

• 📸 Instagram: @superdevacademy

• 🎬 TikTok: @superdevacademy

• 🌐 Website: superdevacademy.com