[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"academy-blogs-en-1-1-all-golang-document-ingestion-pipeline-pdf-word-rag-all--*":3,"academy-blog-translations-ytqbhupna4e0zox":81},{"data":4,"page":67,"perPage":67,"totalItems":67,"totalPages":67},[5],{"alt":6,"collectionId":7,"collectionName":8,"content":9,"cover_image":10,"cover_image_path":11,"created":12,"created_by":13,"expand":14,"id":75,"keywords":76,"locale":47,"published_at":77,"scheduled_at":63,"school_blog":71,"short_description":78,"status":69,"title":79,"updated":80,"updated_by":13,"slug":72,"views":74},"Architecture diagram of a Document Ingestion Pipeline extracting text from PDF and Word files using Golang and upserting it into Qdrant Database for a RAG system.","sclblg987654321","school_blog_translations","\u003Cp>Welcome to EP.157! In our previous episode, we explored the power of Semantic Search and how it allows us to find information based on meaning. However, in the real world, an organization's knowledge base rarely exists as clean, short snippets of text. Instead, it is usually buried inside complex documents like PDFs or Microsoft Word (.docx) files that span dozens of pages.\u003C\u002Fp>\u003Cp>Therefore, our mission as Go backend developers today is to build a \u003Cstrong>Document Ingestion Pipeline\u003C\u002Fstrong>. This automated data pipeline will seamlessly handle the entire workflow: \u003Cstrong>\"Extract Raw Text ➡️ Chunk Text ➡️ Generate Vectors ➡️ Upsert to Qdrant Database.\"\u003C\u002Fstrong>\u003C\u002Fp>\u003Cp>Fire up your favorite code editor, and let's dive in!\u003C\u002Fp>\u003Ch2>Ingestion Pipeline Architecture\u003C\u002Fh2>\u003Cp>To ensure our system is flexible enough to support various file types, we will design a \u003Cstrong>Modular Architecture\u003C\u002Fstrong>. This separates the responsibilities into 4 main components, making it highly maintainable and scalable:\u003C\u002Fp>\u003Cp>Plaintext\u003C\u002Fp>\u003Cpre>\u003Ccode>[PDF \u002F Word File] ──> 1. Text Extractor (Extracts raw text content)\n                               │\n                               ▼\n                      2. Chunking Engine (Splits text with a defined overlap)\n                               │\n                               ▼\n                      3. Embedding Client (Sends API requests to generate []float32)\n                               │\n                               ▼\n                      4. Vector DB Client (Stores Vector embeddings and Metadata in Qdrant)\n\u003C\u002Fcode>\u003C\u002Fpre>\u003Ch2>Choosing Go Libraries for PDF and Word Processing\u003C\u002Fh2>\u003Cul>\u003Cli>\u003Cp>\u003Cstrong>For PDF Files:\u003C\u002Fstrong> We recommend \u003Ca rel=\"noopener noreferrer\" href=\"http:\u002F\u002Fgithub.com\u002Fdslipak\u002Fpdf\">github.com\u002Fdslipak\u002Fpdf\u003C\u002Fa>. It is lightweight, fast, and handles complex scripts (including Thai language tone marks and vowels) remarkably well compared to other alternatives. \u003Cem>(Note: Extraction accuracy can still depend heavily on the font and how the PDF was originally exported).\u003C\u002Fem>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>For Word Files (.docx):\u003C\u002Fstrong> We recommend \u003Ca rel=\"noopener noreferrer\" href=\"http:\u002F\u002Fgithub.com\u002Fnguyenthenguyen\u002Fdocx\">github.com\u002Fnguyenthenguyen\u002Fdocx\u003C\u002Fa>. This library parses the XML structure of Word files directly and rapidly without requiring any external software or Microsoft Office dependencies.\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>Implementation: Building the Pipeline in Go\u003C\u002Fh2>\u003Cp>Here is a complete example of a Go pipeline that extracts text from a PDF file and processes it for a RAG (Retrieval-Augmented Generation) system:\u003C\u002Fp>\u003Cp>Go\u003C\u002Fp>\u003Cpre>\u003Ccode>package main\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com\u002Fdslipak\u002Fpdf\"\n\t\"github.com\u002Fgoogle\u002Fuuid\"\n\t\"github.com\u002Fqdrant\u002Fgo-client\u002Fqdrant\"\n\t\"github.com\u002Fsashabaranov\u002Fgo-openai\"\n)\n\nfunc main() {\n\tctx := context.Background()\n\tpdfPath := \"company_policy.pdf\"\n\n\t\u002F\u002F 1. Text Extraction: Extract text from PDF\n\tfmt.Println(\"⏳ Extracting text from PDF file...\")\n\ttext, err := extractTextFromPDF(pdfPath)\n\tif err != nil {\n\t\tlog.Fatalf(\"Failed to extract text: %v\", err)\n\t}\n\n\t\u002F\u002F 2. Chunking: Split text (using the chunking function from EP.155)\n\t\u002F\u002F Chunk size: 500 characters, Overlap: 100 characters\n\tchunks := ChunkText(text, 500, 100)\n\tfmt.Printf(\"🧩 Successfully split text into %d chunks\\n\", len(chunks))\n\n\t\u002F\u002F Initialize API Clients (Using Environment Variables for security)\n\tapiKey := os.Getenv(\"OPENAI_API_KEY\")\n\tif apiKey == \"\" {\n\t\tlog.Fatal(\"Please set your OPENAI_API_KEY environment variable\")\n\t}\n\topenaiClient := openai.NewClient(apiKey)\n\n\t\u002F\u002F Connect to Qdrant Vector Database\n\tqdrantClient, err := qdrant.NewClient(&qdrant.Config{\n\t\tHost: \"localhost\",\n\t\tPort: 6334, \u002F\u002F Port 6334 is used for gRPC communication with the Go Client\n\t})\n\tif err != nil {\n\t\tlog.Fatalf(\"Qdrant connection failed: %v\", err)\n\t}\n\tdefer qdrantClient.Close()\n\n\t\u002F\u002F 3 & 4. Embedding Generation & Upserting to Qdrant\n\tfmt.Println(\"🚀 Generating embeddings and saving to Qdrant...\")\n\tfor i, chunk := range chunks {\n\t\t\u002F\u002F Generate Vector embedding for the current chunk\n\t\tembReq := openai.EmbeddingRequest{\n\t\t\tInput: []string{chunk},\n\t\t\tModel: openai.SmallEmbedding3Small, \u002F\u002F Standard 1,536-dimension model\n\t\t}\n\t\tembResp, err := openaiClient.CreateEmbeddings(ctx, embReq)\n\t\tif err != nil {\n\t\t\tlog.Printf(\"Error embedding chunk %d: %v\", i, err)\n\t\t\tcontinue\n\t\t}\n\t\tvector := embResp.Data[0].Embedding\n\n\t\t\u002F\u002F Prepare Qdrant Point and attach Metadata Payload\n\t\tpointID := uuid.New().String()\n\t\tpayload := map[string]interface{}{\n\t\t\t\"content\":   chunk,\n\t\t\t\"source\":    pdfPath,\n\t\t\t\"chunk_idx\": i,\n\t\t}\n\n\t\t\u002F\u002F Upsert data into the Collection\n\t\t_, err = qdrantClient.Upsert(ctx, &qdrant.UpsertPoints{\n\t\t\tCollectionName: \"ai_knowledge_base\",\n\t\t\tPoints: []*qdrant.PointStruct{\n\t\t\t\t{\n\t\t\t\t\tId:      qdrant.NewIDUUID(pointID),\n\t\t\t\t\tVectors: qdrant.NewVectorsDense(vector),\n\t\t\t\t\tPayload: qdrant.NewValueMap(payload),\n\t\t\t\t},\n\t\t\t},\n\t\t})\n\t\tif err != nil {\n\t\t\tlog.Printf(\"Error upserting chunk %d to Qdrant: %v\", i, err)\n\t\t}\n\t}\n\n\tfmt.Println(\"✅ Pipeline process completed! Your AI knowledge base is ready.\")\n}\n\n\u002F\u002F Helper function to extract raw text from a PDF file\nfunc extractTextFromPDF(path string) (string, error) {\n\tr, err := pdf.Open(path)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\t\n\tvar buf bytes.Buffer\n\tb, err := r.GetPlainText()\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\t\n\t_, err = buf.ReadFrom(b)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\t\n\treturn buf.String(), nil\n}\n\n\u002F\u002F ChunkText (from EP.155) - Splits text based on runes to fully support multi-byte characters\nfunc ChunkText(text string, chunkSize int, overlap int) []string {\n\trunes := []rune(text)\n\tvar chunks []string\n\t\n\tif len(runes) == 0 {\n\t\treturn chunks\n\t}\n\t\n\tfor i := 0; i < len(runes); {\n\t\tend := i + chunkSize\n\t\tif end > len(runes) {\n\t\t\tend = len(runes)\n\t\t}\n\t\t\n\t\tchunks = append(chunks, string(runes[i:end]))\n\t\tif end == len(runes) {\n\t\t\tbreak\n\t\t}\n\t\t\n\t\ti += (chunkSize - overlap)\n\t}\n\treturn chunks\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\u003Ch2>Scaling Up with Go's Concurrency\u003C\u002Fh2>\u003Cp>When your application grows and users start uploading hundreds or thousands of files simultaneously (such as massive product manuals or financial statements), a sequential \u003Ccode>for\u003C\u002Fcode> loop will rapidly become your bottleneck. Your system would waste time waiting for OpenAI and Qdrant network responses one by one.\u003C\u002Fp>\u003Cp>This is where Go truly shines! We can optimize the code above using a \u003Cstrong>Worker Pool Pattern\u003C\u002Fstrong> powered by \u003Cstrong>Goroutines\u003C\u002Fstrong> and \u003Cstrong>Channels\u003C\u002Fstrong>. By running text extraction, embedding generation, and vector database upserts concurrently, you can slash your data pipeline processing time from minutes down to a few seconds.\u003C\u002Fp>\u003Ch2>🎯 Daily Mission\u003C\u002Fh2>\u003Cp>Try creating a sample PDF containing some text (like a short company policy or a standard operating procedure). Drop it into your project directory and run the code provided above.\u003C\u002Fp>\u003Cp>\u003Cstrong>Bonus Homework:\u003C\u002Fstrong> Refactor the loop using Go's concurrency patterns (e.g., using a \u003Ccode>sync.WaitGroup\u003C\u002Fcode>) to generate embeddings and upsert chunks concurrently. Once executed, don't forget to visit your Qdrant Web UI dashboard at \u003Ca rel=\"noopener noreferrer\" href=\"http:\u002F\u002Flocalhost:6333\u002Fdashboard\">http:\u002F\u002Flocalhost:6333\u002Fdashboard\u003C\u002Fa> to verify that your points and payloads have successfully landed!\u003C\u002Fp>\u003Ch2>FAQ\u003C\u002Fh2>\u003Ch3>What happens if my PDF contains complex tables or images? Can standard text extractors read them?\u003C\u002Fh3>\u003Cp>Standard text extractors only look at raw character strings. If they encounter a complex table layout, the text output might become misaligned or interleaved incorrectly. Furthermore, images will be completely ignored. If your production environment requires high-accuracy parsing of complex tables and images, consider migrating to a \u003Cstrong>Document Intelligence API\u003C\u002Fstrong> (such as Azure Document Intelligence or Google Document AI) or utilizing Vision LLMs to read the documents.\u003C\u002Fp>\u003Ch3>Why do we need to store metadata like \u003Ccode>source\u003C\u002Fcode> and \u003Ccode>chunk_idx\u003C\u002Fcode> inside Qdrant's payload?\u003C\u002Fh3>\u003Cp>This is crucial for \u003Cstrong>Citations\u003C\u002Fstrong>. When the RAG system retrieves information to formulate an answer, you typically want to show users exactly where that information came from. Storing the \u003Ccode>source\u003C\u002Fcode> filename and the \u003Ccode>chunk_idx\u003C\u002Fcode> allows your frontend application to effortlessly point back to the source document, helping users verify facts and boosting trust in your AI system.\u003C\u002Fp>\u003Cdiv data-type=\"horizontalRule\">\u003Chr>\u003C\u002Fdiv>\u003Ch2>Summary & Next Up: EP.158\u003C\u002Fh2>\u003Cp>Outstanding job! You've successfully built an automated pipeline to ingest external documents and structure them into an AI's \"long-term memory.\"\u003C\u002Fp>\u003Cp>In the next episode, we will complete the entire RAG loop by looking into \u003Cstrong>Context Injection\u003C\u002Fstrong>. How do we take the relevant knowledge chunks retrieved from our Vector DB and strategically feed them into an LLM's prompt to force accurate, hallucination-free answers?\u003C\u002Fp>\u003Cp>Stay tuned, and happy coding, Gophers!\u003C\u002Fp>\u003Cp>\u003Cstrong>Follow Superdev Academy on all platforms:\u003C\u002Fstrong>\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cp>\u003Cstrong>🔵 Facebook: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.facebook.com\u002Fsuperdev.academy.th\">\u003Cstrong>Superdev Academy Thailand\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🎬 YouTube: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.youtube.com\u002F@SuperdevAcademy\">\u003Cstrong>Superdev Academy Channel\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>📸 Instagram: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.instagram.com\u002Fsuperdevacademy\u002F\">\u003Cstrong>@superdevacademy\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🎬 TikTok: \u003C\u002Fstrong>\u003Ca target=\"_blank\" rel=\"noopener\" class=\"ng-star-inserted\" href=\"https:\u002F\u002Fwww.tiktok.com\u002F@superdevacademy?lang=th-TH\">\u003Cstrong>@superdevacademy\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003Cli>\u003Cp>\u003Cstrong>🌐 Website: \u003C\u002Fstrong>\u003Ca rel=\"noopener noreferrer\" href=\"https:\u002F\u002Fsuperdevacademy.com\">\u003Cstrong>superdevacademy.com\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\u003C\u002Fli>\u003C\u002Ful>\u003Cp>\u003C\u002Fp>","343vl0ek29et_182s55y6u1.png","https:\u002F\u002Ftwsme-r2.tumwebsme.com\u002Fsclblg987654321\u002Fmwh024gqyuxo74a\u002F343vl0ek29et_182s55y6u1.png","2026-07-06 03:23:38.346Z","76qprkevbgfdps8",{"keywords":15,"locale":41,"school_blog":51},[16,23,28,32,36],{"collectionId":17,"collectionName":18,"created":19,"created_by":13,"id":20,"name":21,"updated":22,"updated_by":13},"sclkey987654321","school_keywords","2026-03-04 08:20:14.253Z","ah6lvy4x8qe08l5","Golang","2026-06-07 06:45:08.193Z",{"collectionId":17,"collectionName":18,"created":24,"created_by":13,"id":25,"name":26,"updated":27,"updated_by":13},"2026-03-04 08:20:11.547Z","ey3puyme01a9bsw","Go","2026-06-07 06:45:07.798Z",{"collectionId":17,"collectionName":18,"created":29,"created_by":13,"id":30,"name":31,"updated":29,"updated_by":13},"2026-07-05 13:45:03.865Z","4kfwr0tzqe7y1ux","Document Ingestion Pipeline",{"collectionId":17,"collectionName":18,"created":33,"created_by":13,"id":34,"name":35,"updated":33,"updated_by":13},"2026-06-11 16:14:22.575Z","gluay8aj98wheus","RAG",{"collectionId":17,"collectionName":18,"created":37,"created_by":13,"id":38,"name":39,"updated":40,"updated_by":13},"2026-03-04 08:44:53.062Z","puutdnxuitnxxgq","Backend","2026-06-07 06:46:40.599Z",{"code":42,"collectionId":43,"collectionName":44,"created":45,"flag":46,"id":47,"is_default":48,"label":49,"updated":50},"en","pbc_1989393366","locales","2026-01-22 11:00:02.726Z","twemoji:flag-united-states","qv9c1llfov2d88z",false,"English","2026-04-10 15:42:46.825Z",{"category":52,"collectionId":53,"collectionName":54,"created":55,"expand":56,"id":71,"slug":72,"updated":73,"views":74},"wqxt7ag2gn7xcmk","pbc_2105096300","school_blogs","2026-07-05 13:45:30.326Z",{"category":57},{"blogIds":58,"collectionId":59,"collectionName":60,"created":61,"created_by":13,"id":52,"image":62,"image_alt":63,"image_path":64,"label":65,"name":66,"priority":67,"publish_at":68,"scheduled_at":63,"status":69,"updated":70,"updated_by":13},[],"sclcatblg987654321","school_category_blogs","2026-03-04 08:33:53.210Z","59ty92ns80w_15oc1implw.png","","https:\u002F\u002Ftwsme-r2.tumwebsme.com\u002Fsclcatblg987654321\u002Fwqxt7ag2gn7xcmk\u002F59ty92ns80w_15oc1implw.png",{"en":66,"th":66},"Golang The Series",1,"2026-03-16 04:39:38.440Z","published","2026-06-07 06:45:03.856Z","ytqbhupna4e0zox","golang-document-ingestion-pipeline-pdf-word-rag","2026-07-06 06:41:05.333Z",124,"mwh024gqyuxo74a",[20,25,30,34,38],"2026-07-06 04:39:34.968Z","Learn how to build a Document Ingestion Pipeline with Go (Golang) to read, extract, and convert PDF\u002FWord documents into vector embeddings for Qdrant, setting a solid foundation for your RAG system.","Golang The Series EP.157: Building a Document Ingestion Pipeline for PDF\u002FWord to Power Your RAG System","2026-07-06 04:39:34.969Z",{"th":72,"en":72}]