Building Production RAG Pipelines with OpenAI and Vector Databases

Retrieval-Augmented Generation (RAG) has gone from a research paper to a production staple in under two years. I've shipped several RAG systems — from a customer support bot handling 50k queries/month to an internal knowledge base for a SaaS product — and the gap between a prototype that impresses in a demo and one that survives real traffic is enormous.

This is the guide I wish existed when I started. No fluff, no toy examples — just the architecture decisions, gotchas, and code patterns that matter in production.

"RAG done right is invisible to the end user. They just get accurate answers. RAG done wrong hallucinates confidently about things that don't exist in your data."

What RAG Actually Is (And What It Isn't)

The core idea: instead of fine-tuning an LLM on your data (expensive, slow, brittle), you retrieve relevant chunks of your data at query time and inject them into the prompt as context. The LLM then answers using both its parametric knowledge and the retrieved context.

The naive version looks like this:

1. User asks a question
2. You embed the question into a vector
3. You search a vector database for the most similar chunks
4. You stuff those chunks into a system prompt
5. GPT-4 returns an answer

This works fine for demos. Production systems need to handle: query routing, chunk quality, context window limits, latency budgets, cost control, and graceful degradation when the retrieval fails.

The Architecture I Use in Production

Here's the high-level flow I've settled on after several iterations:

// Production RAG Flow
User Query
   ↓
Query Preprocessor  // clean, expand, classify
   ↓
Retrieval Layer      // hybrid search (vector + BM25)
   ↓
Reranker             // Cohere rerank or cross-encoder
   ↓
Context Builder     // chunk merging, dedup, trim to token limit
   ↓
LLM Completion      // GPT-4o with structured prompt
   ↓
Response Validator  // groundedness check, citation injection
   ↓
Final Answer

Each layer is independent and testable. This matters because when something breaks in prod, you need to isolate which component failed without re-running the entire chain.

Ingestion Pipeline: Getting Data In Right

Most RAG failures I've seen trace back to poor ingestion. The retrieval is only as good as what you stored. Here's the ingestion pipeline I use:

Chunking Strategy

Fixed-size chunking (chunk_size=512, overlap=50) is the lazy default and it's usually wrong. The chunk boundary will land mid-sentence, splitting context that belongs together. Instead, I use semantic chunking — split on paragraph boundaries, then merge small paragraphs until you approach your target token count.

// utils/chunker.js
const { encode } = require('gpt-tokenizer');

function semanticChunk(text, maxTokens = 400, overlapTokens = 60) {
  const paragraphs = text.split(/\n{2,}/).filter(p => p.trim());
  const chunks = [];
  let current = [];
  let currentTokens = 0;

  for (const para of paragraphs) {
    const tokens = encode(para).length;
    if (currentTokens + tokens > maxTokens && current.length > 0) {
      chunks.push(current.join('\n\n'));
      // carry forward last paragraph for overlap
      current = [current[current.length - 1]];
      currentTokens = encode(current[0]).length;
    }
    current.push(para);
    currentTokens += tokens;
  }
  if (current.length) chunks.push(current.join('\n\n'));
  return chunks;
}

module.exports = { semanticChunk };

Metadata Is Non-Negotiable

Every chunk you store needs metadata: source URL, document title, section heading, creation date, author. Without this you can't cite sources, you can't filter by recency, and you can't debug why a bad chunk got retrieved.

// ingestion/embed.js
async function embedAndStore(chunks, docMeta) {
  const embeddings = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: chunks.map(c => c.text),
  });

  const vectors = chunks.map((chunk, i) => ({
    id: `${docMeta.docId}-chunk-${i}`,
    values: embeddings.data[i].embedding,
    metadata: {
      text: chunk.text,
      source: docMeta.url,
      title: docMeta.title,
      section: chunk.heading || '',
      createdAt: docMeta.createdAt,
      chunkIndex: i,
    },
  }));

  await pinecone.index('knowledge-base').upsert(vectors);
}

Retrieval: Hybrid Search Over Pure Vector

Pure vector search misses exact keyword matches. If a user asks "what is the cancellation policy?" and your docs say "cancellation policy" verbatim, a keyword search will find it reliably. A vector search might surface semantically related content that doesn't directly answer the question.

I use hybrid search in every production RAG system: combine vector similarity with BM25 (or full-text search), then merge the results using Reciprocal Rank Fusion (RRF).

// retrieval/hybrid.js
async function hybridSearch(query, topK = 10) {
  const [vectorResults, ftsResults] = await Promise.all([
    vectorSearch(query, topK),
    fullTextSearch(query, topK),
  ]);

  return reciprocalRankFusion([vectorResults, ftsResults], topK);
}

function reciprocalRankFusion(resultSets, k = 60) {
  const scores = new Map();
  resultSets.forEach(results => {
    results.forEach((doc, rank) => {
      const prev = scores.get(doc.id) || 0;
      scores.set(doc.id, prev + 1 / (k + rank + 1));
    });
  });
  return [...scores.entries()]
    .sort(([,a],[,b]) => b - a)
    .map(([id]) => getDocById(id));
}

Choosing a Vector Database

Here's a pragmatic breakdown of the options I've used:

Database	Best for	Cons
Pinecone	Managed, fast POC → prod path	Cost scales fast; vendor lock-in
pgvector	Existing Postgres infra	Slower at scale (>1M vectors)
Qdrant	Self-hosted, high performance	More ops overhead
Weaviate	Hybrid search built-in	Complex config
Chroma	Local dev and prototyping	Not production-ready

My default: start with pgvector if you're already on Postgres. It's free, you own your data, and it handles millions of vectors fine with HNSW indexing. Switch to Pinecone or Qdrant when you hit real performance ceilings.

Prompt Engineering for RAG

The prompt template is where most teams leave performance on the table. Here's the structure I use:

// prompts/rag-system.js
const buildSystemPrompt = (context) => `
You are a helpful assistant for [Company Name].
Answer questions using ONLY the information in the provided context.
If the context does not contain the answer, say "I don't have that information."
Do NOT make up information. Do NOT reference knowledge outside the context.

CONTEXT:
---
${context}`;

const buildUserPrompt = (query, history) => {
  const historyStr = history
    .map(m => `${m.role.toUpperCase()}: ${m.content}`)
    .join('\n');
  return historyStr ? `${historyStr}\nUSER: ${query}` : query;
};

Managing Latency in Production

A full RAG chain (embed query → vector search → rerank → LLM) can take 3–5 seconds. That's often too slow for interactive use. Here are the optimizations I apply:

• Cache embeddings: The same query gets asked repeatedly. Cache the embedding vector for 24–48h in Redis keyed by query hash.
• Cache retrieval results: For popular queries, skip retrieval entirely and serve cached chunks.
• Stream the LLM response: Use OpenAI's streaming API so the first token appears in ~500ms even if full completion takes 4s.
• Skip reranking for short queries: Single-word or very short queries don't benefit from reranking — skip it and save 200ms.
• Parallelize where possible: Run full-text search and vector search in parallel, not sequentially.

Evaluating RAG Quality

You can't improve what you can't measure. The three metrics that matter most:

1. Context Relevance — Are the retrieved chunks actually relevant to the query? Measure by asking GPT-4 to rate each chunk 1–5 and tracking the average.
2. Groundedness — Is the LLM answer actually supported by the retrieved context? Use an LLM-as-judge approach to check for hallucinations.
3. Answer Correctness — Does the answer match the expected answer? Requires a labeled test set.

I use RAGAS for automated evaluation in CI. Even running it on 50 test queries per deploy gives you enough signal to catch regressions.

The Mistakes I Made So You Don't Have To

• Chunking too small. 128-token chunks lose all context. Minimum 300, usually 400–600 tokens with proper overlap.
• Not storing the raw text in metadata. Once you index millions of vectors, you need the text to reconstruct context without going back to the original source.
• Skipping the reranker. Top-5 vector results are often not the best 5 for the query. A reranker consistently improves accuracy by 15–30%.
• Trusting the LLM to say "I don't know." It won't, unless you explicitly instruct it to. Add groundedness checks as a post-processing step.
• Fetching too many chunks. Stuffing 20 chunks into the context window dilutes relevance. 5–8 high-quality chunks consistently beats 15+ mixed-quality ones.

Wrapping Up

Building a production RAG system is really about building five smaller systems that compose well: an ingestion pipeline, a retrieval layer, a context builder, an LLM integration, and an evaluation framework. None of them are individually hard. The challenge is integrating them reliably and making them observable enough that you can debug failures in production.

If you're starting out, get ingestion right first. Everything downstream is limited by the quality of what you indexed. Then add hybrid search. Then add a reranker. Ship incrementally and measure at each step.

If you're building a RAG system for your product and want a second opinion on the architecture — or if you want someone to build it for you — drop me a message.