Where Does an LLM Get Its Data? RAG Data Sources in Python

The previous article, What Is RAG? The Simplest Explanation of How It Works, established the mental model: the LLM writes, RAG retrieves useful knowledge, the application owns current state, and tools perform actions. This article takes the next step: where does the data actually come from, and what does retrieval look like in Python?

The important surprise is that an “LLM data source” is usually nothing exotic. It can be a text file, a folder of Markdown documents, a SQL database, an API response, a product catalog, a support system, or a vector index derived from those sources. The AI does not magically know these systems. Your application has to load, query, search, or retrieve the relevant data and place the result into the model’s context.

Data source = where information lives. Retrieval = how the application finds useful information. Context = the selected information given to the model. LLM = the component that interprets that context and generates an answer.— The four-part model used throughout this article

Question

How does an LLM use external data such as files, databases or APIs, and how can a small Python program implement the essential RAG steps without hiding them behind a framework?

What This Really Means

When developers say that an LLM is “connected to company data,” several different operations may be hidden behind that sentence. One application may execute SQL. Another may call an API. Another may run full-text search. Another may calculate embedding similarity over document chunks. All of them can provide external information to an LLM, but they are not the same retrieval method and they should not be treated as interchangeable.

This distinction matters because the best retrieval method depends on the shape of the question. “What is our refund policy?” is a document-retrieval problem. “What is order 4711’s current status?” is usually a structured database lookup. “Which paragraph discusses account recovery?” can be keyword or semantic search. RAG is most useful when the system must discover relevant knowledge before generation.

Simplest Example

Start with three strings in ordinary Python. There is no vector database, no framework, and no LLM yet. We only want to make the retrieval step visible.

documents = [
    "The AKM uses 7.62 mm ammunition.",
    "A Med Kit restores health.",
    "A 4x scope can be attached to several compatible weapons."
]

question = "Which ammunition does the AKM use?"

for document in documents:
    if "AKM" in document:
        print(document)

The program prints the first sentence because it contains the term we searched for. This is primitive retrieval, but the architecture is already visible: question → search → relevant text. RAG adds one more major step: pass the retrieved text to a language model together with the question.

A slightly more general version ranks documents by overlapping query terms:

import re

documents = [
    {"id": "weapon-akm", "text": "The AKM uses 7.62 mm ammunition."},
    {"id": "healing-medkit", "text": "A Med Kit restores health."},
    {"id": "scope-4x", "text": "A 4x scope can be attached to several compatible weapons."},
]

def words(text):
    return set(re.findall(r"[a-zA-Z0-9.]+", text.lower()))

def retrieve(question, documents, top_k=2):
    query_terms = words(question)
    ranked = []

    for document in documents:
        score = len(query_terms & words(document["text"]))
        if score > 0:
            ranked.append((score, document))

    ranked.sort(key=lambda item: item[0], reverse=True)
    return [document for _, document in ranked[:top_k]]

question = "Which ammunition does the AKM use?"
hits = retrieve(question, documents)

for hit in hits:
    print(hit["id"], "->", hit["text"])

This is not a production search engine. It ignores morphology, synonyms, spelling variants, document length and many ranking signals. Its value is educational: RAG does not begin with a vector database. It begins with retrieval.

Where the Example Stops Working

Exact or lexical matching becomes weak when the question and the source use different words. A document may say “vehicle maintenance,” while the user asks “how do I repair my car?” A lexical retriever can miss the relationship even though a human sees it immediately. Semantic retrieval addresses this by representing text as vectors and comparing meaning rather than only exact tokens.

Long files create another problem. Searching an entire 80-page manual as one unit is too coarse, but splitting every sentence can destroy useful context. Real RAG systems therefore need decisions about parsing, chunking, metadata, ranking, freshness, permissions and provenance.

The example also says nothing about structured live facts. If the user asks for the current status of order 4711 and the application already has a database key, semantic search is usually the wrong first tool. A deterministic database query is better.

Direct Answer

An LLM data source is any external system from which an application can obtain information for the model: files, databases, APIs, search indexes, vector stores or live application state. RAG is the pattern of retrieving relevant knowledge from such sources before generation.

In Python, the essential pipeline can be very small: load data → create retrievable units → find relevant evidence → assemble context → call the LLM. The retrieval method should match the source and the question. Use SQL for exact structured facts, full-text search for lexical matching, embeddings for semantic similarity, and hybrid retrieval when several signals are valuable.

Why This Is So

A language model does not automatically receive the contents of your filesystem, PostgreSQL database, CRM, private API or newly edited document. The application decides what external information is accessible and what is placed into the model’s current context.

The original Retrieval-Augmented Generation work by Lewis et al. combined a generative model with external non-parametric memory retrieved from a dense vector index. The broader architectural idea survives beyond that specific implementation: external evidence can be retrieved at inference time instead of expecting all useful knowledge to be encoded in model parameters.

This creates a useful separation of responsibilities: the source stores information, the retriever selects evidence, the context carries that evidence into the request, and the model interprets it. Keeping those boundaries visible makes failures much easier to diagnose.

Context: The Main Types of Data Sources

A vector index deserves special attention. In many architectures it is not the canonical source of truth. It is a retrieval index derived from documents or records. The authoritative document may live in object storage, a CMS, Git, PostgreSQL or another system, while embeddings and metadata are stored separately for fast semantic lookup. Some systems do use a vector store as primary storage, but that is an architectural choice rather than a requirement of RAG.

If the boundary between retrieval, persistent memory, current state and model context is still unclear, see AI Agent Memory Is Not RAG. Those layers can use some of the same storage technologies while still having different correctness rules.

Assumptions

• The application is allowed to access the external source.
• The relevant source contains enough information to answer the question.
• The data can be parsed or queried in a form the retrieval layer can use.
• The retrieved information is fresh enough for the requested decision.
• The model receives the selected evidence in its context.
• Authorization is enforced before protected evidence reaches the model.
• The generation model can still be wrong even when retrieval is correct.

These assumptions matter because retrieval cannot compensate for missing evidence, stale source versions, broken parsers or unauthorized access. A RAG pipeline can only be as trustworthy as the evidence path that feeds it.

Variables

Diagnostic / Decision Method

The first decision is not “Which vector database should I install?” It is: What kind of fact am I trying to retrieve?

A useful test is: Do I already know which record I need, or must the system discover which passage is relevant? If the record is known, query it directly. If relevance must be discovered, search becomes more important.

When an answer is wrong, diagnose the pipeline in order instead of immediately changing the LLM:

1. 1. Source coverage: Does the correct information exist in the accessible source set?
2. 2. Freshness: Is that version current enough for the question?
3. 3. Parsing: Was the relevant content extracted correctly?
4. 4. Chunking: Did the evidence stay together with the conditions that give it meaning?
5. 5. Retrieval: Does the correct chunk appear among the candidates?
6. 6. Ranking: Are stronger sources ranked above weaker or conflicting ones?
7. 7. Context assembly: Did the application actually send the selected evidence to the model?
8. 8. Generation: Did the LLM faithfully use the supplied evidence?
9. 9. Attribution: Can each important claim be traced to a source?

For a deeper production-debugging method, see RAG Failed — But Which Layer Actually Failed? A Diagnostic Method, which expands this chain into independently testable failure layers.

Evidence

The RAG paper by Lewis et al. formalized generation that conditions on retrieved external memory rather than relying only on model parameters. That provides the conceptual foundation for separating the generator from a retrievable knowledge source.

Sentence Transformers documents semantic search as embedding the corpus and the query into a vector space and retrieving items with high semantic similarity. Its current API also distinguishes query encoding from document encoding for retrieval tasks.

SQLite FTS5 demonstrates the other side of the spectrum: mature full-text retrieval can rank documents without embeddings. This matters because lexical search remains valuable for identifiers, exact terminology and many hybrid retrieval designs.

OpenAI’s embeddings documentation describes embeddings as numerical vector representations used for relatedness and search. This is one implementation path for semantic retrieval, not the definition of RAG itself.

Real Example 1: A Folder of Text Files

Suppose a directory named knowledge/ contains ordinary text files. Python can load them with no AI library at all.

from pathlib import Path

def load_text_files(folder="knowledge"):
    documents = []

    for path in Path(folder).glob("*.txt"):
        documents.append({
            "source": path.name,
            "text": path.read_text(encoding="utf-8")
        })

    return documents

documents = load_text_files()

for document in documents:
    print(document["source"], len(document["text"]))

The filesystem is the data source. The next question is how much text should become one retrievable unit. For long documents, searching one complete file is often too coarse. This is why RAG pipelines commonly create chunks.

A Very Simple Chunker

def chunk_text(text, max_chars=800):
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]

    chunks = []
    current = ""

    for paragraph in paragraphs:
        candidate = f"{current}\n\n{paragraph}".strip()

        if current and len(candidate) > max_chars:
            chunks.append(current)
            current = paragraph
        else:
            current = candidate

    if current:
        chunks.append(current)

    return chunks

This example groups paragraphs until a rough character limit is reached. It is intentionally understandable rather than optimal. Production systems often chunk by tokens, headings, sections, sentence boundaries or document structure. Tables, source code, contracts and API documentation may need different strategies.

Preserve Provenance While Chunking

def build_chunks(documents):
    chunks = []

    for document in documents:
        for index, text in enumerate(chunk_text(document["text"])):
            chunks.append({
                "id": f'{document["source"]}:{index}',
                "source": document["source"],
                "chunk": index,
                "text": text,
            })

    return chunks

A useful chunk carries more than text. Source name, document ID, URL, timestamp, version or section can later support citation, debugging and freshness checks. If provenance is lost during ingestion, it becomes much harder to explain why a particular answer was produced.

Real Example 2: Structured Data — Use SQL When SQL Is the Right Tool

Not every external fact should go through semantic search. If the question asks for an exact current record, a direct database query is usually clearer and more deterministic.

import sqlite3

def get_order_status(order_id):
    connection = sqlite3.connect("shop.db")
    cursor = connection.cursor()

    cursor.execute(
        "SELECT status, total, currency FROM orders WHERE id = ?",
        (order_id,)
    )

    row = cursor.fetchone()
    connection.close()

    if row is None:
        return None

    return {
        "order_id": order_id,
        "status": row[0],
        "total": row[1],
        "currency": row[2],
    }

print(get_order_status(4711))

If the application already knows that the user is asking about order 4711, embedding the entire orders table and asking semantic search to rediscover that row usually adds complexity without benefit. A strong design rule is: retrieve structured facts with structured queries; retrieve unstructured knowledge with search.

The returned database row can still be placed into the model context so the LLM can explain it in natural language. But direct state or record access is conceptually different from searching a knowledge corpus.

Real Example 3: Full-Text Search Before Embeddings

Between a naive Python loop and vector search lies a mature class of lexical retrieval systems. SQLite includes FTS5 for full-text search, including BM25 ranking.

import sqlite3

connection = sqlite3.connect("knowledge.db")
cursor = connection.cursor()

cursor.execute(
    "CREATE VIRTUAL TABLE IF NOT EXISTS docs USING fts5(title, body)"
)

cursor.execute(
    "INSERT INTO docs(title, body) VALUES (?, ?)",
    ("AKM", "The AKM uses 7.62 mm ammunition.")
)

connection.commit()

query = "AKM ammunition"

rows = cursor.execute(
    "SELECT title, body, bm25(docs) AS score "
    "FROM docs WHERE docs MATCH ? "
    "ORDER BY score LIMIT 5",
    (query,)
).fetchall()

for row in rows:
    print(row)

connection.close()

Lexical search is especially useful when exact terminology, product codes, names, identifiers or domain-specific words matter. Semantic search is not automatically better. Production systems often combine both signals.

Real Example 4: Semantic Retrieval With Embeddings

Embeddings turn text into numerical vectors so semantically related passages can be compared even when they do not use identical wording. Sentence Transformers provides a straightforward local implementation.

# pip install sentence-transformers

from sentence_transformers import SentenceTransformer, util

documents = [
    "The AKM uses 7.62 mm ammunition.",
    "A Med Kit restores health.",
    "Vehicle maintenance includes checking oil, brakes and tires.",
    "Account recovery requires access to the registered email address."
]

model = SentenceTransformer(
    "sentence-transformers/multi-qa-mpnet-base-cos-v1"
)

document_embeddings = model.encode_document(
    documents,
    convert_to_tensor=True
)

question = "How do I repair my car?"

query_embedding = model.encode_query(
    question,
    convert_to_tensor=True
)

hits = util.semantic_search(
    query_embedding,
    document_embeddings,
    top_k=2
)[0]

for hit in hits:
    print(round(float(hit["score"]), 3), documents[hit["corpus_id"]])

The query does not contain the phrase “vehicle maintenance,” but a semantic model can still rank that passage highly because the concepts are related. This is the practical reason embeddings are common in RAG systems.

For small collections, embeddings can stay in memory. Larger systems usually persist them in a vector-capable index or database and perform nearest-neighbor search there. The storage changes, but the logic remains: encode the question, find relevant document representations, return the best evidence.

Real Example 5: Build the Context for the LLM

A retriever should return evidence. The LLM should then receive the question plus that evidence. Keeping retrieval and generation separate makes both easier to inspect and test.

def build_prompt(question, retrieved_documents):
    context = "\n\n".join(
        f'[{doc["id"]}] {doc["text"]}'
        for doc in retrieved_documents
    )

    return f"""
Answer the question using the supplied context.

Rules:
- Do not invent facts that are not supported by the context.
- If the context is insufficient, say so.
- Cite the source IDs you used.

Question:
{question}

Context:
{context}
""".strip()

The instruction does not make the model infallible. It simply creates an explicit evidence boundary. The model can still misunderstand good evidence, ignore a condition or overgeneralize. That is why retrieval quality and generation quality must be evaluated separately.

Real Example 6: A Complete Minimal Pipeline

def answer_question(question, all_documents, call_llm):
    # 1. Retrieve evidence
    retrieved = retrieve(question, all_documents, top_k=3)

    # 2. Build model context
    prompt = build_prompt(question, retrieved)

    # 3. Generate the answer
    answer = call_llm(prompt)

    return {
        "answer": answer,
        "sources": [doc["id"] for doc in retrieved]
    }

The function receives call_llm as a dependency on purpose. Retrieval should not care whether generation is performed by a cloud model, a local model or another provider. The data path belongs to the application.

Optional Generator: OpenAI Responses API

One possible generator is the OpenAI Responses API. Keeping the model name in an environment variable avoids hard-coding a particular model into the RAG architecture.

# pip install openai

import os
from openai import OpenAI

client = OpenAI()

def call_llm(prompt):
    response = client.responses.create(
        model=os.environ["OPENAI_MODEL"],
        input=prompt,
    )
    return response.output_text

The same retrieval pipeline can be connected to a local inference server. This is an important architectural point: RAG is not owned by the LLM provider. The application owns the source, retrieval and context assembly.

The Whole Architecture in One View

USER QUESTION
     |
     v
+-------------+
|  Retriever  |
+-------------+
   |       |
   |       +----> SQL / API / state query
   |
   +------------> keyword / full-text search
   |
   +------------> embedding / vector search
                     |
                     v
              relevant evidence
                     |
                     v
+-----------------------------------+
| question + evidence + instructions |
+-----------------------------------+
                     |
                     v
                   LLM
                     |
                     v
                  answer

This data-flow model is more durable than memorizing one framework. Libraries, databases and model vendors will change; the responsibility boundaries remain.

Common Misconceptions and Failure Modes

“RAG means vector database.”

No. Vector search is one retrieval method. RAG can use full-text search, SQL, APIs, knowledge graphs, vector search or combinations of them. The defining pattern is retrieval of external information for generation.

“If the data is in PostgreSQL, I must embed the whole database.”

No. Structured records should usually remain queryable as structured records. Embeddings are useful for semantic relevance, not as a replacement for deterministic queries.

“More chunks means a better answer.”

Not necessarily. Extra context can introduce noise, conflicting versions and irrelevant material. Retrieval should optimize for useful evidence, not maximum volume.

“A high similarity score proves the answer.”

No. Similarity measures relevance, not truth or applicability. A highly similar passage can be outdated, from the wrong product version or valid only under conditions that do not match the question.

“Once the correct chunk is retrieved, hallucination is solved.”

No. Retrieval improves grounding but does not guarantee faithful reasoning. Generation still needs evaluation, and high-risk workflows may require deterministic validation or human review.

“The model failed, so change the model.”

Not necessarily. The correct source may have been missing, parsed incorrectly, split badly, filtered out, ranked too low or omitted from the assembled context. Model replacement should not be the first diagnostic step.

Edge Cases

• Conflicting documents: two sources may disagree because versions, jurisdictions or products differ.
• Time-sensitive facts: a semantically relevant source may already be stale.
• Permissions: a retriever must not return documents the current user is not authorized to access.
• Multi-language collections: the embedding model and retrieval strategy must support the languages actually used.
• Tables and source code: plain paragraph chunking can destroy structure that is essential to the answer.
• Very short identifiers: semantic retrieval can be weaker than exact matching for SKUs, IDs, error codes or acronyms.
• Long questions requiring several facts: retrieval may need decomposition, several searches or reranking rather than one top-k query.
• Source hierarchy: an official current policy may need to outrank an older but semantically closer discussion document.

Limitations

The Python examples intentionally optimize for transparency, not scale. The keyword retriever is naive, the chunker uses character length, the SQLite examples do not include production connection management, and the semantic example keeps all embeddings in memory.

A production system may require vector indexes, rerankers, hybrid retrieval, document parsers, caching, incremental indexing, source versioning, access-control filters, observability, evaluation datasets and failure handling. None of those additions change the core architecture; they make each boundary more reliable.

RAG also cannot create evidence that is absent from the source set. If the source is wrong, incomplete or stale, a better embedding model cannot turn it into authoritative knowledge.

What Would Change This Answer?

The architecture changes when the task requires more than knowledge lookup. A live order status needs current state. A financial calculation may need deterministic code. A web-research task may need active search. A workflow may need tools that can write data back to another system. An autonomous agent may need planning, permissions and execution control in addition to retrieval.

RAG is therefore best understood as one evidence-acquisition layer inside a larger AI system. It is powerful precisely because it has a narrow job: find useful external information and place it in the model’s working context.

Conclusion

RAG becomes much easier to understand when the technology names are removed. A file is a source. A database is a source. An API is a source. A search function retrieves evidence. A prompt carries that evidence to the model. The LLM then interprets it and produces language.

The hard part of production RAG is not calling an embedding model. It is building a trustworthy evidence path from the original source to the final claim: preserving provenance, selecting the right retrieval method, keeping information current, controlling access, evaluating retrieval separately from generation, and knowing when a direct database or tool call is better than semantic search.

That is the practical continuation of the basic RAG model: first understand the roles, then make the data path explicit.

Primary Sources

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the 2020 paper introducing the RAG formulation that combines generation with retrieved non-parametric memory.
• Sentence Transformers — Semantic Search — official documentation for semantic retrieval, query embeddings and document embeddings.
• OpenAI — Vector Embeddings — official documentation describing embeddings as numerical representations used for relatedness and search.
• SQLite — FTS5 Extension — official documentation for full-text search and BM25 ranking in SQLite.
• OpenAI — SDKs and CLI — official Python SDK example for the Responses API used in the optional generator example.
• What Is RAG? The Simplest Explanation of How It Works — the conceptual first part of this series.