Building IAnversic RAG: Architecture, Trade-offs, and Practical Engineering

From an ERP Assistant to an Independent RAG Platform

Project: IAnversic RAG

When I started designing IAnversic ERP, an institutional assistant was one of my priorities from the beginning.

I've worked with ERP systems for several years, and one recurring problem is documentation. For end users, these systems can feel like black boxes. They need to understand business processes, navigate complex workflows, and find answers across documentation that is not always easy to access or interpret.

My initial idea was to build an assistant that could be integrated into the main ERP workflows, helping users understand processes and find the information they need without having to navigate the entire system.

Eventually, I realized that this capability could be useful beyond a single application. Instead of building an assistant tightly coupled to the ERP, I decided to create an independent service that could be reused across different systems.

That was the starting point for IAnversic RAG.

The goal was to build a knowledge retrieval platform that could run on infrastructure controlled by the organization, provide answers grounded in its documents, and enforce access permissions before information reached the language model.

I wanted the retrieval capability to be a reusable service, not just another feature inside an ERP.

Designing for Real-World Constraints

I live in Ecuador, where reliable connectivity is not always available everywhere. I also wanted to account for organizations that need greater control over their data or cannot depend entirely on external AI services.

Infrastructure costs were another consideration. GPU hardware can be expensive, especially for smaller organizations that want to benefit from AI without making a significant initial investment.

These constraints shaped the design.

I chose a local-first approach, while keeping the architecture flexible enough to support external providers such as OpenAI and Gemini. The idea is to give organizations options based on their infrastructure, privacy requirements, and budget.

For local inference, I started with relatively small models, including llama3.2:3b for generation and BGE-M3 for embeddings.

This approach has trade-offs. Smaller models can reduce hardware requirements, but answer quality, latency, and concurrency depend on the model, the workload, and the available resources. Local deployment is not automatically cheaper or faster in every scenario.

The objective was not to assume that local models would solve every problem. It was to make local deployment a viable option while preserving the ability to evaluate alternatives.

The Architecture: Separating Responsibilities

IAnversic RAG combines a FastAPI backend, PostgreSQL with pgvector, a dedicated document ingestion worker, configurable embedding and generation providers, and a Next.js web interface.

Each component has a specific responsibility.

• FastAPI coordinates API requests and application workflows.
• PostgreSQL and pgvector store application data and support vector similarity search.
• The ingestion worker processes documents outside the main question-answering request.
• The embedding provider converts document content and queries into vectors for retrieval.
• The generation provider produces answers from the retrieved context, using Ollama for local inference or an alternative configured provider.
• Next.js provides the web interface through which users interact with the platform.

The architecture separates document ingestion from question answering because they have different workloads.

Ingestion can involve document processing, text extraction, chunking, and embedding generation. Answering a question requires a different sequence: query processing, retrieval, context preparation, and model inference.

Keeping these responsibilities separate makes the system easier to reason about, test, and evolve.

Architecture overview

This diagram represents the main logical components and their relationships. The exact execution path depends on the configured providers and workflow.

One important design decision was to keep embedding and generation providers configurable. They solve different problems and have different performance and quality characteristics. Separating them makes it possible to evaluate alternatives without redesigning the entire application.

Document Ingestion: Preparing Knowledge for Retrieval

A RAG system is only as useful as the information it can retrieve.

Documents need to be transformed into representations that preserve enough context to support meaningful retrieval. This involves more than splitting text into chunks and generating embeddings.

In IAnversic RAG, the ingestion workflow is designed around a dedicated worker that processes documents separately from the API request that initiates the operation.

At a high level, the workflow is:

• A document is submitted through the application.
• The API validates the request and coordinates the ingestion job.
• The worker processes the document and prepares its content.
• The content is divided into chunks suitable for retrieval.
• The configured embedding provider generates vectors.
• The resulting information is stored for retrieval.

The system supports BGE-M3 embeddings, with a configurable embedding provider. It also supports an optional BGE reranker to refine the ordering of retrieved passages.

These components have different responsibilities. Embeddings represent content in a vector space, vector search identifies semantically similar candidates, and reranking can improve the ordering of those candidates.

However, none of these steps guarantees that the final answer will be correct. Retrieval quality depends on the document representation, the query, the retrieval configuration, and the quality of the evidence returned.

That is why I treat ingestion and retrieval as parts of a larger pipeline rather than as isolated technical features.

Knowledge Base in Practice

The knowledge base interface shows the documents uploaded to IAnversic RAG. For this example, I used the ConvFinQA dataset, sourced from Hugging Face, which contains financial question-answering data.

Knowledge base showing uploaded documents.

Question Answering: Retrieval Is Only One Part of the Process

When a user submits a question, the system needs to identify relevant information and use it to generate a response.

But retrieving the nearest vectors and forwarding them to a language model is not enough for an application that handles organizational knowledge.

The system also needs to account for access permissions, preserve the context of the retrieved information, and determine whether the available evidence is sufficient to answer the question.

The main workflow can be summarized as follows:

The exact checks depend on the configured retrieval and grounding thresholds, but the principle is consistent: the system should not assume that every question can be answered from the available documents.

Access control belongs in the retrieval layer

One of the most important design considerations is ensuring that users can retrieve only the information they are authorized to access.

IAnversic RAG uses SQL-level permission filtering as part of its retrieval and access-control design. The intention is to enforce these restrictions before the retrieved content is passed to the language model.

This is different from relying exclusively on prompt instructions to tell a model which information it should ignore.

If restricted information enters the model's context, the application has already crossed an important security boundary. Access control therefore needs to be enforced by the application and data-access layers, not just by the language model.

Grounded answers and citations

Retrieval alone does not make an answer reliable. The returned passages may be incomplete, only partially relevant, or insufficient to answer the question.

IAnversic RAG includes configurable retrieval and grounding checks, source citations, and abstention behavior when confidence is insufficient.

The objective is to make the relationship between the answer and its supporting evidence more transparent. When the available information does not support a sufficiently grounded response, the system should be able to abstain rather than invent an answer.

Citations are useful, but they are not proof of correctness by themselves. The relevance of the retrieved content and the accuracy of the generated answer still need to be evaluated.

Example: Answer Generation with Ollama

This screenshot shows an example of a response generated by IAnversic RAG using Ollama for local inference.

Example response generated with a locally hosted language model.

Local Models vs. External Providers

Supporting local inference was an architectural decision, not simply a preference for open-source models.

For organizations working with internal documentation, keeping processing within their own infrastructure can help maintain control over sensitive information. The actual privacy properties depend on the complete deployment, including logging, networking, storage, and provider configuration.

Local inference also introduces operational constraints. Model size, available memory, CPU or GPU resources, and concurrent requests all affect performance.

External providers offer a different set of trade-offs, including network latency, provider pricing, availability, and external data-processing requirements.

I do not see local and cloud inference as mutually exclusive approaches. They are options with different characteristics.

By keeping providers configurable, IAnversic RAG can be adapted to different environments instead of assuming that a single model or deployment strategy will fit every organization.

The next step is to compare these options through controlled experiments, measuring not only response time and resource consumption but also retrieval quality and answer grounding.

Language Model Configuration

The application provides settings to configure the language models used for generation, allowing the inference setup to be adapted to different deployment requirements.

Language model settings in IAnversic RAG.

Deployment and Validation

A system that works on a development machine is not necessarily ready for deployment.

IAnversic RAG uses Docker Compose to organize its main services, including the database, migrations, API, ingestion worker, Ollama, and web application.

The project also includes automated tests using stub providers, integration checks, access-control acceptance tests, and scripts for exercising workflows with real models.

These tests serve different purposes.

• Automated tests verify application behavior and core logic under controlled conditions.
• Pipeline checks exercise document ingestion and question answering.
• Access-control tests check organization isolation, role-based permissions, API key access, and related behavior.
• Real-model checks help evaluate the behavior of the configured embedding and generation pipeline.

It is important to distinguish between these forms of validation. Tests using simulated providers can make application behavior easier to verify, but they do not establish real-model quality or production performance. Likewise, a successful smoke test does not replace load testing or a controlled benchmark.

I want the system to be evaluated at multiple levels: retrieval quality, answer grounding, access control, latency, and resource consumption.

Those measurements will be especially important when comparing local inference on different servers with external providers.

What I Learned from the Design Process

Building IAnversic RAG has reinforced a principle I consider important in applied AI engineering: the model is only one component of the system.

The surrounding architecture determines how documents are processed, which information can be retrieved, how context reaches the model, and how the resulting answer is evaluated.

Several design decisions follow from that:

• Document ingestion should be treated as a distinct workflow, not just a preprocessing step.
• Retrieval must account for permissions and preserve the context needed to interpret the evidence.
• Embedding and generation providers should be configurable when deployment requirements vary.
• Local inference needs to be evaluated against real hardware and workloads rather than assumptions about cost or performance.
• Testing should distinguish application correctness from model quality and production-scale behavior.

These considerations are not unique to IAnversic RAG. They are part of the broader engineering work involved in turning an AI prototype into a useful application.

There is still work to do, particularly in systematic performance benchmarking and further evaluation of retrieval and answer quality. I see that as part of the engineering process, not something to hide behind a polished demo.

For me, the goal is not simply to connect documents to a language model. It is to build a system that organizations can adapt to their own infrastructure, security requirements, and operational constraints.

That is the direction behind IAnversic RAG.

---

IAnversic RAG is an ongoing engineering project. This article describes its design goals, architecture, and implemented capabilities; performance and quality claims require separate, controlled validation.