All of RAG infrastructure is already a commodity.
Tons of emerging solutions have made RAG infrastructure directly available off-the-shelf. There are dozens of platforms out there capable of spinning up a production-grade RAG stack in under an hour. To name a few I have worked with : n8n, dify, langflow.
And ready-made RAG solutions cover all enterprise preferences. Self-hosting for the security-conscious, fully managed cloud for the pragmatic, and everything in between.
.jpg)
But a RAG application remains only half as good as the data it’s built on.
This isn’t new: your RAG performance will collapse instantly when fed mediocre data. And in practice, it’s hard to avoid feeding mediocre data. For example, if your knowledge base contains multiple versions (v0, v_final, etc.) of the same file which differ in some aspects, that will inevitably damage your retrieval or end-to-end metrics.
Once your data repositories have been purged, it is surprisingly hard to parse enterprise data (PDFs, PPTX, etc.) effectively. Indeed, when you deal with 5,000+ (or even 500+) documents, the prior you have on the data to ingest is either very limited or will be proven false. And then the mistakes at parsing will compound downstream.
So parsing data is difficult. But at least you can easily get an intuition on whether or not you’ve done a good job so far: look at the chunk, look at the document, and evaluate.
The embedding stage is much more obscure to evaluate. I have frequently found myself incapable of knowing if my indexation was done right. There are no quality metrics on the collection of vectors that will serve the RAG. For example, can you think of a way to anticipate that some vectors are “too close” from one another ?
For RAGs to become a commodity, we need enterprise knowledge bases to converge to a standard structure.
Knowledge Graphs are a high-value promise for Enterprise RAGs. But from my experiments, graph ontologies are notoriously hard to define. And leveraging LLMs for detecting nodes and edges autonomously is not accurate enough. Though I would love to be proven wrong on this point.
Still, I think the use of LLMs to structure messy enterprise data is a promising frontier. Lately, I have come across Ohalo and Deasylabs for similar use cases which sound promising. If you have experimented with these, I would be very curious to know how effective they are in the real world.
If a standard model of enterprise unstructured data emerges (think of knowledge graphs but easier to build), search technology will finally be able to operate on solid ground. For example, can we think of a standard set of 50 metadata fields which can apply to more or less all of your documents ?
Can Agentic AI make that happen ? In practice, that would mean agent-based knowledge curation. Business teams would give instructions and validate the work of a Data Prep Agent that explores repositories, interacts with content owners, removes duplicates, summarizes intermediate versions, and enforces simple consistency rules.
We can assess the feasibility by taking each of the six aspects of data quality : accuracy, completeness, consistency, timeliness, validity and uniqueness. You can probably automate uniqueness and consistency checks. But completeness and validity for sure require external knowledge and an overall understanding of the business.
By their very nature, most RAG systems will require custom work
The essence of RAG is semantic search, ie the following assumption :
two chunk vectors are close ⇔ the two chunks are talking about the same thing.
Well, semantic search fundamentally limits how generic a RAG can ever be. Indeed, being the closest chunk and being the chunk which contains the answer are not the same thing. In other words, finding the “most similar” chunk is by no means a guarantee that they will contain the correct answer.
Find below is an example to illustrate. Chunk 1 is semantically closer, but only chunk 2 contains the answer.
In conclusion, retrieval, which is based on semantic search (or lexical search but same goes), is not perfectly aligned with the objective of finding the answer. That’s why AI teams always have to design custom preprocessing, custom retrieval with reranking or custom reasoning layers.
Even if RAG were a technical commodity, we see RAG systems plagued with adoption issues.
I have also seen RAG applications fail not because of the technical challenge, but because of poor user experience.
For starters, business users don’t know precisely what to expect from a RAG application, or how to prompt it. So they will start by treating the RAG like a human assistant, asking “who are you?” or even “test 123”. The results are likely to feel strange or deceptive.
In my opinion, RAG product design is often misleading. A RAG user interface looks like ChatGPT but behaves nothing like it. ChatGPT is a generalist, it’s good (not excellent) at everything. On the other hand, RAG systems are specialists : they excel at finding the needle in the haystack but are rather limited to that responsibility. Helping business users understand the difference is the key challenge to unlock RAG full value.
Conclusion
To wrap up, RAG applications have become a commodity in the sense that the “time to prototype” is now ridiculously low. You should not write any line of code to get your POC up and running, either vibe code it or use a no code solution.
Yet, to unlock business value, I am willing to bet that your AI team will have at some point to get their hands dirty. Until we have Data Prep agents that work for us to clean and standardize our knowledge bases, I am convinced that making RAG products that actually work will still require manual work.
