RAG Architecture Consulting: How We Built a Grounded Parts-Knowledge Layer for an Industrial Parts Distributor

Ed Hitchcock
Jul 7
6 min read

By Ed Hitchcock, Enterprise AI Systems Architect, SupplyTech Solutions

About the Engagement

The client is a $310M industrial parts distributor operating twelve branches across the Ohio Valley and the Southeast. They sell fluid power, motion control, and industrial process components to roughly 4,200 active buyer accounts. Their catalog runs about 1.1 million unique SKUs, of which approximately 180,000 are stocked. The remaining SKUs are drop-ship or special-order through more than 340 supplier relationships.

The knowledge problem is what pushed them toward RAG architecture consulting in the first place. Every one of those 340 suppliers has technical data sheets, application notes, cross-reference tables, and warranty terms in slightly different formats. Every one of those 1.1 million SKUs has attributes buried in a mix of ERP records, SharePoint folders, supplier PDFs, and email threads with product managers. Inside sales reps average about 4 to 6 years of tenure, and the tribal knowledge of "which cross reference is trustworthy" walks out the door with every retirement.

The client had all the raw material for a working retrieval system. What they did not have was the discipline around retrieval, grounding, and citation that would make sales reps actually trust the outputs. They retained us for a 10-week RAG architecture consulting engagement to fix it.

Why the First Attempt Failed and What RAG Architecture Consulting Actually Fixes

The failed pilot had three problems, and they are the same three problems we see in almost every RAG deployment that dies in month two.

First, the retrieval was single-shot semantic search over a flat vector index. Every question ran a top-K similarity lookup against 40,000 mixed documents, and the top three chunks fed straight into the prompt. When someone asked about a specific hydraulic filter cross reference, the retriever pulled back three chunks that mentioned filters, not the one chunk that answered the actual question.

Second, the grounding logic did not respect document authority. A rep would ask about warranty terms for a specific pump family, and the response would blend answers from the actual supplier warranty PDF, an old sales training deck from 2019, and a customer email thread that quoted incorrect terms. The model averaged three sources with equal weight. Two of the sources were wrong.

Third, there was no citation surface. The tool told reps what the answer was. It did not tell them which document, which page, which supplier, or how recent. Without traceability, no rep with any product experience would trust it on a $40,000 pump quote.

RAG architecture consulting is not about picking a bigger model or a better embedding. It is about designing retrieval, authority, and citation as first-class components of the system. That is what we rebuilt.

The Architecture We Built

We designed a four-layer RAG stack on top of the client's existing Azure environment. The client already ran Dynamics 365, SharePoint Premium, and Microsoft 365 Copilot licensing, so the integration substrate was in place. What was missing was the retrieval intelligence.

The first layer is the source registry. We inventoried every knowledge source and tagged it with three attributes: authority tier, freshness policy, and audience scope. Supplier warranty PDFs and technical data sheets from OEMs are Tier 1. Product manager notes and internal application guides are Tier 2. Historical sales conversations and archived training materials are Tier 3. The retriever weights results by tier and refuses to blend Tier 3 answers into Tier 1 responses without an explicit flag.

The second layer is the document processing pipeline. Every source flows through structured chunking that respects document type. Supplier PDFs get parsed with layout-aware extraction that preserves tables and section headings as retrievable units. Cross-reference tables become row-level chunks with SKU keys attached as metadata. SharePoint pages get chunked by heading section, not by arbitrary token count. Each chunk carries a metadata payload including source document ID, page, section, supplier, product family, authority tier, and effective date.

The third layer is the retrieval logic itself. We deployed Azure AI Search as the vector store, but the retriever is a hybrid that runs three passes in sequence: a metadata filter to constrain the search space by product family and supplier scope, a hybrid vector plus keyword search inside that constrained space, and a re-ranker that scores results against the specific question intent classified from the user prompt. On a well-formed parts question, the retrieval surface averages 8 to 12 candidate chunks, of which the top 3 to 5 make it into the generation prompt.

The fourth layer is the grounded generation and citation surface. Azure OpenAI generates the response with a prompt template that requires inline citations to source chunks. Any claim without a matching source chunk triggers a refusal or a lower-confidence hedged response. The response surface displays every citation as a clickable link back to the source document, the exact page, and the effective date of that document. Reps see what the model saw.

RAG architecture diagram showing the four-layer grounded retrieval stack

The Sequence We Actually Ran

We sequenced the 10-week engagement into four workstreams, all running in parallel after week two.

Weeks 1 and 2 handled discovery and source inventory. We walked every document repository with the product management team and the two lead field application engineers. We built the authority-tier taxonomy, agreed on the freshness rules, and captured the top 40 question patterns that inside sales reps actually ask. Those 40 patterns became our evaluation set.

Weeks 3 through 6 handled the pipeline build. Document parsing, chunking, metadata enrichment, vector indexing, and the initial hybrid retriever went in. We ran the 40-question evaluation set every Friday and tracked citation accuracy, answer accuracy, and refusal rate as three separate metrics.

Weeks 4 through 8 handled the retriever tuning. The re-ranker took the longest to tune. We iterated on the intent classifier, on the metadata filters, and on the chunk-size trade-off for supplier PDFs. By week 7 the evaluation set was scoring 82 percent answer accuracy against a human-graded ground truth, up from 41 percent on the failed pilot.

Weeks 7 through 10 handled the rollout and citation surface polish. We shipped to a controlled group of 14 inside sales reps in week 7, rolled to the full 55-rep team by week 9, and dedicated the final week to the governance framework, refusal-behavior tuning, and operational handoff.

Sequence diagram of the 10-week RAG architecture consulting engagement

What We Measured After 90 Days of Production Use

The client tracks four operational metrics on the RAG surface. All numbers below come from the system logs and from a rep survey conducted in week 12 of production.

Answer accuracy on the 40-question evaluation set stabilized at approximately 84 percent on the initial batch, holding above 80 percent for the full 90-day window as the source corpus grew. The failed pilot never exceeded 44 percent on the same evaluation set.

Rep-verified citation quality, defined as a citation that a rep would confidently forward to a customer, moved from a baseline of roughly 22 percent on the failed pilot to approximately 79 percent on the rebuild.

Average time to answer a technical parts question on the sales floor dropped from a baseline of roughly 8 to 14 minutes (search across systems, ask a peer, look in the ERP) to roughly 45 to 90 seconds through the RAG surface.

Escalations to the two field application engineers dropped by approximately 40 to 55 percent on the covered product families, freeing them for the harder application questions the RAG surface is deliberately designed to refuse.

The engagement cost, including services and the first year of Azure AI Search plus Azure OpenAI consumption, came in at approximately 0.4 percent of revenue. Licensing was already in place.

Metrics chart showing 90-day production results for the RAG system

Three Things We Deliberately Did Not Do

We did not use a single flat vector index. The retrieval failure mode we saw on the failed pilot is nearly always traceable to flat indexing without metadata filtering. Constrain the search space first, then embed.

We did not let the model answer without citations. Every production response either carries inline source citations or refuses the question. Reps trust refusal. Reps do not trust hallucination.

We did not skip the evaluation set. Forty questions took two days to build with the product team and paid back every week. Without a fixed evaluation set, you cannot tell whether a retriever change made the system better or worse. Most failed pilots skip this and cannot debug their own regressions.

Why This Sequence Matters for Anyone Buying RAG Architecture Consulting

RAG works when the retrieval discipline matches the operational reality of the business. For an industrial parts distributor, that means respecting supplier authority, preserving table structure in PDFs, and constraining search by product family before you ever run a vector lookup. Those three moves accounted for most of the accuracy delta between the failed pilot and the working system.

Picking a RAG architecture consulting partner is mostly about whether they will invest in the retrieval layer, refuse to ship without a citation surface, and build the evaluation set on day one. The generation step is a solved problem. The retrieval and grounding steps are where the work lives.

The client is now expanding scope to include supplier warranty automation and a customer-facing parts lookup portal built on the same retrieval spine. We are sequencing those into the next quarter.