LLM Implementation for a Commercial Equipment Distributor: How We Cut a 3-Day Quote Cycle to Under 6 Hours

Ed Hitchcock
1 day ago
7 min read

By Ed Hitchcock, Enterprise AI Systems Architect, SupplyTech Solutions

Most LLM implementation projects we walk into died at pilot. A vendor sold the exec team on a chatbot, the sales team never used it, and now the CFO wants to know what the $180K spend produced. We get called after that happens. This post is a case study on doing it right the second time, at a Commercial Equipment Distributor where LLM implementation was tied to a real revenue metric: quote turnaround time.

What This LLM Implementation Actually Delivered

Our client is a Commercial Equipment Distributor operating in the Southeast US. About $240M in annual revenue across forklifts, aerial work platforms, industrial cleaning equipment, generators, and their aftermarket parts and service. Six branches, 28 outside sales reps, 14 inside quoting specialists. Their product catalog runs to roughly 4,200 SKUs at the base level, but ~78% of orders involve at least one configurable option (attachments, mast heights, tire types, drivetrain, telematics packages), so the effective product surface is much larger.

We were brought in after their first LLM attempt failed. A generic chatbot bolted onto their CMS. It produced plausible-sounding quotes that were wrong roughly one time in five, and sales stopped using it inside three weeks. Twelve weeks after we started, they had a working LLM-backed quoting assistant that six of their fourteen inside quoting specialists were using every day, quote cycle time dropped from an average of 3.2 days to under 6 hours, and their configuration error rate on drafted quotes was sitting near 3.5%.

Numbers up top so the rest of the post has context:

Quote cycle time: 3.2 days average to under 6 hours average

Configuration error rate on drafted quotes: ~19% to ~3.5%

Time-to-first-response on inbound RFQ: ~9 hours to ~40 minutes

Inside quoting specialist hours reclaimed: roughly 14 hours per week per rep, applied to higher-margin custom builds

Engagement cost: fixed hourly across a 12-week build, ~0.09% of annual revenue

What Actually Broke in Their First LLM Implementation

We reviewed the failed pilot before writing a single line of new code. Three problems showed up consistently.

First, the pilot LLM had no access to their real product data. It was pulling from marketing PDFs and a scraped copy of their website. Pricing was 8 to 14 months stale. Configuration compatibility rules did not exist in that dataset at all. A rep would ask “can we put the side-shift attachment on the 5000-series with the 42-inch forks” and the model would confidently say yes, with no source, when the answer depended on hydraulic flow specifications that lived only in ERP.

Second, the pilot had no concept of citations or grounding. It generated fluent paragraphs. Reps learned quickly that they could not trust it, and once trust breaks in a sales tool, it does not come back with a prompt tweak.

Third, and this is the one most LLM implementation vendors skip: the pilot was not wired into the actual quoting workflow. Reps had to open a separate browser tab, type a question, then re-key any useful answer into the ERP quote screen. Every extra system a rep has to touch to complete a task increases the odds they revert to the old way.

Our LLM implementation had to fix all three or it was going to die the same death.

The Design Rules We Wrote Before Touching Code

Before we picked a model or drew an architecture, we wrote the rules the system had to follow. Four of them:

1. Every answer that touches price, compatibility, or spec cites the ERP record it came from. No citation, no answer. The agent says “I do not have a grounded source for that” and stops.

2. The LLM never invents a SKU, price, or option combination. It retrieves and composes. Generation is confined to natural-language framing.

3. The agent lives inside the ERP quote screen as a side panel. Reps do not context-switch. If a rep accepts a suggestion, it writes back to the quote as structured line items, not as pasted text.

4. Every accepted answer creates a training signal. Every rejected answer creates a training signal. The system gets better because reps use it, not because vendors ship model updates.

These rules did most of the architectural work for us.

The Architecture Behind It

We built four layers, each with a specific job. The client already ran Microsoft 365, Dynamics 365 Business Central for ERP, and Azure. We stayed on that stack.

Layer 1: Structured retrieval. The source of truth for price, spec, compatibility, and stock is ERP. We built a nightly extract into an Azure SQL analytics database, keyed by SKU, with option compatibility rules exposed as a lookup table. About 4,200 base SKUs, 11,800 valid configured combinations after compatibility rules apply. The rep-facing agent hits this database first for anything factual.

Layer 2: Unstructured retrieval. Product manuals, application guides, warranty policies, and the OEM technical bulletins we get from their five equipment manufacturers. Roughly 6,400 PDFs. We chunked, embedded, and indexed all of them in Azure AI Search with metadata tags for OEM, product family, revision date, and document type. The agent uses this layer for open-ended technical questions (“what maintenance interval applies to a 2024 aerial platform running in a coastal environment”) that structured data does not cover.

Layer 3: The LLM and tool-use layer. GPT-4-class model behind a system prompt that enforces the four design rules above. The model does not answer factual questions from parametric memory. It calls one of five tools we exposed: `get_product_spec`, `check_compatibility`, `get_price`, `search_technical_docs`, `check_stock`. Every tool call is logged with input, output, and the raw ERP or document reference. When a rep sees an answer, they see the tool calls that produced it, one click to inspect.

Layer 4: The ERP write-back and feedback loop. When a rep accepts a suggested line item, the agent writes it to the Business Central quote through the official API, populating SKU, configured options, quantity, unit price, and the sourced spec notes as a memo. When a rep rejects a suggestion, the reason is captured (wrong SKU, wrong price, wrong option, hallucinated) and fed into a weekly review queue. That queue drives our retrieval improvements and prompt updates.

The point of naming these layers explicitly: an LLM implementation without a retrieval layer is a chatbot. Without a tool-use layer it is a search engine with better prose. Without a write-back layer it is a demo. All four together is a working system.

Four-layer LLM implementation architecture for a Commercial Equipment Distributor: structured retrieval, unstructured retrieval, LLM plus tool-use, ERP write-back and feedback

The 12-Week Build Sequence

We ran this as five phases. Client sponsor sign-off between each phase, fixed hourly billing, no scope creep past the design rules.

Weeks 1-2: Discovery and grounding. Sat with three inside quoting specialists for two full days each. Watched 47 quotes from RFQ arrival to send. Documented the actual questions reps ask ERP, catalogs, manuals, and OEM reps to complete a quote. That question inventory (312 distinct question shapes across configuration, price, availability, and technical spec) became the eval set.

Weeks 3-4: Data foundation. Built the nightly ERP extract, the compatibility rules lookup, the PDF corpus indexing, and the metadata tagging pass. Client's IT team did the ERP side; we did the search and embedding side. Zero LLM code written yet.

Weeks 5-7: Tool-use layer and system prompt. Defined the five tool contracts. Built the system prompt around the four design rules. Started with GPT-4o-mini for the routing layer and GPT-4o for synthesis, evaluated Claude and open-source alternatives against the eval set weekly, ended up staying with the OpenAI stack because latency and tool-calling accuracy were consistently ahead for our patterns. Ran the 312-question eval set at the end of every week.

Weeks 8-10: ERP write-back and side-panel UI. Built the Business Central write-back API integration and the side-panel that lives inside the quoting screen. This is where most vendors give up. It is the least glamorous work and the piece that determines whether reps adopt.

Weeks 11-12: Rollout, feedback loop, and training. Two inside specialists first, then four, then all six of the initial cohort. Weekly 45-minute review sessions where we walked through rejected suggestions and prioritized fixes. Training was two 90-minute sessions per specialist plus a written playbook. No formal go-live. The tool was in the ERP screen, they used it, we watched the metrics.

12-week five-phase build sequence for the LLM implementation, from discovery and grounding through rollout and feedback loop

What We Chose Not To Do

Three decisions worth naming because they came up as pressure during the engagement.

We did not build a customer-facing chatbot. The exec team asked. We refused. A customer-facing LLM without operator hardening is a lawsuit waiting to happen in a regulated equipment sale. We told them we would revisit in year two after the internal system had a full year of stable data.

We did not fine-tune a model. We were pushed twice. Both times we showed data: on our 312-question eval, retrieval quality improvements moved accuracy from ~74% to ~94%. Prompt engineering moved it to ~96%. A fine-tuned model would cost 6-figures to build and maintain and would gain us a marginal point, tops. Money better spent on retrieval and eval expansion.

We did not migrate off Business Central. The client had an ERP consultant lobbying for a full replatform to make LLM implementation “easier.” We shipped against the existing ERP and left that decision to a separate business case.

What We Measured

After 90 days of production use with six specialists:

Median quote cycle: 3.2 days to 5.8 hours

Configuration error rate on drafted quotes: 19% to 3.5%

Inbound RFQ time-to-first-response: ~9 hours to ~40 minutes

Percentage of quotes with agent-suggested line items accepted as-is: ~58%

Percentage of quotes where agent flagged an incompatibility the rep would have missed: ~11% (this one surprised the sales VP more than any other metric)

Reclaimed capacity: reps applied roughly 14 hours per week each to custom-configured or high-margin builds instead of straight repeats

The engagement cost, all-in labor across 12 weeks at fixed hourly, worked out to about 0.09% of annual revenue. The client already owned the Azure, Business Central, and Microsoft 365 licenses.

90-day results after LLM implementation: quote cycle time, configuration error rate, time-to-first-response, and reclaimed specialist hours

Why LLM Implementation Fails Without These Pieces

Most of the LLM implementation work we get called into after a failed pilot has the same fingerprint. A model was chosen before a retrieval layer existed. Tools were not defined, so the model was left to guess. Write-back was skipped. Feedback loops did not exist, so the system got worse over time as new products and rules landed in ERP with no path into the LLM's world.

The model itself is the least interesting part. GPT-4 vs Claude vs Llama moves outcomes by single-digit percentages once you have retrieval and tool-use right. Retrieval quality and workflow integration move outcomes by 20 to 30 points.

If your first LLM implementation stalled at pilot, the fix is almost never a bigger model. It is a serious retrieval layer, explicit tool contracts, and a write-back path into the system your team already uses. Which is the LLM implementation approach we run on every engagement, and the one that got a $240M Commercial Equipment Distributor from a dead pilot to a production system in twelve weeks.