AI Strategy Consulting: How We Turned a Specialty Hardware Distributor's Tribal Knowledge Into a Grounded Retrieval Layer
By Ed Hitchcock, Enterprise AI Systems Architect, SupplyTech Solutions
Most AI strategy consulting engagements we get called into skip the boring layer and go straight to the model. The client wants a copilot, an agent, a chatbot. What they actually need first is a knowledge foundation that any model can retrieve from without hallucinating. We built one for a specialty hardware distributor over 12 weeks, and this is what the sequence looked like in practice.
The client is a specialty hardware distributor with roughly $180M in revenue, 9 branches across the Ohio Valley, 42 inside sales reps, and about 34 years of documentation spread across four file shares, three OneDrive tenants, and a legacy DMS none of the newer hires had ever logged into. Leadership had already licensed Microsoft 365 E5, adopted Copilot for a subset of users, and started a SharePoint tenant that nobody was using. Their ask: give us an AI strategy that ends with a knowledge system our people actually work inside, not another platform we bought and abandoned.
What AI Strategy Consulting Actually Delivered Here
We ran a five-phase engagement over 12 weeks at a fixed hourly billing rate. No packaged product, no license resell margin, no roadmap deliverable that sits on a shared drive and dies. The work centered on building the Knowledge Management System that AI has to be grounded to before any agent, copilot, or automation becomes trustworthy.
The output was a Microsoft 365 native KMS built on SharePoint Online, an internal home dashboard branded to the client, a Copilot Studio agent trained on curated retrieval prompts, a full metadata taxonomy, and a migration of roughly 340,000 legacy files with automated classification applied at ingest. By week 12, 6 departments were operating inside the new system, 92% of high-traffic documents had been migrated and tagged, and the Copilot Studio agent was answering roughly 65% of first-line policy and procedure questions with cited SharePoint sources.
The Problem We Walked Into
The distributor's knowledge was in three failure states at once.
First, information was fragmented. Warranty policy lived in a PDF on a branch manager's desktop. The current version of the return authorization SOP lived in an email thread from 2023. Vendor cross-reference sheets lived in a shared Excel workbook that four people edited simultaneously without version control. The organization did not have a single source of truth. It had 40 partial sources, and every rep had memorized which of those partial sources they personally trusted.
Second, tribal knowledge dominated. New hires took roughly 6 to 9 months to reach full productivity because the answers they needed lived inside three or four senior reps who happened to be sitting near them. When those seniors were on the road, ramp stalled. When those seniors retired, entire domains of institutional memory left the building.
Third, prior AI tools failed silently. The team had piloted a general-purpose chat interface on top of their existing file shares. It generated confident answers with no citations. Reps who tried it once and got burned stopped using it. The pilot did not fail because the model was bad. It failed because there was no structured knowledge for the model to retrieve from.
Why Most AI Strategy Consulting Skips the KMS Step
Buyers of AI strategy consulting typically want the sexy layer. Agents. Copilots. Predictive dashboards. None of it works reliably without a documented, cost-quantified process layer underneath and a structured knowledge layer on top of that.
Our position is direct. If your processes are not documented, AI cannot automate them. If your knowledge is not organized, AI cannot retrieve it. If your metadata does not exist, AI cannot filter or cite. Any consultant selling you an agent strategy without first grounding the agent in a governed knowledge system is selling you a demo, not a system.
The five-phase model puts KMS as Phase 2, immediately after documented and costed processes in Phase 1. Operational AI, ERP modernization, and Enterprise AI all come after. This client had partial Phase 1 work in place, so we scoped tightly around Phase 2.
The 12-Week Sequence We Ran
We split the work into five discrete phases, each with a fixed deliverable and a client sponsor sign-off before we moved forward.
Weeks 1-2: Discovery and Architecture Design. We mapped every existing document repository, interviewed 14 process owners across HQ, warehouse, service, parts, and HR, and produced an Information Architecture Document. That document defined the SharePoint site hierarchy, the library structure inside each department site, the metadata taxonomy every document would be tagged against, and the permissions model. Deliverable: one signed IA document.
Weeks 3-4: SharePoint Home Setup. We built the internal home dashboard, branded to the client, with a tenant-wide search bar, role-based quick links, and a dynamic content zone for recent department updates. Departmental sites went up with libraries pre-configured against the taxonomy. Deliverable: dashboard live, department skeletons in place.
Weeks 5-7: Copilot Studio Agent Build. We built a Copilot Studio agent scoped to the SharePoint knowledge base. We wrote roughly 55 retrieval prompts across common intents: policy lookup, SOP lookup, vendor cross-reference, warranty terms, return authorization, and price book lookup. We defined the citation behavior so every answer either surfaces the source document or refuses. We built a 32-question evaluation set with the operations team over 3 days and used it as our regression harness. Deliverable: agent specification sheet and a working agent embedded in Teams.
Weeks 8-10: Migration and Metadata Automation. We used the SharePoint Migration Tool to move roughly 340,000 legacy files from the four file shares and three OneDrive tenants into SharePoint. SharePoint Premium content understanding models auto-classified document type, department, and effective date at ingest. Below-threshold files were quarantined for department-lead review. Deliverable: migration runbook, governance playbook covering versioning, retention, and ownership.
Weeks 11-12: Training and Rollout. We built written and visual user guides for reps and administrators, ran a live training session in each of the 6 in-scope departments, and did an administrator handoff so the client's internal IT lead could maintain the taxonomy and agent going forward. Deliverable: user training guide, executive summary report with KPIs.
The Architecture Behind It
The stack was intentionally boring and intentionally native. Every layer used tools the client already licensed.
Layer 1: Storage and structure. SharePoint Online, one hub site plus 6 department sites, each with libraries structured by document type. Metadata columns were mandatory at upload, enforced by content types.
Layer 2: Migration and classification. SharePoint Migration Tool for bulk moves. SharePoint Premium models for auto-classification of document type, department, effective date, and vendor. Confidence threshold at 0.85. Anything below went to a human review queue that department leads worked through weekly.
Layer 3: Retrieval and search. Native SharePoint search extended with managed properties tied to our taxonomy. Search results respected permissions, versioning, and document authority tiers so retired policies did not surface above current ones.
Layer 4: Conversational layer. A single Copilot Studio agent, published to Teams, scoped to the SharePoint knowledge base. Retrieval prompts were curated, not open. Citations were mandatory. When the agent could not find grounded content, it refused rather than generating.
Governance layer running across all four: ownership assigned per site, retention policies aligned to compliance, versioning enforced, permissions inherited from department groups in Entra ID.
What We Chose Not To Do
Three decisions we made against the client's initial ask.
We did not build a custom application on top of Microsoft 365. The client's first instinct was a custom rep-facing web app with its own front end. We pushed back. Every custom front end we have seen for a KMS becomes the reason the KMS dies, because someone has to maintain two things instead of one. Teams and the SharePoint home dashboard were sufficient.
We did not migrate everything. Roughly 340,000 files went in. Another 180,000 files were flagged as either duplicative, obsolete, or scoped for archive-only retention. Moving them would have polluted the index and degraded retrieval quality. We wrote an archive policy instead.
We did not skip the evaluation set. The 32-question harness took 3 days to build with the operations team. It saved us weeks in agent refinement because we could measure every prompt change against a fixed rubric instead of vibe-testing.
What We Measured
At 90 days post-launch, using the client's own operational data:
Roughly 65% of first-line policy and procedure questions were being resolved through the Copilot Studio agent without a rep needing to escalate to a senior. Baseline pre-launch was 0%, since no such tool existed.
Onboarding time for new inside sales reps moved from a 6 to 9 month full-productivity ramp to an estimated 4 to 6 month ramp based on the first two cohorts through the new system. Direction and magnitude both matter here. A 15% to 20% ramp improvement compounds fast when the sales headcount is 42 and turnover runs typical for the industry.
Retrieval time for high-frequency lookups (warranty terms, vendor cross-reference, RMA policy) dropped from a self-reported 8 to 14 minutes per lookup to under 60 seconds through the agent, with citations. Reps stopped calling senior colleagues for policy questions.
Total engagement cost landed at roughly 0.15% of annual revenue for the 12-week Phase 2 build. That figure does not include Microsoft licensing, which the client already carried. It reflects consulting labor at a fixed hourly rate.
Why This Sequence Matters for Anyone Buying AI Strategy Consulting
The lesson from this engagement, and from the last dozen we have run against distributors of similar size, is that the AI strategy question and the KMS question are the same question. You cannot ground an agent to knowledge that does not exist in a structured form. You cannot cite documents that live in 40 places. You cannot audit refusals when your taxonomy is inconsistent.
If your AI strategy consulting engagement does not spend real weeks on the knowledge layer, it is not an AI strategy engagement. It is a demo purchase. The demo looks good on the day of the executive review and dies within the quarter, because reps stop trusting answers that arrive without a source.
We built this system on tools the client already owned. SharePoint. Copilot Studio. SharePoint Premium. The Migration Tool. The value was not the software. It was the taxonomy, the migration discipline, the evaluation set, the refusal-first agent behavior, and the governance model. Those are the deliverables that hold up 90 days after launch when the consultant is gone and the reps are the only judges left.
If your legacy knowledge is in that same failure state, message me. The retrieval layer is where AI strategy becomes an operating system instead of a slide deck.
Comments