We want an AI agent to query historical project costing and schedule files to help estimators price new jobs, but our job archives use inconsistent folder structures and fragmented spreadsheets. What data hygiene work must happen first?

Question

Tyler Smith · Accepted Answer

Pointing an AI assistant at unorganized network folders will not generate intelligent pricing. It will hallucinate rates, average out incomparable job scopes, and create catastrophic margin erosion on newly won contracts. AI does not clean up bad data habits; it magnifies them.

Before you invest in any machine learning tooling or custom retrieval models, complete these four non-negotiable data hygiene steps across your project archives:
- Establish a standardized file taxonomy. Rename historical project folders using a strict convention such as Year-Client-ProjectCode-ScopeType, and quarantine all incomplete or abandoned draft files into an unindexed archive.
- Normalize spreadsheet schemas. Ensure historical job costing sheets share identical column headers for labor hours, raw material cost, subcontractor expenses, and final realized gross margin. If 2021 spreadsheets use different cost codes than 2024 sheets, write a script to map them to a single uniform chart of accounts.
- Strip out unstructured internal commentary. Clean out conflicting cell notes, hidden tabs, and informal text overrides that explain why a job blew its budget. Separate standard cost data from anomalous one-off write-downs.
- Verify final actuals against billed totals. Only index completed jobs where final financial reconciliation is verified. Teaching an AI model to quote using preliminary cost estimates rather than closed project actuals guarantees inaccurate margins.

Treat this data sanitation as an operational Rock. Once your historical costing files conform to a rigorous, validated structure, an AI model can reliably surface baseline estimates in seconds. Building this discipline protects your gross margins and builds an auditable, system-dependent estimating asset for prospective acquirers.