What to ask a supply chain AI vendor before signing

Demand forecasting demos are easy to win. They run on clean historical data, against a naive baseline, with hindsight. Production is none of those things. Five questions close most of the gap.

One: what baseline are you beating

An 18% accuracy improvement is meaningless until you know the comparison. Beating a naive last-period forecast is table stakes. Beating your current planners, who apply judgement the model cannot see, is the real bar. Insist the baseline is your incumbent process, not a textbook one.

Two: how does it handle a promotion it has never seen

Most forecasting failures in retail and CPG are promotional. Ask what the model does with a discount depth or mechanic absent from history, because that is exactly what marketing will run next quarter.

Three: the cold-start problem

New products are where planners most want help and where models are weakest. Ask specifically how long a new SKU is handled by fallback logic, and what that logic is. If the answer is that the planner overrides manually for twelve weeks, that is fine, but it changes the business case.

Four and five: who owns it after go-live

Retraining cadence and override design are the two questions most commonly skipped, and the two that decide whether anyone still uses the system in a year. A forecast a planner cannot argue with gets ignored, and a model nobody retrains degrades quietly. Ask who holds both jobs on day 400, and get the answer in writing.