Accuracy
How accurately can the model predict demand?
PGDM — Research & Business Analytics — Applied Research Project
Forecasting Intelligence Lab
Does a better forecast always create a better inventory decision?
Evaluating the Operational Value of Increasing Forecasting Intelligence
An empirical research project comparing classical statistical forecasting, machine learning and small language models across forecast accuracy and downstream inventory performance.
8
forecasting methods
2
retail datasets
10+
accuracy & inventory metrics
3
research layers
The Question
“Does increasing forecasting sophistication—from classical statistical forecasting to machine learning and small language models—necessarily lead to better forecast accuracy and inventory decisions?”
Forecasting Sophistication
Forecast Accuracy
Inventory Policy
Inventory Simulation
Operational Value
Inventory Decision
Why this research matters
Two things are measured, not one.
Forecasting studies often stop at error metrics. This project connects forecast accuracy to the inventory decisions it enables.
Operations
Does the forecast actually improve inventory outcomes?
Intelligence
Does additional model sophistication create additional operational value?
Research design
An experimental pipeline, run end to end.
Select a step to read a short explanation. The full detail lives in the methodology page.
Evidence — instrument 02
A better forecast is not automatically a better inventory decision.
Forecast error versus total inventory cost, plotted per model from the verified synthetic sanity check. The most accurate model (SARIMA) is not the lowest-cost; the lowest-cost model (XGBoost) is not the most accurate.
FORECAST ACCURACY (MAE) × INVENTORY COST — all models
Most accurate ⇄ SARIMA — but 4th in cost ·
Lowest cost ⇄ XGBoost — but 6th in error
Verified synthetic-data sanity check — not final research evidence. Final M5 / Favorita experiments will replace this plot.
Datasets
Two independent retail demand environments.
M5 is the primary dataset; Corporación Favorita is used for replication and divergence across retail environments.
M5 — Walmart Retail Sales
Product × Store × Day
- Daily retail demand at product–store level
- Calendar variables and event days
- Item prices and price changes
- SNAP (supplemental assistance) indicators
- Multiple demand patterns across series
Primary dataset for the full experiment.
Corporación Favorita Grocery Sales
Product × Store × Day
- Daily grocery sales by store and product
- Promotion indicators
- Holiday and event information
- Store and product characteristics
- A second, structurally different retail environment
Independent retail environment used for replication.
M5 and Favorita differ in geography, retailer, product structure, promotions, scale and demand patterns. They are not controlled replications — findings are reported as replication and divergence, not as proof of generalization.
Model progression
From simple forecasts to forecasting intelligence.
Glossover each model to see what it models, why it is included, and how its sophistication changes. Higher levels mean more methodological sophistication — not guaranteed performance.
00 Naïve Baseline Repeats the most recent observed demand as the forecast.
Why included: Establishes the minimum benchmark that more sophisticated models must outperform.
Sophistication: Zero model parameters; minimal “intelligence”.
Role in experiment: The floor of the comparison.
01 Simple Exponential Smoothing Statistical Models a level with exponentially weighted past observations.
Why included: Baseline statistical method for level-based demand without strong trend or seasonality.
Sophistication: One smoothing parameter; univariate.
Role in experiment: First statistical rung.
02 Holt Statistical Adds a trend component to exponential smoothing.
Why included: Tests whether capturing a trend improves retail forecasts.
Sophistication: Two parameters; univariate.
Role in experiment: Introduces trend.
03 Holt-Winters Statistical Adds a seasonal component (level + trend + seasonality).
Why included: Retail demand is often weekly-seasonal; this captures that structure.
Sophistication: Three smoothing parameters + season length.
Role in experiment: Introduces seasonality.
04 ARIMA Statistical Models temporal dependence via autoregressive, differencing and moving-average terms.
Why included: A classical, statistically principled approach with interpretable diagnostics.
Sophistication: Fitted p, d, q order per series; more formal model selection.
Role in experiment: Classical dependence modelling.
05 SARIMA Statistical Extends ARIMA with seasonal components (P, D, Q).
Why included: Combines ARIMA’s power with explicit seasonal structure.
Sophistication: Largest parameter set among statistical models; slowest to fit.
Role in experiment: Most sophisticated statistical forecaster.
06 XGBoost Machine Learning Gradient-boosted trees over lag, rolling-statistic, calendar, price and promotion features.
Why included: Represents nonlinear relationships and uses exogenous drivers unavailable to univariate models.
Sophistication: Feature count + tree ensemble; more data and compute required.
Role in experiment: Primary machine-learning comparison.
07 Small Local LLM Generative AI A small language model run locally (Ollama), evaluated in two distinct roles.
Why included: Explores whether generative AI adds forecasting or decision-support value — without assuming it wins.
Sophistication: Highest complexity, opaque, external model + compute needed.
Role in experiment: Experimental; evaluated, not presumed superior.
The progression represents increasing forecasting sophistication. Whether that sophistication produces better accuracy or better inventory outcomes is exactly what the experiment tests.
Forecasting vs Inventory
A better forecast is not automatically a better inventory decision.
The study evaluates forecasts on both dimensions — and treats them as related but distinct.
Pipeline A · Predictive performance
- Forecast
- MAE — Mean Absolute Error — average absolute error.
- RMSE — Root Mean Squared Error — penalises large errors.
- WAPE — Weighted Absolute Percentage Error — aggregate error, robust to zero demand.
- MASE — Mean Absolute Scaled Error — comparable across demand series.
Pipeline B · Operational performance
- Forecast → inventory policy
- Holding cost
- Stockout / lost-sales cost
- Cycle service level (CSL)
- Fill rate
- Total inventory cost
Predictive Performance + Operational Performance
= Forecasting Value
Inventory simulation
From forecast to cost — through one shared policy.
Every forecast feeds the same periodic-review order-up-to policy, so models are compared fairly on operational outcomes. Lost sales is the primary demand-shortfall model.
Forecast→ Order-up-to policy→ Inventory position→ Demand→ Fulfilled / Lost sales→ Holding + Stockout cost
Interactive conceptual demonstration
Order-up-to level—
Safety stock—
Est. holding (per cycle)—
At-risk units—
Peak stock—
Conceptual only. This uses a simple illustrative service-stock heuristic to show how order-up-to, safety stock, holding and at-risk units respond to your inputs. It is not a research result and does not reproduce the experimental pipeline.
Cost sensitivity
The operationally preferred method may depend on the economics of inventory.
A 2-D holding : stockout cost-ratio sweep will identify the cost-minimal forecasting method in each region of the economic plane. The "winner regions" below are placeholders until actual experimental results exist.
A 2-D holding : stockout cost-ratio sweep will identify the cost-minimal forecasting method per region. This is a planned experiment.
LLM experiment
Where does the LLM actually add value?
A small local language model (Ollama) is evaluated, not assumed to be superior. It is tested in two distinct roles.
Experiment A
LLM as a numerical forecaster
Historical demand→ Prompt→ Structured forecast
Evaluated on→ MAE · RMSE · WAPE · MASE
Run under a controlled protocol (fixed model/version, temperature 0). The expected null is that a small local LLM rarely beats specialised forecasters on retail series — treated as a finding either way.
Experiment B · primary claim
LLM as inventory decision support
Analytical forecast + inventory + lead time + costs + service target → Small local LLM → Structured recommendation
The LLM recommendation is compared against the analytical order-up-to baseline from the same inputs — a measurable "useful?" test, not a vibe check.
Structured output
/* Experiment B — decision support */
{
"action": "ORDER",
"order_quantity": 1260,
"reasoning": "Forecast over lead time plus safety stock exceeds on-hand stock.",
"risk_factors": ["Holding cost if demand runs below forecast"]
}Illustrative example of the output schema — not an experimental result.
Hypotheses
Eight testable claims.
Each hypothesis is paired with its planned statistical test. No hypothesis is marked accepted or rejected until the actual experimental results exist.
| ID | Research claim | Planned test | Status |
|---|---|---|---|
| H1 | Forecasting methods differ significantly in accuracy. | Friedman test across models + post-hoc Nemenyi. | planned |
| H2 | Increasing forecasting sophistication does not necessarily produce proportionally greater forecast accuracy. | Compare accuracy improvement against a complexity/effort increase. | planned |
| H3 | Forecast accuracy is associated with inventory performance. | Spearman rank correlation between error metrics and inventory cost. | planned |
| H4 | The lowest forecast error does not necessarily produce the lowest inventory cost. | Rank models separately by accuracy and by cost; compare ranks. | planned |
| H5 | Operational model ranking changes with the holding : stockout cost structure. | 2-D cost-ratio sweep; identify cost-minimal model per region. | planned |
| H6 | Forecasting performance varies across demand patterns. | Friedman / paired tests within demand segments. | planned |
| H7 | A small local LLM can provide useful inventory decision support when given structured forecast and inventory context. | Compare LLM recommendation against the analytical order-up-to baseline. | planned |
| H8 | Results replicate and diverge across the two retail datasets. | Repeat the identical evaluation framework on M5 and Favorita. | planned |
Research status
Where the project actually stands.
What is proposed, what is implemented and verified, and what experiments are still to be run — from the real repository state.
Proposal
- Research question defined
- Research objectives defined
- Dataset strategy defined
- Methodology defined (v2.0)
Implementation
- Forecasting pipeline scaffolded
- Inventory simulator implemented (lost sales)
- Evaluation metrics implemented
- Streamlit prototype implemented
- Automated tests implemented
Experiment
- M5 full experiment
- Favorita replication
- Cost-sensitivity sweep
- LLM evaluation
- Final statistical testing
The Experiment items are not yet complete. The website reports them as planned — nothing is presented as finished research.
Research contribution
What this project adds.
Academic
An integrated framework connecting forecasting accuracy with downstream inventory performance.
Managerial
Evidence for selecting forecasting methods based on operational objectives, not accuracy alone.
Practical
A reproducible public research artifact combining forecasting, inventory simulation and interactive decision analysis.
Scope & limitations
An experimental framework, not a production system.
The purpose is an experimental research framework. The study deliberately does not attempt:
- Real-time enterprise ERP integration
- Actual purchase-order execution
- Multi-echelon inventory optimisation
- Full supply-chain network optimisation
- Large-scale proprietary foundation-model training
- Autonomous inventory-decision deployment
Public research demo
Explore the Interactive Research Lab
Run or explore forecasting, model comparison and inventory simulations through the interactive research application.
Demo page explains how to run the application locally.