Designing an end-to-end data intelligence platform for a mid-market trading company.
A one-year project to take a growing trading company from scattered operational systems to a unified data platform — Data Lake, Data Warehouse, BI, and predictive models, all built from scratch. This was the foundation engagement where I formalized the planning cadence and sprint discipline I still use today.
From scattered operational data to a single source of analytical truth.
The starting position
The client — a mid-market trading company — was running their business on three operational systems that didn't talk to each other. Reports were assembled manually, leadership decisions ran on stale numbers, and there was no way to look forward: no forecasting, no demand modeling, no predictive view of the business.
The brief was deliberately ambitious. Build the entire analytical stack — ingestion, lake, warehouse, BI, predictive models — inside a single 12-month engagement, while delivering visible value along the way so leadership could feel the program working before it finished.
The challenge
One year. Four phases. Continuous value. Pure waterfall would have given leadership nothing to look at for nine months. Pure agile without a plan would have lost the architectural coherence the platform needed.
The answer was a hybrid: a high-level annual plan locking the macro phases and milestones, combined with two-week sprint execution inside each phase. Strategy at the top. Iteration at the bottom. The plan you'll see below is exactly that — and it's the planning model I still anchor delivery work to today.
Four quarters. Four phase shifts. One continuous build.
Each quarter has its own dominant phase, but the transitions are soft — planning continues into execution, monitoring begins before deployment ends. The boundaries below show emphasis, not isolation.
One concrete milestone per quarter, visible from outside the team.
Internal velocity is not a metric the client can feel. Every quarter therefore had to end on something demonstrable — something a stakeholder could see, click, or query.
Data flowing into the lake
Diagnostic + system spec complete; cloud provider selected; first ingestion pipelines live with data landing in the raw layer.
BI MVP, DW operational
Warehouse built and validated; BI MVP shipped to business users; predictive modeling foundations laid for Q3.
Predictive solution live
Modeling validated, integrated tests passed, predictive solution deployed into production with monitoring.
Operational + opportunity report
Solution in steady-state monitoring; team trained; expansion roadmap delivered; two new features contracted.
Three levels of detail. Each one earns its place.
Coarse-grained items are for the client and the contract. Fine-grained items are for the team and the sprint. The middle layer is where they meet — and where the conversation about scope actually happens.
Three deliverables that set every downstream decision.
Q1 is the most consequential quarter of the project — not because it ships the most, but because every Q2–Q4 decision references the artifacts produced here. Get these wrong and the rest of the year compounds the error.
Sprint one was built tight on purpose.
A first sprint is also a calibration exercise. I planned Sprint 1 with no slack to measure real team capacity — then absorbed the learning into Sprint 2 onwards. The numbers below are the planned envelope; the lesson was always going to be in the variance.
What the team actually does on each day.
A weekly view that shows the rhythm: prep early in the week, stakeholder time mid-week, group discussion and documentation late in the week. Two weeks of this gets the foundational interviews and cloud proposal done end-to-end.
From the project backlog to “done” — six columns, one flow.
The board mirrors the backlog hierarchy. Deliverables and activities sit on the left as a roadmap; refined tasks flow rightward through the sprint columns. The Review column is the quality control mechanism — every task gets a second pair of eyes before it can be marked complete.
What this project still gives me, years later.
Looking back, the value of this engagement wasn't only the platform I shipped — it was the planning model I built doing it. The patterns below are still the ones I anchor delivery work to today.
1. Three-level backlog as the universal translator
The Deliverable / Activity / Task hierarchy works because it speaks to three different audiences at once. The Deliverable layer is the contract — what the client signs off. The Activity layer is the conversation — where scope and effort actually get negotiated with the team. The Task layer is the sprint — what gets done this week. Most delivery friction I've seen since comes from teams collapsing two of these into one.
2. First sprint as a calibration, not a commitment
Sprint 1 was deliberately planned without slack — not because I expected the team to hit it perfectly, but because I needed to measure the gap. Velocity is empirical, and you only get a real number by running a tight first iteration and watching what actually lands. Every engagement I've led since starts the same way.
3. Quarterly milestones that a non-technical stakeholder can verify
“Velocity is up” is not a milestone. “Data is in the raw layer” is. Every quarter ended on something the sponsor could see or touch — a working dashboard, a deployed model, a signed report. The discipline of writing the milestone before designing the work has saved more programs from drift than any process artifact I've used since.
The platform was the deliverable. The planning model was the asset. Every delivery engagement I've led since has been a variation on the framework this project forced me to commit to paper.