Recently I read a LinkedIn Post “Agile is dead.” You’ve probably also heard the phrase thrown around, but the truth is more nuanced. Agile itself isn’t broken - how we’ve applied it is. Originally designed to help teams collaborate and deliver value quickly, Agile often derails in execution. Standups become rituals, sprints feel rushed, and teams find themselves delivering “something” instead of the right thing.
Nowhere is this dysfunction more evident than in data projects. The nature of data work - complex pipelines, siloed access, and misaligned priorities between business teams and data team - exposes Agile’s weaknesses when applied poorly.
When used correctly, Agile solves these challenges by fostering communication, alignment, and value-driven delivery. The problem is data teams often lack the tools and environment to make Agile truly effective.
Latttice is not a replacement for Agile, but a data solution that makes Agile easier, especially in the data space.
Why Agile Stumbles in Data Projects
Data projects are uniquely complex. Business teams want answers yesterday, but data teams face significant hurdles:
Data Silos: Critical data is locked away, slowing progress. According to Gartner (2022), nearly 80% of organizations struggle with siloed data, which hinders agility in data projects.
Disconnect Between Business and Data Teams: Business owners expect results without understanding data complexities. Data teams work in isolation, often solving the wrong problems. Forrester (2022) notes that 56% of data teams are tasked with solving problems not fully understood by business stakeholders.
Repetitive Manual Work: Much of data work involves cleaning and preparing datasets, leaving little time for value-added analysis. DataIQ (2021) found that 60% of data professionals spend most of their time on manual data preparation tasks.
Misaligned Priorities: Agile aims to deliver incremental value, but in data projects, “value” often isn’t clearly defined or agreed upon. This issue often leads to wasted resources and unmet expectations (McKinsey, 2021).
As a result, Agile becomes performative. Standups turn into status updates. Sprints become cycles of frustration. Instead of collaboration, teams experience burnout and growing distrust (Deloitte, 2023).
How Agile Can Work for Data Teams
At its core, Agile is about three things:
Collaboration between teams.
Iterative delivery of outcomes.
Continuous alignment on what matters most.
For data projects, when Agile is done right, it eliminates the chaos.
Standups clarify blockers and priorities.
Sprints deliver incremental insights that the business can act on.
Retrospectives ensure teams learn and adapt to challenges.
Most importantly, Agile keeps data teams and business teams aligned, ensuring that both sides understand the problem being solved and the outcomes being delivered.
A study by the Harvard Business Review (2022) found that organizations using Agile effectively in data projects achieved a 30% improvement in time-to-insight, directly improving business outcomes.
But even when Agile works as intended, data projects can still hit roadblocks - particularly around data access, manual work, and misaligned expectations.
The Importance of Data Products to Define Agile
1. Break Down the Work (Epics, Stories, and Tasks)
Agile Analogy: In agile, you break a large goal into smaller, manageable pieces (epics, user stories, and tasks). For example, if you’re building a software application, you don’t build the entire app in one go; you tackle one feature or function at a time.
Data Product Parallel: A data product is a self-contained, high-quality dataset designed to serve a specific purpose. Instead of trying to create a perfect data product from day one, you identify key features (e.g., schemas, transformations, APIs) and deliver them incrementally.
2. Deliver Incrementally and Iteratively
Agile Analogy: Agile focuses on delivering value early and often by working in sprints. You deliver working software at the end of each sprint, which may not be complete but is functional and can provide feedback.
Data Product Parallel: For a data product, you start by delivering a minimum viable data product (MVDP). This could be a basic version of the dataset with essential fields, pipelines, and access interfaces. Then, you build upon it in future iterations based on feedback.
3. Collaboration and Stakeholder Involvement
Agile Analogy: Agile thrives on close collaboration between developers and stakeholders, ensuring the work being done aligns with the needs of the business.
Data Product Parallel: In a data mesh, the domain team (who owns the data) collaborates with consumers (other teams who use the data). By involving data consumers early, you ensure the data product solves real problems and adds value.
4. Focus on Value and Feedback Loops
Agile Analogy: Agile teams prioritize delivering features that provide the most value to users and regularly seek feedback to adapt their plans.
Data Product Parallel: Data products should be designed to solve a specific business need (value-driven). You gather feedback from data consumers on the usability, quality, and structure of the data product, and adapt it as necessary in future iterations.
5.Continuous Improvement
Agile Analogy: Agile teams reflect on their processes in retrospectives and strive to improve them.
Data Product Parallel: As you build and maintain a data product, you continuously monitor its performance (e.g., data quality, reliability, usability) and iterate to make it better over time.
How This Looks in Practice
Imagine you’re building a Sales Data Product for the marketing team:
Sprint 1 (MVDP): Deliver a basic dataset with total sales by region and customer segments. Provide documentation and APIs for access.
Sprint 2: Add fields for sales trends over time, filters for product categories, and automated quality checks.
Sprint 3: Incorporate feedback from the marketing team to improve data granularity and add new fields like promotion effectiveness.
Sprint 4: Optimize pipelines for faster query performance and improve metadata for discoverability.
By following agile principles, you ensure that the data product evolves to meet user needs while being delivered incrementally.
Why Agile Works for Data Products
Flexibility: Requirements for data products often change as business priorities shift or as users explore new insights.
Value-Driven: Agile ensures you’re focusing on delivering data that drives business impact, rather than building features no one uses.
Efficiency: Incremental delivery ensures quicker time-to-value and avoids the risks of “big bang” data product delivery.
Final Thought: Agile Isn’t Dead - It Just Needs the Right Support
Agile isn’t the problem; it’s how we apply it - especially in data projects where misalignment and access bottlenecks create chaos.
Latttice doesn’t fix Agile. It fixes the issues that derail it. By empowering teams with access, automation, and alignment, Latttice makes Agile easier, enabling data teams to deliver real, measurable value with every sprint.
If you are ready to transform your data projects? Discover how Latttice can support your Agile workflows and keep your teams focused on what matters.
Join the Data Conversation,
Jessie Moelzer.
References
McKinsey & Company. (2021). Why Data Projects Fail—and How to Make Them Succeed. Retrieved from McKinsey.
Gartner. (2022). Data Management Challenges in the Enterprise. Retrieved from Gartner.
Forrester. (2022). The Disconnect Between Data Teams and Business Needs. Retrieved from Forrester.
DataIQ. (2021). Survey on Manual Data Preparation in Organizations. Retrieved from DataIQ.
Harvard Business Review. (2022). Agile in Data Projects: Lessons from High-Performing Teams. Retrieved from HBR.
Deloitte. (2023). Lessons from Failed Agile Transformations. Retrieved from Deloitte.
Comments