The Horse-and-Car Moment for Data Engineering
In this blog, we explore how AI is disrupting traditional data engineering by eliminating bottlenecks, accelerating data access, and transforming roles within data teams. Drawing parallels to the horse-to-car revolution, we show why the future of data lies in AI-powered automation, self-service platforms like Latttice, and strategic roles such as Data Catalysts and AI stewards.
The Problem
The Bottlenecks of "Data Engineering"
Over recent decades, the rise of data engineering as a discipline was meant to streamline how organizations handle data. Yet in practice, it often created new, or exasperated existing, silos. Business stakeholders have grown all too familiar with waiting in line for the data engineering team to deliver pipelines or data.
As one Data Tiles post put it, "Business users can't wait for weeks or months for central teams to deliver requests... the demand on these central teams... is ever increasing." In other words, the very separation of "data engineering" introduced bottlenecks between those who need data and the data itself.
The Ironic Twist: Complexity Breeds More Complexity
It's an ironic twist: data engineering was supposed to solve complexity, but sometimes it added to it. Cameron Price, founder of Data Tiles and creator of Latttice, has compared some of these industry trends to past tech mistakes, noting that they "ended up stifling progress and increasing complexity." Formal processes like data contracts were intended to bring order and reliability, but in reality they can become bureaucratic bottlenecks that slow down innovation.
We've seen this pattern before. Frameworks and committees get erected to manage data, yet months later, decision-makers still struggle to get clear, timely insights. As Zhamak Dehghani (creator of the Data Mesh concept) lamented, data teams often focus on buzzwords and frameworks "to address some of the deepest, darkest problems" in data – only for many organizations to simply rebrand the same issues or form committees, instead of actually solving them.
The Reality Check
The result? Data chaos and delay. In a vivid example, Jessie Moelzer from Data Tiles describes it like this, "room full of clever people" planning elaborate data projects, "frameworks, dashboards, and an 18-month roadmap", yet six months later "the same issues resurface... repackaged in a fresh slide deck."
Despite all the talk, "still no access, still no insights" for the people who need data the most.
This status quo is increasingly untenable for businesses that need agility. Gartner analysts report that organizations empowering domain teams with self-service data access are twice as likely to experience success. Clearly, shuttling every data request through a specialized engineering funnel isn't the answer.
The Revolution
AI: The New Engine Accelerating Data Delivery
Enter Artificial Intelligence, the game-changer for data workflows. AI and automation are doing to data pipelines what the combustion engine did to transportation. Just as early 20th-century businesses had to rethink relying on horses, today we're realizing we no longer need to rely on slow, manual data engineering processes. AI is removing the need for tedious engineering in data.
Cameron Price explores this shift in the context of modern data architectures, showing that AI-powered agents can "transform data mesh architectures by removing manual complexity and enhancing scalability." Instead of humans hand-coding integrations or tweaking pipelines for every new requirement, intelligent agents and algorithms handle these tasks automatically in the background.
This is a horse-and-car moment for data. In the past, if you wanted to move faster, you might add more horses, or more data engineers, and hope to get there sooner. But ultimately, you were still limited by the old paradigm. AI is the Model T of data: a fundamentally different engine. It doesn't just speed up the old process; it renders it obsolete.
Instant Integration
Need to integrate a new data source or fix a broken pipeline? An AI assistant can infer the schema, clean the data, and adjust the pipeline without a human writing code.
Natural Language Analysis
Need an analysis of last quarter's sales? Modern AI-powered platforms let a business user simply ask a question in natural language and get the answer—no SQL or Python required.
No-code, AI-driven data products are turning everyday employees into "citizen data engineers," able to solve data problems with tools that automate the heavy lifting.
From Bottlenecks to Autobahns: The End of Data Plumbing
We already see this acceleration in action. Data Tiles' own platform, Latttice, uses a "zero-code, AI-powered approach" to let domain experts access and combine data instantly, "no more waiting on central IT or data teams." By removing technical barriers and decentralizing control, Latttice "eliminate bottlenecks... and empower teams to make faster, more informed decisions."
It's not just startups saying this; established voices agree that AI is taking over repetitive data chores. As an Andreessen Horowitz report boldly stated, "in short order, the tasks... will be done entirely by AI." Many basic roles composed of "highly repetitive, deterministic tasks" can be "replaced by AI agents, enabling companies to hire fewer employees or move to higher order tasks."
What does it look like when AI truly replaces traditional data engineering? Imagine instant data pipelines automatically built and maintained by AI, error-free and continuously optimized. No more scheduling meetings to hand off requirements to a data engineering team. No more back-and-forth debugging broken ETL/ELT jobs for weeks.
"Boards say they need to 'do something with AI' and hire consultants who talk in buzzwords—yet nothing actually happens. It's all theater."— Cameron Price
AI-driven platforms are about doing, not talking. They connect to sources, govern data, and serve up insights on-demand, without lengthy development cycles.
This shift is comparable to opening up an autobahn for data inside organizations: high speed, no stoplights.
The Transformation
Evolving Roles: Data Engineers as "Data Drivers," Not Mechanics
So, does this mean data engineers are out of a job? Not exactly, but their roles are absolutely evolving.
We're not firing the mechanics. We're handing them a faster car and asking them to drive.
Data engineers will shift from "data plumbers" to focusing where they should: as data pipeline creators. Without the added burden of trying to interpret business needs or deliver every report, they can focus on scalable, reliable pipelines, the foundations of healthy data ecosystems.
The Rise of Strategic Data Leaders
And for some, the shift will go even further. Many will move into strategic roles as AI stewards or as Data Catalysts — a role we champion at Data Tiles. As Cameron Price explores in his blog "The Data Catalyst: Shaping the Future of Data-Driven Alliances", Data Catalysts are cross-functional leaders who guide AI systems, empower business domains, and align data outcomes with strategic priorities. These aren't just new job titles, they represent a new way of thinking about data and leadership.
AI Stewards
Guide and govern AI systems to ensure ethical, effective deployment across the organization
Data Catalysts
Cross-functional leaders who empower business domains and align data outcomes with strategic priorities
Pipeline Architects
Focus on scalable, reliable data foundations without the burden of manual report delivery
"It will be hard to compare data engineering in 2024 and data engineering in 2028 and say 'those are the same things.'"— Tristan Handy, CEO of dbt Labs
AI automation will replace or simplify many of the manual tasks engineers do today, freeing them up to focus on more strategic problems.
It's the classic automation story: AI takes the drudgery. Humans take the lead.
Embracing the New Reality
It's time for a mindset shift. Businesses and data professionals must recognize that we are living through a transformative horse-and-car moment in the data world.
AI isn't just an incremental improvement. It's a disruptive force that changes how we build and deliver data products. Those who embrace it will sprint ahead.
Companies will deliver sophisticated use cases in days, not months. Data experts will lead innovation by orchestrating AI tools, not wrangling CSVs. And the bottlenecks of the pas, the long meetings, the backlog of data tickets, the "maybe next quarter" promises, will finally give way to immediacy and agility.
The optimism is grounded in results. Early adopters are reporting major efficiency gains. AI-driven data platforms are letting a single analyst do what once required a team. "AI can help businesses make decisions by providing better insights faster than ever before," as researchers Thomas Davenport and Rajeev Ronanki observed.
Faster
Better
Smarter
That's the destination.
And the road is already paved. Now it's time to drive.
Join a data conversation,
Lili Marsh.
References
- Andreessen Horowitz – Angela Strange: "The AI Future Is Already Here..."
- dbt Labs – Tristan Handy: "How AI will disrupt data engineering..."
- Thoughtworks – Zhamak Dehghani: Creator of Data Mesh principles
- Gartner (2023) – Analysis on AI-driven mechanisms for scalable, decentralized data management
- Thomas H. Davenport & Rajeev Ronanki: "Artificial Intelligence for the Real World," HBR