Databricks Launches a No-Code Tool for Building Data Pipelines

At its Data+AI Summit today in San Francisco, Databricks launched a number of new features for its data platform, including Lakeflow Designer, a no-code tool for building data pipelines using a drag-and-drop user interface targeted at non-data scientists. Lakeflow Designer is built on top of Lakeflow, the company’s more code-heavy data engineering platform for building these pipelines.

Databricks argues that enterprises are under a lot of pressure because they want to use the data they have to power AI-based applications. But to do that, they need to get this data to the right places — and they don’t always have the right talent to do so.

As Databricks’ VP of marketing Joel Minnick noted in an interview ahead of today’s event, data engineers are now a central part of virtually every data team, and they tend to be code-first and happy to use a platform like Lakeflow. On the other hand, there are now also plenty of no-code tools for business analysts and citizen data scientists.

“Those tools are useful because they do help them be able to build the pipelines they need to get something into their [business intelligence] system or their analytical system,” Minnick said. “But what we find is, generally what those tools don’t do is productionize very well. They don’t have lineage. They don’t have auditing. They’re very hard to troubleshoot. They don’t plug into modern things like CI/CD and aren’t designed for scale and resiliency.”

So what tends to happen is that when these users hand their pipelines over to the engineering teams, they end up rebuilding everything from scratch in code.

Lakeflow was a first stab in this direction, but then the team also looked at how AI has changed the market, Minnick said. So now, with Lakeflow Designer, behind the scenes, the visual interface is translated directly into production-grade code, with governance, lineage tracking, auditability and all the other features of the Lakeflow platform still in place. This way, it’s easy for the business teams to create the pipelines they need (they know the business processes better than any development team, after all), and then hand those off to the development team, which will be able to maintain them in their preferred environment.

It’s worth noting that AI is not going to build these data pipelines for you. Instead, the AI part here is about Databricks’ assistant, which can help answer questions about how to best build these pipelines, but, at least for now, it won’t do it automatically based on a user’s prompt. (Though that’s surely on somebody’s roadmap, too.)

“You’ve got the Databricks Assistant there, kind of riding shotgun, being able to give you advice on, ‘How do I structure these queries? Help me diagnose this error. What advice would you have on how to think about structuring this pipeline?’ And then, at any time, as you publish that Designer pipeline out into production, the data engineering team can look at that exact same pipeline in Lakeflow in a fully code-based environment,” Minnick explained.