Matillion’s Data Productivity Cloud Brings dbt Orchestration

Matillion recently unveiled its Data Productivity Cloud, a platform designed to streamline many of the core processes for pipelining data for applications and analytics.

The cloud solution serves as an ecosystem for users of all types to work within a common framework for transitioning data from source to target systems. Its foundational elements include capabilities around data loading, transformation, synchronization, and orchestration.

Of particular interest is the offering’s support for dbt, which is quickly becoming the script-intensive transformational resource of choice for many. This functionality provides a counterpoint to the almost self-service approach to data pipelines that Matillion has traditionally provided, which “allows people without SQL or command line expertise to do transformations,” commented Mark Balkenende, Matillion VP of Product Marketing.

In addition to now offering resources for both code-savvy and non-savvy code users, Data Productivity Cloud also has a host of options for loading data that simplifies this requisite for data pipelines. Moreover, it provides an environment to accomplish this task for dbt in a single platform for all users, which streamlines processes and makes them more consistent.

dbt Orchestration

Adoption rates for dbt have significantly broadened in a relatively short amount of time for data engineers and professionals engaged in data preparation. The ability to orchestrate dbt within Matillion’s Data Productivity Cloud signifies a pairing of this technical approach to pipelining data with the company’s low-code, no-code method. According to Balkenende, now “the work you’re doing within dbt and the high code user base can also be orchestrated with the rest of the organization that’s using Matillion.” This point is far from academic, particularly for companies with large groups of users — or a significant amount of work being done by users — of low code and script approaches.

Firstly, it heralds an end to the silo ways of working that can fragment organizations, resulting in redundancies, escalating costs, and inefficiencies. “Data teams today are often working in silos,” Balkenende acknowledged. “They’ll have teams that are very high-code types in organizations, versus maybe the line of business data analyst using low code tools, and often not working together.” In addition to redressing this issue, the Data Productivity Cloud’s orchestration of dbt enables users to avail themselves of the platform’s loading capabilities. Point solutions, one-offs, and data silos may also be reduced because of the consistency this element of the cloud platform supplies for dbt’s “high code audiences that would rather do all the scripting and SQL coding themselves,” Balkenende pointed out.

Easing the Load

In addition to helping unify the user base for data pipelines, Data Productivity Cloud’s means of making data loading easy is remarkable. One of the simplest, most efficient, and least error-prone ways involves a software installation like, wizard-based method for rendering customized connectors via REST APIs. On the one hand, its applicability is nearly unlimited as one can employ it to create connectors “to really just about any type of source that you can imagine, all within a no-code solution” Balkenende mentioned. This capability is attributed to coupling Matillion’s universal connectivity with Matillion Data Loader.

The user experience this tandem provides, which is characterized by an ease that’s rare when building connectors for data pipelines, is perhaps an even bigger advantage. With the wizard issuing a simple set of instructions for users to complete to facilitate connections between new sources and targets, one ultimately needs “to know very little about what you’re connecting to, for the most part,” Balkenende reflected. “You really just need the basic access and things like that. It’s pretty easy.”

Unity and Scale

The elimination of silo culture for the data teams with different preferences (and tooling) for transforming data is a palpable benefit of Matillion’s recently announced platform. The support for orchestrating dbt also helps to broaden Matillion’s user base. Still, the scalability of the self-service, wizard-based method for connecting to new data sources, which even non-technical businesspersons can employ, is difficult to top in terms of overall enterprise merit.

Balkenende referenced a hospitality industry example in which users were able to devise these customized connections to sources they hadn’t accessed before within 10 minutes. “There’s upwards of 8,000 Martech solutions out there,” Balkenende noted. “No integration company’s going to build 8,000 connectors. Our goal is to provide this really easy-to-use wizard-based connectivity to allow you to connect to any one of those, quickly and easily.”