MongoDB’s New Tool to Migrate Data from Relational Systems

MongoDB really wants to move your data off of relational databases and onto its document store.

To make it really easy, the company released Relational Migrator, one of many new products launched at MongoDB World 2022 last week.

MongoDB’s new Relational Migrator is a tool that transforms relational schemas into non-relational schemas and allows the user to analyze their original schemas, make decisions on how their schemas will be represented as a MongoDB schema, and migrate the data from their relational database to MongoDB.

“There are other tools that move data from one place to another. Relational Migrator is unique in that the problem of changing a relational schema into a document schema is not one that anyone else is focused on,” said MongoDB Relational Migrator’s Product Manager Tom Hollander, in an interview with The New Stack.

Relational Migrator digs deep into the philosophical differences between relational and non-relational databases and bridging the gap between those two worlds.

Why This and Why Now?

“Clients tend to choose MongoDB for new applications but those same clients also have a sizable data store and it almost always tends to be a relational database,” Hollander said, adding “It’s pretty easy to find a customer who’s had a taste of MongoDB and likes it but are also doing some Oracle, some SQL Server, but don’t find those as good in terms of agility or cost.”

The use cases for Relational Migrator vary greatly as Hollander explains, “Some clients want to change the smallest amount of code needed to get them off of their relational database, but more often than not effectively these applications are end of life so it’s not just the databases that need modernization.”

Because databases support one or more applications, not only does the migration need to consider the schemas and databases themselves but the larger application (applications) as a whole, which tends to make this process a lot more complicated.

In addition to the schema transformation, the tool was also created to show possibility and ease the client’s fears when considering modernization and switching from a relational database to a non-relational database. Due to the unfamiliar territory and consequences associated with any data a loss, users will forego modernization altogether.

Relational Migrator doesn’t solve everything but it does take on the challenge of “What will happen to my data?” as Hollander states. Relational Migrator shows clients that, “your data won’t get lost. Your data won’t be forced to stay in the same schema,” even though those same clients will still have to think through the other tough challenges, he said.

The Engineering Team and Challenges They Faced

While MongoDB wanted to solve the burden of converting relational databases to MongoDB for quite some time, the project really took flight after MongoDB Chief Technology Officer Mark Porter joined MongoDB in 2020.

The Relational Migrator team, based out of Sydney, Australia, started as a team of one — Tom Hollander. Initially, Hollander wasn’t sure if this would be an engineering tool, a partnership, or an acquisition. It took months of research before it was determined that the tool did not exist and would need to be built.

Team member number two, Mark Bereznitsky, the engineering lead, was brought on about 12 months ago. Though not much code was written until more team members were brought on a few months later, a lot of planning, whiteboarding, and document writing took place. The team has grown to nine people total, including a program manager, a designer, and six engineers with another engineer starting soon. Most of the engineering work has taken place over the last six months.

As for some specific technical challenges the team has faced and overcome so far, MongoDB Frontend Engineer, Lavender Chan shared that, “Some of the biggest technical challenges we’ve faced have definitely been around performance. With Relational Migrator you’re dealing with customers that can have one or many databases that they’re working with as well as the tables within those databases.”

The frontend team did put much research into deciding which technologies were best suited to represent the data visually on the screen. There was a lot of testing done to determine how many tables could be displayed on the screen before Relational Migrator became less performant.

Regarding the back end specific technical challenges, since this is data migration, these were primarily centered around preserving the data — not corruption or lost files.

Hollander explained the solutions required, “hard computer science problems, particularly when you consider that networks are unreliable, machines might go down, messages might arrive out of order, and the performance of the back end has to be acceptable.”

Overall, the team is happy with the way the technical challenges were solved and the MVP they launched but no one is under any pretense that the journey is over yet as Hollander excitedly states, “There’s a whole new wave of challenges when we start looking at continuous replication.”

For instance, continuous Replication is a new feature in the pipeline that will update the migrated database when the source database is updated. There is currently no release date on Continuous Replication at this time. Continuous Replication is just one of many new features in the pipeline for Relational Migrator.