Is Apache Spark Too Costly? An Amazon Engineer Tells His Story

RALEIGH, N.C. — Is Apache Spark too costly?

On LinkedIn, Amazon Principal Engineer Patrick Ames is recommended as the “go-to” guy for explaining how things work. You get that kind of reputation when you manage exabyte-scale data transfers from Apache Spark to Ray, a unified framework for scaling AI and Python applications.

Ray, by the way, is the new hotness. According to its GitHub page, open source Ray consists of a core distributed runtime and a set of AI libraries for simplifying machine learning compute.

We talk to a lot of engineers. Sometimes, it’s insightful to learn a bit about who they are.

“I guess I’ve alway approached things with a certain goal in mind,” Ames said in an interview at All Things Open for The New Stack Makers. “I’ve been a person who’s always set out on engineering projects or other pursuits of curiosity with some ends in mind. So, here at Amazon, and in other projects in life, it’s usually just me learning what I need to do to get past some problem that I’m having, or to make something easier in life for myself.

“I’d say I’ve taken a very kind of goal-oriented approach to engineering over the years, where I’m looking at some problem, looking at something that is difficult to do, and trying to find out ways of making it less difficult or easier to approach that sort of problem. So, a goal-driven pursuit of curiosity throughout my life.”

Does that mean Ames is the around-the-house project guy who makes the house more user-friendly? Why, yes.

“You’ve got daily chores, you got daily things you want to do,” he said. “How do you make that easier? How do you make it more painless? How do you make it more fun?”

The Tradeoffs of Spark

Software engineering reduces the act of repeatedly thinking through the same problems by codifying solutions, Ames said. Then, it comes down to revising that approach to the solution and making it more optimal over time.

That’s comparable to large-scale data projects such as moving from Spark to Ray, the talked-about technology that has replaced Spark at Amazon.

But why use Spark in the first place? First of all, it was a popular open source technology that was simple to stand. It merges inserts, updates and deletes with a few simple lines of Spark SQL.

The team at Amazon gave different tools a go, such as AWS Redshift, Apache Hive, and Flink. Sprak entered the picture to cache the results of merging data. Architects worked with Spark, designing it for a decoupled model over data stored in Amazon Simple Storage Service (S3). It wasn’t a coupled model like historic data warehouse databases of storage and compute.

But some tradeoffs led Ames and his team to try Ray.

Spark worked well. The Amazon team went through some growing pains. Using Spark required implementing a serverless job management infrastructure, managing thousands of Spark jobs daily, which means thousands of different Spark clusters daily. But business issues started to surface. Scaling problems popped up.

Recalled Ames, “Once individual partitions of inserts, updates and deletes or unpartitioned tables of these same change data capture logs grew to, say, hundreds of terabytes or petabyte scale,  we were again encountering this familiar scaling problem of, OK, Spark jobs are taking an onerously long time to complete, data consumers are complaining that it’s taking too long to receive their data insights that they need to run their business, and it’s also costing us too much money just to run these gargantuan Spark clusters.”

And so attention eventually turned to Ray. The Amazon team experimented at first and soon found a substantial difference in efficiency.

What was that efficiency? We’ll leave that to you, dear listener, to hear from Ames in this edition of The New Stack Makers.