Custom Logs in Apache Spark

Have you ever felt the frustration of Spark job that runs for hours and it fails due to infra issue.…

Read More »

Apache Spark RDD and Java Streams

A few months ago, I was fortunate enough to participate in a few PoCs (proof-of-concepts) that used Apache Spark. There,…

Read More »

Monitoring Real-Time Uber Data Using Spark Machine Learning, Streaming, and the Kafka API (Part 2)

This post is the second part in a series where we will build a real-time example for analysis and monitoring…

Read More »

Monitoring Real-Time Uber Data Using Spark Machine Learning, Streaming, and the Kafka API (Part 1)

According to Gartner, by 2020, a quarter of a billion connected cars will form a major element of the Internet…

Read More »

Apache Spark: A Quick Start With Python

Spark Overview As per the official website, “Apache Spark is a fast and general engine for large scale data processing”…

Read More »

Predicting Breast Cancer Using Apache Spark Machine Learning Logistic Regression

In this blog post, I’ll help you get started using Apache Spark’s spark.ml Logistic Regression for predicting cancer malignancy. Spark’s…

Read More »

How to Get Started with Spark Streaming and MapR Streams Using the Kafka API

This post will help you get started using Apache Spark Streaming for consuming and publishing messages with MapR Streams and the Kafka…

Read More »

Apache Spark Packages, from XML to JSON

The Apache Spark community has put a lot of effort into extending Spark. Recently, we wanted to transform an XML…

Read More »

Testing Spark Streaming: Unit testing

There is enough evidence to prove the importance of automated testing. Projects in new fields often neglect automated testing, as…

Read More »