Scalable Machine Learning on Big Data using Apache Spark
Keep adding new skills with 10,000+ programs for $239 (usually $399). Save now.
Scalable Machine Learning on Big Data using Apache Spark
Instructor: Romeo Kienzler
23,947 already enrolled
Included with
Ask Coursera
1,252 reviews
1,252 reviews
Skills you'll gain
Tools you'll learn
Details to know
See how employees at top companies are mastering in-demand skills
Build your Machine Learning expertise
- Learn new concepts from industry experts
- Gain a foundational understanding of a subject or tool
- Develop job-relevant skills with hands-on projects
- Earn a shareable career certificate from IBM
There are 4 modules in this course
This course will empower you with the skills to scale data science and machine learning (ML) tasks on Big Data sets using Apache Spark. Most real world machine learning work involves very large data sets that go beyond the CPU, memory and storage limitations of a single computer.
Apache Spark is an open source framework that leverages cluster computing and distributed storage to process extremely large data sets in an efficient and cost effective manner. Therefore an applied knowledge of working with Apache Spark is a great asset and potential differentiator for a Machine Learning engineer. After completing this course, you will be able to: - gain a practical understanding of Apache Spark, and apply it to solve machine learning problems involving both small and big data - understand how parallel code is written, capable of running on thousands of CPUs. - make use of large scale compute clusters to apply machine learning algorithms on Petabytes of data using Apache SparkML Pipelines. - eliminate out-of-memory errors generated by traditional machine learning frameworks when data doesn’t fit in a computer's main memory - test thousands of different ML models in parallel to find the best performing one – a technique used by many successful Kagglers - (Optional) run SQL statements on very large data sets using Apache SparkSQL and the Apache Spark DataFrame API. Enrol now to learn the machine learning techniques for working with Big Data that have been successfully applied by companies like Alibaba, Apple, Amazon, Baidu, eBay, IBM, NASA, Samsung, SAP, TripAdvisor, Yahoo!, Zalando and many others. NOTE: You will practice running machine learning tasks hands-on on an Apache Spark cluster provided by IBM at no charge during the course which you can continue to use afterwards. Prerequisites: - basic python programming - basic machine learning (optional introduction videos are provided in this course as well) - basic SQL skills for optional content The following courses are recommended before taking this class (unless you already have the skills) https://www.coursera.org/learn/python-for-applied-data-science or similar https://www.coursera.org/learn/machine-learning-with-python or similar https://www.coursera.org/learn/sql-data-science for optional lectures
This is an introduction to Apache Spark. You'll learn how Apache Spark internally works and how to use it for data processing. RDD, the low level API is introduced in conjunction with parallel programming / functional programming. Then, different types of data storage solutions are contrasted. Finally, Apache Spark SQL and the optimizer Tungsten and Catalyst are explained.
What's included
6 videos6 readings2 assignments
6 videos•Total 44 minutes
- Introduction to Apache Spark for Machine Learning on BigData•7 minutes
- What is Big Data?•12 minutes
- Data storage solutions•6 minutes
- Parallel data processing strategies of Apache Spark•7 minutes
- Functional programming basics•7 minutes
- Resilient Distributed Dataset and DataFrames - ApacheSparkSQL•6 minutes
6 readings•Total 60 minutes
- Course Syllabus•10 minutes
- Setup of the grading and exercise environment•10 minutes
- Exercise 1 - working with RDD•10 minutes
- Exercise 2 - functional programming basics with RDDs•10 minutes
- Exercise 3 - working with DataFrames•10 minutes
- Programming Lanuage Options for Apache Spark (optional)•10 minutes
2 assignments•Total 30 minutes
- Practice Quiz (Ungraded) - Apache Spark concepts•30 minutes
- Apache Spark and parallel data processing•0 minutes
Applying basic statistical calculations using the Apache Spark RDD API in order to experience how parallelization in Apache Spark works
What's included
8 videos3 readings4 assignments
8 videos•Total 52 minutes
- Averages•6 minutes
- Standard deviation•4 minutes
- Skewness•4 minutes
- Kurtosis•2 minutes
- Covariance, Covariance matrices, correlation•13 minutes
- Plotting with ApacheSpark and python's matplotlib•13 minutes
- Dimensionality reduction•5 minutes
- PCA•6 minutes
3 readings•Total 30 minutes
- Exercise 1 - statistics and transfomrations using DataFrames•10 minutes
- Exercise on Plotting•10 minutes
- Exercise on PCA•10 minutes
4 assignments•Total 30 minutes
- Practice Quiz (Ungraded) - Statistics and API usage on Spark•30 minutes
- Questions on Plotting•0 minutes
- Questions on PCA•0 minutes
- Parallelism in Apache Spark •0 minutes
Understand the concept of machine learning pipelines in order to understand how Apache SparkML works programmatically
What's included
5 videos2 readings3 assignments
5 videos•Total 34 minutes
- How ML Pipelines work•4 minutes
- Introduction to SparkML•21 minutes
- Extract - Transform - Load•4 minutes
- Introduction to Clustering: k-Means•4 minutes
- Using K-Means in Apache SparkML•2 minutes
2 readings•Total 20 minutes
- Exercise 1: Modifying a Apache SparkML Feature Engineering Pipeline•10 minutes
- Exercise 2 - Working with Clustering and Apache SparkML•10 minutes
3 assignments•Total 30 minutes
- Practice Quiz (Ungraded) - ML Pipelines•30 minutes
- Practice Quiz (Ungraded) - SparkML Algorithms•0 minutes
- SparkML concepts •0 minutes
Apply Supervised and Unsupervised Machine Learning tasks using SparkML
What's included
4 videos2 readings2 assignments
4 videos•Total 18 minutes
- Linear Regression•5 minutes
- LinearRegression with Apache SparkML•7 minutes
- Logistic Regression•2 minutes
- LogisticRegression with Apache SparkML•5 minutes
2 readings•Total 20 minutes
- Exercise 1 - Improving Classification performance•10 minutes
- Course Project•10 minutes
2 assignments•Total 30 minutes
- Practice Quiz (Ungraded) - SparkML Algorithms (2)•30 minutes
- Course Project Quiz•0 minutes
Earn a career certificate
Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.
Instructor
Why people choose Coursera for their career
Learner reviews
- 5 stars
41.26%
- 4 stars
24.66%
- 3 stars
17.95%
- 2 stars
7.34%
- 1 star
8.77%
Showing 3 of 1252
Reviewed on May 19, 2020
Great tutorial overall.Room for improvement: Fix the differences int the definition of kurtosis and skew between vide, test, examples (preferable the scipy definition).
Reviewed on Feb 25, 2020
After completing this course you will be able to use Apache Spark to build ML models (e.g., Linear Regression, Gaussian Mixture Model, etc.).
Reviewed on Feb 22, 2020
for the last assignment we should have got the opportunity to code in the notebook instead of just running it and reporting results.
Frequently asked questions
To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.
More questions
Financial aid available,