How to use Is Not Null in PySpark

In data processing, handling null values is a crucial task to ensure the accuracy and reliability of the analysis. PySpark, the Python API for Apache Spark, provides powerful methods to handle null values efficiently. In this article, we will go through how to use the isNotNull method in PySpark to filter out null values from the data.

The isNotNull Method in PySpark

The isNotNull method in PySpark is used to filter rows in a DataFrame based on whether the values in a specified column are not null. This method is particularly useful when dealing with large datasets where null values can impact the accuracy of your results. This method returns a Column type consisting of Boolean values, which are True for non-null values and False for null values. By using isNotNull, you can ensure that only rows with valid data are included in your analysis.

Syntax:

DataFrame.filter(Column.isNotNull())

Simple Example to Implement isNotNull Method in Pyspark

To use the isNotNull the method in PySpark, you typically apply it to a DataFrame column and then use the filter function to retain only the rows that satisfy the condition.

In this example, we are taking a DataFrame with some null values. Then we use the isNotNull method to filter out any rows where the column 'data' contains null.

Output:

+-----+---+
| Name|Age|
+-----+---+
| Anna| 30|
| Julia| 25|
+-----+---+

Another Example to Implement isNotNull Method

Step 1: Initialize Spark Session

First, you need to initialize a Spark session. This is the entry point for using Spark functionality.

Step 2: Create a Sample DataFrame

Next, create a sample DataFrame that contains some null values.

Step 3: Use isNotNull to Filter Data

Now, use the isNotNull method to filter out rows where specific columns have null values. For example, let's filter out rows where the name column is null.

Step 4: Filter Multiple Columns

You can also filter out rows where multiple columns are not null by combining conditions with the & operator.

Complete Code

Here is the complete code combining all the steps:

Output

Original DataFrame:
+----+-------+----+
| id| name| age|
+----+-------+----+
| 1| Alice| 30|
| 2| NULL| 25|
| 3| Bob|NULL|
|NULL|Charlie| 35|
+----+-------+----+

Filtered DataFrame (name is not null):
+----+-------+----+
| id| name| age|
+----+-------+----+
| 1| Alice| 30|
| 3| Bob|NULL|
|NULL|Charlie| 35|
+----+-------+----+

Filtered DataFrame (name and age are not null):
+----+-------+---+
| id| name|age|
+----+-------+---+
| 1| Alice| 30|
|NULL|Charlie| 35|
+----+-------+---+

Q: Can isNotNull be used with multiple columns?

Yes, you can chain multiple isNotNull checks across different columns using logical operators like & (and).

Q: What happens if I use isNotNull on a DataFrame with no null values?

If there are no null values in the column, isNotNull will return the original DataFrame.

Q: Is isNotNull the only way to check for non-null values?

No, PySpark also offers the na.drop() function, which can be used to drop rows based on null values across multiple columns.