Show distinct column values in PySpark dataframe

In this article, we are going to display the distinct column values from dataframe using pyspark in Python. For this, we are using distinct() and dropDuplicates() functions along with select() function.

Let's create a sample dataframe.

Output:

Method 1: Using distinct()

This function returns distinct values from column using distinct() function.

Syntax: dataframe.select("column_name").distinct().show()

Example1: For a single column.

Output:

+-----------+
|Employee ID|
+-----------+
| 3|
| 5|
| 1|
| 4|
| 2|
+-----------+

Example 2: For multiple columns.

Python code to display unique data from 2 columns using distinct() function.

Syntax: dataframe.select("column_name 1, column_name 2 ").distinct().show()

Code:

Output:

+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
| 5| gnanesh|
| 4| rohith|
| 1| sravan|
| 2| ojaswi|
| 3| bobby|
+-----------+-------------+

Method 2: Using dropDuplicates()

This function displays unique data in one column from dataframe using dropDuplicates() function.

Syntax: dataframe.select("column_name").dropDuplicates().show()

Example 1: For single columns.

Output:

+-----------+
|Employee ID|
+-----------+
| 3|
| 5|
| 1|
| 4|
| 2|
+-----------+

Example 2: For multiple columns

Python code to display unique data from 2 columns using dropDuplicates() function

Output:

+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
| 5| gnanesh|
| 4| rohith|
| 1| sravan|
| 2| ojaswi|
| 3| bobby|
+-----------+-------------+