Pandas: A Vital Python Tool for Data Scientists

Pandas aren’t just adorable bears munching on bamboo and rolling around, showing just how much fun being a bear can be. If you work with Python, Pandas is an open source library for data analysis and manipulation. Any data scientist or analyst working in Python will, at some point, depend on Pandas.

The name comes from the term “panel data,” which means data sets that span multiple time periods for the same individual.

Pandas include the following features:

• Series: one-dimensional labeled arrays that are capable of holding any data type.
• DataFrame: two-dimensional labeled data structure with columns that can consist of different types.
• Data cleaning: handles missing data, filtering, and transforming datasets.
• Data aggregation: groups data and calculates summary statistics.
• Time series analysis: support for time series data and allows for date and time manipulations.
• Statistical operations: performs various statistical calculations and analyses.
• Input/output: can read to and from various file formats (such as CSV, Excel, SQL, and JSON).
• Integrations: interacts with other Python libraries such as NumPy, Matplotlib, and Scikit-learn, so it’s also suitable for machine learning.

Pandas is often used for data cleaning, exploratory data analysis and data transformation. Clearly, this library is crucial to science and other fields that depend on data. With Pandas you can do things like calculate statistics, clean data, visualize data and store cleaned data.

Pandas was built on top of NumPy and works quite well within a Jupyter Notebook. To work with Pandas, you’ll want to have a solid understanding of Python first.

Let’s see how Pandas works.

Series vs. DataFrames

The two main components of Pandas are Series and DataFrames. A series is a column and a DataFrame is a multi-dimensional table. Think of a Series as a column from a spreadsheet and a DataFrame as the spreadsheet itself (Figure 1).

Installing Pandas

Before you can work with Pandas, it must be installed. Fortunately, there’s the Python package manager, Pip, to help you out with that. I’m going to demonstrate using Pandas on Ubuntu 22.04. If you have a different OS, you’ll want to make sure you have Pip installed and can use it to install packages.

To install Pandas, log into your OS, open a terminal window, and issue the command:

pip install pandas

You can then ensure it’s the latest version with the command:

pip install --upgrade pandas

Outstanding. You’re now ready to take your first steps with Pandas.

Creating a series

The first thing we’ll discover is how to create a series that transforms a NumPy array into a Pandas series.

First, we must import the two libraries we’ll be working with, using the following statements:

import pandas as pd
import numpy as np

Next, we define our array as data with the line:

data = np.array(['n','e','w', ' ', 's','t','a','c','k'])

Convert data to a series with Pandas using the following line:

ser1 = pd.Series(data)

Print the series with:

print(ser1)

The entire block of code looks like this:

import pandas as pd
import numpy as np

data = np.array(['n','e','w', ' ', 's','t','a','c','k'])

ser1 = pd.Series(data)

print(ser1)

When you run the above code, the output is:

0    n
1    e
2    w
3
4    s
5    t
6    a
7    c
8    k
dtype: object

We can also create a series that includes an index. Typically, Pandas creates default indexes that start with 0 and continue until it runs out of entries. As you can see in the output above, the index is 0 – 8. But what if we wanted an index of, say, even numbers, such that the output would be:

2     n
4     e
6     w
8
10    s
12    t
14    a
16    c
18    k
dtype: object

To do that, we must employ the index keyword like this:

ser1 = pd.Series(data, index=[2,4,6,8,10,12,14,16,18])

You could also use letters as an index, like so:

ser1 = pd.Series(data, index=['A','B','C','D','E','F', 'G','H','I'])

Creating a DataFrame

Let’s create a DataFrame. The idea is similar to that of Series, only you define two series and combine them together. We’ll create a DataFrame for servers and desktops. First, we define our series like so:

data = {
   'servers': [5, 3, 7, 9],
   'desktops': [20, 10, 15, 40]
}

Fairly straightforward, eh?

Next, we pass data to the Pandas DataFrame constructor with the following entry:

inventory = pd.DataFrame(data)

Print it out with:

print(inventory)

The entire block of code looks like this:

import pandas as pd

data = {
   'servers': [5, 3, 7, 9],
   'desktops': [20, 10, 15, 40]
}

inventory = pd.DataFrame(data)

print(inventory)

Run that block of code and the output will be:

servers  desktop
0        5       20
1        3       10
2        7       15
3        9       40

We can change our index as well. Say you want to indicate it by department. That might look like this:

inventory = pd.DataFrame(data, index=['HR','ACCT','MGMT','EMP'])

The output would now look like this:

servers desktops
HR 5       20
ACCT 3       10
MGMT 7       15
EMP 9       40

Pretty cool, huh?

We could also list only the purchases of desktops and servers by the HR department, with the line:

print(inventory.loc['HR'])

The output from that would be:

servers     5
desktop    20
Name: HR, dtype: int64

Reading Data From a File

Considering that we’re talking about data manipulation, hard-coding data might not be the best way to go. Fortunately, Pandas makes it possible to read data in from a file. Say, for example, you have a large CSV file. Let’s say our CSV file is named prices.csv and looks like this:

Date,”price”,”factor_1″,”factor_2″
2024-08-11,1600.20,1.255,1.548
2024-08-12,1610.02,1.258,1.554
2024-08-13,1618.07,1.249,1.552
2024-08-14,1624.40,1.253,1.556
2024-08-15,1626.15,1.258,1.552
2024-08-16,1626.15,1.263,1.558
2024-08-17,1626.15,1.264,1.572

To read that into a Python script with Pandas, we’d use something like this:

import pandas as pd

df = pd.read_csv("prices.csv")
print(df)

The output would be:

         Date    price  factor_1  factor_2
0  2024-08-11  1600.20     1.255     1.548
1  2024-08-12  1610.02     1.258     1.554
2  2024-08-13  1618.07     1.249     1.552
3  2024-08-14  1624.40     1.253     1.556
4  2024-08-15  1626.15     1.258     1.552
5  2024-08-16  1626.15     1.263     1.558
6  2024-08-17  1626.15     1.264     1.572

And there you have it. You’ve taken your first steps with Pandas. This library is fairly complex, so you’ll want to make sure to read the official Pandas documentation to get the most out of it.