 |
VOOZH | about |
|
VOOZH | about |
In Data Science and Machine Learning, Data Cleaning plays an essential role. Data Cleaning is the process of retaining only the crucial information from the output so that only relevant features are sent as input to the machine learning model. It is a very crucial step in data science and it helps in getting better results as all the noise is already removed.
But, have you wondered that such a process can be time taking and coding a pipeline for every new dataset can be quite tiresome? Hence, it is a good idea to automate the whole process by creating a set pipeline for the process and then using it on every piece of data that one needs to clean. Such a pipeline can make the work easier as well as more refined. One will not have to worry about missing a step and all that is needed is to use that same pipeline again.
In this article, we will be working on creating a complete pipeline using multiple libraries, modules, and functions in Python to clean a CSV file.
To understand the process of automating data cleaning by creating a pipeline in Python, we should start by understanding the whole point of data cleaning in a machine-learning task. The user information or any raw data contained a lot of noise (unwanted parts) in it. Such data sent to the model directly can lead to a lot of confusion and henceforth can lead to unsatisfactory results. Hence, removing all the unwanted and unnecessary data from the original data can help the model perform better. This is the reason that data cleaning is an essential step in most of the Machine Learning tasks.
In reference to data cleaning, automating the process, essentially means creating a set of rules (function in terms of code) that align and organize the whole process of data cleaning and make it easier for us to run the same pipeline as per the requirement on different sets of data.
Automating data cleaning in Python means creating a set of rules(function in terms of code) that align and organize the whole process of data cleaning.
The data cleaning process can be done using various libraries and the following are some most popular ones:
Regular Expressions are simple sequences of characters that one matches in a given text. Furthermore, these identified sequences can be removed or extracted from the text using a simple Python module called Regex imported in the code as re.
Here's a sample code for using regular expressions in Python:
Output:
Read the data from httpswwwgfgorg posted by gfggmailcom Emails found: ['gfg@gmail.com']
Here are some examples of the data cleaning tasks that can be done using regular expressions:
Pandas is an extremely popular, well-known, and one of the libraries used in almost all Machine Learning tasks. Pandas are essentially a package of Python used to deal majorly with data frames and manipulate them as per the need. While working on the data cleaning process of a data frame pandas can prove to be a very helpful library.
Below is a sample code for using Pandas in Python for data cleaning:
Here are some things you can do with Pandas to automate the data-cleaning process:
| Task | Function Used | Description |
|---|---|---|
| Remove duplicates | drop_duplicates() | Remove duplicate rows from a dataframe. |
| Drop missing values | dropna() | Remove rows or columns with missing values. |
| Impute missing values | fillna() | Fill in missing values in a dataframe with a specified value or method. |
| Convert data types | astype() | Convert the data type of a column in a dataframe. |
| Rename columns | rename() | Rename columns in a dataframe. |
| Group and aggregate data | groupby(), agg(), apply() | Group and aggregate data in a dataframe. |
| Filter data | query(), loc[], iloc[] | Filter data in a dataframe using various methods |
| Apply functions to data | apply() | Apply a function to a column or row in a dataframe |
| Merge data | merge(), join(), concat() | Merge data from multiple dataframes |
| Pivot data | pivot_table() | The method allows for more advanced features such as multi-index and custom aggregation. |
By using these functions and methods, you can create a powerful data-cleaning pipeline in Pandas to automate the data-cleaning process.
NumPy is another popular library in Python for numerical computing. As its name suggests, it stands for Numerical Python. It provides a powerful array data structure that can be used for efficient data processing and analysis. NumPy has several functions for cleaning data, such as filtering, sorting, and aggregating data.
Here is an example code for using NumPy to filter and sort data:
Output:
[10 9 8 7 6]
Here are some things you can do with Pandas to automate the data-cleaning process:
| Task | Function Used | Description |
|---|---|---|
| Replace missing values | numpy.nan_to_num() | Replaces NaN values with zeros or a specified value |
| Identify missing values | numpy.isnan() | Returns a boolean array indicating where NaN values are found |
| Replace outliers | numpy.clip() | Clips values to a specified range to remove outliers |
| Normalize data | numpy.linalg.norm() | Computes the norm (magnitude) of a vector or matrix |
| Standardize data | numpy.std(), numpy.mean() | Computes the standard deviation and mean of a dataset |
| Scale data | numpy.interp() | Scales a dataset to a specified range |
| Remove duplicate values | numpy.unique() | Removes duplicate values from an array |
| Filter data based on a condition | numpy.where() | Returns values from one array if a condition in another array is met |
| Split data into chunks | numpy.array_split() | Splits an array into equally-sized subarrays |
Now that we have discussed some of the popular libraries for automating data cleaning in Python, let's dive into some of the techniques for using these libraries to clean data. Following is a structure of a basic data-cleaning pipeline that covers the most essential steps:
The above steps include some of the significant and key ones, but, as per the requirement, one can add or remove functions and clean the data using the updated pipeline.
The final pipeline can be implemented as follows using Python. Here the implemented code is tested on a custom-generated dataset as well to see the effect of the data-cleaning process.
Output:
Before Preprocessing: Name Age Gender Income Birthdate Married Children 0 John 30.0 Male 50000 01/01/1990 True 2 1 Jane 25.0 Female 60000 02/14/1996 False 0 2 Bob 40.0 Male 70000 03/15/1981 True 1 3 John 30.0 Male 45000 01/01/1990 False 0 4 Alice NaN Female 80000 06/30/1986 True 3 Removed 1 missing values After preprocessing Name Age Gender Income Birthdate Married Children 0 John 30.0 1 50000 1990-01-01 True 2 1 Jane 25.0 0 60000 1996-02-14 False 0 2 Bob 40.0 1 70000 1981-03-15 True 1
In the above code, a random dataset is created and then it is saved into a CSV file to pass it through the data cleaning pipeline. You can easily use the same pipeline by simply passing a different file path in the "data_cleaning_pipeline functions".
Following is a snippet of the data before the cleaning process:
Output:
Following is a snippet of the data after the cleaning process:
Output:
Automating the data cleaning process, includes, systematically creating all the required functions and using them to solve the purpose as per the need. Some of the key ones and their code is explained in this article and used in the pipeline.
Finally concluding, using a detailed pipeline and standardizing the data-cleaning process can help save time and increase the efficiency of the process. The standard functions can be used as per the requirement once a pipeline has been created leaving a lesser chance for an error.
