S&P 500 Companies Data Analysis Tutorial using R

R is a powerful programming language and environment for statistical computation and data analysis. It is backed by data scientists, accountants, and educators because of its various features and capabilities. This project will use R to search and analyze stock market data for S&P 500 companies.

Tidyverse, ggplot2, and dplyr are just a few of the many libraries provided by R Programming Language that simplify data processing, visualization, and statistical modeling. These libraries allow us to perform many tasks such as data cleaning, filtering, aggregation, and visualization.

In this work, we will analyze the S&P 500 stock market dataset using these packages using R capabilities.

Hey! Hey! Hey! Welcome, adventurous data enthusiasts! Grab your virtual backpacks, put on your data detective hats, Ready to unravel this mysterious project journey with me.

• Dataset introduction - All files contain the following column.
• Date - In format: yy-mm-dd.
• Open - Price of the stock at the market open (this is NYSE data so everything is in USD).
• High - The highest value achieved for the day.
• Low Close - The lowest price achieved on the day.
• Volume - The number of transactions.
• Name - The stock's ticker name.

Installing and Loading the Tidyverse Package

Our adventure begins with the mysterious meta-package called “tidyverse.” With a simple incantation, “library(tidyverse)” we unlock the powerful tools and unleash the magic of data manipulation and visualization.: 

Listing files in the “../input” directory.

As we explore further, we stumble upon a mystical directory known as “../input/”. With a flick of our code wand, we unleash its secrets and reveal a list of hidden files.

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.2 ✔ readr 2.1.4
✔ forcats 1.0.0 ✔ stringr 1.5.0
✔ ggplot2 3.4.2 ✔ tibble 3.2.1
✔ lubridate 1.9.2 ✔ tidyr 1.3.0
✔ purrr 1.0.1 
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<https://conflicted.r-lib.org/>) to force all conflicts to become errors

We stumbled upon a precious artifact—a CSV file named “all_stocks_5yr.csv“. With a wave of our wand and the invocation of “read.csv()“, we summon the data into our realm.

Data Exploration & Analysis

Our adventure begins with a simple task—discovering what lies within our dataset. We load the data and take a sneak peek. With the “head()” function, we unveil the first 10 rows. We revel in the joy of exploring the dimensions of our dataset using “dim()“.

Output:

👁 Screenshot-from-2023-06-16-13-53-10.png

Output:

619040 * 7

Missing values

We’ve stumbled upon missing values! But fret not, we shall fill these gaps and restore balance to our dataset.

Output:

date 0 open 11 high 8 low 8 close 0 volume 0 Name 0

You have summoned the names() function to reveal the secret names of the columns in your dataset. By capturing the output in the variable variable_names and invoking the print() spell, you have successfully unveiled the hidden names of the columns, allowing you to comprehend the structure of your dataset.

Output:

"date" "open" "high" "low" "close" "volume" "Name" 

Next, you cast the str() spell upon your dataset. This spell reveals the data types of each variable (column) granting you insight into their mystical nature. By deciphering the output of this spell, you can understand the types of variables present in your dataset

Output:

👁 Screenshot-from-2023-06-16-13-45-40.png

Handling missing values

We fearlessly remove the voids using na.omit(), rescuing columns from emptiness. With a triumphant flourish, we check for any stragglers using colSums(is.na(data1)). Victory!

Output:

date 0 open 0 high 0 low 0 close 0 volume 0 Name 0

Checking the dimension after removing missing values warriors.

Output:

619029 * 7

The summary reflects the essence of our data, focusing on minimum and maximum values, mean and median, and first and third quartile.

Output:

👁 Screenshot-from-2023-06-16-13-38-11.png

Data Visualization

"Histogram of Opening Prices" displays the frequency distribution of opening prices. The x-axis represents the opening price values, and the y-axis represents the count. The second Histogram "Histogram of Closing Prices" visualizes the distribution of closing prices. I x-axis represents the closing price values and the y -axis represents the count.

Output:

👁 Screenshot-from-2023-06-16-13-34-42.png

The scatter plot is named "High vs. Low Prices." It compares the high and low prices for each data point. The x-axis represents the low prices, and the y-axis represents the high prices.

Output:

👁 pro14.png

Data analysis

Telling the length or we can say the total number of unique companies in the dataset.

Output:

505

Telling the lowest closing price in the dataset.

Output:

1.59

Telling the highest closing price in the dataset.

Output:

2049

Giving the result of the company name with the highest shared volume.

Output:

'VZ'

Giving the result of the company name with the minimum shared volume.

Output:

'BHF'

We calculate the average closing price by taking the mean of the close column in the dataset data1

Output:

83.0433.49787651

We determine the total trading volume by summing up the values in the volume column of the dataset data1.

Output:

2675376439690

We find the company with the longest consecutive days of price increases by identifying the corresponding "Name" value where the condition for consecutive price increases is met.

Output:

'AAL'

We find the company with the longest consecutive days of price decreases by identifying the corresponding "Name" value where the condition for consecutive price decreases is met.

Output:

'AAL'

Feature Engineering

Feature Engineering helps to derive some valuable features from the existing ones. We convert the "date" column to a character format then split the date using "/" as the delimiter then create a new data frame with columns for date, day, month, and year.

Output:

👁 pro5.png

Now in this code, we are filling the column day, month, and year values from the date column.

Output:

👁 pro4.png

Now, we are including day, month, and year columns in the dataset data1.

Output:

👁 pro3.png

We create a new column "is_quarter_end" in data1 to see whether a particular date falls at the end of a quarter. Using the modulo operator (%%), we are checking if the month value is divisible by 3. If it is, we assign 1 to indicate it's a quarter-end otherwise, we assign 0.

Output:

👁 pro2.png

First, we are taking the numeric columns (open, high, low, close) and the date column from "data1" to create a new data frame called "data_num", then we calculate the year from the "date" column using the "lubridate::year" function and add it as a new column in "data_num" then group the data by year and calculate the mean for each numeric column. Create a bar plot for each numeric column, displaying the mean values for each year.

Output:

👁 pro1.png

And that’s a wrap! But remember, this is just the beginning of our data adventure.