WNS Analytics Wizard 2019: Top 3 Winners’ Solutions from our Biggest Data Science Hackathon

9 min read

Overview

• Here’s a unique data science challenge we don’t come across often – a marketing analytics hackathon!
• We bring you the top 3 inspiring winners’ approaches and code from the WNS Analytics Wizard 2019 hackathon

Introduction

Hackathons have shaped my data science career in a huge way. They helped me understand the importance of structured thinking and how to use it when working with tight deadlines. This idea is actually the essence that drives the role of a successful data scientist.

I get a lot of questions from aspiring data science professionals wondering how to stand out from the competition and land a role in this field. This is a multi-layered question but one of the common elements I always point out – start participating in hackathons and gauge where you stand.

👁 marketing_analytics_hackathon

And if you can climb up the leaderboard, even better!

In this article, I am excited to share the top three winning approaches (and code!) from the WNS Analytics Wizard 2019 hackathon. This was Analytics Vidhya’s biggest hackathon yet and there is a LOT to learn from these winners’ solutions.

So bring out a pen and paper, take notes and don’t miss out on any other hackathons! Head straight to the DataHack platform and enroll in the upcoming competitions today.

About the WNS Analytics Wizard 2019 Hackathon

The WNS Analytics Wizard 2019 was the biggest hackathon ever hosted by Analytics Vidhya. Here’s a summary of the numbers behind this historic hackathon:

• Total Registrations: 6,456
• Total Submissions: 11,564
• Prize Money: Worth INR 4 Lacs (and interview opportunities with WNS)

It was a memorable 9-day hackathon with a wide range of data scientists participating from all over the globe.

👁 WNS Analytics Wizard 2019

Problem Statement for the WNS Analytics Wizard 2019 Hackathon

Let’s check out the problem statement for this hackathon.

Zbay is an e-commerce website that sells a variety of products on its online platform. Zbay records the user behavior of its customers and stores it in log form. However, most of the time users do not buy the products instantly and there is a time gap during which the customer might surf the internet and perhaps visit competitor websites.

Now, to improve the sales of products, Zbay has hired Adiza, an Adtech company that built a system where advertisements are shown for Zbay’s products on its partner websites.

If a user comes to Zbay’s website and searches for a product, and then visits these partner websites or apps, his/her previously viewed items or their similar items are shown as an advertisement (ad). If the user clicks this ad, he/she will be redirected to Zbay’s website and might buy the product.

👁 marketing analytics hackathon

In this problem, the task is to predict click probability, i.e., the probability of a user clicking the ad which is shown to them on the partner websites for the next 7 days on the basis of historical view log data, ad impression data and user data.

The participants were provided with:

• View log of users (2018/10/15 – 2018/12/11)
• Product description collected from the Zbay website
• Training data and test data containing details for ad impressions at the partner websites (Train + Test)

The training data contains the impression logs during 2018/11/15 – 2018/12/13 along with the label which specifies whether the ad is clicked or not. The final model was evaluated on the test data which has impression logs during 2018/12/12 – 2018/12/18 without the labels.

Late Submission – Get Your Rank Now !!

The contest is over but you can still make submissions to this contest and get your rank. Download the dataset from here and see if you can beat the top score.

Dataset Description

The participants were provided with the below files:

• train.zip: This contained 3 files and I have provided a description of each file below
• test.csv: The test file contained the impressions for which the participants needed to predict the click rate

train.csv:

Variable	Definition
impression_id	AD impression id
impression_time	Time of the impression at partner website
user_id	user id
app_code	Application Code for a partner website where the ad was shown
os_version	Version of operating system
is_4G	1-Using 4G, 0-No 4G
is_click	(target) Whether user clicked the AD (1-click, 0-no click)

view_log.csv:

Variable	Definition
server_time	Timestamp of the log
device_type	Device type of the user
session_id	Browser session id
user_id	user id
item_id	Item id

item_data.csv:

Variable	Definition
item_id	Item id
item_price	Price of the item
category_1	Category depth 1
category_2	Category depth 2
category_3	Category depth 3
product_type	anonymized item type

Winners of the WNS Analytics Wizard 2019 Hackathon

Winning a hackathon is a remarkably challenging task. There are a lot of obstacles to overcome, not to mention the sheer amount of competition from the top data scientists in the world.

I loved going through these top solutions and approaches provided by our winners. First, let’s look at who won and congratulate them:

• Rank 1: Roman Pyankov
• Rank 2: Sergey Arefiev
• Rank 3: Team AK (predictt.ai)

You can check out the final rankings of all the participants on the Leaderboard.

The top three winners have shared their detailed approach from the competition. I am sure you are eager to know their secrets so let’s begin.

Rank 3: Team AK (predictt.ai) (Aakash Kerawat & Akshay Karangale)

Here’s what Team AK shared with us.

Approach

Our final solution was an ensemble of LightGBM, Neural Networks and CatBoost models.

Taking a cursory look at the datasets, it seemed that the features generated from view_logs would play a significant role in improving the score. But soon we discovered that most of the features generated from view_logs were overfitting the training set.

This was because a higher percentage of training data had recent view_logs than in the test set. So, we focused more on feature engineering on the training set.

The features that worked for us were percentage_clicks_tilldate per user/app_code/user_app_code. These features alone helped us reach 0.73xx on the public leaderboard. Along with these, we used some time-based features and a couple of features from view_logs.

We both had slightly different approaches giving similar public leaderboard scores of around 0.75xx.

Apart from these common features, we used a few different techniques/features in our individual approaches giving scores of around 0.75xx. Our final submission was ranking the average of our best individual models.

Aakash’s approach:

• My most powerful features were target-based, i.e, mean of previous clicks for user_id, for app_code and for user_id/app_code combined. A weighted average of the above also proved to be important (giving higher weights to more recent records). Other features I created were:
  • time_since_prev_ad
  • number_of_impressions (for user)
  • sum_of_prev_clicks
  • impression_time_hour, etc.
• I engineered features for view_logs separately and then joined them based on the most recent server_time. However,  only one feature – number of views – was helpful while others were overfitting
• A strong validation strategy was important for this problem as there was an intersection in the train and view_logs data. I designed the validation set replicating the test set (no intersection between the validation set and view_logs)
• Finally, I trained 20 LightGBM models and 20 fully-connected neural networks (all models having a random state as the only difference), took the arithmetic mean for the LightGBM models and the neural network models. And finally, I took the harmonic mean of the two outputs. This gave a public leaderboard score of 0.754x

Akshay’s approach

My validation strategy was a simple time-based split:

• Since view_logs features were overfitting on the training set, I created an extra dataset from view_logs dataset: train_view_logs (a subset of the original data without overlap on train dataset)
• I used train_view_logs for creating features for the training dataset and used original view_logs for creating features for the test dataset
• The app_code feature was very important so I decided to encode it in a way so as to capture the hidden relationships between them. I used a word2vec model to convert app_codes to 100-dimensional vectors and used them instead of the original app_codes variable
• Apart from these, I created features like:
  • time_since_prev_impression
  • time_to_next_impression
  • last_impression_app_code
  • peak_to_peak_server_time (per user), etc.
• Using these features, I trained 15 LightGBM models with different seeds and a single CatBoost model. I took a simple average of these models to reach 0.754xx on the public leaderboard.

You can check out the full code for this approach here.

Rank 2: Arefev Sergei

Here’s what Arefev shared with us.

Overall Approach

1. 1. Using gradient boosting as the main algorithm
   2. Stratified cross-validation
   3. User standard for click prediction problems features (value counts and mean encoding by ids, the difference between click times, group by id, and length of unique items)
   4. Feature selection and fine-tuning parameters of the model

Let’s look at Arafev’s step-by-step approach now.

Data-preprocessing and Feature Engineering

• Standard features from train.csv:
  • 1. ‘os_version’
    2. ‘is_4G’
  • ‘value_counts_app_code’: Frequency encoding for app code feature
  • ‘mean_target_user_id’: mean value of target for every user_id in the past
• User_id time features:
  • Time from last and from next impression by user_id
  • Minimum time between impressions by user_Ids
• User_id numbers features:
  • Number of unique app_code for every user_id
  • Difference between the number of unique app_code and value_counts for every user_id
• Number of unique items frow view_log.csv which the user was looking for time over a week ago ( impression_time – 7 days)
• Number of user impressions in the past and value_counts by user_ids in view_log
• Group by app_code: For every app_code, calculate the mean value by these features:
  1. ‘mean_target_user_id’
  2. ‘value_counts_user_id’

• Time from next impression by user_id

My final model

• • My final model is a mean rank (sort predictions from one model and give them rank from 0 to the length of the test dataset) of predictions from 5 models. These are trained on different train sets by stratified validation
  • Changing validation from time series to stratified gave me a ~0.01 jump in the score. Also, I dropped features like app_code and user_id, although they gave a better score on validation
  • Now, I had about 50 features. I calculated the feature importance of gradient boosting for the full set of features, sorted them from high to low, and started dropping features one by one. In the end, I got the final subset of the features described above

Key Takeaways

• 1. Use stratified cross-validation if the distribution of the target variable does not change in time
  2. Use time and number statistics by ids
  3. Group by main ids
  4. Drop features if you think they will overfit your model

Things a participant must focus on while solving such problems

1. 1. Choose the right validation
   2. Start with a small set of features
   3. Start Using Gradient Boosting
   4. Do feature selection

Here’s the full code for Arefev’s approach.

Rank 1: Roman Pyankov

Here’s what Roman shared with us.

Approach

And here is the full code for Roman’s winning solution!

End Notes

Phew – take a deep breath. Those were some mind-blowing frameworks that won this hackathon. As I mentioned earlier, it is quite a task winning a hackathon and our three winners really stood out with their thought process.

I encourage you to head over to the DataHack platform TODAY and participate in the ongoing and upcoming hackathons. It will be an invaluable learning experience (not to mention a good addition to your budding resume!).

Login to continue reading and enjoy expert-curated content.

Free Courses

Become an Author

Share insights, grow your voice, and inspire the data community.

• Reach a Global Audience
• Share Your Expertise with the World
• Build Your Brand & Audience

• Join a Thriving AI Community
• Level Up Your AI Game
• Expand Your Influence in Genrative AI

👁 imag