Gesture Controlled Game in Machine Learning

Let us discuss gesture-controlled game using TensorFlow, I hope this article will be much interesting as it provides a holistic gaming experience at the end. We begin by describing the modules necessary to develop this game.

Necessary Libraries

We start our coding by importing the necessary libraries, including cv2 for capturing video from the webcam, then mediapipe for hand tracking, also numpy for numerical operations, then Tensorflow for loading the pre-trained model and pygame for creating the game window and handling game-related functionalities.

• OpenCV
• Mediapipe
• Numpy
• TensorFlow
• Pygame

pip install cv2 mediapipe numpy tensorflow pygame

Below is aby-step step-wise approach for developing gesture controlled game using TensorFlow

STEP 1:Import all the necessary libraries,

For this gesture-controlled game five libraries are required.

Output:

pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/wiki/Contribute

STEP 2: Loading the pre-trained model:

The pre-trained TensorFlow model is loaded using tf.keras.models.load_model(). This model is trained to recognize hand gestures based on hand landmarks. Add "gesture_model.h5" inside the same folder. You can download the pre-trained gesture model using this GitHub link: gesture_model.h5

Output:

Model: "sequential_1"
_________________________________________________________________
 Layer (type) Output Shape Param # 
=================================================================
 conv2d_1 (Conv2D) (None, 100, 100, 8) 136 

 max_pooling2d_1 (MaxPooling (None, 24, 24, 8) 0 
 2D) 

 conv2d_2 (Conv2D) (None, 24, 24, 16) 528 

 max_pooling2d_2 (MaxPooling (None, 12, 12, 16) 0 
 2D) 

 flatten_1 (Flatten) (None, 2304) 0 

 dense_1 (Dense) (None, 512) 1180160 

 dropout_1 (Dropout) (None, 512) 0 

 dense_2 (Dense) (None, 128) 65664 

 dropout_2 (Dropout) (None, 128) 0 

 dense_3 (Dense) (None, 10) 1290 

=================================================================
Total params: 1,247,778
Trainable params: 1,247,778
Non-trainable params: 0
_________________________________________________________________

STEP 3: Initialize MediaPipe Hands model:

The below code initializes the MediaPipe's Hands Model by creating an instance of mp_hands.Hands. This model is used for detecting hand landmarks in each frame captured from the webcam. From the gesture_labels dictionary, it takes 0 as 'Left' and 1 as 'Right'

STEP 4:Initialize Pygame, Setting up Player and Game Loop:

Next Pygame is initialized by calling pygame.init(). The game window dimensions are set, and the window caption is set to "Gesture-Based Game". The player's properties, such as width, height, initial position, and speed, are defined clearly. The player is represented as a white rectangle. The main game loop is set up using a while loop. This loop continues until the game is quit which means the player hits the obstacle.

STEP 5: Control with live video with finger

1. Capturing video from webcam: Video frames are captured from the webcam using cap.read(). The frame is flipped horizontally to create a mirror effect. The captured live gestures are recognized to perform the left and right movements of player.
2. Detecting hand landmarks and recognizing gestures: Then this code processes each frame using the MediaPipe Hands model to detect hand landmarks. It specifically tracks the index finger's position to recognize gestures. Based on the position of the index finger, the code determines whether the gesture is "Left", "Right", or "Neutral".
3. Updating the player's position: In this code,, the player's position is updated based on the recognized gesture. If the gesture is "Left," the player moves left by subtracting the player's speed from its x-coordinate. If the gesture is "Right," the player moves right by adding the player speed to its x-coordinate. If the gesture is "Neutral," the player remains stationary. Using this we can handle the player with ease.
4. Creating player and obstacle: Both the player and obstacle are generated inside pygame using pygame.draw.rect(). If both the player and obstacle hits in the game, then it displays "Game Over" along with a Restart button to continue playing the game again.In order to quit the game, press the close window or press X button. It close all windows and quits the pygame instance.

Full Code Implementation of Gesture Controlled Game :

Here the entire code, combining all the above steps to start playing our gesture controlled game using TensorFlow is ready. The below code combines video processing, gesture recognition, and game mechanics in order to create an interactive gesture-based gaming experience. Further, improve this code with additional features, and more gestures to make this very interesting.Let's start the game now.

How to play this gesture-based game

1. Launch the game: Run the Python code to start the game. The game window will open, displaying a white rectangular player at the bottom of the screen.
2. Position your hand: Position your hand in front of the webcam, ensuring it is within the camera's field of view. Make sure your hand is well-lit and distinguishable from the background.
3. Recognize gestures: The game uses hand gestures to control the player's movement. Extend your index finger to control the player's movement.
4. Left gesture: Move your extended index finger to the left side of the screen to make the player move left.
5. Right gesture: Move your extended index finger to the right side of the screen to make the player move right.
6. Neutral gesture: If your index finger is in the middle area of the screen, the player will stay in its current position.
7. Dodge the falling obstacle: An obstacle in the form of a red rectangular box will fall from the top of the screen. You aim to dodge the obstacle by moving the player left or right using hand gestures.
8. Game over: If the player collides with the falling obstacle, the game ends. The screen will display a "Game Over" message, along with a restart button.
9. Restart the game: To restart the game, simply click the restart button displayed on the screen when the game is over. The player and obstacle positions will be reset, and you can continue playing.
10. Quit the game: To quit the game, close the game window or press the "X" button.

Output:

pygame 2.5.0 (SDL 2.28.0, Python 3.10.6)
Hello from the pygame community. https://www.pygame.org/wiki/Contribute

Launch Gesture Recognition and Gesture Based Game

Pygame starts the game with the player and obstacle, and the gesture recognition shows Neutral to perform the player action.

👁 game2gfg_11zon.jpg

Screenshot 3: Displays Game Over and Option to Restart the game.

The player can restart the game with restart button, to continue playing endlessly.

👁 game3gfg_11zon.jpg

Feel free to support me with your comments below. Happy Playing!