Summary of Game Automation for Mobile Using Python and Scrcpy

The game automation script for a mobile application game uses real-time image processing to capture the mobile screen's condition and make decisions on the next step in gameplay. The key steps and technologies used in the implementation are:

1. Real-Time Screen Capture:

The mobile screen is captured in real time using the Scrcpy software, which streams the screen to a Python script on the PC. The ADB (Android Debug Bridge) is utilized to facilitate communication between the mobile device and the PC.

2. Optimizing Input with Region of Interest (ROI):

The captured screen image is processed using Numpy arrays to slice and focus on a specific region of the screen (ROI). This reduces unnecessary processing and improves efficiency, specifically for detecting objects close to the player's car in the game.

3. Gameplay Logic:

• The game logic revolves around detecting incoming elements (circles or squares) and taking actions accordingly. The logic involves:
  • Detecting Objects: Circles and squares are detected in the ROI using image processing techniques like thresholding and contour detection.
  • Lane Detection: The lane of the incoming object is identified.
  • Lane Switching: The car’s lane is tracked, and if it is in the same lane as a square (to avoid it), or not in the same lane as a circle (to collect it), a lane switch is triggered using an ADB tap command.

4. Real-time Image Processing for Object Detection:

The screenshots are converted to grayscale, and thresholding is applied to enhance object visibility. Contours of the detected objects (squares and circles) are then found, and their properties (area, number of vertices) are analyzed for classification.

5. Final Actuation on the Phone:

• Based on the detected lane and object, the script sends tap commands to the mobile device using ADB to switch lanes in the game. If a square is detected in the current lane, a tap command moves the car to avoid it. Similarly, the car switches lanes to collect the circle if it’s not already in the correct lane.

Technologies Used:

• Scrcpy: Used for streaming real-time screenshots from the mobile device to the Python script.
• ADB (Android Debug Bridge): Facilitates communication and control between the PC and the Android device.
• OpenCV: Used for image processing and object detection (e.g., detecting squares and circles).
• Numpy: Utilized for image array manipulation and processing.