WRIPLE

A cross-platform desktop application that uses Wi-Fi and millimeter wave (mmWave) signal to detect human presence behind obstruction. This system is possible by taking advantage of Wi-Fi CSI capable ESP32 module, a RDM-capable HLK-LD2420 mmWave radar, that the WRIPLE application designed to communicate with. Tools such as simple radar, signal heatmap, detection linechart, and data collection (hidden tab) are available to be used during operation. A Conv-LSTM deep learning model is being used to detect human presence based on the collected Wi-Fi CSI or mmWave RDM. Access the ESP32-WROOM-32U module firmware here. Email the developer for any questions and development that this software has been used.

Table of Contents

• Features
• Data Collection
• Relevant Processes
• Installation
• License
• Acknowledgments

Features

Monitoring Tab

• Status Bar - System status bar indicates the different required components of the system to function properly. This includes the Wi-Fi, Server, ESP32, LD2420 and ML model. Missing one of those components will cause the application functionality dependent on a specific component to not work properly.

• Monitoring Mode - Transmit and receive packets between the station and ESP32. Each packet contains a single sample of CSI data, and an RDM sample once every ~20 packets or 333 ms. Before the system detects human presence, for the first 15 seconds, the application will first perform some calibration to adapt to the new environment, and ensure that the signal noise is below 50 for reliable prediction. To reduce the signal noise, adjust the setup position, LOS and height. A messages such as Noisy, Calibrating, and Restart will be displayed in the presence section depending on the status of the system to assist the user. After that, for every 120 samples or 2 seconds of CSI data, one human presence detection will be performed, and its result is displayed in the Presence section. The total number of packets, loss of signal, signal noise and RSSI were all updated in real-time during monitoring mode.

• mmWave Radar - This mmWave is not capable of detecting human presence behind obstruction, therefore, this radar can only be used for target with direct LOS. The radar detection is highly accurate but the distance estimation is still experimental. For future development, this radar can be used for visualization of other mmWave radar capability in the market.

• Amplitude Heatmap - Provides a real-time heatmap of CSI amplitudes and mmWave RDM. The heatmap data is composed of selected subcarriers, then preprocessed to improve visualization. The heatmap flows from right to left 10x/second. The canvas is limited to store up to 15 seconds of amplitude data since visualizing more data consumes significant processing power. This heatmap can be used by the user to manually identify patterns in case experimentation and unreliable model.

• RDM Heatmap - Provides a real-time heatmap of range-doppler map (RDM). Unlike the Amplitude Heatmap, this heatmap refresh for every 333 ms according to the HLK-LD2420 hardware limitation. The heatmap data is composed of 16 range-bins and 20 doppler-bins that's been normalized to improve visualization. Because of nature of range-doppler map, user can visualize the target position based on the position of hotspot in the heatmap.

• CSI Noise Linechart - Show a line chart that tracks the noise level of the environment and sets up over time for up to 25 seconds. The displayed data is similar to the Noise section of the status bar. Users are recommended to continuously adjust the setup and position of the AP and ESP32 until consistent noise level below 2.0 is achieved. The purpose of this feature is purely for achieving acceptable signal noise level for reliable human presence detection of the model and even during data collection.

• Detection Linechart - Aside from looking at the Presence status in the status bar, detection results can also be tracked using a line chart for the past 50 seconds. This allows the user to leave the station for a short amount of time, or have a better insights of the detection pattern over time.

Data Collection

Data Collection Tool

• Record - Record Wi-Fi CSI signal at 60 samples/second and mmWave RDM signal at 3 samples/second for 4 seconds each recording, just enough to capture a single human breathing cycle. Clicking the button while recording will stop the recording and the CSV file will be incomplete or invalid for dataset. In this case, user have to manually delete the file.
• Parameters Selection - Set the parameters based on the data collection setup before taking the recording.
• CSV Files Selection - Review the selected CSV file metadata to ensure data integrity.

Data Collection Setup

1. Perform the data collection in a controlled environment with no other human presence and less Wi-Fi activity to maintain high quality signal.
2. Immediately perform the movement even before the recording starts.
3. Plan or keep track of group of files that contains different class to avoid confusion during data collection.
4. Keep the amount of recording for the different class and distance the same for balanced dataset classes.
5. Use the ESP32 DFWS prototype LED state as signal if the recording is complete.

Relevant Processes

System Use Case

• The WRIPLE system is composed of DFWS prototype, AP, and the application. To start the system to detect human presence, the user has to first connect all the system components. First, AP must be configured to use WRIPLE as SSID, WRIPLE_ESP32 as password, and 2.4 GHz as AP frequency band to allow the DFWS prototype to automatically connect. Powering the DFWS prototype using a compatible power supply like power bank will automatically connect it to the AP on startup. The application status bar can be used to check for different system components in real-time. Once everything is connected, monitoring and visualization can be started to detect human presence.

Note

If the application cannot establish connection to ESP32, try to enable the device (laptop) location.

UDP Hole Punching

• UDP hole punching method was used for the ESP32 and the Station to both connect to the same AP, establishing a simple two-way communication channel. During the boot of ESP32, once connected to the AP, it starts listening to the broadcast messages of the AP. The Station sends a broadcast message on the local network so the ESP32 can learn its IP address. ESP32 then sends an outbound UDP packet to the discovered Station’s IP address so that Station can learn the ESP32 IP address. Because most operating systems and AP create a temporary state for outbound UDP traffic, that outbound packet implicitly opens a short-lived pinhole that allows the corresponding inbound UDP packets to be received. This approach avoids having to add explicit firewall rules on the Station. However, it is good practice to send periodic packets so the temporary mapping does not time out. Because of this nature, the Station and ESP32 connection might be closed unexpectedly, therefore restarting the ESP32 is needed.

System Data Flow

• The system data flow demonstrates how the system process and collects Wi-Fi CSI and mmWave RDM data. First, the Station will transmit a data request packet to the ESP32 that is forwarded by the AP. ESP32 is configured to respond to every CSI-compatible packet. The moment a request data packet is received, ESP32 begins reading CSI and RDM data, which is then sent back to the Station. After the Station receives the response packet, data is parsed and stored in a separate CSI and RDM data queue for prediction and visualization purposes.

Installation

1. Download and install the latest version of WRIPLE.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Tools

• draw.io: For generating diagrams and visual workflows.
• Jupyter Notebook: For machine learning and deep learning prototyping.
• Phosphor Icons: For tab and button icons.

Frontend

• Tailwind CSS: For CSS framework.
• simpleheat: For real-time heatmaps.
• Chart.js: For real-time line charts.
• D3.js: For 3D scatter plot visualization.

Backend

• uv: For Python dependencies manager and virtual environment.
• Flask-CORS: For handling resource sharing between Python and JavaScript.
• Waitress: For production-ready WSGI server.
• psutil: For system performance monitoring.

Data Science

• Scikit-learn: For machine learning and statistical modeling.
• TensorFlow: For deep learning framework.
• Joblib: For saving and loading trained machine learning models.
• Numpy: For handling different types of arrays.
• Pandas: For data manipulation and analysis.
• SciPy: For scientific computation library.
• Matplotlib: For generating different charts.
• Plotly: For generating interactive charts.

Wrapper and Installer

• PyWebview: For standalone web app wrapper.
• PyQt5: For native desktop window for Linux.
• QtPy: For compatibility layer that PyWebview and PyQt5 used.
• PyQtWebEngine: For providing a chromium-based webview.
• PyInstaller: For packaging the app into executables.
• Inno Setup: For installer.