Pose Estimation & Neural Inference
Relevant source files
The following files were used as context for generating this wiki page:
The Pose Estimation & Neural Inference system transforms Channel State Information (CSI) from standard WiFi signals into dense human pose wireframes. The system achieves 87.2% detection accuracy (AP@50) and operates through solid barriers without line-of-sight requirements. It utilizes a deep learning pipeline to map radio frequency (RF) perturbations to 24 anatomical body regions and 17 COCO-format keypoints.
This module resides in the Neural Network Layer of the project architecture, sitting between the CSI Processing Layer and the API Middleware. It preserves privacy by eliminating the need for cameras while providing real-time tracking at 10–30 FPS with sub-100ms latency.
Sources: plans/overview.md, references/README.md, ui/index.html
Neural Architecture and Data Flow
The inference pipeline converts 1D radio signals into 2D spatial features through a modality translation network. This architecture allows the system to bridge the gap between RF signals and visual pose estimation models like DensePose-RCNN.
Inference Pipeline Stages
The following diagram illustrates the sequential transformation of raw WiFi data into pose wireframes.
The diagram shows the end-to-end flow from hardware signal collection to the final human pose visualization.
Sources: ui/index.html, plans/phase1-specification/functional-spec.md
Modality Translation Network
The Modality Translation Network utilizes a dual-branch encoder architecture to process amplitude and phase data separately. By treating each element in the 3x3 CSI tensor as a holistic scene summary, the network fuses these features to generate spatial representations at 720x1280 resolution.
- Dual-Branch Processing: Separate encoders extract features from CSI amplitude and phase.
- Feature Fusion: Convolutional layers with batch normalization and ReLU activations fuse modality-specific features.
- Upsampling: The network progressively upsamples compact features to full spatial resolution compatible with DensePose models.
Sources: references/README.md, plans/phase1-specification/functional-spec.md
DensePose Estimation Logic
The system identifies anatomical regions and maps 2D detections to 3D human body models using the DensePose-RCNN engine. This engine predicts UV coordinates for surface correspondence across the body.
Detected Regions and Keypoints
The model tracks 24 anatomical regions and 17 major body keypoints.
| Category | Component | Count | Description |
|---|---|---|---|
| Body Regions | Anatomical Parts | 24 | Segmentation masks for specific body sections (e.g., torso, limbs). |
| Keypoints | COCO Keypoints | 17 | Major joints including nose, eyes, shoulders, elbows, wrists, hips, knees, ankles. |
| Output Format | UV Coordinates | - | Mapping for dense correspondence between 2D detection and 3D models. |
Sources: references/script_8.py, plans/phase1-specification/functional-spec.md
Temporal Tracking
The tracking engine maintains person identity across frames using temporal smoothing and motion level classification.
- ID Assignment: Consistent identifiers remain assigned to individuals even during temporary occlusions.
- Trajectory Smoothing: Temporal filters remove jitter from keypoint coordinates.
- Multi-Person Support: The system tracks up to 5 individuals simultaneously.
Sources: plans/phase1-specification/functional-spec.md, plans/phase1-specification/api-spec.md
Performance and Metrics
The inference system is optimized for real-time execution on GPU hardware using PyTorch CUDA support, though it supports graceful degradation for CPU-only environments.
System Performance Specifications
| Metric | Target Value | Environment |
|---|---|---|
| AP@50 (Accuracy) | 87.2% | Same Layout |
| AP@50 (Accuracy) | 51.8% | Different Layout |
| Processing Rate | 10–30 FPS | GPU Accelerated |
| End-to-End Latency | <100 ms | Total Pipeline |
| Inference Latency | <50 ms | Neural Network Stage |
Sources: references/script_8.py, plans/phase1-specification/system-requirements.md, plans/phase1-specification/functional-spec.md
Signal Quality and Hardware Constraints
The accuracy of the neural inference depends on the quality of the incoming CSI data.
- Sampling Rate: The hardware must provide CSI data at 100Hz.
- MIMO Configuration: A minimum 3x3 antenna array is required to provide spatial diversity.
- Phase Sanitization: The system performs phase unwrapping and linear detrending to address hardware-specific noise before inference.
Sources: plans/phase1-specification/system-requirements.md, ui/index.html
Neural Network Model Training
Training the modality translation network involves a transfer learning framework. Image-based DensePose models serve as "teacher" networks, guiding the training of the WiFi-based "student" network. This approach significantly reduces training time and improves convergence stability.
The sequence diagram details the real-time processing loop from signal capture to UI rendering.
Sources: references/README.md, references/script_8.py, plans/phase1-specification/api-spec.md
Summary
Pose Estimation & Neural Inference within the RuView project provides a camera-free alternative for tracking human activity through barriers. By translating WiFi CSI amplitude and phase into visual feature maps, the system leverages established DensePose-RCNN architectures to achieve high precision and low latency. The modular design ensures compatibility with commodity WiFi hardware while maintaining a strict privacy-preserving data pipeline.