Mukesh Sadasivam — Physical AI & Robot Learning

Exploring the full manipulation pipeline in physical AI, from data collection to foundation model integration.

What this is about

I wanted to tackle the manipulation problem in physical AI, so I explored the full pipeline to understand how it all connects:

Data Collection → Policy Training → Foundation Model Integration → Deployment

The pipeline, project by project 🤖 Data Collection — Real Teleoperation

Dual-Arm UR5 Teleoperation Real-time vision-based hand tracking (MediaPipe) controlling two UR5 arms via analytical IK under 2ms latency. ROS2, Docker, ISO 10218-1 safety architecture. Demo: youtu.be/r32MHanCywM

G1 Humanoid Teleoperation Webcam-based per-finger curl control (thumb/index/middle) mapped to real actuated finger joints on the Unitree G1's 3-fingered hand. Real contact-based physical grasping (MuJoCo physics, no kinematic approximation). Palm-down orientation enforced via wrist-override after IK solve to fix Jacobian redundancy instability.

🧠 Policy Training — BC, ACT, Diffusion Policy

G1 Manipulation Pipeline End-to-end IL+RL: GPT-4o vision → Behavioral Cloning (100 demos, 100% success) → Residual PPO on top (20/20, ±4cm generalization). ResMimic architecture.

ACT & Diffusion Policy from scratch (in progress) Implementing Action Chunking Transformers and Diffusion Policy from scratch on the standard PushT benchmark, following the original papers. CVAE encoder + transformer decoder, temporal ensembling, genuine BC vs. ACT vs. Diffusion comparison on the same dataset.

🌍 Foundation Model Integration — World Models & VLMs

Sim_Cortex Open-source replication of Sereact's Cortex 2.0 Plan→Score→Execute architecture. Key question: can simulation replace their $10M+ real-data requirement? Uses NVIDIA Cosmos-Predict2 (2B params, frozen) via mimic-video as a world model backbone. Extracts layer-19 latent representations (task-semantic sweet spot). Generates K=3 imagined futures via cross-attention-conditioned denoising. Trains a 1.6M-parameter PRO scorer (Progress/Risk/Efficiency heads) on simulated failure injection data. Zero real robot data.

👁️ 3D Perception — Object Pose Estimation

3D Bounding Box Prediction End-to-end 3D perception pipeline for robotic bin-picking from point clouds. PointNet-style encoder, 6D continuous rotation (Gram-Schmidt orthonormalization), centroid-centering fix that cut center error 66%. Honest diagnosis of rotational symmetry labeling artifact. ONNX export with verified numerical parity. 3.1cm center error, 0.27 mean 3D IoU.

🏭 Simulation & Multi-Agent RL

Adversarial Warehouse Robots (private, contact for access) PPO adversarial agent disrupting cooperative warehouse robot coordination via communication-layer manipulation. ~44% throughput reduction. Custom Gymnasium environment connected directly to a real warehouse simulator. Thesis work at University of Stuttgart / ITM.

2D Warehouse Detection Custom YOLO-style detector for warehouse logistics equipment (pallets, pallet trucks, stillages). Grid-based OBB detection, class imbalance fix (pos_weight=15), augmentation pipeline. 84.8% recall, honest diagnosis of single-scale grid limitation for dense scenes.

Skills at a glance Area Tools / Experience Policy Learning BC, ACT, Diffusion Policy, Residual PPO, Stable-Baselines3, LeRobot Foundation Models Cosmos-Predict2, mimic-video, GPT-4o vision, VLA fine-tuning Simulation MuJoCo (deep), Isaac Sim (ROS2 integration, PPO training) Teleoperation MediaPipe hand tracking, analytical IK, ROS2, real contact grasping Perception PointNet, 3D pose estimation, ONNX deployment Infrastructure PyTorch, Docker, Git, ROS2, Gymnasium What this is really about

I wanted to understand the full manipulation pipeline end to end, not just one piece of it. So I built all of it: teleoperation systems for data collection, BC and ACT and Diffusion Policy for policy training, world models for failure-aware planning, and 3D perception for scene understanding. Each project was genuinely trying to answer a real question, not just stack technologies.

Physical AI for real robots is where I want to spend my career. Everything here points at that.

📧 sadrevreddy2000@gmail.com | LinkedIn | GitHub