Adaptive Fighting Robot Training with Reinforcement Learning

Scope: simulation study. No physical robot was built or tested.

The simultaneous process of balance and adversarial combat automation of an intrinsically unstable system—represented by an inverted pendulum mechanics model—has been implemented with a model-free reinforcement-learning approach using a Deep Q-Network. A 4-phase design framework based on progressive difficulty calibration was executed, initiating from a baseline linear control (LQR) reference. Symmetric self-play competition across paired self-play agents was executed to reduce the risk that one policy becomes dominant during self-play and narrows the diversity of training scenarios.

⚠️ For PoC Projects: The agent profile formulated via self-play architecture demonstrates a quantifiable potential to maintain higher disturbance tolerance (robustness) within environments containing deterministic anomalies, when juxtaposed directly against agents calibrated via rigid analytical inputs (LQR references).

---

Project Portfolio

Current Situation and Problem

Context: Inverted pendulum structures function as mechanically unstable systems. In scenarios demanding an external mechanical conflict (combat) vector, coordinating stabilization simultaneously with reactive action planning complicates the optimization problem. Critical Issues: Calibration logic bounded purely by static limits (such as LQR) exhibits an inherent tendency to fail within flexible operational domains where definitive system equations cannot be assumed. Optimizing models over static parameters (overfitting) empirically generates degraded performance under disturbances not represented in training.

Solution Architecture and Action

Architectural Approach: To examine performance degradation in under-defined control environments, a 4-phase training framework encompassing a variable difficulty curve was designed.

Applied Methodology:

Phase 1: LQR Baseline (Reference Data Extraction)

Purpose: To map foundational system dynamics and catalog baseline responses for establishing a comparative testing platform.

• A native LQR controller block was built independent of external library functions.
• A customized test physics engine was computed leveraging the CTMS Michigan structural model.
• The formulated output matrices (state → action) were archived to serve as the reference model benchmark.

Phase 2: Self-Balancing (Standalone Stabilization)

Purpose: Optimizing the capability of the system to maintain stability via native error functions without applying a preemptive input map (supervised learning).

• Training parameters were designated by migrating structural mechanics to a Deep Q-Network (DQN).
• Experience replay and a target network were used to stabilise training.
• Specific constraint mechanisms (Reward Shaping) were applied: The system was filtered by calculating target axis deviation, axial position error, and momentum expenditure.

Phase 3: Disturbance Resistance and Attack

Purpose: The activation of physical anomalies within the given simulation scope and a discrete action space to test steady-state stability.

• Supplementary external forces (disturbance) mapped under a Poisson distribution were generated to simulate non-deterministic stochastic physical impacts.
• The computing structure subsequently weighted parameters commanding planned combat movements while preserving structural balance.
• The primary “Balance force” vector and the independent “Attack force” vector were processed across fully isolated phase spaces.

Phase 4: Self-Play Fighting (Adversarial Training)

Purpose: Testing robustness under mutual adversarial pressure, building on the isolated training phases.

• To guarantee a flawless measurement baseline across the array, two agent instances were initialised from the same neural-network starting weights.
• During each independent epoch of the routine, dual modules executed logic disrupting the opponent’s balance function while calculating their own internal stabilization.
• The modules were cross-evaluated symmetrically against a dynamic clone reacting directly to mutual behaviors, explicitly discarding static functional parameters.

Architectural Decision: Employing two segregated neural network blocks invariably triggered asymmetric superiority deviations, categorized structurally within early epochs as “model dominance”. Unifying the calculation into a singular common network topology (YSA) reduced this instability and constrained variance growth during training.

Dual Mode Operational Conditions:

1. Isolated Mode: During early epoch cycles, competitive routines remain inactive, prioritizing exclusively Cartesian balance assessment.
2. Combined Mode: As stabilization gradients hit operational maturity, adversarial policies (Q-Values) are activated simultaneously alongside the balance vectors.

To prevent control disruption scaling within the system, the maximum threshold limits dictating combat actions were held to a fractional ratio of ~15% of the associated balance boundaries. (Balance Tolerance: [-10, +10] N, Attack Tolerance: [-1.5, +1.5] N).

---

Results and Operational Gains

Test Results

Simulation Visuals

Demo: Self-Play Combat Simulation

Related Links

🔗 Detailed Article: Control Strategies in Non-Linear Systems: LQR and Deep RL Comparison 
📄 Source Paper PDF: Makina Öğrenmesi Teknikleri Kullanılarak Bir Dövüşen Robotun Eğitilmesi (Turkish) 
📂 Source Code: Github/neural-adaptive-control-simulation

---

This research was conducted within the ITU Control and Automation Engineering program and presented under the graduation project titled: “Self-adaptive training architectures using machine learning methodologies”.

Last Updated: January 2026