Adaptive Fighting Robot Training with Reinforcement Learning

Comparative simulation evaluation of classical LQR and Deep Q-Network architectures on inverted pendulum dynamics: stabilization, model fragility, and adversarial resilience.

The simultaneous process of balance and adversarial combat automation of an intrinsically unstable system, represented by an inverted pendulum mechanics model, has been implemented with a model-free reinforcement-learning approach using a Deep Q-Network. A 4-phase design framework based on progressive difficulty calibration was executed, initiating from a baseline linear control (LQR) reference. Symmetric self-play competition across paired self-play agents was executed to reduce the risk that one policy becomes dominant during self-play and narrows the diversity of training scenarios.

ℹ️ Scope Note: This study is an academic simulation research project; no physical robotic chassis was constructed, and algorithmic robustness was validated using a dynamic physics engine.

Dual cart-pendulum system simulation visualization

Project Portfolio

Parameter	Value
Category	Solutions Engineering
Delivery Type	Academic Research
Status	Proof of Concept
Scale / Scope	4-Phase Training Pipeline, Self-Play Adversarial Training

Current Situation and Problem

Context: Inverted pendulum structures function as mechanically unstable systems. In scenarios demanding an external mechanical conflict (combat) vector, coordinating stabilization simultaneously with reactive action planning complicates the optimization problem. Critical Issues: Calibration logic bounded purely by static limits (such as LQR) exhibits an inherent tendency to fail within flexible operational domains where definitive system equations cannot be assumed. Optimizing models over static parameters (overfitting) empirically generates degraded performance under disturbances not represented in training.

Problem	Detail
Structural Instability	The persistent requirement for an endless closed-loop feedback array to maintain inverted pendulum continuity
Multiple Optimization	Computing orientation positioning simultaneously while preserving native center-of-gravity stabilization
Undefined Model	Operating without the provision of a pre-calculated external dynamic system transfer function
Robustness Under Disturbance	The critically low tolerance of static algorithms to unpredictable, non-deterministic physical impacts

Solution Architecture and Action

Architectural Approach: To examine performance degradation in under-defined control environments, a 4-phase training framework encompassing a variable difficulty curve was designed.

Applied Methodology:

Phase 1: LQR Baseline and Reference Data Extraction

Purpose: To map foundational system dynamics and catalog baseline responses for establishing a comparative testing platform.

A native LQR controller block was built independent of external library functions.
A customized test physics engine was computed leveraging the CTMS Michigan structural model.
The formulated output matrices (state → action) were archived to serve as the reference model benchmark.

Phase 2: Self-Balancing and Standalone Stabilization

Purpose: Optimizing the system’s capability to maintain stability via reward-driven error functions without relying on a predefined input map or analytical reference.

Training parameters were designated by migrating structural mechanics to a Deep Q-Network (DQN).
Experience replay and a target network were used to stabilise training.
Reward shaping design: The reward function penalized angular deviation, cart displacement error, and actuation effort to guide the agent toward stable equilibrium.

Phase 3: Disturbance Resistance and Attack

Purpose: The activation of physical anomalies within the given simulation scope and a discrete action space to test steady-state stability.

Supplementary external forces (disturbance) mapped under a Poisson distribution were generated to simulate non-deterministic stochastic physical impacts.
The computing structure subsequently weighted parameters commanding planned combat movements while preserving structural balance.
The primary “Balance force” vector and the independent “Attack force” vector were processed across fully isolated phase spaces.

Phase 4: Self-Play Fighting and Adversarial Training

Purpose: Testing robustness under mutual adversarial pressure, building on the isolated training phases.

To guarantee a measurement baseline across the array, two agent instances were initialised from the same neural-network starting weights.
During each independent epoch of the routine, dual modules executed logic disrupting the opponent’s balance function while calculating their own internal stabilization.
The modules were cross-evaluated symmetrically against a dynamic clone reacting directly to mutual behaviors, explicitly discarding static functional parameters.

Architectural Decision: Employing two segregated neural network blocks invariably triggered asymmetric superiority deviations, categorized structurally within early epochs as “model dominance”. Unifying the calculation into a singular common network topology reduced this instability and constrained variance growth during training.

Dual Mode Operational Conditions:

Isolated Mode: During early epoch cycles, competitive routines remain inactive, prioritizing exclusively Cartesian balance assessment.
Combined Mode: As stabilization gradients hit operational maturity, adversarial action channels were activated simultaneously alongside primary balance outputs.

To prevent control disruption scaling within the system, the maximum threshold limits dictating combat actions were held to a fractional ratio of ~15% of the associated balance boundaries. (Balance Tolerance: [-10, +10] N, Attack Tolerance: [-1.5, +1.5] N).

Results and Operational Gains

Focus	Verified Impact
Concurrent Optimization	Reaction vectoring variables were processed within identical operating cycles alongside mechanical stabilization curves.
Robustness Improvement	Implementing self-play weight updates explicitly restricted errors spawned directly by closed-loop static system assumptions that do not hold under disturbance.
System Robustness	Under mapped adversarial pressure scenarios, the implementation extracted more sustainable flexibility limits opposed to classic analytic LQR benchmarks.
Model Elasticity	Command control limits were accurately established internally without necessitating ideal, pre-formulated system equations from external sources.

🎯 Test Results

Metric	Value
Test Episode Count	300 Episodes
Average Simulation Time	~320 Frames/Steps
Maximum Observed Peak	700 Frames/Steps
Exploration Multiplier	0.0 (Zero exploration, pure policy)

📊 Simulation Visuals

📥 Downloads

📄 Source Paper PDF: Makina Öğrenmesi Teknikleri Kullanılarak Bir Dövüşen Robotun Eğitilmesi (PDF, Turkish)

🔗 Companion Article: Classical Control is Not Dead: The Balance Between LQR and Deep Reinforcement Learning
📂 Source Code: GitHub / neural-adaptive-control-simulation

This research was conducted within the ITU Control and Automation Engineering program and presented under the graduation project scope.