Deep Q-Networks for Autonomous Lunar Lander: A Reinforcement Learning Approach
Overview :
The Lunar Lander DQN project involves training a reinforcement learning agent to control a lander in the Lunar Lander environment from OpenAI's Gym. The goal is to land the spacecraft safely between two designated flags on a lunar surface while minimizing fuel usage and avoiding crashes. The DQN algorithm is a popular approach to reinforcement learning, where a neural network is used to approximate the Q-value function, which predicts the total expected rewards for taking certain actions from given states.
Please visit the official link below for additional information on Lunar Lander.
Key Components of the DQN Implementation:
-
Environment Setup:
-
The environment used is LunarLander-v2 from OpenAI Gym, which provides a simulation of a lunar lander with discrete actions (do nothing, fire left orientation engine, fire main engine, fire right orientation engine) and an 8-dimensional observation space representing the state of the lander.
-
-
Neural Network (Q-Network):
-
The Q-network is a simple feedforward neural network with two hidden layers (each with 128 neurons and ReLU activation) and an output layer corresponding to the action space size. This network predicts the Q-values for each action given a state input.
-
-
DQN Agent:
-
The agent interacts with the environment by selecting actions based on an epsilon-greedy policy (choosing a random action with probability epsilon, otherwise choosing the action with the highest Q-value).
-
The agent learns from experiences stored in a replay buffer by sampling mini-batches and performing gradient descent to minimize the difference between predicted and target Q-values (using Mean Squared Error loss).
-
-
Replay Buffer:
-
A replay buffer stores experiences (state, action, reward, next state, done) to stabilize learning. By sampling random mini-batches from the buffer, the agent learns from a diverse set of experiences, reducing correlation and improving performance.
-
-
Training Loop:
-
The agent is trained over a number of episodes. In each episode, it performs a series of actions in the environment until it either lands successfully or crashes. After each action, the experience is stored, and the agent periodically updates its Q-network using experiences from the replay buffer.
-
Soft updates to the target network are performed to slowly adjust its weights towards the evaluation network's weights, which stabilizes training.
-
-
Testing and Visualization:
-
After training, the agent is tested to see how well it can perform. The code includes functionality to render the environment and record the agent's performance, creating visualizations like GIFs to showcase its learning progress.
-
This graph depicts the performance of the DQN agent over 5000 episodes in the Lunar Lander environment. The y-axis represents the scores, showing the agent's improvement in landing the spacecraft safely as training progresses.
Results Interpretation:
-
Training Score History:
-
The first image shows the score history over 5000 episodes. The scores represent the agent's performance, with higher scores indicating better landings (closer to the target and using less fuel). Over time, the agent learns to avoid crashing, leading to a trend where scores gradually increase and stabilize around a higher average.
-
-
Agent's Performance:
-
The second image (GIF) likely shows the agent's gameplay in the Lunar Lander environment, where it attempts to land the spacecraft. The trained agent should demonstrate improved control over the lander, achieving successful landings more consistently.
-
Please visit the github website for code.