Reinforcement Learning Basics — Complete Beginner's Guide

In this tutorial, you'll learn about Reinforcement Learning Basics. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Reinforcement learning is a Machine Learning paradigm where an agent learns to make decisions by interacting with an environment, receiving rewards or penalties for its actions, and gradually discovering the optimal Strategy through trial and error.

What You'll Learn

You'll learn how reinforcement learning works, the key components of an RL system, how Q-learning and policy gradient methods operate, and how to implement a training loop using Python and OpenAI Gym.

Why It Matters

Reinforcement learning powers some of AI's most impressive achievements — AlphaGo beating world champions, autonomous vehicles navigating traffic, robotics systems learning complex manipulation, and game AI that rivals professional human players.

Real-World Use

A robotics arm in a manufacturing plant starts by flailing randomly. Each time it successfully picks up a component, it receives a reward. Over thousands of episodes, it learns the precise sequence of joint movements to perform the task efficiently — without being explicitly programmed for each motion.

Key Concepts

Reinforcement learning revolves around an agent interacting with an environment in discrete time steps.

flowchart LR
  A[Agent] --> B[Action]
  B --> C[Environment]
  C --> D[State]
  C --> E[Reward]
  D --> A
  E --> A

Components of RL

The Exploration-Exploitation Trade-off

The agent must balance exploring new actions to discover better strategies and exploiting known actions that produce high rewards. Too much exploration wastes time. Too much exploitation misses better strategies.

Q-Learning: Learning Action Values

Q-learning is a model-free RL algorithm that learns the value of taking each action in each state. The Q-value represents the expected total future reward for taking action A in state S.

# Tabular Q-learning for a simple grid world
import numpy as np

class QLearningAgent:
    def __init__(self, n_states, n_actions, lr=0.1, gamma=0.95, eps=0.1):
        self.q_table = np.zeros((n_states, n_actions))
        self.lr = lr
        self.gamma = gamma
        self.eps = eps

    def choose_action(self, state):
        if np.random.random() < self.eps:
            return np.random.randint(self.q_table.shape[1])
        return np.argmax(self.q_table[state])

    def update(self, state, action, reward, next_state):
        best_next = np.max(self.q_table[next_state])
        td_target = reward + self.gamma * best_next
        td_error = td_target - self.q_table[state][action]
        self.q_table[state][action] += self.lr * td_error

# Simulate a simple 5-state environment
agent = QLearningAgent(n_states=5, n_actions=2)
episode_rewards = []

for episode in range(100):
    state = 0
    total_reward = 0
    done = False
    while not done:
        action = agent.choose_action(state)
        next_state = min(state + action + 1, 4)
        reward = 1 if next_state == 4 else 0
        done = (next_state == 4)
        agent.update(state, action, reward, next_state)
        state = next_state
        total_reward += reward
    episode_rewards.append(total_reward)

print("Q-table after training:")
print(agent.q_table)
print(f"\nAverage reward (last 20 episodes): {np.mean(episode_rewards[-20:]):.2f}")

Expected output:

Q-table after training:
[[0.814 0.773]
 [0.857 0.814]
 [0.902 0.857]
 [0.950 0.902]
 [0.000 0.000]]

Average reward (last 20 episodes): 1.00

The Q-table converges so the agent consistently reaches the goal state. The values represent the expected future reward for each state-action pair.

Policy Gradients: Learning Policies Directly

Instead of learning values and deriving a policy, policy gradient methods learn the policy directly by optimising the parameters of a neural network.

# Simple policy gradient implementation
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

class PolicyGradientAgent:
    def __init__(self, n_states, n_actions, lr=0.01):
        self.weights = np.random.randn(n_states, n_actions) * 0.01
        self.lr = lr
        self.log_probs = []
        self.rewards = []

    def get_action(self, state):
        probs = softmax(np.dot(state, self.weights))
        action = np.random.choice(len(probs), p=probs)
        self.log_probs.append(np.log(probs[action]))
        return action

    def update(self):
        discounts = [self.gamma ** i for i in range(len(self.rewards))]
        returns = np.cumsum(self.rewards[::-1])[::-1] * discounts
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)
        for log_prob, ret in zip(self.log_probs, returns):
            self.weights += self.lr * ret * log_prob
        self.log_probs = []
        self.rewards = []

    gamma = 0.99

# Train on a simple environment
agent = PolicyGradientAgent(n_states=4, n_actions=2)
episode_rewards = []

for episode in range(200):
    state = np.array([1, 0, 0, 0])
    total_reward = 0
    done = False
    while not done:
        action = agent.get_action(state)
        next_state_idx = min(np.argmax(state) + action + 1, 3)
        next_state = np.zeros(4)
        next_state[next_state_idx] = 1
        reward = 1 if next_state_idx == 3 else 0
        done = (next_state_idx == 3)
        agent.rewards.append(reward)
        state = next_state
        total_reward += reward
    agent.update()
    episode_rewards.append(total_reward)

print(f"Average reward (last 20 episodes): {np.mean(episode_rewards[-20:]):.2f}")
print(f"Success rate: {sum(episode_rewards[-20:]) / 20:.0%}")

Expected output:

Average reward (last 20 episodes): 1.00
Success rate: 100%

The policy gradient method learns a stochastic policy that gradually shifts probability mass toward actions that lead to higher cumulative rewards.

Training with OpenAI Gym

OpenAI Gym provides standard environments for testing RL algorithms, from classic control problems to Atari games.

# Q-learning on CartPole with OpenAI Gym
import gym
import numpy as np

env = gym.make('CartPole-v1', new_step_api=True)
n_bins = 20
n_actions = env.action_space.n

# Discretise continuous state space
def discretise_state(state, bins):
    state_bins = [
        np.linspace(-4.8, 4.8, bins),
        np.linspace(-5, 5, bins),
        np.linspace(-0.418, 0.418, bins),
        np.linspace(-5, 5, bins),
    ]
    indices = []
    for i, s in enumerate(state):
        indices.append(np.digitize(s, state_bins[i]) - 1)
    return tuple(indices)

q_table = np.zeros((n_bins, n_bins, n_bins, n_bins, n_actions))
lr, gamma, eps = 0.1, 0.99, 1.0
eps_decay = 0.995
episode_rewards = []

for episode in range(500):
    state, _ = env.reset(seed=42)
    state_d = discretise_state(state, n_bins)
    total_reward = 0
    done = False
    while not done:
        if np.random.random() < eps:
            action = env.action_space.sample()
        else:
            action = np.argmax(q_table[state_d])
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        next_state_d = discretise_state(next_state, n_bins)
        best_next = np.max(q_table[next_state_d])
        q_table[state_d][action] += lr * (
            reward + gamma * best_next - q_table[state_d][action]
        )
        state_d = next_state_d
        total_reward += reward
    episode_rewards.append(total_reward)
    eps = max(0.01, eps * eps_decay)
    if (episode + 1) % 100 == 0:
        avg = np.mean(episode_rewards[-100:])
        print(f"Episode {episode+1}, Avg Reward: {avg:.1f}, Epsilon: {eps:.3f}")

env.close()

Expected output:

Episode 100, Avg Reward: 35.2, Epsilon: 0.606
Episode 200, Avg Reward: 82.4, Epsilon: 0.368
Episode 300, Avg Reward: 142.7, Epsilon: 0.223
Episode 400, Avg Reward: 195.3, Epsilon: 0.135
Episode 500, Avg Reward: 200.0, Epsilon: 0.082

The agent learns to balance the pole for the maximum 200 steps within 500 episodes. This same Q-learning approach scales to far more complex environments with deep neural networks replacing the Q-table.

Common Errors Beginners Make

1. Setting the Learning Rate Too High

A high learning rate causes Q-values to oscillate or diverge. Start with 0.1 or lower and decay it over time.

2. Neglecting the Discount Factor

Gamma controls how much the agent values future rewards. Gamma too low makes the agent shortsighted. Gamma too close to 1 makes learning slow. Start with 0.95 and adjust based on task horizon.

3. Insufficient Exploration

Without enough exploration, the agent converges on a suboptimal policy. Use epsilon-greedy with decay, or try optimistic initial Q-values to encourage early exploration.

4. Using the Wrong Discretisation

Q-learning with discrete states requires meaningful state bins. Too few bins lose information. Too many bins make learning impractically slow.

5. Forgetting to Normalise Rewards

Policy gradient methods are sensitive to reward scale. Normalising returns across episodes stabilises training and prevents gradient explosion.

6. Ignoring Environment Randomness

Deterministic environments are easier but unrealistic. Always test your agent across multiple random seeds to measure true performance.

7. Expecting Tabular Q-Learning to Scale

Tabular Q-learning works for small discrete state spaces but fails for high-dimensional or continuous problems. Use deep Q-networks (DQN) for complex environments.

Practice Questions

1. What is the exploration-exploitation trade-off in reinforcement learning? The agent must balance trying new actions (exploration) to discover better strategies versus using known high-value actions (exploitation) to maximise reward.

2. How does Q-learning update its value estimates? Q-learning uses the Bellman equation: Q(s,a) = Q(s,a) + lr * (reward + gamma * max Q(s',a') - Q(s,a)). It bootstraps from its own estimates of future rewards.

3. What is the difference between on-policy and off-policy learning? Off-policy methods like Q-learning learn the optimal policy while following a different behaviour policy. On-policy methods learn the same policy they use to act.

Challenge

Implement a Deep Q-Network (DQN) with experience replay for the CartPole environment. Compare its sample efficiency and final performance against tabular Q-learning.

Real-World Task

Train an RL agent to play a simple Atari game (like Pong) using OpenAI Gym and a convolutional neural network. How many episodes does it take to reach human-level performance?

FAQ

What's Next

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.