Reinforcement Learning

On a basic level, Reinforcement Learning involves the iterative interplay between an Agent and an Environment:

Learning Controller - coordinates execution of the learning processes
Environment - the set of elements being trained
Agent - the set of elements causing Environment changes
Actions - processes to alter the Environment
States - numeric quantifiers of Environment aspects
Rewards - elements that cause Environment changes

A More Detailed Architecture

The diagram below shows the Reinforcement Learning architecture at a more detailed level.

Key elements include:

Learning Controller - coordinates execution of the learning processes
Episode - the processing of one set of timesteps
Episode Control - iterates through episodes
Timestep - the processing of one cycle of Agent-Environment interaction
Timestep Control - iterates through timesteps
Environment - the set of elements being trained
Agent - the set of elements causing Environment changes
Episode Timesteps - tracking of episode timesteps
Timestep Terminate - control of termination of a set of timesteps
Actions - processes to alter the Environment
States - numeric quantifiers of Environment aspects
Rewards - elements that cause Environment changes
Observer - monitor and modifier of States and Rewards
Policy - defines Agent Actions for specific States
Algorithm - methodology for updating Policy
Update - changes to Policy

In mathematical terms, it looks like this:

$\\max\left ( S_{o} \right )=C_{i=0}^{e}\left ( C_{i},T_{t=0}^{m} \left ( T_{t},n_{t},s_{t},r_{t},E\left (T_{t},N,S,R,O \right ),A\left (P,U,G \right ) \right )\right ) \\\\\mathrm{\textup{where:}} \\\\S_{o}\,\,\mathrm{\textup{state to be optimized}} \\C\,\,\,\,\mathrm{\textup{episode controller}} \\T\,\,\,\,\mathrm{\textup{timestep control}} \\E\,\,\,\,\mathrm{\textup{environment}} \\A\,\,\,\,\mathrm{\textup{agent}} \\N\,\,\,\mathrm{\textup{actions}} \\S\,\,\,\,\,\mathrm{\textup{states}} \\R\,\,\,\,\mathrm{\textup{rewards}} \\O\,\,\,\,\mathrm{\textup{observer}} \\P\,\,\,\,\mathrm{\textup{policy}} \\U\,\,\,\,\mathrm{\textup{updates}} \\G\,\,\,\,\mathrm{\textup{algorithm}} \\i\,\,\,\,\,\,\,\mathrm{\textup{eposode number}} \\e\,\,\,\,\,\,\mathrm{\textup{eposode end number}} \\m\,\,\,\,\mathrm{\textup{timestep number}} \\n\,\,\,\,\,\,\mathrm{\textup{action number}} \\s\,\,\,\,\,\,\,\mathrm{\textup{states number}} \\r\,\,\,\,\,\,\,\mathrm{\textup{reward number}}$

Python Example

The example uses the OpenAI Gym CartPole environment which trains against 4 state variables:

Cart Position
Cart Velocity
Pole Angle
Pole Velocity at Tip

Values of these state variables are shown below the code.

The objective is to maximize the absolute value of the Cart Position distance from the 0 point while preventing the Cart Pole from falling.

To download the code below, click here.

"""
reinforcement_learning_using_tensorforce.py
trains a model using reinforcement learning
"""

# Import needed libraries.
# TensorFlow and Tensorforce must be installed.
from tensorforce import Agent, Environment

# Define parameters.
environment_type = 'gym'
environment_level = 'CartPole'
environment_max_episode_timesteps = 40
agent_name = 'tensorforce'
agent_memory = 10000
agent_update_unit = 'timesteps'
agent_update_batch_size = 64
agent_optimizer_type = 'adam'
agent_optimizer_learning_rate = 3e-4
agent_policy_network = 'auto'
agent_objective = 'policy_gradient'
agent_reward_estimation_horizon = 20
number_of_episodes = 10

# Instantiate an environment to train.
environment = Environment.create(
    environment=environment_type,
    level=environment_level,
    max_episode_timesteps=environment_max_episode_timesteps
)

# Instantiate an agent for training.
agent = Agent.create(
    agent=agent_name,
    environment=environment,  # alternatively: states, actions, (max_episode_timesteps)
    memory=agent_memory,
    update=dict(unit=agent_update_unit, batch_size=agent_update_batch_size),
    optimizer=dict(type=agent_optimizer_type, learning_rate=agent_optimizer_learning_rate),
    policy=dict(network=agent_policy_network),
    objective=agent_objective,
    reward_estimation=dict(horizon=agent_reward_estimation_horizon)
)

# Train the environment using the agent timesteps within episodes.
best_cart_distance = 0
for episode in range(number_of_episodes):

    # Print the episode number.
    print('states within episode: ' + str(episode))

    # Initialize the environment episode.
    states = environment.reset()
    print(states)

    # Process episode timesteps.
    timesteps = 0
    terminate_timesteps = False
    while not terminate_timesteps:

        # Process an individual timestep.
        timesteps += 1
        actions = agent.act(states=states)
        states, terminate_timesteps, reward = environment.execute(actions=actions)
        agent.observe(terminal=terminate_timesteps, reward=reward)
        print(states)

    # Print the episode summary data.
    print('timesteps: ' + str(timesteps))
    cart_distance = abs(states[0])
    best_cart_distance = max(best_cart_distance, cart_distance)
    print('best cart final timesteps distance of all episodes: ' + str(best_cart_distance))
    print(' ')

# Close the environment and agent.
agent.close()
environment.close()

Results are below:

states within episode: 0
[-0.00026061 -0.01103473  0.04581026 -0.00464354]
[-0.0004813  -0.20678271  0.04571739  0.3021339 ]
[-0.00461695 -0.40252542  0.05176006  0.6088774 ]
[-0.01266746 -0.59833129  0.06393761  0.91740354]
[-0.02463409 -0.79425672  0.08228568  1.22947602]
[-0.04051922 -0.99033531  0.1068752   1.54676344]
[-0.06032593 -1.18656599  0.13781047  1.87079154]
[-0.08405725 -1.38289924  0.1752263   2.20288746]
[-0.11171523 -1.57922116  0.21928405  2.54411426]
timesteps: 8
best cart final timesteps distance of all episodes: 0.11171523316788975

states within episode: 1
[-0.0228279   0.02221037  0.0176807  -0.03375597]
[-0.0223837  -0.17316061  0.01700558  0.26445254]
[-0.02584691 -0.3685211   0.02229463  0.56245031]
[-0.03321733 -0.5639487   0.03354363  0.86207293]
[-0.04449631 -0.75951094  0.05078509  1.16511126]
[-0.05968652 -0.95525584  0.07408732  1.47327445]
[-0.07879164 -1.15120113  0.10355281  1.78814786]
[-0.10181566 -1.3473218   0.13931576  2.11114314]
[-0.1287621  -1.54353531  0.18153863  2.44343828]
[-0.15963281 -1.73968452  0.23040739  2.78590681]
timesteps: 9
best cart final timesteps distance of all episodes: 0.1596328054527629

states within episode: 2
[ 0.04953899 -0.03442587  0.02211645 -0.00488439]
[ 0.04885047  0.16037203  0.02201876 -0.29050808]
[ 0.05205792  0.3551732   0.0162086  -0.57616602]
[ 0.05916138  0.55006424  0.00468528 -0.86369906]
[ 0.07016266  0.7451221  -0.0125887  -1.15490516]
[ 0.08506511  0.94040593 -0.0356868  -1.45150867]
[ 0.10387323  1.13594769 -0.06471698 -1.75512426]
[ 0.12659218  1.33174091 -0.09981946 -2.06721278]
[ 0.153227    1.52772705 -0.14116372 -2.38902682]
[ 0.18378154  1.72377931 -0.18894425 -2.72154443]
[ 0.21825712  1.91968406 -0.24337514 -3.06539148]
timesteps: 10
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 3
[-0.03586108 -0.01346205 -0.02185208  0.01710818]
[-0.03613032 -0.20826391 -0.02150992  0.30281721]
[-0.0402956  -0.4030728  -0.01545357  0.58863952]
[-0.04835705 -0.59797498 -0.00368078  0.87641471]
[-0.06031655 -0.79304671  0.01384751  1.16793817]
[-0.07617749 -0.98834606  0.03720628  1.46493015]
[-0.09594441 -1.18390343  0.06650488  1.76899932]
[-0.11962248 -1.37971019  0.10188487  2.08159819]
[-0.14721668 -1.57570491  0.14351683  2.40396805]
[-0.17873078 -1.77175712  0.19159619  2.73707224]
[-0.21416592 -1.96764853  0.24633763  3.08151786]
timesteps: 10
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 4
[ 0.00462236  0.04730914  0.02341448 -0.02013573]
[ 0.00556854 -0.14814064  0.02301176  0.2798418 ]
[ 0.00260573 -0.34358317  0.0286086   0.57969284]
[-0.00426594 -0.5390941   0.04020245  0.88124901]
[-0.01504782 -0.73473842  0.05782743  1.1862947 ]
[-0.02974259 -0.93056064  0.08155333  1.49652884]
[-0.0483538  -1.12657383  0.1114839   1.81352152]
[-0.07088528 -1.32274677  0.14775433  2.13866262]
[-0.09734021 -1.5189889   0.19052759  2.47310036]
[-0.12771999 -1.71513291  0.23998959  2.81766921]
timesteps: 9
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 5
[-0.01108862  0.01700961 -0.0420929  -0.03565462]
[-0.01074843  0.21270912 -0.04280599 -0.34131552]
[-0.00649424  0.40841313 -0.0496323  -0.64718376]
[ 0.00167402  0.60419018 -0.06257598 -0.95507361]
[ 0.01375782  0.80009551 -0.08167745 -1.26674179]
[ 0.02975973  0.99616042 -0.10701229 -1.58384518]
[ 0.04968294  1.1923802  -0.13868919 -1.90789279]
[ 0.07353055  1.38870027 -0.17684705 -2.24018934]
[ 0.10130455  1.58500002 -0.22165083 -2.58176896]
timesteps: 8
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 6
[0.00122252 0.00248242 0.02053107 0.0383462 ]
[ 0.00127217 -0.19292784  0.021298    0.33743553]
[-0.00258638 -0.38834629  0.02804671  0.63675786]
[-0.01035331 -0.58384792  0.04078186  0.93813962]
[-0.02203027 -0.77949529  0.05954466  1.24335321]
[-0.03762017 -0.97532868  0.08441172  1.55407849]
[-0.05712675 -1.17135486  0.11549329  1.8718584 ]
[-0.08055385 -1.36753392  0.15293046  2.19804618]
[-0.10790452 -1.56376384  0.19689138  2.53374224]
[-0.1391798  -1.75986277  0.24756623  2.87972029]
timesteps: 9
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 7
[ 0.00587895 -0.04401325 -0.03611816 -0.00374312]
[ 0.00499869  0.15160758 -0.03619302 -0.30759942]
[ 0.00803084  0.34722605 -0.04234501 -0.61147339]
[ 0.01497536  0.54291347 -0.05457447 -0.9171871 ]
[ 0.02583363  0.73872914 -0.07291822 -1.22651024]
[ 0.04060821  0.93471011 -0.09744842 -1.54111947]
[ 0.05930242  1.13085948 -0.12827081 -1.86255214]
[ 0.0819196   1.32713296 -0.16552185 -2.19215195]
[ 0.10846226  1.52342322 -0.20936489 -2.53100466]
[ 0.13893073  1.71954197 -0.25998499 -2.87986343]
timesteps: 9
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 8
[-0.04839189 -0.01577572  0.04301474  0.02458942]
[-0.0487074  -0.2114873   0.04350652  0.33052768]
[-0.05293715 -0.40720068  0.05011708  0.63660685]
[-0.06108116 -0.60298442  0.06284921  0.94464197]
[-0.07314085 -0.79889413  0.08174205  1.25639184]
[-0.08911873 -0.99496177  0.10686989  1.57351671]
[-0.10901797 -1.19118373  0.13834022  1.89753041]
[-0.13284164 -1.38750688  0.17629083  2.22974412]
[-0.16059178 -1.58381257  0.22088571  2.57120028]
timesteps: 8
best cart final timesteps distance of all episodes: 0.21825712417700763

states within episode: 9
[-0.02113959  0.01149438 -0.02149935  0.03416971]
[-0.0209097  -0.18331277 -0.02081595  0.31999259]
[-0.02457595 -0.37813218 -0.0144161   0.60603895]
[-0.0321386  -0.57304962 -0.00229532  0.89414653]
[-0.04359959 -0.76814037  0.01558761  1.18610705]
[-0.0589624  -0.96346096  0.03930975  1.48363493]
[-0.07823162 -1.1590396   0.06898245  1.78833033]
[-0.10141241 -1.35486458  0.10474905  2.10163396]
[-0.1285097  -1.55087048  0.14678173  2.42477123]
[-0.15952711 -1.7469216   0.19527716  2.75868472]
[-0.19446554 -1.94279316  0.25045085  3.10395523]
timesteps: 10
best cart final timesteps distance of all episodes: 0.21825712417700763

Reinforcement Learning

A More Detailed Architecture

Python Example

References