{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import gym\n",
"import numpy as np\n",
"import random\n",
"import math\n",
"from time import sleep\n",
"from IPython import display\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"\n",
"\n",
"env = gym.make('CartPole-v0')\n",
"\n",
"low = env.observation_space.low\n",
"high=env.observation_space.high\n",
"\n",
"input_dim = env.observation_space.shape[0]\n",
"output_dim = env.action_space.n\n",
"\n",
"discount_factor = 0.99 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import Keras and define the neural network"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In policy gradient, the neural network outputs a softmax probability distribution for each action available, i.e. $\\pi_{\\theta}(a|s)$ where $\\theta$ represents the weights of the neural network.\n",
"Remember that the cost function for policy gradient is the expected return over a trajectory $\\tau$:\n",
"$$ \\bar{R}=\\sum_{\\tau} P_{\\theta}(\\tau) R(\\tau) $$\n",
"where $P_{\\theta}(\\tau$ is the probability of a certain trajectory and $R(\\tau)$ is the associated cumulative reward for that particular trajectory.\n",
"Policy gradient theorem states that if we update the weights of the neural direction following:\n",
"$$ \\Delta \\theta= \\eta \\frac{\\partial \\bar{R} }{\\partial \\theta} $$\n",
"where:\n",
"$$ \\frac{\\partial \\bar{R} }{\\partial \\theta} = \\sum_t \\mathbb{E} \\left[ R \\frac{\\partial \\ln \\pi_{\\theta} (a_t|s_t)}{\\partial \\theta}\\right]$$ \n",
"
\n",
"To implement all of this in Keras, we can use the \"categorial cross-entropy trick\":\n",
"\n",
"
\n",
"Basically, for the training, we need all the state of an episode and the cumulative rewards **starting** from the state onwards, $G_t$:\n",
"$$G_t=\\sum_{t'=t}\\gamma^{t'} r_{t'}$$\n",
"Then, the corresponding label for each timestep t:\n",
"$$ x[t]=s_t$$\n",
" \n",
"$$ y[t]=\\begin{cases} G_t & \\mbox{for each action taken at time }t \\\\ 0 & \\mbox{for every other action}\\end{cases}$$\n",
"\n",
"With this choice, the cost function $C=-\\sum_a P(a) \\ln \\pi_{\\theta} (a|s)$ becomes:\n",
"$$C=- G_t \\ln \\pi_{\\theta} (a_t|s_t) $$\n",
"which reproduces the right policy gradient update rule."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
}
],
"source": [
"import keras as K\n",
"\n",
"model = K.Sequential()\n",
"model.add(K.layers.Dense(64, input_dim=input_dim, activation='relu'))\n",
"model.add(K.layers.Dense(32, activation='relu'))\n",
"model.add(K.layers.Dense(output_dim, activation='softmax'))\n",
"\n",
"model.compile(loss=\"categorical_crossentropy\", optimizer=K.optimizers.Adam())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You need to define a neural network with two hidden layers of respectively 64 and 32 neurons, while the last layer must have a _softmax_ activation function."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Policy gradient algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You only need to compute the discounted_rewards list, which contains all the discounted cumulative rewards ($G_t$) of an episode for each timestep. \n",
"**Example** discounted_rewards=$[..., 1+0.99 \\times1+(0.99^2) \\times 1, 1+0.99 \\times 1, 1]$\n",
" \n",
"**Note** To reverse a list, you can simply do list.reverse()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def policy_gradient(states, actions, rewards):\n",
" episode_length = len(states)\n",
"\n",
" discounted_rewards = []\n",
" discounted_rewards.append(rewards[0])\n",
" for t in range(len(rewards)-1):\n",
" discounted_rewards.append( pow(discount_factor,t+1)*rewards[t]+discounted_rewards[t] )\n",
" discounted_rewards.reverse()\n",
"\n",
" x = np.zeros((episode_length, input_dim))\n",
" y = np.zeros((episode_length, output_dim))\n",
"\n",
" for i in range(episode_length):\n",
" x[i] = states[i]\n",
" y[i][actions[i]] = discounted_rewards[i]\n",
"\n",
" model.fit(x, y, epochs=1, verbose=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Rule for selecting an action"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_np.random.choice()_ is a useful function to choose among different options, according to a probability distribution $p$."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def select_action(state):\n",
" policy = model.predict(state.reshape(1,input_dim)).flatten()\n",
" return np.random.choice(output_dim, 1, p=policy)[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simulation"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Episode: 0 Total reward: 9.0\n",
"Episode: 1 Total reward: 22.0\n",
"Episode: 2 Total reward: 11.0\n",
"Episode: 3 Total reward: 18.0\n",
"Episode: 4 Total reward: 34.0\n",
"Episode: 5 Total reward: 21.0\n",
"Episode: 6 Total reward: 11.0\n",
"Episode: 7 Total reward: 11.0\n",
"Episode: 8 Total reward: 13.0\n",
"Episode: 9 Total reward: 11.0\n",
"Episode: 10 Total reward: 10.0\n",
"Episode: 11 Total reward: 26.0\n",
"Episode: 12 Total reward: 15.0\n",
"Episode: 13 Total reward: 17.0\n",
"Episode: 14 Total reward: 14.0\n",
"Episode: 15 Total reward: 15.0\n",
"Episode: 16 Total reward: 11.0\n",
"Episode: 17 Total reward: 12.0\n",
"Episode: 18 Total reward: 11.0\n",
"Episode: 19 Total reward: 13.0\n",
"Episode: 20 Total reward: 19.0\n",
"Episode: 21 Total reward: 23.0\n",
"Episode: 22 Total reward: 18.0\n",
"Episode: 23 Total reward: 9.0\n",
"Episode: 24 Total reward: 27.0\n",
"Episode: 25 Total reward: 19.0\n",
"Episode: 26 Total reward: 21.0\n",
"Episode: 27 Total reward: 51.0\n",
"Episode: 28 Total reward: 13.0\n",
"Episode: 29 Total reward: 16.0\n",
"Episode: 30 Total reward: 27.0\n",
"Episode: 31 Total reward: 28.0\n",
"Episode: 32 Total reward: 9.0\n",
"Episode: 33 Total reward: 18.0\n",
"Episode: 34 Total reward: 10.0\n",
"Episode: 35 Total reward: 14.0\n",
"Episode: 36 Total reward: 14.0\n",
"Episode: 37 Total reward: 33.0\n",
"Episode: 38 Total reward: 32.0\n",
"Episode: 39 Total reward: 13.0\n",
"Episode: 40 Total reward: 14.0\n",
"Episode: 41 Total reward: 13.0\n",
"Episode: 42 Total reward: 8.0\n",
"Episode: 43 Total reward: 14.0\n",
"Episode: 44 Total reward: 27.0\n",
"Episode: 45 Total reward: 21.0\n",
"Episode: 46 Total reward: 24.0\n",
"Episode: 47 Total reward: 20.0\n",
"Episode: 48 Total reward: 34.0\n",
"Episode: 49 Total reward: 17.0\n",
"Episode: 50 Total reward: 35.0\n",
"Episode: 51 Total reward: 15.0\n",
"Episode: 52 Total reward: 11.0\n",
"Episode: 53 Total reward: 16.0\n",
"Episode: 54 Total reward: 17.0\n",
"Episode: 55 Total reward: 11.0\n",
"Episode: 56 Total reward: 49.0\n",
"Episode: 57 Total reward: 24.0\n",
"Episode: 58 Total reward: 10.0\n",
"Episode: 59 Total reward: 12.0\n",
"Episode: 60 Total reward: 56.0\n",
"Episode: 61 Total reward: 13.0\n",
"Episode: 62 Total reward: 20.0\n",
"Episode: 63 Total reward: 17.0\n",
"Episode: 64 Total reward: 10.0\n",
"Episode: 65 Total reward: 16.0\n",
"Episode: 66 Total reward: 20.0\n",
"Episode: 67 Total reward: 10.0\n",
"Episode: 68 Total reward: 36.0\n",
"Episode: 69 Total reward: 14.0\n",
"Episode: 70 Total reward: 54.0\n",
"Episode: 71 Total reward: 19.0\n",
"Episode: 72 Total reward: 11.0\n",
"Episode: 73 Total reward: 12.0\n",
"Episode: 74 Total reward: 51.0\n",
"Episode: 75 Total reward: 16.0\n",
"Episode: 76 Total reward: 19.0\n",
"Episode: 77 Total reward: 30.0\n",
"Episode: 78 Total reward: 10.0\n",
"Episode: 79 Total reward: 21.0\n",
"Episode: 80 Total reward: 23.0\n",
"Episode: 81 Total reward: 29.0\n",
"Episode: 82 Total reward: 17.0\n",
"Episode: 83 Total reward: 23.0\n",
"Episode: 84 Total reward: 12.0\n",
"Episode: 85 Total reward: 30.0\n",
"Episode: 86 Total reward: 14.0\n",
"Episode: 87 Total reward: 19.0\n",
"Episode: 88 Total reward: 14.0\n",
"Episode: 89 Total reward: 14.0\n",
"Episode: 90 Total reward: 23.0\n",
"Episode: 91 Total reward: 16.0\n",
"Episode: 92 Total reward: 13.0\n",
"Episode: 93 Total reward: 11.0\n",
"Episode: 94 Total reward: 15.0\n",
"Episode: 95 Total reward: 14.0\n",
"Episode: 96 Total reward: 29.0\n",
"Episode: 97 Total reward: 36.0\n",
"Episode: 98 Total reward: 13.0\n",
"Episode: 99 Total reward: 16.0\n",
"Episode: 100 Total reward: 33.0\n",
"Episode: 101 Total reward: 21.0\n",
"Episode: 102 Total reward: 22.0\n",
"Episode: 103 Total reward: 48.0\n",
"Episode: 104 Total reward: 20.0\n",
"Episode: 105 Total reward: 9.0\n",
"Episode: 106 Total reward: 12.0\n",
"Episode: 107 Total reward: 18.0\n",
"Episode: 108 Total reward: 19.0\n",
"Episode: 109 Total reward: 16.0\n",
"Episode: 110 Total reward: 11.0\n",
"Episode: 111 Total reward: 20.0\n",
"Episode: 112 Total reward: 21.0\n",
"Episode: 113 Total reward: 26.0\n",
"Episode: 114 Total reward: 13.0\n",
"Episode: 115 Total reward: 12.0\n",
"Episode: 116 Total reward: 22.0\n",
"Episode: 117 Total reward: 50.0\n",
"Episode: 118 Total reward: 17.0\n",
"Episode: 119 Total reward: 59.0\n",
"Episode: 120 Total reward: 13.0\n",
"Episode: 121 Total reward: 23.0\n",
"Episode: 122 Total reward: 14.0\n",
"Episode: 123 Total reward: 14.0\n",
"Episode: 124 Total reward: 32.0\n",
"Episode: 125 Total reward: 26.0\n",
"Episode: 126 Total reward: 22.0\n",
"Episode: 127 Total reward: 19.0\n",
"Episode: 128 Total reward: 19.0\n",
"Episode: 129 Total reward: 17.0\n",
"Episode: 130 Total reward: 18.0\n",
"Episode: 131 Total reward: 12.0\n",
"Episode: 132 Total reward: 14.0\n",
"Episode: 133 Total reward: 40.0\n",
"Episode: 134 Total reward: 19.0\n",
"Episode: 135 Total reward: 32.0\n",
"Episode: 136 Total reward: 31.0\n",
"Episode: 137 Total reward: 22.0\n",
"Episode: 138 Total reward: 12.0\n",
"Episode: 139 Total reward: 23.0\n",
"Episode: 140 Total reward: 16.0\n",
"Episode: 141 Total reward: 23.0\n",
"Episode: 142 Total reward: 20.0\n",
"Episode: 143 Total reward: 20.0\n",
"Episode: 144 Total reward: 25.0\n",
"Episode: 145 Total reward: 18.0\n",
"Episode: 146 Total reward: 16.0\n",
"Episode: 147 Total reward: 10.0\n",
"Episode: 148 Total reward: 16.0\n",
"Episode: 149 Total reward: 15.0\n",
"Episode: 150 Total reward: 21.0\n",
"Episode: 151 Total reward: 31.0\n",
"Episode: 152 Total reward: 34.0\n",
"Episode: 153 Total reward: 21.0\n",
"Episode: 154 Total reward: 18.0\n",
"Episode: 155 Total reward: 19.0\n",
"Episode: 156 Total reward: 18.0\n",
"Episode: 157 Total reward: 19.0\n",
"Episode: 158 Total reward: 18.0\n",
"Episode: 159 Total reward: 26.0\n",
"Episode: 160 Total reward: 22.0\n",
"Episode: 161 Total reward: 20.0\n",
"Episode: 162 Total reward: 13.0\n",
"Episode: 163 Total reward: 17.0\n",
"Episode: 164 Total reward: 12.0\n",
"Episode: 165 Total reward: 23.0\n",
"Episode: 166 Total reward: 51.0\n",
"Episode: 167 Total reward: 13.0\n",
"Episode: 168 Total reward: 16.0\n",
"Episode: 169 Total reward: 9.0\n",
"Episode: 170 Total reward: 19.0\n",
"Episode: 171 Total reward: 12.0\n",
"Episode: 172 Total reward: 24.0\n",
"Episode: 173 Total reward: 15.0\n",
"Episode: 174 Total reward: 38.0\n",
"Episode: 175 Total reward: 25.0\n",
"Episode: 176 Total reward: 30.0\n",
"Episode: 177 Total reward: 12.0\n",
"Episode: 178 Total reward: 20.0\n",
"Episode: 179 Total reward: 28.0\n",
"Episode: 180 Total reward: 25.0\n",
"Episode: 181 Total reward: 11.0\n",
"Episode: 182 Total reward: 12.0\n",
"Episode: 183 Total reward: 35.0\n",
"Episode: 184 Total reward: 27.0\n",
"Episode: 185 Total reward: 41.0\n",
"Episode: 186 Total reward: 40.0\n",
"Episode: 187 Total reward: 44.0\n",
"Episode: 188 Total reward: 25.0\n",
"Episode: 189 Total reward: 112.0\n",
"Episode: 190 Total reward: 23.0\n",
"Episode: 191 Total reward: 18.0\n",
"Episode: 192 Total reward: 19.0\n",
"Episode: 193 Total reward: 32.0\n",
"Episode: 194 Total reward: 22.0\n",
"Episode: 195 Total reward: 25.0\n",
"Episode: 196 Total reward: 59.0\n",
"Episode: 197 Total reward: 19.0\n",
"Episode: 198 Total reward: 50.0\n",
"Episode: 199 Total reward: 20.0\n",
"Episode: 200 Total reward: 13.0\n",
"Episode: 201 Total reward: 27.0\n",
"Episode: 202 Total reward: 24.0\n",
"Episode: 203 Total reward: 13.0\n",
"Episode: 204 Total reward: 30.0\n",
"Episode: 205 Total reward: 17.0\n",
"Episode: 206 Total reward: 15.0\n",
"Episode: 207 Total reward: 27.0\n",
"Episode: 208 Total reward: 61.0\n",
"Episode: 209 Total reward: 32.0\n",
"Episode: 210 Total reward: 48.0\n",
"Episode: 211 Total reward: 45.0\n",
"Episode: 212 Total reward: 49.0\n",
"Episode: 213 Total reward: 44.0\n",
"Episode: 214 Total reward: 75.0\n",
"Episode: 215 Total reward: 30.0\n",
"Episode: 216 Total reward: 17.0\n",
"Episode: 217 Total reward: 54.0\n",
"Episode: 218 Total reward: 47.0\n",
"Episode: 219 Total reward: 34.0\n",
"Episode: 220 Total reward: 25.0\n",
"Episode: 221 Total reward: 31.0\n",
"Episode: 222 Total reward: 23.0\n",
"Episode: 223 Total reward: 63.0\n",
"Episode: 224 Total reward: 16.0\n",
"Episode: 225 Total reward: 27.0\n",
"Episode: 226 Total reward: 37.0\n",
"Episode: 227 Total reward: 20.0\n",
"Episode: 228 Total reward: 26.0\n",
"Episode: 229 Total reward: 28.0\n",
"Episode: 230 Total reward: 35.0\n",
"Episode: 231 Total reward: 96.0\n",
"Episode: 232 Total reward: 40.0\n",
"Episode: 233 Total reward: 39.0\n",
"Episode: 234 Total reward: 32.0\n",
"Episode: 235 Total reward: 21.0\n",
"Episode: 236 Total reward: 46.0\n",
"Episode: 237 Total reward: 22.0\n",
"Episode: 238 Total reward: 28.0\n",
"Episode: 239 Total reward: 54.0\n",
"Episode: 240 Total reward: 43.0\n",
"Episode: 241 Total reward: 23.0\n",
"Episode: 242 Total reward: 57.0\n",
"Episode: 243 Total reward: 22.0\n",
"Episode: 244 Total reward: 26.0\n",
"Episode: 245 Total reward: 15.0\n",
"Episode: 246 Total reward: 69.0\n",
"Episode: 247 Total reward: 31.0\n",
"Episode: 248 Total reward: 82.0\n",
"Episode: 249 Total reward: 30.0\n",
"Episode: 250 Total reward: 47.0\n",
"Episode: 251 Total reward: 107.0\n",
"Episode: 252 Total reward: 38.0\n",
"Episode: 253 Total reward: 126.0\n",
"Episode: 254 Total reward: 61.0\n",
"Episode: 255 Total reward: 76.0\n",
"Episode: 256 Total reward: 18.0\n",
"Episode: 257 Total reward: 176.0\n",
"Episode: 258 Total reward: 26.0\n",
"Episode: 259 Total reward: 155.0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Episode: 260 Total reward: 174.0\n",
"Episode: 261 Total reward: 158.0\n",
"Episode: 262 Total reward: 39.0\n",
"Episode: 263 Total reward: 200.0\n",
"Episode: 264 Total reward: 106.0\n",
"Episode: 265 Total reward: 87.0\n",
"Episode: 266 Total reward: 155.0\n",
"Episode: 267 Total reward: 64.0\n",
"Episode: 268 Total reward: 149.0\n",
"Episode: 269 Total reward: 122.0\n",
"Episode: 270 Total reward: 45.0\n",
"Episode: 271 Total reward: 85.0\n",
"Episode: 272 Total reward: 200.0\n",
"Episode: 273 Total reward: 115.0\n",
"Episode: 274 Total reward: 184.0\n",
"Episode: 275 Total reward: 28.0\n",
"Episode: 276 Total reward: 48.0\n",
"Episode: 277 Total reward: 151.0\n",
"Episode: 278 Total reward: 114.0\n",
"Episode: 279 Total reward: 34.0\n",
"Episode: 280 Total reward: 177.0\n",
"Episode: 281 Total reward: 74.0\n",
"Episode: 282 Total reward: 200.0\n",
"Episode: 283 Total reward: 200.0\n",
"Episode: 284 Total reward: 62.0\n",
"Episode: 285 Total reward: 36.0\n",
"Episode: 286 Total reward: 112.0\n",
"Episode: 287 Total reward: 111.0\n",
"Episode: 288 Total reward: 84.0\n",
"Episode: 289 Total reward: 110.0\n",
"Episode: 290 Total reward: 114.0\n",
"Episode: 291 Total reward: 155.0\n",
"Episode: 292 Total reward: 70.0\n",
"Episode: 293 Total reward: 63.0\n",
"Episode: 294 Total reward: 131.0\n",
"Episode: 295 Total reward: 200.0\n",
"Episode: 296 Total reward: 200.0\n",
"Episode: 297 Total reward: 190.0\n",
"Episode: 298 Total reward: 200.0\n",
"Episode: 299 Total reward: 200.0\n",
"Episode: 300 Total reward: 97.0\n",
"Episode: 301 Total reward: 104.0\n",
"Episode: 302 Total reward: 200.0\n",
"Episode: 303 Total reward: 174.0\n",
"Episode: 304 Total reward: 200.0\n",
"Episode: 305 Total reward: 191.0\n",
"Episode: 306 Total reward: 132.0\n",
"Episode: 307 Total reward: 111.0\n",
"Episode: 308 Total reward: 95.0\n",
"Episode: 309 Total reward: 183.0\n",
"Episode: 310 Total reward: 29.0\n",
"Episode: 311 Total reward: 188.0\n",
"Episode: 312 Total reward: 126.0\n",
"Episode: 313 Total reward: 108.0\n",
"Episode: 314 Total reward: 180.0\n",
"Episode: 315 Total reward: 200.0\n",
"Episode: 316 Total reward: 200.0\n",
"Episode: 317 Total reward: 197.0\n",
"Episode: 318 Total reward: 200.0\n",
"Episode: 319 Total reward: 166.0\n",
"Episode: 320 Total reward: 200.0\n",
"Episode: 321 Total reward: 200.0\n",
"Episode: 322 Total reward: 118.0\n",
"Episode: 323 Total reward: 183.0\n",
"Episode: 324 Total reward: 200.0\n",
"Episode: 325 Total reward: 118.0\n",
"Episode: 326 Total reward: 200.0\n",
"Episode: 327 Total reward: 200.0\n",
"Episode: 328 Total reward: 200.0\n",
"Episode: 329 Total reward: 200.0\n",
"Episode: 330 Total reward: 151.0\n",
"Episode: 331 Total reward: 200.0\n",
"Episode: 332 Total reward: 200.0\n",
"Episode: 333 Total reward: 200.0\n",
"Episode: 334 Total reward: 75.0\n",
"Episode: 335 Total reward: 200.0\n",
"Episode: 336 Total reward: 47.0\n",
"Episode: 337 Total reward: 200.0\n",
"Episode: 338 Total reward: 200.0\n",
"Episode: 339 Total reward: 200.0\n",
"Episode: 340 Total reward: 200.0\n",
"Episode: 341 Total reward: 200.0\n",
"Episode: 342 Total reward: 200.0\n",
"Episode: 343 Total reward: 128.0\n",
"Episode: 344 Total reward: 200.0\n",
"Episode: 345 Total reward: 125.0\n",
"Episode: 346 Total reward: 162.0\n",
"Episode: 347 Total reward: 189.0\n",
"Episode: 348 Total reward: 166.0\n",
"Episode: 349 Total reward: 172.0\n",
"Episode: 350 Total reward: 200.0\n",
"Episode: 351 Total reward: 200.0\n",
"Episode: 352 Total reward: 200.0\n",
"Episode: 353 Total reward: 200.0\n",
"Episode: 354 Total reward: 200.0\n",
"Episode: 355 Total reward: 200.0\n",
"Episode: 356 Total reward: 200.0\n",
"Episode: 357 Total reward: 200.0\n",
"Episode: 358 Total reward: 200.0\n",
"Episode: 359 Total reward: 200.0\n",
"Episode: 360 Total reward: 200.0\n",
"Episode: 361 Total reward: 200.0\n",
"Episode: 362 Total reward: 200.0\n",
"Episode: 363 Total reward: 200.0\n",
"Episode: 364 Total reward: 200.0\n",
"Episode: 365 Total reward: 200.0\n",
"Episode: 366 Total reward: 200.0\n",
"Episode: 367 Total reward: 200.0\n",
"Episode: 368 Total reward: 200.0\n",
"Episode: 369 Total reward: 198.0\n",
"Episode: 370 Total reward: 121.0\n",
"Episode: 371 Total reward: 167.0\n",
"Episode: 372 Total reward: 200.0\n",
"Episode: 373 Total reward: 115.0\n",
"Episode: 374 Total reward: 103.0\n",
"Episode: 375 Total reward: 168.0\n",
"Episode: 376 Total reward: 200.0\n",
"Episode: 377 Total reward: 189.0\n",
"Episode: 378 Total reward: 194.0\n",
"Episode: 379 Total reward: 200.0\n",
"Episode: 380 Total reward: 200.0\n",
"Episode: 381 Total reward: 200.0\n",
"Episode: 382 Total reward: 200.0\n",
"Episode: 383 Total reward: 111.0\n",
"Episode: 384 Total reward: 95.0\n",
"Episode: 385 Total reward: 110.0\n",
"Episode: 386 Total reward: 103.0\n",
"Episode: 387 Total reward: 200.0\n",
"Episode: 388 Total reward: 200.0\n",
"Episode: 389 Total reward: 132.0\n",
"Episode: 390 Total reward: 186.0\n",
"Episode: 391 Total reward: 130.0\n",
"Episode: 392 Total reward: 127.0\n",
"Episode: 393 Total reward: 115.0\n",
"Episode: 394 Total reward: 123.0\n",
"Episode: 395 Total reward: 134.0\n",
"Episode: 396 Total reward: 103.0\n",
"Episode: 397 Total reward: 108.0\n",
"Episode: 398 Total reward: 129.0\n",
"Episode: 399 Total reward: 37.0\n",
"Episode: 400 Total reward: 120.0\n",
"Episode: 401 Total reward: 112.0\n",
"Episode: 402 Total reward: 61.0\n",
"Episode: 403 Total reward: 48.0\n",
"Episode: 404 Total reward: 50.0\n",
"Episode: 405 Total reward: 111.0\n",
"Episode: 406 Total reward: 107.0\n",
"Episode: 407 Total reward: 124.0\n",
"Episode: 408 Total reward: 36.0\n",
"Episode: 409 Total reward: 107.0\n",
"Episode: 410 Total reward: 57.0\n",
"Episode: 411 Total reward: 45.0\n",
"Episode: 412 Total reward: 109.0\n",
"Episode: 413 Total reward: 51.0\n",
"Episode: 414 Total reward: 113.0\n",
"Episode: 415 Total reward: 39.0\n",
"Episode: 416 Total reward: 127.0\n",
"Episode: 417 Total reward: 43.0\n",
"Episode: 418 Total reward: 115.0\n",
"Episode: 419 Total reward: 110.0\n",
"Episode: 420 Total reward: 118.0\n",
"Episode: 421 Total reward: 120.0\n",
"Episode: 422 Total reward: 133.0\n",
"Episode: 423 Total reward: 139.0\n",
"Episode: 424 Total reward: 125.0\n",
"Episode: 425 Total reward: 115.0\n",
"Episode: 426 Total reward: 126.0\n",
"Episode: 427 Total reward: 138.0\n",
"Episode: 428 Total reward: 174.0\n",
"Episode: 429 Total reward: 150.0\n",
"Episode: 430 Total reward: 159.0\n",
"Episode: 431 Total reward: 139.0\n",
"Episode: 432 Total reward: 129.0\n",
"Episode: 433 Total reward: 95.0\n",
"Episode: 434 Total reward: 131.0\n",
"Episode: 435 Total reward: 124.0\n",
"Episode: 436 Total reward: 158.0\n",
"Episode: 437 Total reward: 142.0\n",
"Episode: 438 Total reward: 136.0\n",
"Episode: 439 Total reward: 200.0\n",
"Episode: 440 Total reward: 200.0\n",
"Episode: 441 Total reward: 137.0\n",
"Episode: 442 Total reward: 125.0\n",
"Episode: 443 Total reward: 200.0\n",
"Episode: 444 Total reward: 200.0\n",
"Episode: 445 Total reward: 200.0\n",
"Episode: 446 Total reward: 200.0\n",
"Episode: 447 Total reward: 200.0\n",
"Episode: 448 Total reward: 200.0\n",
"Episode: 449 Total reward: 200.0\n",
"Episode: 450 Total reward: 200.0\n",
"Episode: 451 Total reward: 200.0\n",
"Episode: 452 Total reward: 200.0\n",
"Episode: 453 Total reward: 200.0\n",
"Episode: 454 Total reward: 200.0\n",
"Episode: 455 Total reward: 200.0\n",
"Episode: 456 Total reward: 200.0\n",
"Episode: 457 Total reward: 200.0\n",
"Episode: 458 Total reward: 200.0\n",
"Episode: 459 Total reward: 200.0\n",
"Episode: 460 Total reward: 200.0\n",
"Episode: 461 Total reward: 142.0\n",
"Episode: 462 Total reward: 106.0\n",
"Episode: 463 Total reward: 200.0\n",
"Episode: 464 Total reward: 113.0\n",
"Episode: 465 Total reward: 200.0\n",
"Episode: 466 Total reward: 200.0\n",
"Episode: 467 Total reward: 33.0\n",
"Episode: 468 Total reward: 144.0\n",
"Episode: 469 Total reward: 200.0\n",
"Episode: 470 Total reward: 200.0\n",
"Episode: 471 Total reward: 200.0\n",
"Episode: 472 Total reward: 200.0\n",
"Episode: 473 Total reward: 146.0\n",
"Episode: 474 Total reward: 200.0\n",
"Episode: 475 Total reward: 200.0\n",
"Episode: 476 Total reward: 200.0\n",
"Episode: 477 Total reward: 130.0\n",
"Episode: 478 Total reward: 200.0\n",
"Episode: 479 Total reward: 200.0\n",
"Episode: 480 Total reward: 140.0\n",
"Episode: 481 Total reward: 200.0\n",
"Episode: 482 Total reward: 200.0\n",
"Episode: 483 Total reward: 200.0\n",
"Episode: 484 Total reward: 200.0\n",
"Episode: 485 Total reward: 200.0\n",
"Episode: 486 Total reward: 200.0\n",
"Episode: 487 Total reward: 200.0\n",
"Episode: 488 Total reward: 200.0\n",
"Episode: 489 Total reward: 200.0\n",
"Episode: 490 Total reward: 200.0\n",
"Episode: 491 Total reward: 200.0\n",
"Episode: 492 Total reward: 200.0\n",
"Episode: 493 Total reward: 200.0\n",
"Episode: 494 Total reward: 200.0\n",
"Episode: 495 Total reward: 200.0\n",
"Episode: 496 Total reward: 200.0\n",
"Episode: 497 Total reward: 200.0\n",
"Episode: 498 Total reward: 200.0\n",
"Episode: 499 Total reward: 200.0\n",
"Episode: 500 Total reward: 200.0\n",
"Episode: 501 Total reward: 200.0\n",
"Episode: 502 Total reward: 200.0\n",
"Episode: 503 Total reward: 200.0\n",
"Episode: 504 Total reward: 108.0\n",
"Episode: 505 Total reward: 200.0\n",
"Episode: 506 Total reward: 200.0\n",
"Episode: 507 Total reward: 200.0\n",
"Episode: 508 Total reward: 200.0\n",
"Episode: 509 Total reward: 200.0\n",
"Episode: 510 Total reward: 200.0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Episode: 511 Total reward: 200.0\n",
"Episode: 512 Total reward: 200.0\n",
"Episode: 513 Total reward: 200.0\n",
"Episode: 514 Total reward: 200.0\n",
"Episode: 515 Total reward: 200.0\n",
"Episode: 516 Total reward: 125.0\n",
"Episode: 517 Total reward: 200.0\n",
"Episode: 518 Total reward: 200.0\n",
"Episode: 519 Total reward: 200.0\n",
"Episode: 520 Total reward: 200.0\n",
"Episode: 521 Total reward: 200.0\n",
"Episode: 522 Total reward: 200.0\n",
"Episode: 523 Total reward: 200.0\n",
"Episode: 524 Total reward: 200.0\n",
"Episode: 525 Total reward: 200.0\n",
"Episode: 526 Total reward: 200.0\n",
"Episode: 527 Total reward: 200.0\n",
"Episode: 528 Total reward: 200.0\n",
"Episode: 529 Total reward: 200.0\n",
"Episode: 530 Total reward: 200.0\n",
"Episode: 531 Total reward: 200.0\n",
"Episode: 532 Total reward: 107.0\n",
"Episode: 533 Total reward: 200.0\n",
"Episode: 534 Total reward: 200.0\n",
"Episode: 535 Total reward: 200.0\n",
"Episode: 536 Total reward: 200.0\n",
"Episode: 537 Total reward: 176.0\n",
"Episode: 538 Total reward: 200.0\n",
"Episode: 539 Total reward: 184.0\n",
"Episode: 540 Total reward: 200.0\n",
"Episode: 541 Total reward: 200.0\n",
"Episode: 542 Total reward: 200.0\n",
"Episode: 543 Total reward: 200.0\n",
"Episode: 544 Total reward: 200.0\n",
"Episode: 545 Total reward: 200.0\n",
"Episode: 546 Total reward: 200.0\n",
"Episode: 547 Total reward: 200.0\n",
"Episode: 548 Total reward: 200.0\n",
"Episode: 549 Total reward: 199.0\n",
"Episode: 550 Total reward: 186.0\n",
"Episode: 551 Total reward: 200.0\n",
"Episode: 552 Total reward: 200.0\n",
"Episode: 553 Total reward: 162.0\n",
"Episode: 554 Total reward: 200.0\n",
"Episode: 555 Total reward: 174.0\n",
"Episode: 556 Total reward: 200.0\n",
"Episode: 557 Total reward: 200.0\n",
"Episode: 558 Total reward: 200.0\n",
"Episode: 559 Total reward: 200.0\n",
"Episode: 560 Total reward: 200.0\n",
"Episode: 561 Total reward: 200.0\n",
"Episode: 562 Total reward: 200.0\n",
"Episode: 563 Total reward: 200.0\n",
"Episode: 564 Total reward: 200.0\n",
"Episode: 565 Total reward: 200.0\n",
"Episode: 566 Total reward: 200.0\n",
"Episode: 567 Total reward: 200.0\n",
"Episode: 568 Total reward: 200.0\n",
"Episode: 569 Total reward: 200.0\n",
"Episode: 570 Total reward: 200.0\n",
"Episode: 571 Total reward: 200.0\n",
"Episode: 572 Total reward: 200.0\n",
"Episode: 573 Total reward: 200.0\n",
"Episode: 574 Total reward: 200.0\n",
"Episode: 575 Total reward: 200.0\n",
"Episode: 576 Total reward: 200.0\n",
"Episode: 577 Total reward: 200.0\n",
"Episode: 578 Total reward: 200.0\n",
"Episode: 579 Total reward: 200.0\n",
"Episode: 580 Total reward: 200.0\n",
"Episode: 581 Total reward: 200.0\n",
"Episode: 582 Total reward: 200.0\n",
"Episode: 583 Total reward: 200.0\n",
"Episode: 584 Total reward: 200.0\n",
"Episode: 585 Total reward: 200.0\n",
"Episode: 586 Total reward: 200.0\n",
"Episode: 587 Total reward: 200.0\n",
"Episode: 588 Total reward: 200.0\n",
"Episode: 589 Total reward: 200.0\n",
"Episode: 590 Total reward: 200.0\n",
"Episode: 591 Total reward: 200.0\n",
"Episode: 592 Total reward: 200.0\n",
"Episode: 593 Total reward: 200.0\n",
"Episode: 594 Total reward: 200.0\n",
"Episode: 595 Total reward: 200.0\n",
"Episode: 596 Total reward: 200.0\n",
"Episode: 597 Total reward: 200.0\n",
"Episode: 598 Total reward: 200.0\n",
"Episode: 599 Total reward: 200.0\n",
"Episode: 600 Total reward: 200.0\n",
"Episode: 601 Total reward: 200.0\n",
"Episode: 602 Total reward: 200.0\n",
"Episode: 603 Total reward: 200.0\n",
"Episode: 604 Total reward: 200.0\n",
"Episode: 605 Total reward: 200.0\n",
"Episode: 606 Total reward: 200.0\n",
"Episode: 607 Total reward: 200.0\n",
"Episode: 608 Total reward: 200.0\n",
"Episode: 609 Total reward: 148.0\n",
"Episode: 610 Total reward: 200.0\n",
"Episode: 611 Total reward: 200.0\n",
"Episode: 612 Total reward: 200.0\n",
"Episode: 613 Total reward: 200.0\n",
"Episode: 614 Total reward: 200.0\n",
"Episode: 615 Total reward: 200.0\n",
"Episode: 616 Total reward: 166.0\n",
"Episode: 617 Total reward: 200.0\n",
"Episode: 618 Total reward: 200.0\n",
"Episode: 619 Total reward: 200.0\n",
"Episode: 620 Total reward: 200.0\n",
"Episode: 621 Total reward: 200.0\n",
"Episode: 622 Total reward: 200.0\n",
"Episode: 623 Total reward: 200.0\n",
"Episode: 624 Total reward: 200.0\n",
"Episode: 625 Total reward: 200.0\n",
"Episode: 626 Total reward: 200.0\n",
"Episode: 627 Total reward: 173.0\n",
"Episode: 628 Total reward: 200.0\n",
"Episode: 629 Total reward: 200.0\n",
"Episode: 630 Total reward: 200.0\n",
"Episode: 631 Total reward: 106.0\n",
"Episode: 632 Total reward: 200.0\n",
"Episode: 633 Total reward: 200.0\n",
"Episode: 634 Total reward: 200.0\n",
"Episode: 635 Total reward: 200.0\n",
"Episode: 636 Total reward: 200.0\n",
"Episode: 637 Total reward: 200.0\n",
"Episode: 638 Total reward: 165.0\n",
"Episode: 639 Total reward: 85.0\n",
"Episode: 640 Total reward: 200.0\n",
"Episode: 641 Total reward: 200.0\n",
"Episode: 642 Total reward: 200.0\n",
"Episode: 643 Total reward: 159.0\n",
"Episode: 644 Total reward: 198.0\n",
"Episode: 645 Total reward: 150.0\n",
"Episode: 646 Total reward: 200.0\n",
"Episode: 647 Total reward: 200.0\n",
"Episode: 648 Total reward: 188.0\n",
"Episode: 649 Total reward: 200.0\n",
"Episode: 650 Total reward: 200.0\n",
"Episode: 651 Total reward: 200.0\n",
"Episode: 652 Total reward: 200.0\n",
"Episode: 653 Total reward: 200.0\n",
"Episode: 654 Total reward: 200.0\n",
"Episode: 655 Total reward: 200.0\n",
"Episode: 656 Total reward: 86.0\n",
"Episode: 657 Total reward: 200.0\n",
"Episode: 658 Total reward: 200.0\n",
"Episode: 659 Total reward: 200.0\n",
"Episode: 660 Total reward: 200.0\n",
"Episode: 661 Total reward: 196.0\n",
"Episode: 662 Total reward: 161.0\n",
"Episode: 663 Total reward: 155.0\n",
"Episode: 664 Total reward: 200.0\n",
"Episode: 665 Total reward: 146.0\n",
"Episode: 666 Total reward: 200.0\n",
"Episode: 667 Total reward: 200.0\n",
"Episode: 668 Total reward: 200.0\n",
"Episode: 669 Total reward: 200.0\n",
"Episode: 670 Total reward: 200.0\n",
"Episode: 671 Total reward: 151.0\n",
"Episode: 672 Total reward: 200.0\n",
"Episode: 673 Total reward: 200.0\n",
"Episode: 674 Total reward: 200.0\n",
"Episode: 675 Total reward: 200.0\n",
"Episode: 676 Total reward: 200.0\n",
"Episode: 677 Total reward: 200.0\n",
"Episode: 678 Total reward: 200.0\n",
"Episode: 679 Total reward: 200.0\n",
"Episode: 680 Total reward: 85.0\n",
"Episode: 681 Total reward: 24.0\n",
"Episode: 682 Total reward: 194.0\n",
"Episode: 683 Total reward: 200.0\n",
"Episode: 684 Total reward: 191.0\n",
"Episode: 685 Total reward: 179.0\n",
"Episode: 686 Total reward: 181.0\n",
"Episode: 687 Total reward: 177.0\n",
"Episode: 688 Total reward: 200.0\n",
"Episode: 689 Total reward: 160.0\n",
"Episode: 690 Total reward: 200.0\n",
"Episode: 691 Total reward: 200.0\n",
"Episode: 692 Total reward: 143.0\n",
"Episode: 693 Total reward: 200.0\n",
"Episode: 694 Total reward: 200.0\n",
"Episode: 695 Total reward: 200.0\n",
"Episode: 696 Total reward: 170.0\n",
"Episode: 697 Total reward: 54.0\n",
"Episode: 698 Total reward: 56.0\n",
"Episode: 699 Total reward: 200.0\n",
"Episode: 700 Total reward: 23.0\n",
"Episode: 701 Total reward: 85.0\n",
"Episode: 702 Total reward: 118.0\n",
"Episode: 703 Total reward: 68.0\n",
"Episode: 704 Total reward: 195.0\n",
"Episode: 705 Total reward: 200.0\n",
"Episode: 706 Total reward: 62.0\n",
"Episode: 707 Total reward: 200.0\n",
"Episode: 708 Total reward: 200.0\n",
"Episode: 709 Total reward: 28.0\n",
"Episode: 710 Total reward: 200.0\n",
"Episode: 711 Total reward: 197.0\n",
"Episode: 712 Total reward: 200.0\n",
"Episode: 713 Total reward: 200.0\n",
"Episode: 714 Total reward: 200.0\n",
"Episode: 715 Total reward: 200.0\n",
"Episode: 716 Total reward: 200.0\n",
"Episode: 717 Total reward: 200.0\n",
"Episode: 718 Total reward: 200.0\n",
"Episode: 719 Total reward: 200.0\n",
"Episode: 720 Total reward: 200.0\n",
"Episode: 721 Total reward: 55.0\n",
"Episode: 722 Total reward: 200.0\n",
"Episode: 723 Total reward: 200.0\n",
"Episode: 724 Total reward: 200.0\n",
"Episode: 725 Total reward: 107.0\n",
"Episode: 726 Total reward: 200.0\n",
"Episode: 727 Total reward: 200.0\n",
"Episode: 728 Total reward: 200.0\n",
"Episode: 729 Total reward: 188.0\n",
"Episode: 730 Total reward: 200.0\n",
"Episode: 731 Total reward: 200.0\n",
"Episode: 732 Total reward: 200.0\n",
"Episode: 733 Total reward: 200.0\n",
"Episode: 734 Total reward: 200.0\n",
"Episode: 735 Total reward: 200.0\n",
"Episode: 736 Total reward: 200.0\n",
"Episode: 737 Total reward: 200.0\n",
"Episode: 738 Total reward: 200.0\n",
"Episode: 739 Total reward: 200.0\n",
"Episode: 740 Total reward: 97.0\n",
"Episode: 741 Total reward: 200.0\n",
"Episode: 742 Total reward: 200.0\n",
"Episode: 743 Total reward: 175.0\n",
"Episode: 744 Total reward: 200.0\n",
"Episode: 745 Total reward: 200.0\n",
"Episode: 746 Total reward: 200.0\n",
"Episode: 747 Total reward: 200.0\n",
"Episode: 748 Total reward: 200.0\n",
"Episode: 749 Total reward: 169.0\n",
"Episode: 750 Total reward: 173.0\n",
"Episode: 751 Total reward: 105.0\n",
"Episode: 752 Total reward: 174.0\n",
"Episode: 753 Total reward: 134.0\n",
"Episode: 754 Total reward: 128.0\n",
"Episode: 755 Total reward: 160.0\n",
"Episode: 756 Total reward: 130.0\n",
"Episode: 757 Total reward: 197.0\n",
"Episode: 758 Total reward: 174.0\n",
"Episode: 759 Total reward: 133.0\n",
"Episode: 760 Total reward: 174.0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Episode: 761 Total reward: 128.0\n",
"Episode: 762 Total reward: 154.0\n",
"Episode: 763 Total reward: 113.0\n",
"Episode: 764 Total reward: 200.0\n",
"Episode: 765 Total reward: 200.0\n",
"Episode: 766 Total reward: 200.0\n",
"Episode: 767 Total reward: 200.0\n",
"Episode: 768 Total reward: 200.0\n",
"Episode: 769 Total reward: 200.0\n",
"Episode: 770 Total reward: 200.0\n",
"Episode: 771 Total reward: 200.0\n",
"Episode: 772 Total reward: 200.0\n",
"Episode: 773 Total reward: 200.0\n",
"Episode: 774 Total reward: 200.0\n",
"Episode: 775 Total reward: 200.0\n",
"Episode: 776 Total reward: 200.0\n",
"Episode: 777 Total reward: 200.0\n",
"Episode: 778 Total reward: 200.0\n",
"Episode: 779 Total reward: 200.0\n",
"Episode: 780 Total reward: 200.0\n",
"Episode: 781 Total reward: 200.0\n",
"Episode: 782 Total reward: 200.0\n",
"Episode: 783 Total reward: 200.0\n",
"Episode: 784 Total reward: 200.0\n",
"Episode: 785 Total reward: 200.0\n",
"Episode: 786 Total reward: 200.0\n",
"Episode: 787 Total reward: 200.0\n",
"Episode: 788 Total reward: 200.0\n",
"Episode: 789 Total reward: 200.0\n",
"Episode: 790 Total reward: 200.0\n",
"Episode: 791 Total reward: 200.0\n",
"Episode: 792 Total reward: 200.0\n",
"Episode: 793 Total reward: 200.0\n",
"Episode: 794 Total reward: 200.0\n",
"Episode: 795 Total reward: 200.0\n",
"Episode: 796 Total reward: 200.0\n",
"Episode: 797 Total reward: 200.0\n",
"Episode: 798 Total reward: 200.0\n",
"Episode: 799 Total reward: 123.0\n",
"Episode: 800 Total reward: 184.0\n",
"Episode: 801 Total reward: 200.0\n",
"Episode: 802 Total reward: 200.0\n",
"Episode: 803 Total reward: 200.0\n",
"Episode: 804 Total reward: 200.0\n",
"Episode: 805 Total reward: 200.0\n",
"Episode: 806 Total reward: 200.0\n",
"Episode: 807 Total reward: 200.0\n",
"Episode: 808 Total reward: 200.0\n",
"Episode: 809 Total reward: 200.0\n",
"Episode: 810 Total reward: 200.0\n",
"Episode: 811 Total reward: 200.0\n",
"Episode: 812 Total reward: 200.0\n",
"Episode: 813 Total reward: 200.0\n",
"Episode: 814 Total reward: 200.0\n",
"Episode: 815 Total reward: 200.0\n",
"Episode: 816 Total reward: 200.0\n",
"Episode: 817 Total reward: 200.0\n",
"Episode: 818 Total reward: 200.0\n",
"Episode: 819 Total reward: 200.0\n",
"Episode: 820 Total reward: 181.0\n",
"Episode: 821 Total reward: 200.0\n",
"Episode: 822 Total reward: 200.0\n",
"Episode: 823 Total reward: 200.0\n",
"Episode: 824 Total reward: 200.0\n",
"Episode: 825 Total reward: 200.0\n",
"Episode: 826 Total reward: 200.0\n",
"Episode: 827 Total reward: 200.0\n",
"Episode: 828 Total reward: 200.0\n",
"Episode: 829 Total reward: 200.0\n",
"Episode: 830 Total reward: 200.0\n",
"Episode: 831 Total reward: 200.0\n",
"Episode: 832 Total reward: 200.0\n",
"Episode: 833 Total reward: 200.0\n",
"Episode: 834 Total reward: 200.0\n",
"Episode: 835 Total reward: 200.0\n",
"Episode: 836 Total reward: 200.0\n",
"Episode: 837 Total reward: 200.0\n",
"Episode: 838 Total reward: 200.0\n",
"Episode: 839 Total reward: 200.0\n",
"Episode: 840 Total reward: 200.0\n",
"Episode: 841 Total reward: 200.0\n",
"Episode: 842 Total reward: 200.0\n",
"Episode: 843 Total reward: 200.0\n",
"Episode: 844 Total reward: 200.0\n",
"Episode: 845 Total reward: 200.0\n",
"Episode: 846 Total reward: 200.0\n",
"Episode: 847 Total reward: 200.0\n",
"Episode: 848 Total reward: 200.0\n",
"Episode: 849 Total reward: 200.0\n",
"Episode: 850 Total reward: 200.0\n",
"Episode: 851 Total reward: 200.0\n",
"Episode: 852 Total reward: 200.0\n",
"Episode: 853 Total reward: 200.0\n",
"Episode: 854 Total reward: 200.0\n",
"Episode: 855 Total reward: 200.0\n",
"Episode: 856 Total reward: 200.0\n",
"Episode: 857 Total reward: 200.0\n",
"Episode: 858 Total reward: 200.0\n",
"Episode: 859 Total reward: 200.0\n",
"Episode: 860 Total reward: 200.0\n",
"Episode: 861 Total reward: 200.0\n",
"Episode: 862 Total reward: 200.0\n",
"Episode: 863 Total reward: 200.0\n",
"Episode: 864 Total reward: 200.0\n",
"Episode: 865 Total reward: 200.0\n",
"Episode: 866 Total reward: 200.0\n",
"Episode: 867 Total reward: 200.0\n",
"Episode: 868 Total reward: 200.0\n",
"Episode: 869 Total reward: 200.0\n",
"Episode: 870 Total reward: 200.0\n",
"Episode: 871 Total reward: 200.0\n",
"Episode: 872 Total reward: 200.0\n",
"Episode: 873 Total reward: 200.0\n",
"Episode: 874 Total reward: 200.0\n",
"Episode: 875 Total reward: 200.0\n",
"Episode: 876 Total reward: 200.0\n",
"Episode: 877 Total reward: 200.0\n",
"Episode: 878 Total reward: 200.0\n",
"Episode: 879 Total reward: 200.0\n",
"Episode: 880 Total reward: 200.0\n",
"Episode: 881 Total reward: 200.0\n",
"Episode: 882 Total reward: 200.0\n",
"Episode: 883 Total reward: 200.0\n",
"Episode: 884 Total reward: 200.0\n",
"Episode: 885 Total reward: 200.0\n",
"Episode: 886 Total reward: 200.0\n",
"Episode: 887 Total reward: 200.0\n",
"Episode: 888 Total reward: 200.0\n",
"Episode: 889 Total reward: 200.0\n",
"Episode: 890 Total reward: 200.0\n",
"Episode: 891 Total reward: 200.0\n",
"Episode: 892 Total reward: 200.0\n",
"Episode: 893 Total reward: 200.0\n",
"Episode: 894 Total reward: 200.0\n",
"Episode: 895 Total reward: 200.0\n",
"Episode: 896 Total reward: 200.0\n",
"Episode: 897 Total reward: 200.0\n",
"Episode: 898 Total reward: 200.0\n",
"Episode: 899 Total reward: 200.0\n",
"Episode: 900 Total reward: 200.0\n"
]
}
],
"source": [
"max_episodes = 1000\n",
"max_t = 250\n",
"combo_goal = 100\n",
"timesteps_goal= 195\n",
"\n",
"combo = 0\n",
"R=[]\n",
"\n",
"\n",
"for episode in range(max_episodes):\n",
" all_states, all_actions, all_rewards = [], [], [] \n",
" state = env.reset()\n",
" cumulative_reward = 0 \n",
" for t in range(max_t): \n",
" action = select_action(state)\n",
" next_state, reward, done, info = env.step(action)\n",
" if done:\n",
" if (t >= 150): \n",
" combo += 1\n",
" else: \n",
" combo = 0\n",
" next_state = None \n",
" cumulative_reward += reward \n",
" all_states.append(state); all_actions.append(action); all_rewards.append(reward)\n",
" state = next_state\n",
" if done:\n",
" policy_gradient(all_states, all_actions, all_rewards)\n",
" break\n",
" \n",
" print('Episode: {}'.format(episode), 'Total reward: {}'.format(cumulative_reward))\n",
" R.append(cumulative_reward)\n",
" if combo > combo_goal:\n",
" break"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXucHFWZ939P91wzmckkk8k9IVdIAoSEDJBwMxCQS1BgRQWVm0DEhV19ddcFvK67KOoqyq6iQRCRiyKguIisEIGAEGCCGAIBciGBhCQzkJDJbW7d5/2jq7pPVdepOnXr6/PNJ5/prjp1zunqrl899ZznPIeEEGAYhmEql0SxO8AwDMPECws9wzBMhcNCzzAMU+Gw0DMMw1Q4LPQMwzAVDgs9wzBMhcNCzzAMU+Gw0DMMw1Q4LPQMwzAVTk2xOwAAI0eOFJMnTy52NxiGYcqKVatWvSuEaPcqVxJCP3nyZHR2dha7GwzDMGUFEW3WKceuG4ZhmAqHhZ5hGKbCYaFnGIapcFjoGYZhKhwWeoZhmArHU+iJaCIRPU5ErxLRK0T0OWP7CCJ6lIjWGX+HG9uJiG4iovVEtJqIjoz7QzAMwzBqdCz6QQBfFELMBrAAwFVENBvANQCWCyFmAFhuvAeAMwDMMP4vBXBz5L1mGIZhtPGMoxdCbAOwzXi9h4jWAhgP4GwAi4xivwTwBIB/M7bfITJrFK4kolYiGmvUw5QZg6k0HnhxK86bPwEAcN+LW3DuvPGoTSbw5Bvd2LWvH+fMGw8A6NrTi2VPbkQyQWhprEVLQw0uXDgZALBm627cuXIzEgnC9Pah2PjuXjTV16DnwCAAgQQR2prqsGv/AC45bjK27DqAVZt2OvZp7qRW7O1LoSZB2LJrP/b3p3DWnLF4bG0XzpozFhOGDwn8eTs37cRzb+5EOi0wkErn7R82pA6b3t2HCcMbsa8/ha27DqCuhjCQEmhrqkN9jbPt1NM7CCKguT5zye3rT2EglUZrY62l3Nb3ezF2WAM2dO/F9FFDQcb2hroktu46AAFgZFNd4M9n8s7uXowb1oDuvf1oaaxBfTKR3d5Ym0R9TQJD6pIY2lCDvoE0Zo5twSNrtmN8a0O23IThjZg3aTgefXU7Rgypy9YJALPGtmBP7yAWTG3DrU9vxL7+FKaMbMKh41owZlgD9vWlcN+qLdjbN4gxLfVorE1mPz8ADB9Si97BFEYMyXzW7T29EAJob65HS2MtLj1uMuprktnPs+KNbnQqfi9O1CYTSCQIfQOpvH3de/swrLEOdUlyODJ6Dh7TjLPmjIu1DV8TpohoMoB5AJ4DMFoS7+0ARhuvxwN4Wzpsi7HNIvREtBQZix+TJk3y2W2mUNz21zfxrYdfQ0oIJBOEL923Gu/u7cM/LpqOi297HgCyQn/3c2/h50+/aTn+6CltOGRMM75w70t4Y8derTZ/tXIzprU3YUP3PpDtWlMtcfyX17qwestuvNm9D985b46/DynxqVufQ+9ATuDl9nWWV7b3V+c48xinckTq7UHx006h8Nv+3ImtWDC1Lfv+P//4Kt7YsVfrvNjbUX3HYc6xH86aM650hJ6IhgK4H8DnhRA9JJ0FIYQgIl8/EyHEMgDLAKCjo4NXKC9R3tvXDwDYtb8ftYmM1bdzb7+ljBACRITBVP7XuKd3IFOP7RgnDhvfgjVbewAAAymBc+aOww/Pn2cp8+//+wp+8ddNecf2D2bEefeBAc923JBF/rnrFmN0S0P2/ed//Tf8/qV3XI9/89tL8rZ19fTi6G8tBwC89LVTUV+TxKyvPQIAeOpLJ2HiiMwTyBfufQkPvLg1r75v/OEV3P7Mpuy2Wy7qwKmzRyMoX/n9y7hz5Vs4YsIw/H3L7mw7P3psHW587I1sOXm/ycQRjZjc1oSn1r3rWPfSE6eidyCFO57VmrCZ5c1vL8FHf/oMXti0y7L9qS+dhPVde3Hp7S9Yttt/a4NpgbPmjMX/fMJ7SPCh1e/g6rv/BgD4yxc/gKntQ7P7/vbWLpz7k2eyfaoUtKJuiKgWGZG/SwjxgLF5BxGNNfaPBdBlbN8KYKJ0+ARjG1OGEPTNmjoHt0W/4f5ob673PD4hGQ+ptEAikd/2pBHObpl0DOaovXWKyMSTq9Gpsqk+aXmfDBkrZ57npO382usN8nmjtoKdflMAkBICQgg8/loXUmkB+Pj65d90re1D1yQqMxBRJ+qGANwKYK0Q4gfSrj8AuNh4fTGAB6XtFxnRNwsA7Gb/fPkj66j9mjL32S8aIGOZA0DrkNq8fXZkjUgLgaSDatjFKVfes3r/2JoKLGKysNsqlcVUdVOtS1qFPhFSTc2j7aJmv7HWKM61GwnyYxp4U5tM5N9xAaTTAo++ugOX3v4Clq3YCAH9G5NcrCZpv9kVyF9TYHRcN8cBuBDAy0T0krHtOgA3ALiXiC4DsBnAx4x9DwM4E8B6APsBXBppj5mC4tf6tDNguFR0hJjsFr1Dg6ouxGPRk+v7QPWQ7ZzK5RTV19ZEK0aksuhtHXB6ovKsG/6fBL79D4cr99XVJBzPeloIvGu4A9/auS/jPvTRRxP7za62QAOwhUYn6uZpqK+vxQ7lBYCrQvaLKSNE9m++2JqRKykNpZf1IS2cXTcqNTR13qkPQbE3FVRf7TdLWfhVoi9TZ3tSCiv05g3Uy5oNatH7xe2IumTC8caRMQQyr9NpGBa9ZntSuajPbalSmQ4pJjKcfvZ241kYG5yMatNHP6gh9HYfvZMvWnUduvUhKPk++vD1EOzi7iz6MnbxdHJp+eqPcbhd1PLaCST0gbvlSG3S+TkqLXJPfBl/vfsNw0qupP1m5+R+rAQq81MxkeFHN4WDypo++lQ6PybdjiwSqbSzj17lPonDR2+3JIP6xi1+eJsPWyX6UJQBorDoM3/tFru93kCDz0S+b4hmeafPr+pDWuRcS+m0gIBgH70LLPRMaLKuGwex7RvMTEhxCr20I1+o8oVsLeN0XDw+envzUVn08g1Dx0ef16+YfPT2eoM0kyD1DcsLldvN6bxYXDc+LXq5XG3CHnXDQs9UIX5+9k6Xad+AvutGbktt0Tsfl/PRR4dbhIyveuw+est7b9eNfXNYMTLbsQ9E2s93EBdRMB+9+zFO+2XXTVoY37+2jz5XMC/SiF03TDUju2Xslpe5y8mqNrfpDMbKItE7mHJ8jHa26CkWiz4vvDJwNWR5rRZ3vRbChleqBmPtN5AgN7YE6T+Z6KK06BM5Hz2g/yThVopdN0xVorI+ZUzhd9JaIYDegVR25qobsoEpVK4bJz8uYgqvtDUVWGBtFr1il7ZAhg6vVNRjP98q49btBmAfg/DVIR+70yL3xCGMyVP6ri/1vkoNr2ShZ7TQ0VFH181gCjO/+gi2vn/A83it6BLbpq9/aLYlT0pJRt24HBfEao4svNJD2J1ubPMnDXccdDexu6biIu0UXql5rNutiC16pipxuijywysVOwDs78/PDqhsS2PCjn3LuNbGTNO53mi357c/cUiATp1RRf/k6sv8Tdpnxmq0c8NH3BPGRRlHv+JfT1IWSIlclE02vDLIaKyNqk2BwDCAnsUUVmLt9TsOxjqIEYGyVuZja7uwvksvS6bf/gQejLXUYa/T+/gxwxos76OKDLHXkyf0tv0Hjx6KhlprOgY7hODnSWZofQ0mtQ0x6nQejDWtbyGM8Er20SthoWdc8XPNOvnJ/Yi//Rpz8hHn3QwSmY3yWO/qLe/7aFVNEFF2rsc6GGvZpyFOH5w9GqcdmstWGZeP3o5qt5uQB8l143VjcNotu25SaX8WfVTJ6coJFnpGC1nD5ZS58j7VYKwuOq4b+5O1KSzxxNFH4zJxHXB1qdJcxISIcNIho3L9CCn05pny+jyq8Erz6clHhopQOFUpR92YySu1hT6ynpUPLPRMZDhJ7U+f3KB9fJ5FrzEzNmHMxEzHkr7S3nbA4+SoG5d9dh7/l0WO28OmQMjW43H1e1m+Tk8EiUwyH184FZcHfJ36kZYmSJkTpnQbrkKDnoWe0Ud1gbiFV/qr3x4F4uSjR16ZjI8+XNvO/bG+D2pJW/PZqJ8SVAPNdqLyI3tZ9F7NyMebZTM6H7+Syk9wmdf64ZWF6F+pwULPuOLnkvATy+yEXVh0XCVmOF9h0hQHrMfNog9QX1RC79diN0+xUwoF87vKDMb67YfxVzPXD2D45Y3X6TT8pUCoPp33t2YsU724pf+V0w8Qgkff2C90Z4veVibrow/YqFt/8gZjwytEFAO8cbludD+v6VaR+xFFl+TfGLk86QBm2gNhvBaR+uh/flEHDmoLvsB8KcJCz7ijcfXkwugz+UeCWtd5A60acfTJRCalgBDeM2/94sef7lqPbNHnxeb79ytHFeptv2HYvzavG0EyKYtx7hbvP+rG//6cX9587SO80qPBU0Ksx1uq6CwleBsRdRHRGmnbb4joJeP/JnPlKSKaTEQHpH0/jbPzTOHQeTQWItxkHidrPb9M/jHxWfR2f3rAelzOnNyE20eQRTiqST1egucnKsd6MwvVLQfyKzRDKrOvfbTLrhtnbgfwPwDuMDcIIT5uviai7wOQl4rfIISYG1UHmeKiuib+8tqO7OvcI7RhbepPhnVtyzmO3sG9Q8658MOS708POBgbsbCE1XmzO95x9O4WvyULpMNcgVHN9eja06fRH/8nyOKjFz599L5bK388fzJCiBUAdjrtMxYO/xiAeyLuF1Nktu/uxcW3PY89vYMA8q3NT9/emXeMgPM6r7roxK3nRcIY0XyF8NEHt+j129AhrI9e91R5fZfyzNrsgKrtmDkThvnpmiNO3RBCZF2EuaRm0bhuKpGwPvoTAOwQQqyTtk0hor8B6AHwFSHEUyHbYIrAf/9lHZ58oxuvvNOT3abOXmn89bWcWz75M2O9ffSZOPp40hTnfd4YBMLJml08cxROmDFS2XShpul7LbziFHUjR14RAR8/aiJWb9kNNzx99Lb3yQRl89sARq4b9yp8tVeJhHX2XQCrNb8NwCQhxDwAXwBwNxG1OB1IREuJqJOIOru7u0N2g4mLd/caj94uQirvitKi14q6SZAle6Vf1m7rwbceXqvl+gn6ydzT+uZvWzC1DZccNyVQfVGiCq80kV1Icsmoeyd/3t9euRBJIvzsyY14en1GN9Zs7cGe3kGeGetCYKEnohoA/wDgN+Y2IUSfEOI94/UqABsAHOx0vBBimRCiQwjR0d7eHrQbTAmRFiLc9Pw8V4mO6ybc9JdP3LISy1ZsxO4DA55lo0iBoFOns6siUNOh8IyzV+3PxtQH/2YsM2Ol7UdNHoH+VBqDaYE7V75lbZZnxioJY9GfAuA1IcQWcwMRtRNR0ng9FcAMABvDdZEpeaRcN+F0XsOit71PJsJZuGmp715EEV6Zt0/xWsXHOyYG64QGp8wajYVT27LvvVIkyDd11emL4gYVfTRN9Sm9TnjlPQCeBXAIEW0hosuMXecjfxD2RACrjXDL+wBcKYRwHMhlyg/PFAjQHxDTqb/OQWniys2u4+MPPBjr03Xjhj9vtD8a65JYdtH87Hs/g7Ey5lZ9gY5GeLWjbqpP570HY4UQFyi2X+Kw7X4A94fvFlNq6MhL2MFYO80N+T9Px8HYEG1kBxE1ysbhG3eq03lb5E07Yl3PVj/80vJkEnFfo3bJVKHOc64bJjyWFAhhLHrb+6FOQu8Q+RFGWMxDdbJfFkxsC9OMZ9teYZyy0DudPV23jVMr6gXUXerh8EolLPSMFlprxoZMamY/trm+1rNM5n14d1FKK+omXoEwe1DMwVhLqgVbP+xdqEkqLHr4F+ko0G2qQheRcoWFntFGdX1EFUdvx9GiV+SjD07m4MFUfD56vxTXopdi433OnM3W4fMD+M11c8UJ6tBTrfaq0HnDQs+ERmRnKIaz4OwXYFO9w/qkEYdXmlqmMxhbDD954dvOvbYLeX7Ek/tgbFwoXS+RR+dUDiz0jBY60R5eKRDOPHyM6/H2Q+tr8oXeXjuRyp2jh1l2UMNHHzbCR5dCCtGYlgZctPAgx31erhsvH75uhJDXrVp3XYBqtNR14TTFjD4eKRDSHq6bhdNGoq2pHr9audmzqe9+ZI7jdicrM8wFbh5biKUIdSmkXK28brG1bReL3o4quVrkUTd5N/JwLiO26BkmBBnXjUvMuI+6PCZdSu9DRt34sOgLFq3h0E7BBmN9DKRaXDfyS5/nSZW0TLU/pOemKi1/FnomNLnwSnc18rIQdRJ35T3GUzgL2Dw2peW6CdGQj76UOqaQW77PGG9E2q4btuiVsNAzWrhZlFmBF+rHecDfBaYb0UEIGbtvHKsj9MXUh7mTWgFk0hTECWlY5tmlBD3ufEJoppbw0Sen97l6dOPotYpVFOyjZ7Txuj7SHsu5Ebysfu/QPvvWqFIg6MTRh0rYFpKZY1qw/vozUOOVgCYkpHjtRK2iL3ELadiwTnbdMIwCVxm0zIxVF/MjAOrMiPl1RuGj79ZaCam4xC3ygL8Zqc31znaiLKSHj/deeMQzjt7jvdd2v+1VIiz0TGR4TZgi459yvyXiQ11HXp1hhN6o7zO/WoVd+/rdyxYqvLIgrXi37fW05DShzc4RE1vxtbNm+++H2w1Hd6ReAc+MZRgF7j763F9XMSR3141FZFSuG7tFnwgZXikdOu8/HtUuGwelEOBp9dFn/o5qrncs65R0Tj5OeJSTjvC1P6xFX/xns8LDQs9ooxK6bNSNR64bP/50levG6TE+iqRmOhRqwlQxsVjSxt8mw0VjX4WruSE/F5F8XHR9sr4P7aOv/K8xDxZ6RguvFZh6B1LoHUh7uG7csbhulJNx7OGV+fa8n5hzP+6YqtMH49yolln0sujjInTUTYR9KRc46obR4p7n31KmMBAQmPnVRwAA00cNVdZBpC/CKqstb8FqhzJ+JrlGEfIZNaXgwgEcnp5sn7+pLhr58Jww5dEvt3qcy1Wf1LNFz2izavMuzzKuFr1ndEWugHLClMNjvP3C1VnoO9emD+K2VOOt3jf2c20/r4kEYc4EI6pG9u3DfBIw6wkbAmt/ilOU060vVG/KE52lBG8joi4iWiNt+wYRbSWil4z/Z0r7riWi9UT0OhGdFlfHmdJBvv7drF4/g6bqevIv+jCuGz9lq8FHL+OdbAyY1p7/BOc7TbHP/Zzrxj86Fv3tAE532H6jEGKu8f9hACCi2cisJXuoccxPzMXCmfJHdeHLWhkmjl4noZbjZrvl6cP50TeY1i5bbfpgj55R7S9UP7za1V5hquq+SQ2hF0KsAKC7wPfZAH4thOgTQrwJYD2Ao0P0j6kwdCVYHUdvL5d/2frx0fcNprTLVpsl6Ofjjm9tBACMVIRiurbjlQPJ1hPVU5iuy67avkcgnI/+aiJabbh2hhvbxgN4WyqzxdjGlBlOF4M6vFLONKi+iuxW+tUnTccJM0bmjpX2qX30Dq6bPB+9sgt51PmYbVp1rhvj4zbWZh7KD2prsu6XvrFPHz8FP7twPpYcPlb6Pgo7rFyoDJ/lSFChvxnANABzAWwD8H2/FRDRUiLqJKLO7u7ugN1g4sLpotm2u9fzOLdZh3adHNvagLkTW53r0cx1Qw7bdFaLMpk5tgUA0NZU51m2ynQ+K+TjWxtxy0Ud+OH5c637pfORJMJph45xDHf1bsdjv2aFut96tX2PQEChF0LsEEKkhBBpALcg557ZCmCiVHSCsc2pjmVCiA4hREd7e3uQbjAlgqyr7hNjydXqki1zXR+905qxfgw786agStCl6l8cZBcHj7UVYFhjreWvEqkjp84ejRbbBCm5n04zasPguq6BYpfuDb7ansyAgHH0RDRWCLHNeHsuADMi5w8A7iaiHwAYB2AGgOdD95IpG3Rz2XihnhnrFHUTPLzS9OcPpr0HZStFHi45djIaapM4/6iJruUK9Xn9DNK7ofu1V6HOews9Ed0DYBGAkUS0BcDXASwiornIGCGbAHwGAIQQrxDRvQBeBTAI4CohhP5oF1P2eKS60UZ3mTqnFab8+GrNJQRLac3YuKlJJvCpBc7rxMqYVrVb1I05O7ahNhdcF/VZ0h2D0XbdVMwtWx9PoRdCXOCw+VaX8tcDuD5Mp5jywuq68bLoXZKaBQ2vtOHHR2+WTaV0lhK0vv/kMZMwccQQ3PCn17Tb06FUxhS9TzXh306fifGtjTj90Nys6aiXEtSGo26U8MxYJlLcZ8bqX2G6Swk61evLohd+LHrr+9pkAld+YJp+Yx6Umv7ouFSa6mvwmQ9Mcxw8jyoKRve86Fv01QcLPRMaeYJSGNeNLOLKOHqnsE/be18WveGa11lK0N5SpbhyVAT9ePbjajwSwHvOwNWcMKX9tVf21+YICz0TGovrxqWcH4vej+smjN6aN4UBjcHYaluwIpezxllBVafDvn3JnLG44oQpmDmmOVQ/vNCdEV2NPnoWesaR4Naci48e7laXJS7bl+vG+j6Ij17nEPvNp8INeg3Xjd5jV20ygS8vma0O5/QZdaOeGetej6q+aoCFngmNfH3J+pyXUtjHBebLotecIu+Exhhsrk98tVjw+jqjGlRmH314OB89Eyn2VMPplJ7/PnNsDt2ZsU7ouNsHUmlc/8e16Orxnu2bbdvjA/z8og7tusqBoBPElC4d1QOA3/pD+uirMR89Cz0TGqFw0mcuKGseHG3XTQgfvY7rZvnaHbj9mU2e5WRUfTI5ZfZoX/WVOuan9Zu9Ui3oQf2BesV0ffTVNtYCsOuGiQBLmmLptc5qUCrUF6ODj95HvSYHBvzP48vz0Qdo141CpUDQJeiMVb+C7mRhW9eu1VZ6vfZK5gwXDhZ6JlJkMcwfvPQKo5OO1VxhymljWsN30+8jD322T1WmD+b3FzQe3h6tE1W4pgrtQfgq+x4BFnomAlRJzZwsYP3Ha30fvX2bTguy0NfX6F0GqptP1JTLzFiVZezfpZOP25qxKjjqRg0LPeOIPyvOecDVK+omL0GZ9FrlD3e6Adg3rd3Wg8df61L2FrCuLNUihf1dfdJ05TFxT5AqNf0xP67yp+BzcDVulwlH3ahhoWcixTK7NWEXch8TpjSTmmXqtfKnNdtx6e0vuNbfn8oJvZmYC1DH7wPhwkXLEw9XW1StBI3Xt8FRN2pY6BlH/FwLKteN3Sr39Hy4uH1yRdwH7nTpHZAseinPuqvQV5mTPqxP3a67gevTLKc/M7b6YKFnIsV1QJX0rS71UoIO23Q7JzGQcnbduGl5pee2sZMNr1SlQPBxM3Zvx2uQXq8e9tGrYaFnQqMKr6z16bqxJjXTvxrNosMaa10tchl5MFZ23XTv6VMeU2UGfXAXh3IwNlh9Ufv2ObySYQIgh7XJ13KtLZrFXwoE/bLmhUtkXez7lXd2K4+xWPSS62ZP76BLn9Thoh8+Ypx+h8uEQslhVBa2bg57tugZJgBy4kf5GrKnp80Pg1THWeta5nLFBGDMsIbs5iU3Pa08RLboWySLvi+ljq9XPWVsumEJbrpgnmZny4ewBr1ddwPrq8uB/7gotx5AqYSlliKeQk9EtxFRFxGtkbZ9j4heI6LVRPQ7Imo1tk8mogNE9JLx/6dxdp4pDWSLXhZD+4LbXo/uR00e7lnWyWgzSxIRxrQ05BeQ6OkdwOvb91iibuRl8NwmUtkjgaI2DEtNqKLOYBo0141bP46dNjL7WtdHX21jLYCeRX87gNNt2x4FcJgQYg6ANwBcK+3bIISYa/y/MppuMoVk175+bN11QLu8vGiHxXWTJ/Tu9Rw2bphnW06zH0my6Nub612P/9TPn8NpP1xhEXRdoffKdRMVpSJDQX3ZUfTfmgLBmcuPn4LjZ4zENz40G4CPqJtSOcEFRGfN2BVENNm27c/S25UAzou2W0wxWfDt5ZYJRV6khPNwbE0yP7zS9VLUuACdjpd99A217rbL6i0Zv73so2+UjnET+kLFX5eMZR/xx1VW57DDMjNWcd5rDEPCdPNxBgQ1UfjoPw3gT9L7KUT0NyJ6kohOUB1EREuJqJOIOru7uyPoBhMVfkQesOaWcbPodS6xU2aN8tW2tU1CnUM6gwP9KeywpSOWBb2xTrLoU2nl+ECUUTdD6/NtrFIToKD9ycbR5+W6ifYJIWvBmzl5dOurQpM+VJpiIvoygEEAdxmbtgGYJIR4j4jmA/g9ER0qhOixHyuEWAZgGQB0dHSUjBHD+EfOIZawCL09SsW9HgLhJ5+cj/396sgXr8iKumQyb9vHlz2L1Vt2Y9MNS7LbBqQ8+bLr5thpbbjj00c71p13AwihF89/ebFW3vxiYgqi36Rmfn3xYePuVYO/6uOrj8BCT0SXADgLwGJhXH1CiD4AfcbrVUS0AcDBADrDd5UpVSw+eukyyvPRa9RVV5NAXU2dcr+j64Zyf50setNdI2Ox6CWh//wpB7tY9NFJxJC60l8KIrBFH3Xcu606dbSWPx/9KbMqa/0ANwL92ojodABfAvABIcR+aXs7gJ1CiBQRTQUwA8DGSHrKlCxCEUdfk8iPutFdeETdlsNxpo8e+U8RKuSoG9l1wykQckTt4fCb1TK7X5Ul09hu3oA11nc32iP89ZqT0dakNigqDU+hJ6J7ACwCMJKItgD4OjJRNvUAHjUe71YaETYnAvgmEQ0ASAO4UgixM6a+MyVCSjVhyu66iaAtOebd3iZRflROSuEfUUXduFFlOh8Yz6yXAeuzY1r2ZHuvw/jWxpC9Ki90om4ucNh8q6Ls/QDuD9spprywhlfKUTdWi97L9aGjo6NaGnDnZcfgU7c+53A8YdAm7FfeucqxHnmFqUZtobffuKpD+f0IaBD8nsU8H3128Dea/lQiPDOWCY1iyVjHwdgoRGPm2GZbvbnwylTKWv+jr+5wrOOtnVmPow+LvjqE3aRcbmRmP1nn1bDQM6FRWfS1DknlDxrRlH09qtk6i1U37M1eiqS/KU2zTu6zV+y9Cbtu9PAbvui9xKT1fZ6xwBa9J6U/9M+UPGnFsm+1NfkW/VUnTcPhE1pARFh0cDtWb3nf8Vg37MIgv1X55N0oFYu+UnTKZxCMRn16Lr+4XUzlDAs9Exprrpvc9vwQlL7qAAAgAElEQVTwSkJNMoGTZ4YLa1Na9ESBhD7omrHV4smJKo4+aHnPJSijHv2tQNh1w4RGTvhocd3YhN7L4tIViLwLX9rQNtQ9140T+TN4nYnbdVNq943oBzmjnRlr3886r4YteiY0KZXrxjYYqxvn7IU9midn0WcW937y9S783WGSlJ07LzsG+/oHS8Z1U2oUasKUp5DbzrtqwpRuPvpqhC16JjQ9BwZyb1wmTHkNlOoKRJ0iK6Y5M/ZjR010PG7bbmtGzkkjhuC0Q8dotQn4zJFfQfj1fRf6fthk5A0aJi0JyVhhoS9j1u3YgyP+/c/YvrvXu3CMfOX32aUKbCkQrFd8EP+5E/mzX82ZsZm/qnTCf3mty/JenhGrQ76vuMIJ+AF9x8V7zox159RZo/H1D83Gv50x02fL1QMLfRlzx7ObsfvAAP786vZidyWLbPTaXSyeQq/to3ceFDX/6qYq8Cv01ea6CUtcM2PtT36JBOHS46aURf6gYsFCz0SKW5riwaic9DZMC37ze5lJUCpBtnuOdGfE2tth3PGf08Yrjp7Pe1hY6MuYUowbdnPdeOl80OvZfh5UQTTffOhVWzm/E3t8Fa8Y/I9x6t1og1KKv/tSh4W+Aigl/XGz6L0HYwO2aX+UVyiynMjszMP1B2Gz7bhM1KpECpUCodLPYynATi3GQldPuIFdS1IzyWKeNGIIjpvWFqpuFXYLrxR96T86f25eFJKdEux2INQrTEVUf0mZNuUBC30FEOWD7H/8cW2o4y356CXXzTc+PDtvcDb/2GguYDkzZalw9tzxnmW+eOoh6B9M46PzJxSgR/HhPxullXmTWnH9OYdH1R0GLPRlTSlaNnKPvCzYuNjbq16KMCxXnDAFa7ftwdPr34287uFNdfjueUdEXq9fvrJkFkZKM4z9GhJhb9jXnD4Ts8e1hKqDscI+esZC2NmF1hWmcm90bkpB5cHe5T0xCv2Xl8zGMVNGACjNG20UXH7CVJwzb3zBXEl+bwwXHD0RJx7cjsuPnxpTjyoPtuiZSJH947qpBaJmb9+AZ5kwIs0xH+6ocs8EzU5qp3VInXIBd8YZLYueiG4joi4iWiNtG0FEjxLROuPvcGM7EdFNRLSeiFYT0ZFxdZ4pPeRLtKWxxnmH6lgf2vvPJ0/H0hOtFp3pbrho4WT9ipjIUS/9pyhvL8c5ayJH13VzO4DTbduuAbBcCDEDwHLjPQCcgcyi4DMALAVwc/huMuWCbI3V18Rn0X/hg4fgujNnWbb95zmHAQAmjhiCq06aFlvbVUfMaYqZ+NESeiHECgD2Rb7PBvBL4/UvAZwjbb9DZFgJoJWIxkbRWaa8SFp89N4Edac46VAhjMJKF7SoP16Fn66SJsxg7GghxDbj9XYA5moS4wG8LZXbYmyzQERLiaiTiDq7u7tDdIOJkrD6qByMrSBVZM+CO9k1XDXPUwX9NEqWSKJuRMap5uvnL4RYJoToEEJ0tLe3R9ENpgSQB2N1k4uZ8AVfmvhOORDye6wko6BUCCP0O0yXjPHXzAG7FYCcEHyCsY0pA8JeYtY4en+um6CYlqMvfWAt8SSo4PqfMMVfRtyEEfo/ALjYeH0xgAel7RcZ0TcLAOyWXDxMifDUum7s68uPN4/CdTN3YiuOmDCsJFMRREllf7rwRJ187IoTpkRaXzWhFUdPRPcAWARgJBFtAfB1ADcAuJeILgOwGcDHjOIPAzgTwHoA+wFcGnGfmZC8vXM/Lrz1eZx5+Bj85JPzI62biPD7q44DYF3RSUfzy+W+wNkT3fH7JGAv7hReuemGJWG6VPVoCb0Q4gLFrsUOZQWAq8J0iomXff0ZS3591978nSE1TL5oWxvrwlWmyTnzxuGxtTswe2zwafNtTXVo4aXoHPE7+Fwm9+uqgmfGljGlaFnK/tbGuiSOmTICz725UzMFQjCJOGvOOJw1Z1ygY01WffXUUMdXIlE/YZXLE1slwrluKoBIr5/QERPRdKMsqJIP6z+pmXFc6dkhVQsLPWMlrOtGtb2SfPRVImBRLQTjWd6+Jmy5/BDKCBZ6JlLKJdKmPHpZGvjNPVMmP4GqgoWeiRTlwtA6x0bYjyoxumOlYGmK+bYbOyz0TKRU0yVbTZ/VD+o0xXpnjLNXRg8LPRMtIUz6KH2zcWoFy5AHPr9GdvXEDws9YyFsyKbP9DYW+HovTXxH3YT8JnkwNnpY6KuQOH2iqrrZD1uOOH9n5sS0kUOjmRDHuh4/PGGKiZQwF20hL/gorMZKF6jWIZmZwjPHNFu2/8tph+CDh47BoeOGOR5HKic9UzRY6JlIqXDty1Alg4XT2ofiN0sX4IiJrZbttckE5h80XHkcZ68sPdh1U8VEqVdnHj4GgNt6od6NlZtvthoE6pipbb4XeTfXISjFFB3VCgs9k+WJ17vw8MvbAx07cfgQAOUn1kz0ZD03Np1XBmRpZK9kwsFCz2T59fNvexfygHWeyea6sW1n/S4eLPRVSJyP1KXizojzM7JeuWM+1ela5vZfDD8VRg8LfRWSTsdXd7lco1F0s1w+a6FRzozl81U0AkfdENEhAH4jbZoK4GsAWgFcAaDb2H6dEOLhwD1klAR9FFZZu2GsYLZyGZNE1qLXK883gPgJLPRCiNcBzAUAIkoiswD475BZOvBGIcR/RdJDJnLi9JXyNcuEFW4ejI2eqFw3iwFsEEJsjqg+xg8+ryzVdVQq/vVSh3XIHf+/I/7dxU1UQn8+gHuk91cT0Woiuo2I1DMrmNj5ziOv4Z7n37Jsi8N1E5SaRPkOE7E8ORPWoufB2OgJfZURUR2ADwP4rbHpZgDTkHHrbAPwfcVxS4mok4g6u7u7nYowEXDzExtw7QMvW7alY9DzoI/bV5w4JeKeMMVGN17eazsTHVGYU2cAeFEIsQMAhBA7hBApIUQawC0AjnY6SAixTAjRIYToaG9vj6AbjC6mKMdhv/u9aIfUFScLB4tLfLALsPSIQugvgOS2IaKx0r5zAayJoA3GDZ/WNLuYw8FT+93x643j20L8hDKniKgJwKkAPiNt/i4RzUVGTzbZ9jEREtQq5aiGaOCnAmfYoi89Qgm9EGIfgDbbtgtD9YiJnTh0XhnJw9d81eF38Rn74CsbItFTviEPTGBUl9H/vbIjcJ3JZOZiHUhZay/aNctaUTT45l56sNBXIWmHsJvegVSoOofUZh4OD/SHq6ccYIPTC39Kz7lu4oeFvgqJQ6eG1GVylu+vAqE3YUFyRuW6Yd998WChr0LSMZikjYbQHxgYjLzuOGDJiQ+/N0C+X8YPC301Yui8POgVVvtNi74aXDeMO6zbpQcLfRXipOlhrfys0If09ZcD7KJ3J+HXoudbQ+zw4uBVxCvv7EZ9TcJR1L3Eq6kuiX0u1nqjMcO1mnz0jDPsiik92KIvY/wa4Utuehqn/GCF43FeFv0r3zzddf/ssS0AgI8fNdFfp4pEGKuco26ihW8M8cMWfYXy2vYetDbWOe5ztOhDild7cz023bAkXCVMRZDQmDF1yqzReGxt8HkbjD9Y6CsBB5Po9B8+pSzupOnVNhuRjcj4UJ5bacdnF03F2m092Pr+gUJ0qeph103MrO/ag57egdjb2dC9F7v3a7bjoOlx6XwlP5ZX8mcLg95gLJ+8QsJCHzOn/GAFPnHLytjbWfz9J3HW/6iteBkn100csfXFJM5Pw9kr3dHRedm7wzfM+GGhLwBrtvbE24Ah0m/v1HsMNjXdlKtUWmBD974YOsZUI1r2PKt7QWEfvY3uPX0YWl+TnelZDvi1L+3lb1q+Dj9avi6q7jBVjo6I+81wyYSDLXobR13/GM776TPF7oYv/Hpd7G6aFzbtjLA31QNP9HFGpfMDg+lcGT53BYWF3oFX3sm4WgZS6VADqYWKZNFpR85OKRcXQmCX7iBuAErV9R/KdVCin6lUUJ3ZN9/NuQedTv/RU0YAAEY118fQq+qGhd6Fq+56EXO+8efAxxdK5HSaWfS9J3LlpY4tW7ERa7fFPIbAVBWqqJtZxqQ6wFnov3DqIXjiXxZh8simuLpWtYQWeiLaREQvE9FLRNRpbBtBRI8S0Trj7/DwXS08f3413ISOQkWy6DSzvac3V17avvy1rug7xFQ1qoel73xkTq6Mg92fTBCLfExEZdGfJISYK4ToMN5fA2C5EGIGgOXG+6IhhAjlRgl6rMP6HoHa9mrf92CsdEANj4oFhgNHnFG5xRrrkjiobYhRppA9YuJy3ZwN4JfG618COCemdjzZfWAAU659GLc8tTFwHVOufRiPrNnu+7iwFn1Pb6bvy1a4993vjUjuV02SvXd+YRe9O24ibv70/Ga4ZMIRxVUuAPyZiFYR0VJj22ghxDbj9XYAoyNoJxDde/oAAL9+/u1Q9TyyZpt3IRuX/OL5UG1m+/6Ce99T0qPDqs27POuVhYoteiZq3H5R5mQz1vnCEoXQHy+EOBLAGQCuIqIT5Z0iY27mGUFEtJSIOomos7u7O4JueODxw1q1eRcmX/PHSJtcubEwYYspyUJ/8g3vcyk/ASSrVOir81MXBjdr3fzp8fkvLKGFXgix1fjbBeB3AI4GsIOIxgKA8TdvxE8IsUwI0SGE6Ghvbw/bDbceapX68yv+XTMyj6zZhsnX/BHv7e0LVY+MrkcmlfLnTMjWK4DaZGVecoUIba3MMxceHdcNz4wtLKGEnoiaiKjZfA3ggwDWAPgDgIuNYhcDeDBMO2HQtiBC/u5ue3oTAGBd195wFVkwHnM9SqVsouY9eCv56BPx+uiLfT1/4OB2XLTwoEjrrLZMn37RmQxV7N9FtRE2BcJoAL8z7s41AO4WQjxCRC8AuJeILgOwGcDHQrYTGPOS9LIgopqpF+XvN6snikrN3Wk5vEcIz2iftDFBceO7+7Dx3eLnuHnyXxdh03v7Y6n7uOltsdTLqHG36DM/Ttm9w/fN+Akl9EKIjQCOcNj+HoDFYeqOGi8B9rIwivlb9GvRpzyUvtSuq4PamnBQW3zx0ywkhcVV6M0yBekJY8KxdQbl+MO7+7m3AACDsrATeYZ1Bg37/Omn5uObZx8a6NhSIOpxZ3Y/OOP2dJzz0ReoMwyAKhB6XU0LG9cbR45yN7dT32Aud02+68ajLwG7evphY3DI6OZgB5cAl58wNfu6IUR2Un5CcMfthmpeJxxHX1gqX+g143ZL8XdnCrZT13bu68++HrS5auS3R0xszTuWF84Arj1jZug6OAOjM27jYXyTLA4VL/Smv9rroozqko0ybCzbd4cqb39mU/Z1Oq320dc7zHwNk5qhUsLimhtqi92FikXHRVYhP6OyoeKF3oww8fxheRSIyhLp6R3AZ+9cZbHIVaTT6n0/ezKXFsHuo5fD/+pq8r/iarOqquzjFh1Xi974y66bwlL5Qq+paoX62d218i38ac12/OzJDZ5lU0L9NDJXcslYPqMQFoveUehDSJ9XRE8pE9V3XL5noPjwYGxxqHiht4ceqgg9GGuuw+rRntmMTq9k18223Qfw9QfXYDCVMfOHD8m5HgZTah99vYPQ+9Xqfzs958/2K/Sl9vTwlSWzcNslHd4FNWCxCkK+8cLnMX4qfs1Yu/9aRVQ/Nq8bi9mMzuxK2VK/7oGX8fjr3Vg8azROPLgdA5K429tMe7hu/KZ7+Oyiacq2ShXbQ04WOfKGKTy57JXF7Ue1UfkWfdYq9vfLksMX3di2+wBufPQNaZaqe/msRa+hl3LfzfuVKbT9qXReOROL0DsMxj617l3vxpV98viANoplrS09cSqOndaGj86fKHWmOH1hcmR/mfxdFJSKt+hTLiGKALC3bxDLVmzM23/fqi2W9ypd/uydL+Klt99Hc0ONpb0okJ9GzCyT6bRAT+8Ann8zlxnTLvRePvowpGw6/+njpmDC8MZI24iCUS0NuPuKBQCA8+ZPwMMvb8PFCydHUneZPNSUJMJl3ImJj4oQ+kfWbMOYYY2WAUoTr4vy2G8vR0/vIKbaljCz+71V7NqfiZ4xLWcvV5H5A3cqtXztDjQ31GYXSZZvUuYYQiot8N1HXrMcZxd6+TNHL/RWpf/ah2ZHWn8ctDfX43//6fhid4OBHHVT1G5UHRUh9Ffe+SIAYNMNS/L22UVw1ead2H1gACfPzKyF0tM7CAAY8OmSMOkbyBz3nhEu6TVY6ea6ueyXnQByn0MejDUvjLQQGBh0z20jv6+NeAUpOZTzC6ceHGndTPVQKfMxyoXK99GLnFgCwEdufhafvr0T/YNpiyDaHyXtv0PV4Kndl69y3dzx7CZLe398+R3P+tNS303XTSqdb6XLi5gLWH30UV9O5mf40BHj8M+LZ0Rce3lw/tETUZsknHbomGJ3pezgwdjiUBEWvRtpxezSLbv2Y1xrzrcc1MDoHbA+CahcN1978BUkiLKWzI6ePnTt6cWo5gYAQM+BwbxjZH94whR6ITzdMTpzB2oSlJc6QQdT6Ct0vRItDh7djHXXn1nsbpQl7KMvDpVv0StSIKTSAgOSktrj6O0/w4dWb0PvQAo79/XjMcmC1rXogdwasCZrtu7Ovn53X/7KVHLfk0b/Xty8y1XohQDuf3ErAODHnzhSWW54U51ynxvmzSEZ84IlTGXC49jFoeKvVpV1O5ASllh0HYv+0l+8gJO//wQuv6MTPb0DRv3WMm4++t7BlOUGIj8N9A7kh3M6uW5uf2aTY8ikyYN/34qbn8jMunV7PJYnXNn50umHKPeZn48XFWcCYc6MrXjlKS0q/nSburuxey9e374nu91u0efJloPyP7vxPby/PyPwqqgcN7dJb3/KUq0s7n2D+YPBOYve+sSxoVu9XOGufQPZ14kEKRf/Hlrv7LX77ZUL8Y+Lpivrz7puNH03nzzmIADAjFHlm96YiY5s6u2i9qL6CCz0RDSRiB4noleJ6BUi+pyx/RtEtJWIXjL+F9WZuXLjewCAff0pnPbDFdntvYMpPPF6bs1yuz57/RBVgm6PM5fpHUgHsuhBZLHOH1q9TdnG3r6crz9BhBqF9T+s0dmi9zLUj5qcCf0887Cx7gUNPnTEOGy6YQnam+u1yjOVjRx0cMmxkwEAw4cEcyMy+oQZjB0E8EUhxIvGAuGriOhRY9+NQoj/Ct89b2RXyd6+QWx6dx8OGz8MQGbA9Y5nNzsed9PydZYZonbfupcrR2nRG/155Z3deft6B1OWsDKVRX+gP4XXd+yxfDaVZe5GMqF2sUxrH4rHX+922OPeziFjmh3DWBlGh6xFT4QrTpyKK07klBSFILBFL4TYJoR40Xi9B8BaAOOj6pgusvvlil924qz/fjqb+MvJSjbZ2G1dFNvvbMcBhemeMrJHLrnp6bx9+/qskTW9gyns7RvEGzv2oE/q63W/exnn/Piv2N7TC8Bw3QQQ+rpkUnmDmDzSeY1We/Gz547z3S7DqMg+pBa3G1VHJD56IpoMYB6A54xNVxPRaiK6jYiGK45ZSkSdRNTZ3e1kWeohhwiu2rwLQC4PTI1LZIhdAP2m7lWFJqbSAj0HBhz3vb9/wOLy6R1I49O3v4AP3rgCByShN6NxzCgdIuCt9/bn1ee1rF9dTUJp0bcqBmPtE1lu/Nhc1zYYJgg8X6qwhBZ6IhoK4H4AnxdC9AC4GcA0AHMBbAPwfafjhBDLhBAdQoiO9vb2wO0PyiGSxqcxZ6u6DVrahd4+MdYrznf77l7sclg8JC1ENiLHzq79/RZ3zN/e2pXNWbN7f+6YRmM9U3Pg90B/Ck+vz09E9uDVx7n2sa4mkf2cY4c1WPapBmPt94UgTxIMAwAdB+XbeMIhTTETP6EmTBFRLTIif5cQ4gEAEELskPbfAuChUD30YEdPLv7cjEzpG0zj7Z37sykFnLDrlz3qxcviuOCWlY7bU2mB3QqLfpfNopfHCOQVpxprM0LftafXsW8mTrnmZeqSOYt++qih2La7N7tPLfR8ATLhefkbH0R9Tf4C7GcePhYPvLgVNdU8464IhIm6IQC3AlgrhPiBtF0OxzgXwJrg3fNGjqQxRap/MJ31b6uwW/T9mmmJvXAX+n6ly2fnfknoDYt+y64Drm155Qupq0kgaUTdTBwxBJNGDMnua6jNXYRLDteLoGEYXZobah0n9n3nI3PQ+ZVTIs/BxLgT5mwfB+BCACfbQim/S0QvE9FqACcB+H9RdFSHQcP/0jeYwv5+d+G2PzoO2KJoVGLtRVoIbHvf+SYjBLCnNz/VAWC16M1ZsJsNv7zOIiVO1NckUGumTkgJy2xYWeh/eP7c7EXJFj0TJ7XJBEYO5VDbQhPYdSOEeBrOg+cPB+9OOEz/d99gGjsdUgrIHLBF5NijaG74kzUVsC4H+tN4cXOXcv877ztb6V2SC6rTGFQ2UbluvJB99APpNAakekY01eHFr54KIQRqkwlMGjEE67v28iAZw1QgFZXUbNAi9O4W+f7+FOqSiWyETpAEX07c+NgbrvsffOkdx+079uSeAuxPE7Jv3Q91yUTWFyrPBP7JJ4/ECFuuGzOtQjkv/s0wjDMV5SgzPRx9AynsUUS+mBzoH/QczCwk7yjcPV7MHJMJsXz+y4vz9mUs+pyAmze1WWNb8srWGufCLMPBNgxTOZS1Ra/yXX/i58/h8uOnuB67fyCVmXrt7uEpGKm0wGHjW/DFUw9B155e/OKvm/CalJtHxR2XHY1d+way6Y5l6moSGNOS2X5Q2xC8aLiEnAbJ6gzL33TvrLxusWPqZIZhyo/SMWkDYB9Aldnb5y5SQmTcFSuvzbeE4+I3Sxc4bjeFd83WHpw0cxQ+ftQkNCnCH+2Mam7AIWOcJ07VJAhHTxmBOy87Bp8/5WBMGzUUADCkNj/szYyCMM/pqOYGTDfKMwxT3pS50KsHKf+0Zrvn8bU1hDHD8i3huJjani+cf/rcCeh3GGwNGmkjY4ZfHj9jJGqTCfz4k0fi7iuOccxFP7U9kxLBDO1kGKZyKGvXjV0gG2oT2YyQOuGRUcbyEnnny2luyD/dU9ubssdef+5h2e32lav8cvflx+Rta2moxbHTRjqW/8qS2Th+ejvmO8xmZBimvKkoi753II2pimRdThzwiLV342MdEyzvj5jQ6lr+F5ce5Tj4Ky8iMndiro7RLfmxxuN8PH0cO91Z0FU01CZx+mG8BirDVCJlLfS68eX/fcE8S/51MyOjavKSDt869/Ds6x+dPxe3XXKUY7nvnTcHd11+DE46ZJRlJusQw0Uib2uuz/XxBw7JxNo8Jpo8+a+L8B9nH4pfKPrCMEx1UtauGycf/cZ39+Vtm3/QcBw+flg2MdiFCw7Cgy+9Eyhm/J4rFmBIXdKyoMfZc/OzM48cWoerTpqOjxw5wZIYbNmF8zFlZBPqa5J5OeuHSq6d4U11uO/KhXhjx15c97uXM20vXYDfvbgFX33wFce+HdTWhAsX6j/RMAxTHZS10Jsx36fMGoWp7UOxbMVGAMBtl3TgnuffxpNvdGf9+PJNwbTuzcVGHvqn47F8bZfjZKefXTgfo5rrce5PngEALJzWlt131+XHOKYM+N55c7BgahsmSrllTD54aM49Mqkts799aD269vShqd46ENoxeUR2ZaYJwxsxtL4GFy6cjLqaBI6Z0gaGYRgdylroBwYzQn3+UZNwyuzROGJCK9qb63H0lBE4eeZonPGjp7B2Ww/SQmSjSf7p5OloMYTeXA3qsPHDcNj4YVg8axTWde3BrLEtePCldzC9fShOM4R52YXz0TbUGq1ynMIP/tGOib4+x72fWYhnNrznmO3PHDCWs01+/KhJvupnGKa6KWuhb2mswbnzxmNcayMAYMkcaxbGn1/cgf/9+zsY39qI7543B/e+8DauOml61rc/1BYFYwo+AMw83Tp7VLbEVdx52TGeM3KdmDyySbni09hhDbjuzJk4Q3ONVoZhGDsURbx2WDo6OkRnpzp3fBz87MkNOGnmKBzssUoTwzBMqUJEq4QQHV7lytqiD8NnPjCt2F1gGIYpCGUdXskwDMN4w0LPMAxT4cQm9ER0OhG9TkTrieiauNphGIZh3IlF6IkoCeDHAM4AMBvABUQ0O462GIZhGHfisuiPBrBeCLFRCNEP4NcAzo6pLYZhGMaFuIR+PIC3pfdbjG0MwzBMgSnaYCwRLSWiTiLq7O7uLlY3GIZhKp64hH4rADkPwARjWxYhxDIhRIcQoqO9vT2mbjAMwzCxzIwlohoAbwBYjIzAvwDgE0IIx7SLRNQNYHOIJkcCeDfE8ZUEnwsrfD5y8LmwUgnn4yAhhKelHMvMWCHEIBFdDeD/ACQB3KYSeaN8KJOeiDp1pgFXA3wurPD5yMHnwko1nY/YUiAIIR4G8HBc9TMMwzB68MxYhmGYCqdShH5ZsTtQQvC5sMLnIwefCytVcz5KIk0xwzAMEx+VYtEzDMMwCspa6KstcRoRTSSix4noVSJ6hYg+Z2wfQUSPEtE64+9wYzsR0U3G+VlNREcW9xPEAxEliehvRPSQ8X4KET1nfO7fEFGdsb3eeL/e2D+5mP2OAyJqJaL7iOg1IlpLRAur9fdBRP/PuE7WENE9RNRQrb+NshX6Kk2cNgjgi0KI2QAWALjK+MzXAFguhJgBYLnxHsicmxnG/6UAbi58lwvC5wCsld5/B8CNQojpAHYBuMzYfhmAXcb2G41ylcaPADwihJgJ4AhkzkvV/T6IaDyAfwbQIYQ4DJkw7/NRrb8NIURZ/gewEMD/Se+vBXBtsftV4HPwIIBTAbwOYKyxbSyA143XPwNwgVQ+W65S/iMz63o5gJMBPASAkJkEU2P/nSAzr2Oh8brGKEfF/gwRnothAN60f6Zq/H0gl29rhPFdPwTgtGr9bZStRY8qT5xmPFrOA/AcgNFCiG3Gru0ARhuvq+Ec/RDAlwCkjfdtAN4XQgwa7+XPnD0fxv7dRvlKYQqAbgC/MFxZPyeiJlTh7wxjQZIAAAH1SURBVEMIsRXAfwF4C8A2ZL7rVajS30Y5C33VQkRDAdwP4PNCiB55n8iYJFURSkVEZwHoEkKsKnZfSoQaAEcCuFkIMQ/APuTcNACq5/dhjEOcjczNbxyAJgCnF7VTRaSchd4zcVolQkS1yIj8XUKIB4zNO4horLF/LIAuY3uln6PjAHyYiDYhs+bBycj4qFuNfEuA9TNnz4exfxiA9wrZ4ZjZAmCLEOI54/19yAh/Nf4+TgHwphCiWwgxAOABZH4vVfnbKGehfwHADGMUvQ6ZgZY/FLlPsUJEBOBWAGuFED+Qdv0BwMXG64uR8d2b2y8yoisWANgtPcKXPUKIa4UQE4QQk5H5/v8ihPgkgMcBnGcUs58P8zydZ5SvGOtWCLEdwNtEdIixaTGAV1Gdv4+3ACwgoiHGdWOei6r8bRR9kCDMfwBnIpMlcwOALxe7PwX4vMcj89i9GsBLxv8zkfElLgewDsBjAEYY5QmZyKQNAF5GJgKh6J8jpnOzCMBDxuupAJ4HsB7AbwHUG9sbjPfrjf1Ti93vGM7DXACdxm/k9wCGV+vvA8C/A3gNwBoAvwJQX62/DZ4ZyzAMU+GUs+uGYRiG0YCFnmEYpsJhoWcYhqlwWOgZhmEqHBZ6hmGYCoeFnmEYpsJhoWcYhqlwWOgZhmEqnP8PPYIeZ/CGXHgAAAAASUVORK5CYII=\n",
"text/plain": [
"