Page 9 - jamshid

Network Architecture
The convolution network used in the experiment is exactly the one
proposed by proposed by Mnih et al. (2015), we only provide de-
tails here for completeness. The input to the network is a 84x84x4
tensor containing a rescaled, and gray-scale, version of the last four
frames. The first convolution layer convolves the input with 32 fil-
ters of size 8 (stride 4), the second layer has 64 layers of size 4
(stride 2), the final convolution layer has 64 filters of size 3 (stride
1). This is followed by a fully-connected hidden layer of 512 units.
All these layers are separated by Rectifier Linear Units (ReLu). Fi-
nally, a fully-connected linear layer projects to the output of the
network, i.e., the Q-values. The optimization employed to train the
network is RMSProp (with momentum parameter 0.95).

Hyper-parameters
In all experiments, the discount was set to γ = 0.99, and the learn-
ing rate to α = 0.00025. The number of steps between target net-
work updates was τ = 10, 000. Training is done over 50M steps
(i.e., 200M frames). The agent is evaluated every 1M steps, and
the best policy across these evaluations is kept as the output of the
learning process. The size of the experience replay memory is 1M
tuples. The memory gets sampled to update the network every 4
steps with minibatches of size 32. The simple exploration policy
used is an -greedy policy with the decreasing linearly from 1 to
0.1 over 1M steps.
Supplementary Results in the Atari 2600
Domain
The Tables below provide further detailed results for our experi-
ments in the Atari domain.