深度强化学习：PPO + AI2THOR问题总结

一、PPO代码

选用的是如下repo中的PPO代码：

/MorvanZhou/Reinforcement-learning-with-tensorflow/tree/master/contents/

二、网络结构

1.网络变量

项目中的网络输入针对的是向量，但AI2THOR环境下的观测状态及神经网络需要的输入是一个三维张量，因此需要修改项目中的网路结构，S为[None, 84, 84, 3]，Q_tar为[None, 1]，Q_eval为[None, 1]，pi为[None, action_size]，a为[None, ]

2.Actor和Critic网络

#s的输入顺序：s->conv1->relu->conv1->relu->fc->relu->out_put_fcwith tf.variable_scope("base_conv", reuse=reuse):# WeightsW_conv1, b_conv1 = self._conv_variable([8, 8, 3, 16], "base_conv1")W_conv2, b_conv2 = self._conv_variable([4, 4, 16, 32], "base_conv2")# Nodesh_conv1 = tf.nn.relu(self._conv2d(self.tfs, W_conv1, 4) + b_conv1) # stride=4h_conv2 = tf.nn.relu(self._conv2d(h_conv1, W_conv2, 2) + b_conv2) # stride=2with tf.variable_scope("base_fc", reuse=reuse):W_fc1, b_fc1 = self._fc_variable([2592, 256], "base_fc1")# Nodesconv_output = tf.reshape(h_conv2, [-1, 2592])# (-1,9,9,32) -> (-1,2592)conv_output_fc = tf.nn.relu(tf.matmul(conv_output, W_fc1) + b_fc1)#out_put_fc经过fc、softmax后得到piwith tf.variable_scope("base_policy", reuse=reuse):# Weight for policy output layerW_fc_p, b_fc_p = self._fc_variable([256, A_DIM], "base_fc_p")# Policy (output)base_pi = tf.nn.softmax(tf.matmul(conv_output_fc, W_fc_p) + b_fc_p)#out_put_fc经过fc后得到vwith tf.variable_scope("base_value", reuse=reuse):# Weight for policy output layerW_fc_p, b_fc_p = self._fc_variable([256, 1], "base_fc_pc")# Policy (output)v = tf.matmul(conv_output_fc, W_fc_p) + b_fc_p

3.变量赋值

s = s[np.newaxis,，；，：] #s是[84,84,3],tfs是[None, 84,84,3],需要给s增加一个维度pi = self.sess.run(self.pi, feed_dict={self.tfs:s})

4.批训练

变量维度中的None则对应着batch

5.Tensorflow中变量的类型要是numpy,不能是python中的list,否则会出现报错或错误的矩阵运算结果, 影响神经网络以及训练的效果.

def update(self, s, a, r):self.sess.run(self.update_oldpi_op)s = np.array(s)a = np.array(a)r = np.array(r)r = r[:, np.newaxis]adv = self.sess.run(self.advantage, {self.tfs: s, self.tfdc_r: r})#s如果不转化为numpy则求得的v的shape变为(32, 32)# update actor[self.sess.run(self.atrain_op, {self.tfs: s, self.tfa: a, self.tfadv: adv}) for _ in range(A_UPDATE_STEPS)]# update critic[self.sess.run(self.ctrain_op, {self.tfs: s, self.tfdc_r: r}) for _ in range(C_UPDATE_STEPS)]