我正在尝试创建一个带有 2d 卷积层的简单 Deep Q 网络。
我无法弄清楚我做错了什么，我唯一能看到的似乎不正确的是，当我在优化器步骤之后获得状态的模型预测时，它似乎并没有更接近目标.
我在 OpenAI 的健身房中使用来自 pong 的像素，其中包含单通道 90x90 图像、32 个批量大小和重放内存。
例如，如果我尝试使用 1 的批量大小，并尝试self(states)在优化器步骤之后立即再次运行，则输出如下：
current_q_values -> -0.16351485 0.29163417 0.11192469 -0.08969332 0.11081569 0.37215832
q_target -> -0.16351485 0.5336551 0.11192469 -0.08969332 0.11081569 0.37215832
self(states) -> -0.8427617 0.6415581 0.44988257 -0.43897176 0.8693738 0.40007943
这看起来像一个步骤所期望的吗？
带有损失和优化器的网络：
self.in_layer = Conv2d(channels, 32, 8)
self.hidden_conv_1 = Conv2d(32, 64, 4)
self.hidden_conv_2 = Conv2d(64, 128, 3)
self.hidden_fc1 = Linear(128 * 78 * 78, 64)
self.hidden_fc2 = Linear(64, 32)
self.output = Linear(32, action_space)
self.loss = torch.nn.MSELoss()
self.optimizer = torch.optim.Adam(
self.parameters(), lr=learning_rate) # lr is 0.001
def forward(self, state):
in_out = fn.relu(self.in_layer(state))
in_out = fn.relu(self.hidden_conv_1(in_out))
in_out = fn.relu(self.hidden_conv_2(in_out))
in_out = in_out.view(-1, 128 * 78 * 78)
in_out = fn.relu(self.hidden_fc1(in_out))
in_out = fn.relu(self.hidden_fc2(in_out))
return self.output(in_out)
然后是学习块：
self.optimizer.zero_grad()
sample = self.sample(self.batch_size)
states = torch.stack([i[0] for i in sample])
actions = torch.tensor([i[1] for i in sample], device=device)
rewards = torch.tensor([i[2] for i in sample], dtype=torch.float32, device=device)
next_states = torch.stack([i[3] for i in sample])
dones = torch.tensor([i[4] for i in sample], dtype=torch.uint8, device=device)
current_q_vals = self(states)
next_q_vals = self(next_states)
q_target = current_q_vals.clone()
q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()
loss = fn.smooth_l1_loss(current_q_vals, q_target)
loss.backward()
self.optimizer.step()
```