网络工程 - Aruba 交换机 - 速率限制导致 TCP 数据包重新排序？ - 吾爱随笔录

我们刚刚完成了对由 Aruba 交换机聚合的洲际 WAN 链路上的一个非常有趣的带宽问题的调试。根据 tcpdump 捕获，在 ProCurve 3500 交换机的端口上使用任何速度的速率限制器将导致交换机随机重新排序通过它发送的 TCP 数据包！

开关是最新的固件版本，我找不到关于这个特性的勘误表。对数据包重新排序会导致 TCP 快速重传每隔几十个数据段启动一次，因此即使在该链路的速率限制器设置为 100Mbps 的情况下，洲际 TCP 连接在好日子也只能达到大约 100Kbps。

使用 UDP 流进行的测试表明交换机在原始数据流的实际带宽限制方面按预期运行。交换机仅开始重新排序高于某个最小阈值带宽的数据包，在我们的测试中，这大约是 4Mbps。在不更改任何其他内容的情况下清除速率限制允许在高延迟链路上进行全速 TCP 传输，并且 TCP 快速重传次数要少得多（更典型）。

使用 iperf3 完成了测试，并广泛使用了 -b 标志，以使测试数据流远低于配置的速率限制。这样做是为了避免在受限端口的实际拥塞期间拖累速率限制器行为的额外异常。

有没有其他人看过这个？这种行为对于此类中的开关是否正常？ 出于各种原因，我们更喜欢端口限速架构；这些交换机用于与具有分层数据速率和 IP 过滤器/ACL 的消费者系统接口，但显然，如果它们正在重新排序数据包，则需要更换它们。

任何帮助表示赞赏！

编辑：

一旦明确交换机导致带宽问题，两台独立的主机直接连接到交换机上的两个端口，并在两台主机上使用 iperf3 和 tcpdump 来检查交换机上的数据流。一直看到的是几个数据包的“块”被延迟/乱序移动，但还不够快速重传开始的地方。基本上如下：

HOST A (receiver)       HOST B (sender)
SEQ 1 <         < SEQ 1
SEQ 2 <         < SEQ 2
SEQ 3 <         < SEQ 3
ACK 1 >         > ACK 1
ACK 2 >         > ACK 2
SEQ 4 <         < SEQ 4
ACK 3 >         > ACK 3
ACK 4 >         > ACK 4
 ... now the interesting part ...
SEQ 7 <         < SEQ 5
ACK 5 >         < SEQ 6
SEQ 8 <         < SEQ 7
ACK 5 >         < SEQ 8
SEQ 5 <         < SEQ 9
SEQ 6 <         > ACK 5

注意从接收者的角度来看，几个数据包是如何在传输序列中更早移动的；还要注意在快速重传开始之前接收到的序列号是如何向后跳的。为了清楚起见，我省略了序列的其余部分，因为一旦发送者开始处理多个 ACK 数据包，快速重传无论如何都会开始，并且事情看起来很正常恢复路径。

配置是沼泽标准。交换机重置为出厂默认设置，速率限制应用于其中一个测试端口，如下所示

config
int 28 rate-limit all out kbps 100000
int 28 rate-limit all in kbps 100000
write mem

这足以导致下降。两台主机都是 GbE 在已知良好的短电缆上，撤消速率限制配置（无 int 28 ...）会立即恢复性能并停止重新传输。

编辑2：

捕获了新的转储，即使 NTP 处于活动状态，主机之间可能仍存在少量时间偏差，但我不确定如何将其考虑在内。

问题从序列号 10515814 开始——注意在 10515814 的三个 ACK 甚至离开接收主机之前接收到 10515814 数据包。除非我误解了什么，这意味着快速重新传输甚至没有机会启动，显示重新排序与直接下降？

接收主机端口上的交换机配置速率限制为 100Mbps，发送主机端口不限制。iperf3 以 40Mbps 的传输限制运行（凭经验选择，因为在这些条件下更容易在转储中找到重新排序，同样的事情分别发生在 10Mbps 和 4Mpbs 上）。

接收者：

21:24:07.302170 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10494094:10501334, ack 1, win 229, options [nop,nop,TS val 1242980393 ecr 1549151364], length 7240
21:24:07.302182 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10501334:10502782, ack 1, win 229, options [nop,nop,TS val 1242980393 ecr 1549151364], length 1448
21:24:07.302188 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10501334, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980393], length 0
21:24:07.302343 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10502782:10507126, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 4344
21:24:07.302358 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10507126, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980393], length 0
21:24:07.302375 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10507126:10510022, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 2896
21:24:07.302388 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10510022, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.302553 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10510022:10515814, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 5792
21:24:07.302568 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10517262:10518710, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302575 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13478, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10518710}], length 0
21:24:07.302727 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10518710:10520158, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302734 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13504, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10520158}], length 0
21:24:07.302876 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302883 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10520158, win 13496, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.303094 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 2896
21:24:07.303101 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10523054, win 13504, options [nop,nop,TS val 1549151366 ecr 1242980394], length 0
21:24:07.303256 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.344514 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395], length 0
21:24:07.344674 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10527398:10530294, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 2896
21:24:07.344679 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395,nop,nop,sack 1 {10527398:10530294}], length 0
21:24:07.344870 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10527398, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 2896

发射机：

21:24:07.313522 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10510022:10511470, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313525 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10511470:10512918, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313527 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10512918:10514366, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313533 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10514366:10515814, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313537 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10510022, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.313545 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313548 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10517262:10518710, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313728 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13478, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10518710}], length 0
21:24:07.313741 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10518710:10520158, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313744 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10521606, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313747 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10521606:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313753 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313756 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10525950, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313885 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13504, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10520158}], length 0
21:24:07.313896 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314085 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10520158, win 13496, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.314096 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10521606, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314099 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10521606:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314263 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10523054, win 13504, options [nop,nop,TS val 1549151366 ecr 1242980394], length 0
21:24:07.314273 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.314276 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10525950, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.314281 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10525950:10527398, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.355648 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395], length 0
21:24:07.355661 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10527398:10528846, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 1448

编辑3：

开关配置。如前所述，非常基本：

Running configuration:

; J8693A Configuration Editor; Created on release #K.15.16.0019m
; Ver #0f:01.80.00.00.34.00.00.00.00.74.fc.7f.ff.3f.ef:62
hostname "TEST"
module 1 type j86yya
module 2 type j86xxa
no timesync
no telnet-server
no web-management
interface 28
   rate-limit all in kbps 100000
   rate-limit all out kbps 100000
   exit
snmp-server community "public" unrestricted
vlan 1
   name "DEFAULT_VLAN"
   untagged 1-48
   ip address dhcp-bootp
   exit
no tftp client
no tftp server
no autorun
no dhcp config-file-update
no dhcp image-file-update