パフォーマンスチェック　Infiniband or 10G

Infiniband

何とか、ConnectX2を、 Infiniband と 10G ether で使える状態になったので、パフォーマンスチェックをやってみた。

＜壱號機：eth2 ConnectX2 10G Mode Port＞
eth2 Link encap:Ethernet HWaddr 00:02:c9:0e:7f:49
inet addr:172.22.1.112 Bcast:172.22.1.255 Mask:255.255.255.0
inet6 addr: fe80::2:c900:10e:7f49/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:131037059 errors:0 dropped:0 overruns:0 frame:0
TX packets:1932936974 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19505299558 (19.5 GB) TX bytes:2925739799190 (2.9 TB)

ib0 Link encap:UNSPEC HWaddr 80-0C-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:172.22.0.112 Bcast:172.22.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c903:e:7f49/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:184444030 errors:0 dropped:0 overruns:0 frame:0
TX packets:193080077 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:7896001718587 (7.8 TB) TX bytes:8519499301671 (8.5 TB)


＜弐號機：eth2 ConnectX2 10G Mode Port＞
eth2 Link encap:Ethernet HWaddr 00:02:c9:0e:c5:3b
inet addr:172.22.1.114 Bcast:172.22.1.255 Mask:255.255.255.0
inet6 addr: fe80::2:c900:10e:c53b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2986024748 errors:0 dropped:3585 overruns:3585 frame:3585
TX packets:2100897385 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4347705345828 (4.3 TB) TX bytes:2928382445700 (2.9 TB)

ib0 Link encap:UNSPEC HWaddr 80-14-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:172.22.0.114 Bcast:172.22.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c903:e:c53b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:131130226 errors:0 dropped:0 overruns:0 frame:0
TX packets:60575118 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:8443599834853 (8.4 TB) TX bytes:2666134749 (2.6 GB)


◎パフォーマンスチェック
★10G Ethernet<壱號機側:iperf server>
% iperf -s

                                                                                                                      • -

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)

                                                                                                                      • -

[ 4] local 172.22.1.112 port 5001 connected with 172.22.1.114 port 60915
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 10.3 GBytes 8.85 Gbits/sec<弐號機側:iperf client>
% iperf -c node2 -i 1

                                                                                                                      • -

Client connecting to node2, TCP port 5001
TCP window size: 64.0 KByte (default)

                                                                                                                      • -

[ 3] local 172.22.1.114 port 60915 connected with 172.22.1.112 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 1.03 GBytes 8.88 Gbits/sec
[ 3] 1.0- 2.0 sec 1.03 GBytes 8.87 Gbits/sec
[ 3] 2.0- 3.0 sec 1.04 GBytes 8.92 Gbits/sec
[ 3] 3.0- 4.0 sec 1.04 GBytes 8.93 Gbits/sec
[ 3] 4.0- 5.0 sec 1.04 GBytes 8.90 Gbits/sec
[ 3] 5.0- 6.0 sec 1.02 GBytes 8.79 Gbits/sec
[ 3] 6.0- 7.0 sec 1.02 GBytes 8.79 Gbits/sec
[ 3] 7.0- 8.0 sec 1.05 GBytes 9.02 Gbits/sec
[ 3] 8.0- 9.0 sec 976 MBytes 8.19 Gbits/sec
[ 3] 9.0-10.0 sec 1.09 GBytes 9.32 Gbits/sec
[ 3] 0.0-10.0 sec 10.3 GBytes 8.86 Gbits/sec


★Infiniband<壱號機側:iperf server>
% iperf -s

                                                                                                                      • -

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)

                                                                                                                      • -

[ 4] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47753
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 14.7 GBytes 12.6 Gbits/sec<弐號機側:iperf client>
% iperf -c 172.22.0.112 -i 1

                                                                                                                      • -

Client connecting to 172.22.0.112, TCP port 5001
TCP window size: 193 KByte (default)

                                                                                                                      • -

[ 3] local 172.22.0.114 port 47753 connected with 172.22.0.112 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 1.46 GBytes 12.5 Gbits/sec
[ 3] 1.0- 2.0 sec 1.47 GBytes 12.6 Gbits/sec
[ 3] 2.0- 3.0 sec 1.47 GBytes 12.7 Gbits/sec
[ 3] 3.0- 4.0 sec 1.48 GBytes 12.7 Gbits/sec
[ 3] 4.0- 5.0 sec 1.48 GBytes 12.7 Gbits/sec
[ 3] 5.0- 6.0 sec 1.47 GBytes 12.6 Gbits/sec
[ 3] 6.0- 7.0 sec 1.47 GBytes 12.6 Gbits/sec
[ 3] 7.0- 8.0 sec 1.47 GBytes 12.7 Gbits/sec
[ 3] 8.0- 9.0 sec 1.47 GBytes 12.7 Gbits/sec
[ 3] 9.0-10.0 sec 1.47 GBytes 12.7 Gbits/sec
[ 3] 0.0-10.0 sec 14.7 GBytes 12.6 Gbits/sec


マルチセッション（2セッション並列）にしてみると<壱號機側:iperf server>
% iperf -s

                                                                                                                      • -

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)

                                                                                                                      • -

[ 4] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47754
[ 5] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47755
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 9.09 GBytes 7.80 Gbits/sec
[ 5] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec
[SUM] 0.0-10.0 sec 21.9 GBytes 18.8 Gbits/sec<弐號機側:iperf client>
% iperf -c 172.22.0.112 -i 1 -P 2

                                                                                                                      • -

Client connecting to 172.22.0.112, TCP port 5001
TCP window size: 193 KByte (default)

                                                                                                                      • -

[ 4] local 172.22.0.114 port 47754 connected with 172.22.0.112 port 5001
[ 3] local 172.22.0.114 port 47755 connected with 172.22.0.112 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 1.70 GBytes 14.6 Gbits/sec
[ 3] 0.0- 1.0 sec 512 MBytes 4.30 Gbits/sec
[SUM] 0.0- 1.0 sec 2.20 GBytes 18.9 Gbits/sec
[ 4] 1.0- 2.0 sec 1.58 GBytes 13.6 Gbits/sec
[ 3] 1.0- 2.0 sec 633 MBytes 5.31 Gbits/sec
[SUM] 1.0- 2.0 sec 2.20 GBytes 18.9 Gbits/sec
[ 4] 2.0- 3.0 sec 1.23 GBytes 10.5 Gbits/sec
[ 3] 2.0- 3.0 sec 987 MBytes 8.28 Gbits/sec
[SUM] 2.0- 3.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 4] 3.0- 4.0 sec 304 MBytes 2.55 Gbits/sec
[ 3] 3.0- 4.0 sec 1.89 GBytes 16.3 Gbits/sec
[SUM] 3.0- 4.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 4] 4.0- 5.0 sec 656 MBytes 5.51 Gbits/sec
[ 3] 4.0- 5.0 sec 1.56 GBytes 13.4 Gbits/sec
[SUM] 4.0- 5.0 sec 2.20 GBytes 18.9 Gbits/sec
[ 4] 5.0- 6.0 sec 435 MBytes 3.65 Gbits/sec
[ 3] 5.0- 6.0 sec 1.77 GBytes 15.2 Gbits/sec
[SUM] 5.0- 6.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 3] 6.0- 7.0 sec 1.80 GBytes 15.5 Gbits/sec
[ 4] 6.0- 7.0 sec 400 MBytes 3.36 Gbits/sec
[SUM] 6.0- 7.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 4] 7.0- 8.0 sec 432 MBytes 3.62 Gbits/sec
[ 3] 7.0- 8.0 sec 1.77 GBytes 15.2 Gbits/sec
[SUM] 7.0- 8.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 4] 8.0- 9.0 sec 619 MBytes 5.20 Gbits/sec
[ 3] 8.0- 9.0 sec 1.59 GBytes 13.6 Gbits/sec
[SUM] 8.0- 9.0 sec 2.19 GBytes 18.8 Gbits/sec
[ 4] 9.0-10.0 sec 1.81 GBytes 15.5 Gbits/sec
[ 4] 0.0-10.0 sec 9.09 GBytes 7.81 Gbits/sec
[ 3] 9.0-10.0 sec 378 MBytes 3.17 Gbits/sec
[SUM] 9.0-10.0 sec 2.18 GBytes 18.7 Gbits/sec
[ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec
[SUM] 0.0-10.0 sec 21.9 GBytes 18.8 Gbits/sec
２セッション以上にしても、ほとんど変化はなかった。<壱號機側:iperf server>のみ
% iperf -s

                                                                                                                      • -

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)

                                                                                                                      • -

[ 4] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47760
[ 5] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47761
[ 6] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47762
[ 7] local 172.22.0.112 port 5001 connected with 172.22.0.114 port 47763
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 8.05 GBytes 6.91 Gbits/sec
[ 5] 0.0-10.0 sec 3.28 GBytes 2.81 Gbits/sec
[ 6] 0.0-10.0 sec 5.81 GBytes 4.98 Gbits/sec
[ 7] 0.0-10.0 sec 4.94 GBytes 4.24 Gbits/sec
[SUM] 0.0-10.0 sec 22.1 GBytes 18.9 Gbits/sec