YARP  2.3.70.2
Yet Another Robot Platform
Setting up your network for YARP

Along with TCP, YARP supports UDP and multicast communication on a local network.

These are unreliable network protocols, but can be quite efficient. We discuss here steps that are useful for making your network friendly to UDP and multicast. But if you find yourself spending a lot of time on this, you may need to reconsider whether you have made the right choice of protocol.

NOTE: if you are just getting started with YARP, you don't need to look at any of this yet! It only matters when you are working with a cluster of computers using unreliable protocols, or run into bandwidth problems. But it is important to remember that this page is here for when you need it.

iperf is your friend

If you are doing any non-trivial networking, you need to learn how to test your network and debug throughput issues. A key resource in this battle is the "iperf" program.

Please make sure you have iperf available on all your machines before going any further.

Checking TCP communication

TCP is the simplest protocol to work with, so let's start with it. Let's check the throughput between two of your machines. Pick two machines, and choose one (arbitrarily) to be the iperf server. Determine its IP address of domain name – let's say it is 10.0.0.26. Run the following commands:

 [On server, e.g. 10.0.0.26] iperf -s
 [On client, e.g. 10.0.0.27] iperf -c 10.0.0.26

This will run for a while and then describe your TCP throughput, for example:

 ------------------------------------------------------------
 Client connecting to 10.0.0.26, TCP port 5001
 TCP window size: 16.0 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.0.0.25 port 34160 connected with 10.0.0.26 port 5001
 [  3]  0.0-10.0 sec  1.08 GBytes    924 Mbits/sec
 ...

Checking UDP communication

UDP is an unreliable protocol, and there are lots of possible pitfalls in using it. YARP will try to send messages broken into rather large UDP packets (around 60K), and your network may or may not actually pass those packets through.

To test UDP behavior on your network with iperf, add the "-u" flag to the client and server. To the server, we can add "-i 1" to get reports at regular intervals. To the client, we can specify target bandwidth, e.g. "-b 50m" for a target of 50 MBits/sec.

 [On server, 10.0.0.26] iperf -s -u -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 50m

To test a gigabit network:

 [On server, 10.0.0.26] iperf -s -u -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 1000m

On a healthy gigabit network you should get a bandwidth of ~950 Mbits/sec.

Example result on the client side:

------------------------------------------------------------
Client connecting to 10.0.0.26, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.221 port 52990 connected with 10.0.0.222 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 6.1 sec    700 MBytes    956 Mbits/sec
[  3] Sent 499273 datagrams
[  3] Server Report:
[  3]  0.0- 6.1 sec    699 MBytes    955 Mbits/sec  0.014 ms  575/499272 (0.12%)
[  3]  0.0- 6.1 sec  1 datagrams received out-of-order

Result on the server side:

------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.26 port 5001 connected with 10.0.0.27 port 52990
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    114 MBytes    954 Mbits/sec  0.012 ms   88/81245 (0.11%)
[  3]  1.0- 2.0 sec    114 MBytes    955 Mbits/sec  0.011 ms   92/81263 (0.11%)
[  3]  2.0- 3.0 sec    114 MBytes    955 Mbits/sec  0.014 ms   90/81287 (0.11%)
[  3]  3.0- 4.0 sec    114 MBytes    955 Mbits/sec  0.012 ms   98/81273 (0.12%)
[  3]  4.0- 5.0 sec    114 MBytes    954 Mbits/sec  0.012 ms   99/81263 (0.12%)
[  3]  5.0- 6.0 sec    114 MBytes    955 Mbits/sec  0.012 ms   95/81288 (0.12%)
[  3]  0.0- 6.1 sec    699 MBytes    955 Mbits/sec  0.014 ms  575/499272 (0.12%)
[  3]  0.0- 6.1 sec  1 datagrams received out-of-order

A key test for YARP is that you can send big datagrams okay. This is necessary for efficient transmission of large images using UDP or MCAST. To check 63KB datagrams, do:

 [On server, 10.0.0.26] iperf -s -u -l 63K -w 200K -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -u -l 63K -w 200K -b 1000m

Make sure you try this in both directions! For example, we've seen an asymmetric problems going from a Linux machine to a Windows machine on a gigabit network with a particular switch. If you see problems, reduce the "-l" size until things work just to understand your system (but if you want to send large images you'll need to tweak your system until large packets work).

It could also be worth checking several machines since different network cards can have different performance.

Checking multicast communication

For testing multicast, make sure you have the most recent version of iperf, and try this:

 iperf -s -u -B 224.0.55.55 -i 1 -T 32 -l 60k
 iperf -c 224.0.55.55 -u -T 32 -t 100 -i 1 -l 60k -b 1000m

Note the IP addresses are multicast-style, they don't correspond to individual machines.

On the client ("-c") side, you should see something like:

------------------------------------------------------------
Client connecting to 224.0.55.55, UDP port 5001
Sending 61440 byte datagrams
Setting multicast TTL to 32
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.221 port 60992 connected with 224.0.55.55 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    114 MBytes    958 Mbits/sec
[  3]  1.0- 2.0 sec    114 MBytes    956 Mbits/sec
[  3]  2.0- 3.0 sec    114 MBytes    956 Mbits/sec
[  3]  3.0- 4.0 sec    114 MBytes    956 Mbits/sec

Server side:

------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 224.0.55.55
Joining multicast group  224.0.55.55
Receiving 61440 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 224.0.55.55 port 5001 connected with 10.0.0.221 port 60992
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    111 MBytes    931 Mbits/sec  0.380 ms   50/ 1945 (2.6%)
[  3]  1.0- 2.0 sec    111 MBytes    934 Mbits/sec  0.442 ms   46/ 1947 (2.4%)
[  3]  2.0- 3.0 sec    112 MBytes    941 Mbits/sec  0.280 ms   31/ 1946 (1.6%)
[  3]  3.0- 4.0 sec    112 MBytes    935 Mbits/sec  0.362 ms   43/ 1946 (2.2%)
[  3]  4.0- 5.0 sec    110 MBytes    920 Mbits/sec  0.459 ms   75/ 1947 (3.9%)
[  3]  5.0- 6.0 sec    111 MBytes    934 Mbits/sec  0.418 ms   45/ 1946 (2.3%)
[  3]  6.0- 7.0 sec    111 MBytes    933 Mbits/sec  0.380 ms   48/ 1947 (2.5%)
[  3]  7.0- 8.0 sec    111 MBytes    933 Mbits/sec  0.365 ms   49/ 1947 (2.5%)
[  3]  8.0- 9.0 sec    112 MBytes    936 Mbits/sec  0.319 ms   41/ 1946 (2.1%)

YARP's suggestion

If MCAST/UDP packets drop when using YARP, YARP may suggest this:

 The UDP/MCAST system buffer limit on your system is low.
 You may get packet loss under heavy conditions.
 To change the buffer limit on linux: sysctl -w net.core.rmem_max=8388608
 (Might be something like: sudo /sbin/sysctl -w net.core.rmem_max=8388608)
 To change the limit use: sysctl for Linux/FreeBSD, ndd for Solaris, no for AIX

This controls the maximum allowed size for a RECEIVER buffer, so that a number of packets adding up to a large amount of data can be received and stored while your program is doing something else. YARP will try to get a large buffer, and your operating system may not comply unless you do the above. This configuration step is not necessary on MS-Windows.

There are a lot more suggestions for Linux here:

YARP connection problems

If you are confident of your network, but are having trouble with making a YARP connection of some kind, read about that kind of connection in Configuring YARP Connections and see if there's a tweak that can be made. There are some parameters you can control for the udp carrier and mcast (multicast) carrier.

Relevant source code

In case you need to fix something, the relevant source code in YARP is:

We appreciate patches to these files. We only have a few networks to test on, and this topic can be quite network-dependent and subtle.