openssh / nagle / too much buffering

Recently I tried to open a connection to a remote server over SSH at a new location. The connection opened just fine, but it seemed that a few bytes kept getting buffered.

It looked like this first animated gif you see.

After a long wait, you realise that the data you’re wating just won’t come. First after pressing a key, you get the data.

This isn’t workable…

Enumerating the possible culprits, there could really only be the wifi-nat-modem — a Thomson TG789vn, Telia device — doing extra buffering, possibly conflicting with the Nagle algorithm (TCP_NODELAY).

Attempt #1: try to switch off any/all buffering options in the modem: the administrator user has too few powers! Boo, Telia.se.

Attempt #2: try to switch off nagle in openssh: eek, there is no option to do that!

Once again, LD_PRELOAD comes to the rescue.

Using this simple custom library, we can override setsockopt — which normally handles the setting of the TCP_NODELAY option — to do nothing at all.

/* gcc nosetsockopt.c -fPIC -shared -ldl -o nosetsockopt.so
 * LD_PRELOAD=./nosetsockopt.so ssh DEST
 */
#include <sys/socket.h>
#include <stdio.h>
int setsockopt(int sockfd, int level, int optname,
               const void *optval, socklen_t optlen) {
  printf("SETSOCKOPT: %d: %d: %d=%p (%d)\r\n",
         sockfd, level, optname, optval, optlen);
  return 0;
}

Compiled an ran, we get this extra output:

$ gcc nosetsockopt.c -fPIC -shared -ldl -o nosetsockopt.so
$ LD_PRELOAD=./nosetsockopt.so ssh dummy@wjd.nu
SETSOCKOPT: 3: 1: 9=0x7fffbb2eeef4 (4)
dummy@wjd.nu's password:
SETSOCKOPT: 3: 6: 1=0x7fffbb2ef188 (4)
SETSOCKOPT: 3: 0: 1=0x7fffbb2ef1b8 (4)
Linux wjdsys.wjd.nu 3.2.0-4-amd64..
...

The values we see are as follows:

• The file descriptor is 3.
• The level is either SOL_SOCKET (1), SOL_TCP (6) or SOL_IP (0). See /etc/protocols for those last two.
• The corresponding options are: SO_KEEPALIVE (9), TCP_NODELAY (1) and IP_TOS (1).

Preloading that lib, meant that all those options are not really set anymore. And guess what? The shell behaved normally again. This fixed behaviour is what you see on the right hand side of the second animation. The two windows use a shared screen, so the behaviour should have been identical.

For bonus points, you can alter the lib to call the real setsockopt for all calls except Nagle, as can be seen in this example. But the above version does the trick just fine.

Move the files to /usr/local and put this in your ~/.bash_aliases:

echo "TEMP ALIAS FOR NAGLE"
alias ssh='LD_PRELOAD=/usr/local/lib/nosetsockopt.so ssh'