Thomas’ Lab Notes

Stuff worth not forgetting

Bizarre Packet Filtering on OVH Kimsufi Server

Context

I am leasing a Kimsufi dedicated server from OVH, ks3269175.kimsufi.com aka 5.39.82.72. Since early January 2015, TCP connections to that machine (and in particular SSH connections) are sporadically hanging.

Analysis of the issue

This machine is on a network whose default router is 5.39.82.254 (vss-gw-6k.fr.eu).

As far as I was able to determine, this is a virtual router, load balanced using GLBP . The two actual routers are:

  • vss-9b-6k.fr.eu, MAC address 00:07:b4:00:01:01
  • vss-9a-6k.fr.eu, MAC address 00:07:b4:00:01:02

When attempting to establish an SSH connection from the outside to that machine, the first data packet in the connection appears to be dropped if sent through 00:07:b4:00:01:01.

This does not appear to be related to any kind of stateful firewalling system. As an experiment, I wrote a simple Scapy script that loops sending identical TCP segments, one per second, through both of the above MAC addresses, to a remote address outside OVH.

A tcpdump on the dedicated server shows the stream of outgoing packets:

1
2
3
4
5
6
23:04:15.390421 00:22:4d:83:36:80 > 00:07:b4:00:01:01, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2122 > 194.98.77.4.60347: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:04:15.406734 00:22:4d:83:36:80 > 00:07:b4:00:01:02, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2222 > 194.98.77.4.60347: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:04:16.424437 00:22:4d:83:36:80 > 00:07:b4:00:01:01, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2122 > 194.98.77.4.60348: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:04:16.441460 00:22:4d:83:36:80 > 00:07:b4:00:01:02, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2222 > 194.98.77.4.60348: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:04:17.459641 00:22:4d:83:36:80 > 00:07:b4:00:01:01, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2122 > 194.98.77.4.60349: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:04:17.476966 00:22:4d:83:36:80 > 00:07:b4:00:01:02, ethertype IPv4 (0x0800), length 115: 5.39.82.72.2222 > 194.98.77.4.60349: Flags [P.], seq 0:49, ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49

Now observing traffic on the remote machine, we see only those packets that went through 00:07:b4:00:01:02:

1
2
3
4
5
6
23:05:13.322004 IP 5.39.82.72.2222 > 194.98.77.4.60403: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:05:14.355176 IP 5.39.82.72.2222 > 194.98.77.4.60404: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:05:15.390245 IP 5.39.82.72.2222 > 194.98.77.4.60405: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:05:16.426968 IP 5.39.82.72.2222 > 194.98.77.4.60406: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:05:17.456869 IP 5.39.82.72.2222 > 194.98.77.4.60407: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49
23:05:18.494918 IP 5.39.82.72.2222 > 194.98.77.4.60408: Flags [P.], ack 0, win 8192, options [TS val 123 ecr 456,eol], length 49

Resolution

None so far. OVH has been notified of the problem (TICKET#2015010719008317) and all analysis elements in my possession have been conveyed to them, to no avail so far: the machine has been essentially unusable for the past two months and counting.

Update 2015-03-20 OVH say they identified a problem and are working on a fix. No more SSH failure observed since 2015-03-18 in the afternoon, so apparently the fix did work. Still waiting for a post-mortem explanation as to what went wrong, and why it took them so long to ackonwledge, investigate, and resolve the problem.

Update 2015-04-01 Service remains stable, in that failures are not observed anymore. OVH indicates they are still discussing the underlying issue with Cisco, and the fix is not completed yet.

Update 2015-06-17 Service remains stable. OVH indicates that they have identified the origin of the issue, a fix is available, and they have scheduled its deployment.

Update 2015-09-17 At long last, OVH confirmed that the problem is indeed resolved on their side, and agreed to extend my subscription by 3 months at no cost in compensation, which is decent of them.

Acknowledgements

Thanks to Pierre Beyssac for hinting at GLBP, and to Fabian at OVH support for following up internally on this issue.

Upgrading to FreeBSD 10.1-RELEASE

I endeavoured to upgrade my old FreeBSD 9.3-RELEASE system to brand new 10.1. As expected, this was a quite bumpy process, and below are a few things I had to find out the hard way.

Ports

Neither pkg update nor portupgrade can update all ports. I had to dump all origins, remove all ports, then reinstall everything manually.

Linuxulator

New CentOS-6 based linux_base requires sysctl:

1
compat.linux.osrelease=2.6.18

X11

New Xorg

As of Apr. 16, 2014, the X server has been upgraded to a new release. From https://wiki.freebsd.org/Graphics/WITH_NEW_XORG:

linenos:false
1
2
3
4
5
6
7
Note that there's a know regression with syscons and kernel video
drivers: you can't switch back to a console once an X.Org session is
started. A new console driver called vt(4) fixes this issue while
bringing nice features. It's available in FreeBSD 9.3-RELEASE and
10.1-RELEASE but isn't enabled by default. To enable it, put the
following line in your /boot/loader.conf:
    kern.vty=vt

It is a real shame that users essentially have no choice but switching from the default syscons to the “new” (unfinished, far from functionally complete) vt console driver.

The X mouse cursor occasionnally disappears for some unidentified reason. Alt-Tab brings it back.

DRI

Both GDM and the GNOME desktop now require DRI access. At least for ATI video cards, this means that user gdm, as well as anyone logging in to a GNOME session, must have access to /dev/dri/card0:

/etc/devfs.rules linenos:false
1
add path 'dri/card0 mode 0666

GDM

Gdm won’t work out of the box (black screen): gdm_lang cannot be set to a non-UTF-8 locale anymore (if the month name in the current date contains an accent, the greeter will abort). Time to bite the UTF-8 bullet, then.

Oh, and I can’t just remove the variable altogether, see below.

Keymap

Interesting issue for GNOME users. I found out that the GDM login screen would always revert to US layout, no matter what. Initially I thought the X server had an incorrect keymap due to HAL device enumeration, so I added the following to my setup:

/usr/local/share/hal/fdi/policy/99local/10-x11-keymap.fdi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<?xml version="1.0" encoding="ISO-8859-1"?>

<deviceinfo version="0.2">
  <device>
    <match key="info.capabilities" contains="input.keyboard">
      <merge key="input.x11_options.XkbLayout" type="string">fr</merge>
      <merge key="input.x11_options.XkbOptions" type="string">terminate:ctrl_alt_bksp,compose:rctrl</merge>
      <merge key="input.xkb.layout" type="string">fr</merge>
      <merge key="input.xkb.options" type="string">terminate:ctrl_alt_bksp,compose:rctrl</merge>
    </match>
  </device>
</deviceinfo>

<!-- Legacy X11 options:
  Option  "XkbRules"    "xorg"
  Option  "XkbModel"    "pc105"
  Option  "XkbLayout"   "fr"
  Option  "XkbOptions"  "terminate:ctrl_alt_bksp" 
  Option  "XkbOptions"  "compose:rctrl" 
  -->

However this happened to be a total red herring, as by default the port configures Xorg to use devd, not HAL. For devd, I found out this is achieved using xorg.conf options:

/usr/local/share/hal/fdi/policy/99local/10-x11-keymap.fdi
1
2
3
4
5
6
Section "InputClass"
        Identifier "Keyboard Defauls"
        Driver "keyboard"
        MatchIsKeyboard "on"
        Option "XkbLayout" "fr"
EndSection

But all of this was mostly irrelevant for my setup since I add AutoAddDevices turned off in the X server setup, and the correct layout was hardcoded in xorg.conf. And indeed, starting it with startx yields the expected French layout.

However, it appears that gnome-shell considers that whatever keymap is configured in the X server probably must be unsuitable, and changes it on its own to a better default based on the current locale (or “us” if no locale is set for gdm).

Printing

I am using an HP MFP1217nfw network printer, which requires the proprietary print/hplip and print/hplip-plugin packages. These install print/cups as a dependency. print/cups-filters is not installed as a dependency, but is required anyway, or all print operations will fail with:

/usr/local/share/hal/fdi/policy/99local/10-x11-keymap.fdi
1
D [02/Mar/2015:22:24:01 +0100] Print-Job client-error-document-format-not-supported: Unsupported format "application/pdf".

Setting Up Octopress

Objective

To set up Octopress in its own, isolated Ruby environment

Prerequisite

Install rbenv to manage Ruby virtual environments.

Create isolated environment:

1
2
3
$ export GEM_HOME=$HOME/octopress.gems
$ gem install bundler
$ export PATH=$GEM_HOME/bin:$PATH

GEM_HOME and PATH need to be set to do any Octopress work.

Clone Octopress:

1
2
$ git clone git://github.com/imathis/octopress.git octopress
$ cd octopress 

Install dependencies and default theme:

1
2
$ bundle install
$ rake install

warning If --path is passed to bundle install, the value is cached in .bundle/config, and $GEM_HOME is subsequently ignored.

Emoji support

Debian Jessie on Dell Precision M4800

The testing ISO image works on a USB key.

Using the nvidia (non-free) driver

To install the nVidia driver:

  • make sure installed kernel headers (linux-headers) match kernel (linux-image), otherwise DKMS won’t build.
  • install nvidia-driver (note: this will build a kernel module, so requires a working compiler)
  • run nvidia-xconfig --query-gpu-info by hand and make note of PCI BusID
  • run nvidia-xconfig --busid=PCI:x:x:x to generate xorg.conf

BIOS setup

  • Video -> Switchable graphics -> uncheck Enable Switchable Graphics

GNOME 3

Disable gnome-keyring:

1
2
3
$ cd /etc/xdg/autostart
$ mv gnome-keyring-ssh.desktop gnome-keyring-ssh.desktop-
$ mv gnome-keyring-gpg.desktop gnome-keyring-gpg.desktop-

Remember NumLock state:

1
$ gsettings set org.gnome.settings-daemon.peripherals.keyboard remember-numlock-state true

Support for smart-card reader

apt-get install pcscd

Network

Disable IPv6 Privacy Extensions

/etc/sysctl.conf
1
2
3
# Disable IPv6 "Privacy extensions" (random SLAAC addresses)
net.ipv6.conf.all.use_tempaddr=0
net.ipv6.conf.default.use_tempaddr=0