Set ups without using the daemon

netns/.gitignore

@@ -0,0 +1 @@
+mtu1280-netns\.env

netns/Makefile

@@ -0,0 +1,19 @@
+.PHONY: uninstall
+
+install:
+	install -m 644 -D mtu1280-netns.env /etc/mtu1280/mtu1280-netns.env
+	install -m 644 -b -D mtu1280-netns-httpd.override.conf /etc/systemd/system/httpd.service.d/mtu1280-netns-httpd.override.conf
+	install -m 644 -D mtu1280-netns.service /usr/local/lib/systemd/system/mtu1280-netns.service
+	install -m 644 -D pmtu-flush-cache/mtu1280-netns-flush-cache.service /usr/local/lib/systemd/system/mtu1280-netns-flush-cache.service
+	install -m 755 -D pmtu-flush-cache/mtu1280-netns-flush-cache.sh /usr/local/bin/mtu1280-netns-flush-cache.sh
+	install -m 755 -D mtu1280-netns.sh /usr/local/libexec/mtu1280/mtu1280-netns.sh
+	systemctl daemon-reload
+
+uninstall:
+	rm -f \
+		/etc/systemd/system/httpd.service.d/mtu1280-netns-httpd.override.conf \
+		/usr/local/lib/systemd/system/mtu1280-netns.service \
+		/usr/local/lib/systemd/system/mtu1280-netns-flush-cache.service \
+		/usr/local/libexec/mtu1280/mtu1280-netns.sh \
+		/usr/local/bin/mtu1280-netns-flush-cache.sh
+	systemctl daemon-reload

netns/README.md

@@ -0,0 +1,111 @@
+# MTU1280 using Kernel Network Namespace
+This set up is the "true" PMTU mismatch soft simulation using Linux kernel
+netns. Each netns has its own virtual routing table and PMTU cache. The
+bottleneck MTU segment is achieved through the MTU attribute of static route
+entries.
+
+```
+                     ROUTER A         ROUTER B
+
+<---------- mtu1280-a ------ mtu1280-b ------ mtu1280-c ---------->
+
+to internet           |  MTU 1280 segment   |               to httpd
+```
+
+<pre>
+$ tracepath -n fd12:34::3:1500:0
+ 1?: [LOCALHOST]                        0.025ms pmtu 1500
+ 1:  fd12:34::1:1280:0                                     0.128ms
+ 1:  fd12:34::1:1280:0                                     0.156ms
+ 2:  fd12:34::1:1280:0                                     0.244ms <b>pmtu 1280</b>
+ 2:  fd12:34::2:1280:0                                     0.235ms
+ 3:  fd12:34::3:1500:0                                     0.188ms reached
+     Resume: pmtu 1280 hops 3 back 3
+$ tracepath -n 10.12.80.34
+ 1?: [LOCALHOST]                      pmtu 1500
+ 1:  10.12.80.2                                            0.138ms
+ 1:  10.12.80.2                                            0.042ms
+ 2:  10.12.80.2                                            0.045ms <b>pmtu 1280</b>
+ 2:  10.12.80.18                                           0.055ms
+ 3:  10.12.80.34                                           0.051ms reached
+     Resume: pmtu 1280 hops 3 back 3
+</pre>
+
+3 netns are create to hide the MTU 1280 links from both internet facing gateway
+and httpd. This forces httpd to announce its MSS as 1440 octets and PMTUD is
+carried out for each endpoint from the internet. The PMTU caching in
+**mtu1280-c** is disabled so that the test clients can get the same result at
+all times. The makefile recipe installs a "drop-in" service unit config to
+override `NetworkNamespacePath` so httpd is run in the mtu1280-c netns.
+
+The set up should perfectly simulate the kernel's behaviour. A few down sides
+over the original mtu1280d approach are
+
+1. more overhead as all 3 netns along the path need to be "simulated" -
+   routing table, neighbor cache, PMTU cache ...
+1. somewhat difficult to integrate netns to most programs as they're not
+   designed with namespace in mind
+1. complexity making it difficult to troubleshoot
+
+## IPv4
+The script was originally written to test the PMTUD of IPv6, but the IPv4
+support has been added to test the PMTUD in IPv4 networks as well. As most
+operating system does not do IPv4 PMTUD by default, the use of special system
+API is required to set the DF flag. It's useful when any middlebox of the ISP
+blocks `Fragmentation Needed` or fragmented packets in general.
+
+## Network Set Up
+See [/um/README.md#network-set-up](/um/README.md#network-set-up).
+
+## INSTALL
+Copy and edit the env file [mtu1280-netns.env.sample](mtu1280-netns.env.sample):
+
+```sh
+cp mtu1280-netns.env.sample mtu1280-netns.env
+vi mtu1280-netns.env
+```
+
+Set `MTU1280NS_ADDR6_*` to the addresses of the routers. For IPv4, set
+`MTU1280NS_ADDR4_*` as well. If you plan to use private IPv4 addresses, remember
+to set port forwarding.
+
+- Set `MTU1280NS_NO_OFFLOAD` to "true" to disable offloading features that
+  can interfere with the packet capture result when debugging
+
+```sh
+# Stop http as the recipe will override some settings in httpd.service
+sudo systemctl stop httpd.service
+
+sudo make install
+
+sudo systemctl enable --now mtu1280-netns.service httpd.service
+```
+
+```sh
+# To see if the mtu1280-netns.sh script is alive and httpd is running
+systemctl status mtu1280-netns.service httpd.service
+
+# To understand and debug the magic yourself
+sudo ip netns
+sudo ip -6 addr
+sudo ip -6 route
+sudo ip -6 -n mtu1280-a addr
+sudo ip -6 -n mtu1280-a route
+sudo ip -6 -n mtu1280-b addr
+sudo ip -6 -n mtu1280-b route
+sudo ip -6 -n mtu1280-c addr
+sudo ip -6 -n mtu1280-c route
+
+# The listening ports will show up in the ns, not the default so the ports won't
+#  show up in the ss command run without the ip or nsenter command
+sudo ip netns exec mtu1280-c ss -tlnp
+```
+
+## Other Useful Info
+All settings are netns-local.
+
+- Use `ip -4 route show cache` and `ip -6 route show cache` to show PMTU cache
+- Use `ip -6 route flush cache` and `ip -4 route flush cache` to flush the PMTU
+  cache
+- `sysctl -w net.ipv6.route.mtu_expires=0` and `sysctl -w
+  net.ipv4.route.mtu_expires=0` disables PMTU caching

netns/mtu1280-netns-httpd.override.conf

@@ -0,0 +1,6 @@
+[Unit]
+Requires=mtu1280-netns.service
+After=mtu1280-netns.service
+
+[Service]
+NetworkNamespacePath=/run/netns/mtu1280-c

netns/mtu1280-netns.env.sample

@@ -0,0 +1,10 @@
+MTU1280NS_ADDR6_A=fd12:34::1:1280:0/128
+MTU1280NS_ADDR6_B=fd12:34::2:1280:0/128
+MTU1280NS_ADDR6_C=fd12:34::3:1500:0/128
+MTU1280NS_ADDR4_A0=10.12.80.1/28
+MTU1280NS_ADDR4_A1=10.12.80.2/28
+MTU1280NS_ADDR4_B0=10.12.80.17/28
+MTU1280NS_ADDR4_B1=10.12.80.18/28
+MTU1280NS_ADDR4_C0=10.12.80.33/28
+MTU1280NS_ADDR4_C1=10.12.80.34/28
+MTU1280NS_NO_OFFLOAD=false

netns/mtu1280-netns.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=MTU1280 network namespace set up
+Documentation=https://github.com/si-magic/mtu1280d/tree/master/netns
+Requires=network.target
+After=network.target
+
+[Service]
+Type=notify
+EnvironmentFile=/etc/mtu1280/mtu1280-netns.env
+ExecStartPre=-/usr/local/libexec/mtu1280/mtu1280-netns.sh rmns_q
+ExecStart=/usr/local/libexec/mtu1280/mtu1280-netns.sh daemon
+KillSignal=SIGCONT
+
+[Install]
+WantedBy=multi-user.target

netns/mtu1280-netns.sh

@@ -0,0 +1,170 @@
+#!/bin/sh
+[ -z "$MTU1280NS_NO_OFFLOAD" ] && MTU1280NS_NO_OFFLOAD=false
+
+safe_run () {
+	if [ -z "$1" ]; then
+		return
+	fi
+
+	shift
+	$@
+}
+
+mkns () {
+	set -e
+	local ll
+
+	do_make_ns () {
+		ip netns add "$1"
+		ip netns exec "$1" sysctl -qw \
+			"net.ipv4.conf.all.forwarding=1" \
+			"net.ipv4.conf.default.forwarding=1" \
+			"net.ipv4.ip_forward=1" \
+			"net.ipv6.conf.default.forwarding=1" \
+			"net.ipv6.conf.all.forwarding=1"
+	}
+
+	get_ll_addr () {
+		local ll
+
+		while true
+		do
+			if [ -z "$2" ]; then
+				ll=$(ip -br -6 addr show dev $1 scope link | grep -Eoi 'fe[^\s/]+')
+			else
+				ll=$(ip -br -6 -n $2 addr show dev $1 scope link | grep -Eoi 'fe[^\s/]+')
+			fi
+
+			if [ -z "$ll" ]; then
+				sleep 0.1
+			else
+				echo "$ll"
+				break
+			fi
+		done
+	}
+
+	filter_cidr () {
+		echo "${1%/*}"
+	}
+
+	no_offload () {
+		if ! "$MTU1280NS_NO_OFFLOAD"; then
+			return
+		fi
+
+		if [ -z "$2" ]; then
+			ethtool -K "$1" tcp-segmentation-offload off generic-segmentation-offload off generic-receive-offload off
+		else
+			ip netns exec "$2" ethtool -K "$1" tcp-segmentation-offload off generic-segmentation-offload off generic-receive-offload off
+		fi
+	}
+
+	do_make_ns mtu1280-a
+	do_make_ns mtu1280-b
+	do_make_ns mtu1280-c
+
+	# link up system default ns -> mtu1280-a
+	ip link add veth-mtu1280-a type veth peer name veth-mtu1280-gw netns mtu1280-a
+	no_offload veth-mtu1280-a
+	no_offload veth-mtu1280-gw mtu1280-a
+	# link up mtu1280-a -> mtu1280-b
+	ip -n mtu1280-a link add veth-mtu1280-b type veth peer name veth-mtu1280-a netns mtu1280-b
+	no_offload veth-mtu1280-b mtu1280-a
+	no_offload veth-mtu1280-a mtu1280-b
+	# link up mtu1280-b -> mtu1280-c
+	ip -n mtu1280-b link add veth-mtu1280-c type veth peer name veth-mtu1280-b netns mtu1280-c
+	no_offload veth-mtu1280-c mtu1280-b
+	no_offload veth-mtu1280-b mtu1280-c
+
+	safe_run "$MTU1280NS_ADDR4_A0" ip -4 addr add "$MTU1280NS_ADDR4_A0" dev veth-mtu1280-a
+
+	safe_run "$MTU1280NS_ADDR4_A1" ip -n mtu1280-a -4 addr add "$MTU1280NS_ADDR4_A1" dev veth-mtu1280-gw
+	safe_run "$MTU1280NS_ADDR4_B0" ip -n mtu1280-a -4 addr add "$MTU1280NS_ADDR4_B0" dev veth-mtu1280-b
+
+	safe_run "$MTU1280NS_ADDR4_B1" ip -n mtu1280-b -4 addr add "$MTU1280NS_ADDR4_B1" dev veth-mtu1280-a
+	safe_run "$MTU1280NS_ADDR4_C0" ip -n mtu1280-b -4 addr add "$MTU1280NS_ADDR4_C0" dev veth-mtu1280-c
+
+	safe_run "$MTU1280NS_ADDR4_C1" ip -n mtu1280-c -4 addr add "$MTU1280NS_ADDR4_C1" dev veth-mtu1280-b
+
+	safe_run "$MTU1280NS_ADDR6_A" ip -n mtu1280-a -6 addr add "$MTU1280NS_ADDR6_A" dev veth-mtu1280-gw
+	safe_run "$MTU1280NS_ADDR6_B" ip -n mtu1280-b -6 addr add "$MTU1280NS_ADDR6_B" dev veth-mtu1280-a
+	safe_run "$MTU1280NS_ADDR6_C" ip -n mtu1280-c -6 addr add "$MTU1280NS_ADDR6_C" dev veth-mtu1280-b
+
+	ip link set veth-mtu1280-a up
+	ip -n mtu1280-a link set lo up
+	ip -n mtu1280-a link set veth-mtu1280-gw up
+
+	ip -n mtu1280-a link set veth-mtu1280-b up
+	ip -n mtu1280-b link set lo up
+	ip -n mtu1280-b link set veth-mtu1280-a up
+
+	ip -n mtu1280-b link set veth-mtu1280-c up
+	ip -n mtu1280-c link set lo up
+	ip -n mtu1280-c link set veth-mtu1280-b up
+
+	# default route from mtu1280-a to system
+	safe_run "$MTU1280NS_ADDR4_A0" ip -n mtu1280-a -4 route add default dev veth-mtu1280-gw via $(filter_cidr "$MTU1280NS_ADDR4_A0")
+	ip -n mtu1280-a -6 route add default dev veth-mtu1280-gw via $(get_ll_addr veth-mtu1280-a)
+	# default route from mtu1280-b to mtu1280-a (mtu 1280 segment)
+	safe_run "$MTU1280NS_ADDR4_B0" ip -n mtu1280-b -4 route add default dev veth-mtu1280-a via $(filter_cidr "$MTU1280NS_ADDR4_B0") mtu 1280
+	ip -n mtu1280-b -6 route add default dev veth-mtu1280-a via $(get_ll_addr veth-mtu1280-b mtu1280-a) mtu 1280
+	# default route from mtu1280-c to mtu1280-b
+	safe_run "$MTU1280NS_ADDR4_C0" ip -n mtu1280-c -4 route add default dev veth-mtu1280-b via $(filter_cidr "$MTU1280NS_ADDR4_C0")
+	ip -n mtu1280-c -6 route add default dev veth-mtu1280-b via $(get_ll_addr veth-mtu1280-c mtu1280-b)
+
+	# static route from system default ns to inner
+	ll=$(get_ll_addr veth-mtu1280-gw mtu1280-a)
+	safe_run "$MTU1280NS_ADDR4_B0" ip -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_B0")" dev veth-mtu1280-a via "$(filter_cidr "$MTU1280NS_ADDR4_A1")"
+	safe_run "$MTU1280NS_ADDR4_B1" ip -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_B1")" dev veth-mtu1280-a via "$(filter_cidr "$MTU1280NS_ADDR4_A1")"
+	safe_run "$MTU1280NS_ADDR4_C0" ip -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_C0")" dev veth-mtu1280-a via "$(filter_cidr "$MTU1280NS_ADDR4_A1")"
+	safe_run "$MTU1280NS_ADDR4_C1" ip -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_C1")" dev veth-mtu1280-a via "$(filter_cidr "$MTU1280NS_ADDR4_A1")"
+	safe_run "$MTU1280NS_ADDR6_A" ip -6 route add "$MTU1280NS_ADDR6_A" dev veth-mtu1280-a via $ll
+	safe_run "$MTU1280NS_ADDR6_B" ip -6 route add "$MTU1280NS_ADDR6_B" dev veth-mtu1280-a via $ll
+	safe_run "$MTU1280NS_ADDR6_C" ip -6 route add "$MTU1280NS_ADDR6_C" dev veth-mtu1280-a via $ll
+	# static route from mtu1280-a to inner (mtu 1280 segment)
+	ll=$(get_ll_addr veth-mtu1280-a mtu1280-b)
+	safe_run "$MTU1280NS_ADDR4_C0" ip -n mtu1280-a -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_C0")" dev veth-mtu1280-b via "$(filter_cidr "$MTU1280NS_ADDR4_B1")" mtu 1280
+	safe_run "$MTU1280NS_ADDR4_C1" ip -n mtu1280-a -4 route add "$(filter_cidr "$MTU1280NS_ADDR4_C1")" dev veth-mtu1280-b via "$(filter_cidr "$MTU1280NS_ADDR4_B1")" mtu 1280
+	safe_run "$MTU1280NS_ADDR6_B" ip -n mtu1280-a -6 route add "$MTU1280NS_ADDR6_B" dev veth-mtu1280-b via $ll mtu 1280
+	safe_run "$MTU1280NS_ADDR6_C" ip -n mtu1280-a -6 route add "$MTU1280NS_ADDR6_C" dev veth-mtu1280-b via $ll mtu 1280
+	# static route from mtu1280-b to inner
+	ll=$(get_ll_addr veth-mtu1280-b mtu1280-c)
+	safe_run "$MTU1280NS_ADDR6_C" ip -n mtu1280-b -6 route add "$MTU1280NS_ADDR6_C" dev veth-mtu1280-c via $ll
+
+	# disable PMTU caching in mtu1280-c
+	ip netns exec mtu1280-c sysctl -qw "net.ipv4.route.mtu_expires=0" || true
+	ip netns exec mtu1280-c sysctl -qw "net.ipv6.route.mtu_expires=0" || true
+}
+
+rmns () {
+	ip link del veth-mtu1280-a type veth
+	ip -n mtu1280-a link del veth-mtu1280-b type veth
+	ip -n mtu1280-b link del veth-mtu1280-c type veth
+
+	ip netns del mtu1280-a
+	ip netns del mtu1280-b
+	ip netns del mtu1280-c
+}
+
+rmns_q () {
+	rmns $@ 2> /dev/null > /dev/null
+}
+
+daemon () {
+	mkns
+
+	systemd-notify --status="Stopped and holding netns"
+	systemd-notify --ready
+	kill -STOP 0
+
+	systemd-notify --stopping
+	systemd-notify --status="Deleting netns"
+	rmns
+
+	systemd-notify --status=""
+}
+
+cmd="$1"
+shift
+"$cmd" $@

netns/pmtu-flush-cache/README.md

@@ -0,0 +1,19 @@
+# Flush mtu1280-c netns pmtu cache every second
+`net.ipv4.route.mtu_expires` in a netns is a [recent
+invention](https://github.com/torvalds/linux/commit/1de6b15a434c0068253fea5d719f71143e7e3a79).
+With the system kernel without the patch, the following error will be shown when
+staring `mtu1280-netns.service`:
+
+```
+sysctl: cannot stat /proc/sys/net/ipv4/route/mtu_expires: No such file or directory
+```
+
+As a cheap and quick hack, `mtu1280-netns-flush-cache.service` can be used to
+achieve `net.ipv4.route.mtu_expires=0`.
+
+```sh
+systemctl enable --now mtu1280-netns-flush-cache.service
+```
+
+The service starts a shell script that flushes the mtu cache every second until
+the netns is gone.

netns/pmtu-flush-cache/mtu1280-netns-flush-cache.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Flush pmtu cache in mtu1280-c netns every second
+Requisite=mtu1280-netns.service
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/mtu1280-netns-flush-cache.sh
+
+[Install]
+WantedBy=multi-user.target

netns/pmtu-flush-cache/mtu1280-netns-flush-cache.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+a=0
+b=0
+
+while [ "$a" -eq 0 ] || [ "$b" -eq 0 ]
+do
+	sleep 1
+
+	ip -n mtu1280-c -4 route flush cache 2> /dev/null >/dev/null
+	a=$?
+	ip -n mtu1280-c -6 route flush cache 2> /dev/null >/dev/null
+	b=$?
+done

um/.gitignore

@@ -0,0 +1 @@
+build/*