Sharing the same IPv6 subnet across different networks without using Prefix Delegation
Table of Contents
A story about how my fediverse instance wasn't completely ready for IPv6 and how Linux saved the day.
An IPv6-only fediverse instance
The other day, I was just cruising on the Internet, as one does, when I came across IPv6.camp: an IPv6-only Mastodon instance. I checked my personal instance to see if I already federated with it, but no, I didn't yet. As I was visiting this instance, I saw some interesting profiles that I wanted to follow, but when I tried to search for their profile from my instance, all I got was an ominous error.
Well, that's unfortunate, let's try again... Same thing? Hmmmmmm. What are saying the network logs? "500 - Internal Error". Huh. And the Sharkey logs? What do you mean "unreachable"?
And at this moment, I just realized that I didn't configure IPv6 on this virtual machine, thus making my Sharkey instance unable to contact this IPv6-only one.
A reverse proxy
As unallocated IPv4 pools have been exhausted for a while now, hosting companies are trying to discourage their customers from keeping addresses that they don't actually need or use. This is usually achieved by making users pay a recurring fee for each address they have allocated or by limiting the number of addresses they are allowed to rent.
One of the hosting machines used in my infrastructure is a dedicated server hosted at Hetzner, for which I pay around 2€ per additional IPv4 allocated, with a default maximum of 6 additional IPv4 per dedicated server. In order to avoid renting resources that I don't actually need, I have a reverse-proxy virtual machine for the applications that are just web-based. This machine also serves as a router, using NAT to give internet access to the VMs that are behind it. It also doesn't do any port-forwarding, as any service needing more that a web interface gets its own public IPv4 address to open ports as needed.
In this configuration, my Sharkey instance was accessible from both IPv4 and IPv6 networks through the reverse proxy, but couldn't contact IPv6-only instances by itself, as it didn't have any IPv6 connectivity.
It was time to fix that!
A /64 to rule them all
The problem with this architecture is that these internal virtual machines located behind the reverse proxy are doing so through a dedicated network bridge, without direct layer 2 access to the actual IPv6 router. This meant that adding IPv6 connectivity to these machines wasn't as straightforward as just giving each of them an address and calling it a day; it first required a bit of network configuration.
This is due to the fact that, just like ARP in IPv4, IPv6-enabled entities can use the Neighbor Discovery Protocol to gather some network information, such as auto-discovering routers or discovering their link neighbors. This is cool and all, but as I said, the internal machines and the router are kind of separated by another machine splitting the link into two network domains, thus making it impossible for the IPv6 router to know that these internal machines exist at all.
So, okay, we can't just give them an address and what? Can't we just route an IPv6 subnet to the reverse-proxy-router machine and ask it to do some forwarding shenanigans?
Well, yes, but actually, no.
By default, Hetzner only provides a /64 IPv6 subnet to dedicated routers. Granted, that's far better than some other cloud providers who just provide a single /128 address and then say they're IPv6-friendly, but it means that you can't split it in subnets without losing some features such as SLAAC, which is used for auto-configuring an address. In my case, losing this feature wouldn't be so much of a problem, as everything is manually configured and managed anyway, but I would still prefer not to have to break this /64 subnet in smaller ones unless necessary. Of course, I could just request an additional /56 IPv6 subnet, which could then be split into 256 /64 subnets, but I'd also rather not do that, as I don't feel it'd be worth it for me to request a manual intervention from a Hetzner technician just for these few VMs.
So, I'd need to find a way to somehow use the same /64 subnet across different network links, even out of direct reach from my IPv6 router. But how could I do that? I would need some kind of proxy... Some kind of Neighbor Discovery Protocol proxy...
And a proxy?
Well, good news everyone, it already exists and it is already included in the Linux kernel (and available by default in the Debian kernel)!
First of all, we need to enable this NDP proxy using a sysctl call on the
machine sitting in the middle of both network spaces: sysctl net.ipv6.conf.all.proxy_ndp=1.
Additionally, we also need to allow IPv6 packet forwarding with the command
sysctl net.ipv6.conf.all.forwarding=1.
This former command instructs the kernel to enable its built-in NDP proxy for all interfaces, which isn't always desired. In order to enable it only for specific interfaces, you can use the
net.ipv6.conf.<interface name>.proxy_ndpproperty instead.
Once enabled, we need to tell the intermediary machine to reply to Neighbor
Solicitations for the relevant addresses on both sides. For this, we need to use
the ip -6 neigh add proxy <address> dev <interface> command. For this, let's
imagine that the IPv6 router is reachable at the address 2001:db8::1/64 and
that the internal machines are going to be addressed starting from 2001:db8::2/64
to 2001:db8::5/64. Let's also imagine that the IPv6 router is located on eth0
while that the internal machines are on eth1. We will need to run the following
commands on the intermediary machine:
# reply internally for the router
ip -6 neigh add proxy 2001:db8::1 dev eth1
# reply externally for the internal addresses
ip -6 neigh add proxy 2001:db8::2 dev eth0
ip -6 neigh add proxy 2001:db8::3 dev eth0
ip -6 neigh add proxy 2001:db8::4 dev eth0
ip -6 neigh add proxy 2001:db8::5 dev eth0
If you don't like adding these neighbors by hand, you can also use an automated and configurable proxy like NDPPD.
Once done, both sides will be aware of the other side when using the Neighbor Discovery Protocol, but even though packets are already being forwarded, this doesn't seem to work yet: they are leaving the network interface but never coming back.
This is caused by the middle machine's routing table, which is currently:
# ip -6 route
2001:db8::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via 2001:db8::1 dev eth0 metric 1024 onlink pref medium
As we can see, our intermediary machine has a route to send its 2001:db8::/64
traffic to its external interface, which is expected as it also has a public
address in the same subnet. When packets come back for the internal machines,
it examines its routing table and re-sends those packets to the external network,
which is both normal and the exact opposite of what we need to achieve. So let's
tell it to route traffic destined to the internal machines on the internal
interface eth1:
ip -6 route add 2001:db8::2 dev eth1
ip -6 route add 2001:db8::3 dev eth1
ip -6 route add 2001:db8::4 dev eth1
ip -6 route add 2001:db8::5 dev eth1
And now we should be able to ping an IPv6 address!
# ping -c 5 ipv6.camp
PING ipv6.camp (2602:fa6d:70:1:6666:6666:6666:6666) 56 data bytes
64 bytes from sea01.as21903.net (2602:fa6d:70:1:6666:6666:6666:6666): icmp_seq=1 ttl=51 time=147 ms
64 bytes from sea01.as21903.net (2602:fa6d:70:1:6666:6666:6666:6666): icmp_seq=2 ttl=51 time=147 ms
64 bytes from sea01.as21903.net (2602:fa6d:70:1:6666:6666:6666:6666): icmp_seq=3 ttl=51 time=147 ms
64 bytes from sea01.as21903.net (2602:fa6d:70:1:6666:6666:6666:6666): icmp_seq=4 ttl=51 time=147 ms
64 bytes from sea01.as21903.net (2602:fa6d:70:1:6666:6666:6666:6666): icmp_seq=5 ttl=51 time=147 ms
--- ipv6.camp ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 146.745/146.834/146.901/0.070 ms
Great success! Now we can put all that in configuration files to persist this
setup across reboots. Firstly, we can save the sysctl configuration in a file
named /etc/sysctl.d/99-ip-forward.conf:
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.proxy_ndp = 1
And as for the ip commands, let's create a script to add those dynamically
when the eth1 interface it brought up, located at /etc/network/if-up.d/eth1
and containing the following code:
#!/bin/bash
if [ "$IFACE" = eth1 ]; then
ip -6 neigh add proxy 2001:db8::1 dev eth1
PROXIED_ADDRESSES=("2001::db8:2" "2001::db8:3" "2001::db8:4" "2001::db8:5")
for PROXIED_ADDRESS in ${PROXIED_ADDRESSES[@]}; do
ip -6 neigh add proxy ${PROXIED_ADDRESS} dev eth0
ip -6 route add ${PROXIED_ADDRESS} dev eth1
done
fi
Let's not forget to make it executable with chmod u+x /etc/network/if-up.d/eth1,
and that's it!
We successfully deployed public IPv6 addresses on internal machines without a direct access to their router!
Additional links worth reading: