FreeBSD: IPv6 en jail con VNET

Introduction

Fellow human @LeJax@bsd.network has been trying to setup IPv6-enabled VNET Jails in FreeBSD.

This, turns out, can be a bit frustrating, because all the tools are there, but the tricks are not widely documented.

In an attempt to fix that, I am going to document all the tricks and sources I have found (and remember) that were necessary until I got a working and reproducible setup. Hopefully this will not only help LeJax, but also it will review my notes and allow them (or someone else) to prepare a nice general-use write up on this topic that might be useful to others.

I use cdist and iocage extensively, but the general tricks should be valid without any or either of those.

Table of Contents

General notes

Reminder about FreeBSD jails

Jails on FreeBSD are a very useful OS tool that is tightly integrated with the rest of the Operating System, which means that, for the most part, issues regarding IPv6 in jails are actually issues with general networking and not specific to jails.

Note on iocage

iocage is a helper to create and manage jails, some like it, some don’t. It itself doesn’t do much magic, but has some ZFS integrations that I find useful. These notes should mostly apply to jails created with jail(8) as well.

Note on cdist

cdist is a “usable configuration management system”. It’s basically shell on steroids to manage servers. If I cite a type, the manifest and gencode-remote (optionally the explorer/*) files should be reviewed. These should contain enough comments to be understandable and to identify the relevant bits.

Networking options for IPv6 VNET jails

VNET: virtualised network stack

From vnet(9):

VNET is the name of a technique to virtualize the network stack.  The
basic idea is to change global resources most notably variables into per
network stack resources and have functions, sysctls, eventhandlers, etc.
access and handle them in the context of the correct instance.  Each
(virtual) network stack is attached to a prison, with vnet0 being the
unrestricted default network stack of the base system.

With VNET(9), our jails basically become a separated machine network-wise. We can (and maybe want to) also run a firewall on the jail itself.

It also means we have to take care of networking somehow; this is usually the tricky bit. For IPv4 that means: static addresses or DHCP, for IPv6 that might mean SLAAC.

Because of IPv6 nature, the usual tricks for IPv4 of using NAT / port forwarding are a subpar solution, so we shouldn’t necessarily just try to replicate that.

Option A: jail host acts as a router

Depending on how that is happening, our host might need to act as a router (the sysctl nodes: net.inet.ip.forwarding / net.inet6.ip6.forwarding set to 1, dhcpd(8) and rtadvd(8) running, …, see: FreeBSD router).

This might be the case if, e.g. we assign a /64 to the jails in a given host.

| jail |      | jail host  |      | vnet |
| host | <--> | bridge     | <--> | jail |
              | (optional) |

Has a           Receives            uses SLAAC
routed          rtadv               to gain IPv6
/64             for the /64         connectivity
Runs
rtadvd(8)

If we do this, some important things to take into account:

• The jail host will need to forward packages (ipv6_gateway_enable=YES in /etc/rc.conf, which results in sysctl net.inet6.ip6.forwarding=1)
• The jail host will have to run rtadvd(8) in a way that reaches the jails
• Depending on the setup, ipv6_cpe_wanif should be set in /etc/rc.conf if the jail host needs to accept route advertisements on some interface.

In any case rc.conf(5) has more information on these settings.

Option B: bridge to public interface

This might be easier to setup if it matches our security needs and we already have a working router with SLAAC.

                                   ·--> | jail host |  firewall passes
                                  /     | interface |  traffic on bridge
| router | <--> | jail host | <--·                     (switch-like)
                | bridge    | <--·
                                  \     | vnet |       uses SLAAC to
Pre-existing                       ·--> | jail |       gain IPv6
Does SLAAC                                             Connectivity

Option C: no SLAAC, static routing

This gets a tad complicated to manage and I haven’t tested it.

The network schemes might look a lot like Option A and Option B depending on how we actually do it, but it doesn’t have to.

The main difference being that instead of SLAAC + ND + DAD, we are potentially routing things somehow else.

IPv6: accept_rtadv and auto_linklocal

This is actually the key point to get everything to work.

There is a bug in the latest iocage release (1.2 in January 2021), in which newly created jails are not properly prepared for IPv6. This is fixed in the development branch (see commit).

For regular jails, and before iocage has release beyond 1.2, we have to manually ensure that the interface is properly setup.

In any case, we do need linklocal addresses, in order for Neighbour Discovery (ND) to work.

# Inside the Jail
# Partial contents of /etc/rc.conf

# Get an IPv4 on jail start with DHCP
ifconfig_epair0b="SYNCDHCP"
# Ensure we accept route advertisements and have a linklocal address
ifconfig_emailr0b_ipv6="inet6 accept_rtadv auto_linklocal"

Firewall – Passing the devices

Nowadays it should be safe to use, e.g. pf(4) on VNET jails (TODO: citation needed, it’s somewhere on the freebsd-net/freebsd-questions ML).

We basically setup the firewall as usual, except that, by default, the devfs.rules(5) do not expose the necessary packet filter devices.

Something similar might apply for ipfw(4).

# In the Jail Host:
# Partial contents of /etc/devfs.rules
[devfsrules_iocage=5]
add include $devfsrules_jail
add path pf unhide
add path pflog unhide
add path pfsynv unhide

Changing this file requires a service devfs restart.

And we have to make sure that our jails use this set of devfs rules.

iocage set devfs_ruleset=5 ${JAIL}

In any case, care should be taken not to block too much (e.g. blocking all icmp6 will utterly break IPv6 connectivity).

Using my cdist types

I wrote a couple cdist types that take care of all this in a reliable fashion. You can find them and the source for other useful cdist types on my cdist repository.

• __evilham_iocage: Setup the jail host with iocage to be ready for IPv6-enabled jails.
• __evilham_iocage_jail: Create and maintain an IPv6-enabled jail.

And the manifest for the jail host looks like this:

# Setup the jail host
__evilham_iocage --zpool jails
# Create/update the necessary jails
require="__evilham_iocage" __evilham_iocage_jail \
    "jail.example.org" \
    --bridge "bridge0"

Then I can setup DNS and ssh -6 jail.example.org, or use a cdist manifest to set up the service.

Conclusion

Hopefully this contains enough pointers to help others figure out IPv6-enabled jails, and to document this in a more generic fashion for everyone else.

If this was useful for you, do let me know.