FreeBSD eXO router -- Get to ping6

Introduction

eXO – expansió Xarxa Oberta (open network expansion) is an non-profit association based in Barcelona and since 2019, it’s also a RIPE member.

eXO […] promotes open telecommunication networks, technological sovereignty, access to wholesale Internet services and reduces the digital gap.

They are also pretty much the only decent way to access native IPv6 in Barcelona and since RIPE ran out of IPv4 in 2019, it’s also pretty much the only decent Internet Service Provider around.

Also, the fact that you are not a customer but a member who can make things work out if necessary, kicks ass over <rant> Customer Service that keeps you waiting for hours and then insults your time and intelligence, or a service that forces you to use crappy routers, uses CGNAT or one that basically mobs you into joining them by aggressively calling you multiple times a week. (Who does these terrible things? Search for “biggest ISPs in Spain”) </rant>

Also: you get to pick your router and, besides something running OpenWRT (default), do it your self and e.g. run OpenBSD or FreeBSD.

This is roughly about how internet with eXO works and mainly about how to setup a FreeBSD system as a router for it.

PS: eXO does more than just fiber, there are also community-owned wireless mesh networks, but that’s for another day.

This is part of a series on home routing with FreeBSD.

FreeBSD eXO router – Get to ping6 (this one)

FreeBSD home router – Access Point and IPv6-only

FreeBSD home router – Legacy IPv4: NAT+DHCP

Introduction
Table of Contents
Hardware
Software and installation
- Hardware-specific bits
- External network bits
Remaining work
- Optional
- Internal network bits

Hardware

Fiber network

Since Barcelona is one of those regions, particularly cities, that missed the “let’s own our digital infrastructure” wagon, and it’s prohibitively expensive to install your own, the physical layer gets leased from a wholesale provider which, in turn, pays to the owner of the network (see “biggest ISPs in Spain”) for its use.

Depending on where in the metropolitan area, that provider changes, in the case of the city of Barcelona, it’s Sarenet.

Once Sarenet has done its job (which was extremely quick!), there is an optical fiber cable arriving at its future usage location, eXO gets notified and receives some details like the connection’s required VLAN and they also setup users and assign a fixed IP block from their pool to the connection.

Going Ethernet

Before we can use our regular router, an Optical Network Terminal is needed. In this case eXO offered me a Huawei EchoLife EG8010H which is kind of cute and does the job. Plus side: now I own the li’l box (no BS equipment rental or crappy hardware).

Router

I’m re-purposing electronics, whose intended initial use has developed beyond its capabilities.

In this case, that’s an APU2d4 from PCEngines, with a 1T mSATA drive for home storage and booting off a 32G SD card. Some more drives will be attached but are not relevant.

I decided to boot off a 32G SD card, because if anything happens, I can have it back up in no time. And it slightly simplifies the management of the data drive.

Software and installation

Since, because of the somewhat large mSATA, I’ll be using the APU as home storage as well, I will prefer FreeBSD over OpenBSD for this use-case, as the former will give me the wonders of ZFS, minus some of the easy routing awesomeness of the latter, while still being good at it.

Hardware-specific bits

Installing FreeBSD on a PCEngines APU2 is somewhat easy with the right equipment and a couple important tips.

Kamila covered that when she did her router, so do read that.

Once the OS is installed and before rebooting into the newly installed system, I prepare a few things for provisioning over local network:

Setup my root SSH key in /root/.ssh/authorized_keys
Ensure password logins are disabled in SSH
Enable root login over SSH
Add pf_enable="YES" to /etc/rc.conf and kldload pf
Create an empty /etc/pf.conf
Make sure things like sendmail and the like are fully disabled
I make sure I can use the router as a client over a local network with

# In /etc/rc.conf
ifconfig_igb2="DHCP"
ifconfig_igb2_ipv6="inet6 accept_rtadv"

External network bits

I’ve been switching to cdist for provisioning, so this might be easier for you once I finish publishing the types.

Since I’ll be doing the provisioning over local network on igb2 (mangement), igb0 is assumed to be the egress, wan interface and igb1 will be treated as a client-facing interface.

Wholesale provider requirements (PCP + VLAN)

The wholesale provider requires a specific IEEE 802.1p Priority Code Point and VLAN for the connection.

The magical man pages are man 8 ifconfig and man 4 vlan as well as the Chapter on VLANs from the FreeBSD Handbook.

This is achieved one time with:

# VLAN 24 -- Called exovlan because of ppp
# We also setup IEEE 802.1p priority code point
ifconfig igb0.24 create name exovlan vlan 24 vlanpcp 3

And permanently by adding following to /etc/rc.conf:

# Wholesale provider requirements
#   Create VLAN inteface
#   It doesn't look like ppp likes dots in interfaces, let's give this
#   interface a friendly name.
vlans_igb0="exovlan"
#   Setup the VLAN number
create_args_exovlan="vlan 24"
#   IEEE 802.1p priority code point
ifconfig_exovlan="vlanpcp 3"

PPPoE

This is what is actually used by eXO to provide the service.

The magical man pages are man 8 ppp and the Chapter on PPP from the FreeBSD Handbook.

And this is what the resulting /etc/ppp/ppp.conf looks like:

default:
  # An address is required
  set ifaddr 0 0 255.255.255.255

exo:
  # I'll be using my own DNS, otherwise use "enable dns"
  disable dns
  # Reminder that ppp does not like dots in interface names
  # so we created the VLAN 24 as exovlan and not igb0.24
  set device PPPoE:exovlan
  # Set a deterministic name
  iface name exoppp
  # Replace with your own data
  set authname ${EXO_USER}
  set authkey  ${EXO_PASS}
  set dial
  set login
  # Use LQR and echo
  set echoperiod 15
  set lqrperiod 15
  # mssfixup should be enabled by default
  enable echo lqr mssfixup
  set timeout 0
  # Pause of 5, increment +5, -5 times
  set redial 5+5-5 0
  # Wait random (1-30) to reconnect
  set reconnect random 0
  add! default HISADDR
  add! default HISADDR6

Now PPPoE can be used once with:

ppp -ddial exo

And permanently by adding following to /etc/rc.conf:

# PPPoE bits
ppp_enable="YES"
ppp_mode="ddial"
ppp_profile="exo"

Now pinging some IPv4 address should work :-).

IPv6 connectivity

IPv6 connectivity is slightly more difficult as eXO must delegate the /56 prefix to us.

In this case, I’ll want to quickly assign a /64 to the remaining ethernet interface, igb1.

Sadly dhclient in the base system does not include this feature, and that’s the kind of situation when ports are needed.

Good thing we now have some legacy (v4) network connectivity!

Let’s get the dhcp6 port:

# The first time pkg runs, it will bootstrap itself, since it's
# actually maintained as a port.
pkg install dhcp6

Since this does not belong to the base system, its settings are under /usr/local/etc and there is indeed a dhcp6c.conf.sample with nearly all we need.

By reviewing that, man 8 dhcp6c and specially man 5 dhcp6c.conf, following results:

# This is /usr/local/etc/dhcp6c.conf

interface exoppp {
    # This will solicit a non-temporary address on the wan interface
    send ia-na 0;
    # This will solicit a prefix on the exoppp interface
    send ia-pd 0;
};

# An id-assoc is always needed, even for NAs
id-assoc na 0 {
};

# This will use that delegated prefix to assign it the igb1 ethernet.
id-assoc pd 0 {
    prefix-interface igb1 {
        # This is how many bits we have "free"
        # Since IPv6 addresses are 128 bits and eXO delegates a /56,
        # we have 8 bits of freedom until a /64.
        # The remaining 64 bits, will be used by SLAAC locally
        sla-len 8;
        # This *decimal* will be turned binary to a sla-len bits
        # representation and be used along with the delegated prefix.
        sla-id 0;
    };
};

In order to use this, following must be added to /etc/rc.conf:

# Just enabling dhcp6c has issues when reconnecting
#dhcp6c_enable="YES"
dhcp6c_interfaces="exoppp"

And it can be tested out with

service dhcp6c start

Now ifconfig igb1 should have a globally routable unicast IPv6 in a /64 under your control and ping6 should work (real-time update: happy 2020!).

Update (2020-01-28): Just enabling dhcp6c has issues with IPv6 on reconnect after connection loss. Bright side of this is that we detected an anomaly on eXO‘s ppp server that will be solved in the upcoming planned maintenance.

In order to reconnect properly, let’s use ppp’s hook functionality.

On /etc/ppp/ppp.linkup:

MYADDR:
  # This should block as little as possible, we use !bg to keep going
  # "in the background" so ppp is not blocked.
  !bg /usr/sbin/service dhcp6c onerestart

On /etc/ppp/ppp.linkdown:

MYADDR:
  # dhcp6c can take a bit to close, if the link came back up
  # meanwhile things can be messy. We won't use !bg here.
  ! /usr/sbin/service dhcp6c onestop

Remaining work

Create a bridge interface for ethernet and wlan.
Setup the wireless network
Setup SLAAC
Secure the firewall
Create cdist types for all the things

Optional

Setup dhcpd to hand out local IPv4s
Setup NAT

Internal network bits

Monitoring (Prometheus)
DNS (unbound)
Traffic dashboards (taking into account the percentiles)

Evilham.com