Trying out Kubernetes: Network and VMs

Introduction

As part of trying out Kubernetes: planning, we are going to set up the network and VMs!

Quick reminder that this will be virtualised Linux on a FreeBSD physical host. This part is FreeBSD-specific.

Table of contents

Network

Given that bhyve(8)‘s networking uses tap(4) devices, which we can bridge and route as needed, we will only need basic FreeBSD networking knowledge.

This actually is somewhat similar to the setup I use for IPv6-VNET jails.

That is, we will need a bridge and a tap device for each network segment:

# Adding following to /etc/rc.conf

# k8s testing
# Create tap and bridge interfaces
cloned_interface="${cloned_interfaces} tap0 bridge0 tap1 bridge1"
# Rename them
ifconfig_bridge0_name="etcdbridge"
ifconfig_tap0_name="etcdtap"
ifconfig_bridge1_name="k8sbridge"
ifconfig_tap1_name="k8stap"

# Save our network prefix data
_k8s_v6="2XXX:XXXX:XXXX"
_k8s_v4="10.2"

# Define etcd network
ifconfig_etcdtap="fib 1"
ifconfig_etcdbridge_descr="etcd cluster and k8s control-panel"
ifconfig_etcdbridge="inet ${_k8s_v4}.1.1/24 addm etcdtap up fib 1"
ifconfig_etcdbridge_ipv6="inet6 ${_k8s_v6}:1::/64 auto_linklocal"

# Define k8s network
ifconfig_k8stap="fib 1"
ifconfig_k8sbridge_descr="k8s nodes and control-panel"
ifconfig_k8sbridge="inet ${_k8s_v4}.2.1/24 addm k8stap up fib 1"
ifconfig_k8sbridge_ipv6="inet6 ${_k8s_v6}:2::/64 auto_linklocal"

Remarkable is that we are assigning /64 subnets for IPv6 and /24 for IPv4, and that we will use static IP assignment since this is just a handful of machines.

I am also using a different routing table (fib 1) in order not to mix traffic.

Note that besides valid IPv6 routing, there are NAT rules taking care of IPv4:

# k8s NAT
nat on $ext_if inet from {10.2.1.0/24, 10.2.2.0/24} to any -> ($ext_if:0)

IP assignment

We will perform static IP assignment due to the small scale:

Now I should be able to create the Virtual Machines on each network bridge.

Virtual Machines

As mentioned before, the virtualisation will be done with bhyve(8), particularly with vm-bhyve(8), as it is very lightweight and takes care of lower level details for us.

vm-bhyve switches

We add following to VM_BHYVE/.config/system.conf:

switch_list="etcdbridge k8sbridge"

type_etcdbridge="manual"
bridge_etcdbridge="etcdbridge"

type_k8sbridge="manual"
bridge_k8sbridge="k8sbridge"

Which makes sure that vm-bhyve will add the network interfaces to the appropriate bridges.

Template

Then we create the vm-bhyve(8) template:

# This lives in VM_BHYVE/.templates/k8s.conf
cat k8s.conf
loader="grub"
cpu=1
memory=1G

network0_type="virtio-net"
network0_switch="k8sbridge"

# We use sparse zvols for storage, which enables us to snapshot and clone the
# whole disk while only allocating space as we start using it.
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"

Since we will later clone this VM, we’ll add network adapters / change bridges.

OS: Debian

We’ll use Debian’s latest release: bookworm:

curl -L \
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso > \
  VM_BHYVE/.iso/debian-12.2.0-amd64-netinst.iso

Creating the base VM

vm create -t k8s -s 240G k8sbase

Now we install the OS using the serial console and the downloaded .iso:

vm install -f k8sbase debian-12.2.0-amd64-netinst.iso

I use an unassigned IP for the base VM, so I can apply provisioning for the final touches.

Still over serial, we finish up basic things like enabling SSH but only with public keys, and installing my keys.

# In /etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
UsePAM no
KbdInteractiveAuthentication no

$ service sshd restart
$ mkdir .ssh
$ chmod 750 .ssh
$ wget https://evilham.com/ssh.pubkey -O .ssh/authorized_keys
$ chmod 640 .ssh/authorized_keys

Cloning the VMs

# We create a snapshot first
$ vm snapshot -r k8sbase@clean
# And then actually clone it, we start with one VM of each type
$ vm clone k8sbase@clean k8s-etcd-1
$ vm clone k8sbase@clean k8s-cp-1
$ vm clone k8sbase@clean k8s-node-1

Now, we can edit each VM’s VM_BHYVE/VM/VM.conf file in order to assign it to the corresponding network(s):

# In VM_BHYVE/k8s-etcd-1/k8s-etcd-1.conf:
network0_type="virtio-net"
network0_switch="etcdbridge"

# In VM_BHYVE/k8s-cp-1/k8s-cp-1.conf:
network0_type="virtio-net"
network0_switch="etcdbridge"
network1_type="virtio-net"
network1_switch="k8sbridge"

# In VM_BHYVE/k8s-node-1/k8s-node-1.conf:
network0_type="virtio-net"
network0_switch="k8snode1"

And using provisioning against the unassigned IP one VM at a time, we:

• Reset the SSH keys and hostid so they are not shared (this is important for etcd)
• Apply the assigned static IP configuration on all interfaces
• Reboot and check that SSH on the new VM is possible
• Snapshot the VM

Conclusion

FreeBSD’s networking and virtualisation stack are great and it often surprises me how effortless it is to get things done.

Networking is set up, one of each VM type is set up with outgoing dual stack internet and two separated network segments.

Next up, setting up etcd. These were the steps according to the plan

• [X] we start by setting up the network segments as bridges on the physical host, taking care of NAT, firewall and IPv6 routing
• [X] then we set up the etcd cluster
• [ ] finally, we actually setup the Kubernetes cluster