FreeBSD jails: networking: vnet

From wikinotes

Instead of aliasing an existing network interface, vnet lets you create a virtual network interface.
It is assigned it's own ip address, a separate loopback address, and is generally more secure.

Documentation

bridge docs https://www.freebsd.org/doc/handbook/network-bridging.html

Tutorials

vivek freebsd-11 https://www.cyberciti.biz/faq/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/
reddit freebsd-12 https://www.reddit.com/r/freebsd/comments/ahdbbq/howto_jails_freebsd_12_vnet_zfs/

Overview

Outside Jail:

  1. Install kernel source code
  2. Recompile kernel with VIMAGE enabled
  3. Create a single bridge netwk-iface from netwk-iface providing internet
  4. Create an epair netwk-iface connecting bridge to jail (for each jail)
  5. Add jail epairs to a ifconfig interface-group, so you can manage pf rules for them

Inside Jail:

  1. Define vnet interface you'll be using to connect to internet (in place of em/ix)
  2. Configure internet like you would for a normal host within rc.conf
  3. Run pf rules specific to that jail only

Setup

Recompile Kernel with VIMAGE

Verify if VIMAGE is already compiled into kernel

# if file exists, VIMAGE is already enabled
test -d /usr/obj/usr/src/amd64.amd64/sys/VIMAGE && echo "VIMAGE installed" || echo "VIMAGE missing"

# alternatively, `uname -v` should include VIMAGE
uname -v | grep VIMAGE

Compile kernel with VIMAGE

# Fetch kernel sourcecode, extract to /usr/src
fetch -o /tmp ftp://ftp.freebsd.org/pub/`uname -s`/releases/`uname -m`/`uname -r | cut -d'-' -f1,2`/src.txz
tar -C / -xvf /tmp/src.txz

# copy VIMAGE configuration
cp -v /usr/src/share/examples/jails/VIMAGE /usr/src/sys/amd64/conf/VIMAGE

# recompile kernel
cd /usr/src
sudo make -j 16 KERNCONF=VIMAGE kernel  # '-j' number of async jobs
sudo reboot

# install required tools
sudo cp -v /usr/src/share/examples/jails/{jib,jng} /usr/sbin

create filesystem

Create a thin or thick jail like you normally would.
You do not need to compile VIMAGE in the jail kernels.

create devfs ruleset

See FreeBSD devfs for explanation.

# /etc/devfs.conf

# NOTE: replace '5' with a ruleset number not
#       taken in /etc/defaults/devfs.conf
#       or       /etc/devfs.conf
[devfsrules_jail_rsnapshot=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'tun*' unhide
add path zfs unhide

/etc/jail.conf

# /etc/jail.conf

$uplink_dev = "em0";    # source of internet
$vnet_iface = "jail0";  # iface within jail
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
path = "/usr/local/jails/${host.hostname}/filesystem";


testjail {
    host.hostname = "testjail";

    # vnet setup
    $epair = "epair1";
    $ipaddr = "192.168.1.111";

    mount.devfs;
    devfs_ruleset = "5";
    allow.raw_sockets;   # if want ping
    vnet;
    vnet.interface  = $vnet_iface;

    exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || ( ifconfig bridge0 create up && ifconfig bridge0 addm $uplink_dev )";
    exec.prestart += "ifconfig ${epair} create up              || echo 'Skipped creating epair (exists?)'";
    exec.prestart += "ifconfig bridge0 addm ${epair}a          || echo 'Skipped adding bridge member (already member?)'";
    exec.prestart += "ifconfig ${epair}a group jail_epairs";
    exec.prestart += "pfctl -t jail_ips -T add 1.1.1.1";

    exec.poststop = "ifconfig bridge0 deletem ${epair}a";
    exec.poststop += "ifconfig ${epair}a destroy";
    exec.poststop += "ifconfig ${epair}a -group jail_epairs";
    exec.poststop += "pfctl -t jail_ips -T delete ${ipaddr}";

    exec.clean;

    exec.created = "ifconfig ${epair}b name ${vnet_iface} || echo 'Skipped renaming ifdev to ${vnet_iface} (looks bad...)'";
}

usage within jail

# /etc/rc.conf

hostname = "testjail";
ifconfig_jail0="inet 192.168.1.240/24"
defaultrouter="192.168.1.1"

# disable sendmail in your jail if you don't need it
# (slows jail startup)
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# enable ssh
sshd_enable="YES"

allow traffic from host

Adding the following rule will allow all traffic to your jail. The jail can then run it's own pf, and manage a simpler ruleset.

Notes on PF rules concerning epairs


  • whether or not the bridge has an ip address is irrelevant
  • only packet source/dest count in pf, passing through various interfaces (ex: bridge) does not require special rules.
  • pf rules are associated with epair side a or b.
    A is always attached to bridge, B is always attached to jail.

You have 3x options here.

1. Dynamically manage members of an ifconfig interface group (best)

I like this best, since pf can remain blissfully ignorant of the actual interfaces. There is no duplication at all, nearly all jail logic is managed in the jail itself.

# /etc/jail.conf

testjail {
  # ...
  exec.prestart += "ifconfig ${epair}a group jail_epairs";
  exec.prestart += "pfctl -t jail_ips -T add 1.1.1.1";

  exec.poststop += "ifconfig ${epair}a -group jail_epairs";
  exec.poststop += "pfctl -t jail_ips -T delete 1.1.1.1";
}
# /etc/pf.conf

ext_if="em0"
vnet_bridge="bridge0"
table <jail_ips> persist

block log all
pass out all

pass in on { $vnet_bridge, jail_epairs, $ext_if } to <jail_ips>
pass in on { $vnet_bridge, jail_epairs } from <jail_ips>


2. keep duplicate list of used epairs in a variable within pf.conf (not DRY)

jail_epairs = { epair1a, epair2a, ... }
pass in on $jail_epairs all


3. pass in any traffic to all epairs (insecure)

All epairs are added automatically to the epair interface group.
I don't like this, since I may want to use epairs for other tasks in the future.

pass in on epair all

test jail network

nc -uvz 192.168.1.1 53  # confirm nameserver's port-53 acessible via UDP (DNS)
nc -vz  google.com 80   # confirm external network accessible


NOTE jls for vnet jails will not list ip addresses. (the following is normal).

JID  IP Address      Hostname                      Path
  5  192.168.1.111   non-vnet-jail                 /usr/local/jails/non-vnet-jail/filesystem
  8                  vnet-jail                     /usr/local/jails/vnet-jail/filesystem