FreeBSD jails: networking: vnet
Instead of aliasing an existing network interface, vnet lets you create a virtual network interface.
It is assigned it's own ip address, a separate loopback address, and is generally more secure.
Documentation
bridge docs | https://www.freebsd.org/doc/handbook/network-bridging.html |
Tutorials
vivek freebsd-11 | https://www.cyberciti.biz/faq/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/ |
reddit freebsd-12 | https://www.reddit.com/r/freebsd/comments/ahdbbq/howto_jails_freebsd_12_vnet_zfs/ |
Overview
Outside Jail:
- Install kernel source code
- Recompile kernel with VIMAGE enabled
- Create a single bridge netwk-iface from netwk-iface providing internet
- Create an epair netwk-iface connecting bridge to jail (for each jail)
- Add jail epairs to a ifconfig interface-group, so you can manage pf rules for them
Inside Jail:
- Define vnet interface you'll be using to connect to internet (in place of em/ix)
- Configure internet like you would for a normal host within rc.conf
- Run pf rules specific to that jail only
Setup
Recompile Kernel with VIMAGE
Verify if VIMAGE is already compiled into kernel
# if file exists, VIMAGE is already enabled test -d /usr/obj/usr/src/amd64.amd64/sys/VIMAGE && echo "VIMAGE installed" || echo "VIMAGE missing" # alternatively, `uname -v` should include VIMAGE uname -v | grep VIMAGECompile kernel with VIMAGE
# Fetch kernel sourcecode, extract to /usr/src fetch -o /tmp ftp://ftp.freebsd.org/pub/`uname -s`/releases/`uname -m`/`uname -r | cut -d'-' -f1,2`/src.txz tar -C / -xvf /tmp/src.txz # copy VIMAGE configuration cp -v /usr/src/share/examples/jails/VIMAGE /usr/src/sys/amd64/conf/VIMAGE # recompile kernel cd /usr/src sudo make -j 16 KERNCONF=VIMAGE kernel # '-j' number of async jobs sudo reboot # install required tools sudo cp -v /usr/src/share/examples/jails/{jib,jng} /usr/sbincreate filesystem
Create a thin or thick jail like you normally would.
You do not need to compile VIMAGE in the jail kernels.create devfs ruleset
See FreeBSD devfs for explanation.
# /etc/devfs.conf # NOTE: replace '5' with a ruleset number not # taken in /etc/defaults/devfs.conf # or /etc/devfs.conf [devfsrules_jail_rsnapshot=5] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path 'tun*' unhide add path zfs unhide/etc/jail.conf
# /etc/jail.conf $uplink_dev = "em0"; # source of internet $vnet_iface = "jail0"; # iface within jail exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; path = "/usr/local/jails/${host.hostname}/filesystem"; testjail { host.hostname = "testjail"; # vnet setup $epair = "epair1"; $ipaddr = "192.168.1.111"; mount.devfs; devfs_ruleset = "5"; allow.raw_sockets; # if want ping vnet; vnet.interface = $vnet_iface; exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || ( ifconfig bridge0 create up && ifconfig bridge0 addm $uplink_dev )"; exec.prestart += "ifconfig ${epair} create up || echo 'Skipped creating epair (exists?)'"; exec.prestart += "ifconfig bridge0 addm ${epair}a || echo 'Skipped adding bridge member (already member?)'"; exec.prestart += "ifconfig ${epair}a group jail_epairs"; exec.prestart += "pfctl -t jail_ips -T add 1.1.1.1"; exec.poststop = "ifconfig bridge0 deletem ${epair}a"; exec.poststop += "ifconfig ${epair}a destroy"; exec.poststop += "ifconfig ${epair}a -group jail_epairs"; exec.poststop += "pfctl -t jail_ips -T delete ${ipaddr}"; exec.clean; exec.created = "ifconfig ${epair}b name ${vnet_iface} || echo 'Skipped renaming ifdev to ${vnet_iface} (looks bad...)'"; }usage within jail
# /etc/rc.conf hostname = "testjail"; ifconfig_jail0="inet 192.168.1.240/24" defaultrouter="192.168.1.1" # disable sendmail in your jail if you don't need it # (slows jail startup) sendmail_enable="NONE" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO" # enable ssh sshd_enable="YES"allow traffic from host
Adding the following rule will allow all traffic to your jail. The jail can then run it's own pf, and manage a simpler ruleset.
Notes on PF rules concerning epairs
- whether or not the bridge has an ip address is irrelevant
- only packet source/dest count in pf, passing through various interfaces (ex: bridge) does not require special rules.
- pf rules are associated with epair side
a
orb
.
A is always attached to bridge, B is always attached to jail.You have 3x options here.
1. Dynamically manage members of an ifconfig interface group (best)
I like this best, since pf can remain blissfully ignorant of the actual interfaces. There is no duplication at all, nearly all jail logic is managed in the jail itself.
# /etc/jail.conf testjail { # ... exec.prestart += "ifconfig ${epair}a group jail_epairs"; exec.prestart += "pfctl -t jail_ips -T add 1.1.1.1"; exec.poststop += "ifconfig ${epair}a -group jail_epairs"; exec.poststop += "pfctl -t jail_ips -T delete 1.1.1.1"; }# /etc/pf.conf ext_if="em0" vnet_bridge="bridge0" table <jail_ips> persist block log all pass out all pass in on { $vnet_bridge, jail_epairs, $ext_if } to <jail_ips> pass in on { $vnet_bridge, jail_epairs } from <jail_ips>
2. keep duplicate list of used epairs in a variable within pf.conf (not DRY)
jail_epairs = { epair1a, epair2a, ... } pass in on $jail_epairs all
3. pass in any traffic to all epairs (insecure)
All epairs are added automatically to the
epair
interface group.
I don't like this, since I may want to use epairs for other tasks in the future.pass in on epair alltest jail network
nc -uvz 192.168.1.1 53 # confirm nameserver's port-53 acessible via UDP (DNS) nc -vz google.com 80 # confirm external network accessible
NOTE jls for vnet jails will not list ip addresses. (the following is normal).JID IP Address Hostname Path 5 192.168.1.111 non-vnet-jail /usr/local/jails/non-vnet-jail/filesystem 8 vnet-jail /usr/local/jails/vnet-jail/filesystem