Installing FreeBSD 6.0 with PF Firewall

Document by: David Pierron © 2005
This work is licensed under a Creative Commons License.
v1.0 :: Thu Nov 17 05:34:39 EST 2005 :: Initial write
v1.1 :: Thu Dec 1 06:57:19 EST 2005 :: Detailed FBSD installation info
v1.2 :: Sat Dec 3 06:57:39 EST 2005 :: Got bridge working (wow! freaky! only 20 seconds off from 2 days ago)
Covering installation of FreeBSD 6.0

WARNING:

This document is still "Under Construction" ... It will currently get a working bridge, but everything I've set out to cover is not done yet.
If you have any suggestions, please email me at himself@davidpierron.com

Objective
Requirements
Install FreeBSD
Configuring the new install
Get Current Sources
Install PHP and Apache2
Install Midnight Commander
Complete PHP and Apache2 Installation
Compiling a Custom Kernel
Editing files for PF
Working on the PF Rules
Getting FTP servers to work behind PF
PF Commands
PF Monitoring
Install pftop
Usage Graphs

Reference Links

— Objective

This document is first and foremost my notes on how I installed this particular firewall and will also serve as instruction for anyone else wanting to set up a like machine to protect their network. Sometimes it's hard to find all the information all in one place. I am attempting to provide that here with links to where all of the information was gleaned.

I also do not need a router. Many HOW-TOs are instructions on how to set up a FreeBSD router doing NAT and also happened to use PF. I only need PF. I am also providing services to the Internet behind this firewall. FTP, SMTP, DNS, HTTP, POP3, IMAP, HTTPS, SMTP-AUTH, IRC and game servers.

If you follow all of the steps in this document, you should end up with a FreeBSD PF firewall running an IPless bridge on two NICs plus a third NIC for maintenance and monitoring. This machine will have PHP and Apache2 installed with applications installed to produce graphs showing firewall activity. You can skip the third NIC and anything else that's associated with it. Everything is run from the CLI (Command Line Interface).

— Requirements

You must use FreeBSD 6.0 or greater for this firewall. The reason for this is that the if_bridge option is not available in any prior version of FreeBSD. I need an IPLess bridge for my needs. I do not do NAT (Network Address Translation) and use all real world (routable) IP addresses.

You will need a copy of FreeBSD. You can get installable media from FreeBSD.Org. The ISO image you're looking for is: 6.0-RELEASE-i386-bootonly.iso which is about 45M. Faster download than a full 650M ISO.

You will need a PC with at least two NICs. (I am going to use three NICs). The PC need not be a powerhouse. You could use the simplest of PCs you have sitting around and see what your results are. I've used a P2 350mHz with 128M of RAM for years.

You will need this machine connected to the Internet. The faster the connection, the better.

That's all you really need, an Internet connection, a PC and a copy of FreeBSD 6.0 installation media. I chose to use a fairly robust machine for this installation since I've been bitten in the past using a machine without a third NIC. Without the third NIC you receive no nightly emails from this machine. You will not know if there is a problem developing. You will not be able to easily download new sources and upgrade this machine's kernel, release or ports. (The only way to accomplish this would be to disable the firewall and set one NIC to talk to the Internet while you downloaded sources and recompiled. In my situation, there can be no substantial downtime.) You will not be able to backup your configuration. Over the course of time, your rules may become fairly complex. It's impossible to remember all the changes you've made to the file, and you always can't find that piece of paper you wrote the changes on. Or, assembling all of the different papers from all of the different times you've made changes. There's no way around this, I have convinced myself I need a third NIC. =)

For even more protection, I am using a SCSI RAID solution. The RAID is performed by the hardware so the operating system doesn't even know this is going on. I chose RAID mirroring. The server is a Compaq Proliant ML530 with 1G of RAM. It's a dual 800mHz P3 Xeon. I'm using 3 Intel Pro/100 Ethernet NICs. This is the machine I am documenting in this text. I will surely downsize the machine once I am satisfied with the quality of this document.

— Install FreeBSD

Let's begin.

I wrote a FreeBSD installation HOW-TO with screen shots a while back covering the now extinct 4.x release. Most of it applies to 6.0 except for the kernel configuration at the beginning of the install. (DO NOT use my FreeBSD Installation HOW-TO for this install, only use it as a reference. The install covered in this document is done differently. More specifically, this install is via FTP while the other is done from a complete ISO.)

You may have different choices than I have listed below. What is documented is the choices I made for my particular installation.

Start with a fresh hard drive and boot from the burned ISO CD (boot only) mentioned above.

Select a Standard installation.
Allocate the entire hard drive to FreeBSD by pressing "A" followed by "Q" to quit/save.
Select the Standard BootMgr
In the DiskLabel Editor, select "A" for an automatic configuration followed by "Q" to quit/save.
Choose the Minimal Distribution by pressing "space" and "tab" to "OK" then press "ENTER"
Choose FTP for Installation Media (Passive if behind an existing firewall)
Choose a mirror close to you. You will be asked to configure your network card.
IPV6? No ... DHCP? No and fill in the remaining fields pertaining to your NIC.
I have 3 of the same network cards in this machine. I didn't know which was which so I connected each to the switch so any one I chose would work. I picked fxp2 as my Internet facing NIC. Once configured, it could take a minute or two to connect to the FTP server. Sometimes it's immediate. This will happen right after your drives are formatted. You will see "base" being downloaded. It's about 57.7M.
The quick questions:
1. Network Gateway? No
2. Configure inetd? No
3. enable SSH? No
4. anonymous FTP? No
5. NFS Server? No
6. NFS Client? No
7. customize console? No
8. Time Zone? YES
  - CMOS set to UTC? No
  - 2 - America
  - 45 - United States
  - 1 - Eastern Time
  - EST? Yes
9. Linux Compatibility? No
10. Mouse? No
11. Packages? No
12. Initial User Accounts? Yes
  - Select User
  - Fill in: Login ID, Password, Full Name, Member Group=wheel
  - "tab" to OK - Select EXIT
13. Set root password
14. Visit Config menu? No
15. "tab" to EXIT INSTALL press "ENTER"
16. You sure? Yes
Machine will reboot from hard drive, remove CD quick!

— Configuring the new install

I like to do a quick customization of the box to make myself feel "at home". I'm not a vi kind of guy, so I use ee until I have another means of editing files.

Set the date/time: ntpdate ip.address.of.timeserver

Edit /etc/crontab to automate the running of ntpdate:

	0 * * * * root ntpdate ip.address.of.timeserver 1>/dev/null 2>/dev/null

Save the file

Edit /root/.cshrc
Add a new alias:

	ls ls -G

Alter the set prompt to:

	set prompt = "[`whoami`@`hostname -s` `pwd`]# "

On the next line add an alias:

	alias cd 'cd \!* ; set prompt = "[`whoami`@`hostname -s` `pwd`]# "'

Save the file

Edit /etc/aliases
Uncomment the master root line and set the email address where you want informative messages delivered.
Save the file and run: newaliases

— Get Current Sources

Now we have a barren FreeBSD installation. We need to eventually rebuild the kernel for the PF firewall (and in my case for multiple CPUs) and we need the ports tree to install the applications that we want to use on this machine. The first thing we're going to do is install a package called cvsup to retrieve all of this goodness for us. Since we're not running a window manager, there is a special package just for us:

	pkg_add -r cvsup-without-gui

This will download and install the cvsup utilty in binary form.

	rehash
	cd /usr/share/examples/cvsup
	cp ports-supfile /usr/local/etc
	cp standard-supfile /usr/local/etc
	cd /usr/local/etc

We rehash to be able to access cvsup in our path. Then we copy the ports and source supfiles to our /usr/local/etc directory. We need to edit these files (line 51 in both) and point them to a cvsup mirror near us.

	ee ports-supfile
	ee standard-supfile
	cd
	cvsup -g -L 2 /usr/local/etc/ports-supfile ; cvsup -g -L 2 /usr/local/etc/standard-supfile

After editing, I just like switching to root's home with the "cd" command and then we issue 2 commands with the last statement downloading all of the sources to our computer. We could have done them one at a time, but you don't have to wait for all the ports to come down and then re-issue the command for the kernel source. It'll do them easily in succession. Take a walk, watch a TV show ... It will take a little bit of time, definitely under a half hour.

— Install PHP and Apache2

If you're not going to have external access to this machine, then you have no need installing PHP or Apache2. If you're going to install a window manager with plans of accessing everything through local host, then you may want to continue with this step.

	cd /usr/ports/lang/php4
	make install clean

Switch to the directory with the source make files. When you choose to make the PHP port, it will ask you what options you want for the install. I tick Apache2 and untick everything else.

	cd ../php4-extensions
	make install clean

The PHP4 extensions are required for many things PHP4 related. Mostly the MySQL extension. When you choose to install this port, you will be presented with a screen allowing you to choose what extensions you want. I leave all the defaults and additionally tick GD.

— Install Midnight Commander

I use Midnight Commander (mc) to hop around the hard drive, view and edit files. Sometimes it's a bit quicker than the command line. It makes other tasks easier as well for me such as copying files, etc. It runs without a window manager right from the command line and in FreeBSD it's in color! I use the default configuration but also untick SUBSHELL and SAMBA.

	cd /usr/ports/misc/mc
	make install clean
	rehash

— Complete PHP and Apache2 Installation

Once all this is done compiling (and it will take a while), you should edit /usr/local/etc/apache2/httpd.conf
I use mc (Midnight Commander) to surf to the apache2 directory and use mc's editor (F4) to edit httpd.conf. Using the arrow down key to browse the file and see what needs to be changed, the first thing is Listen. I change that to the IP address of the machine.

You can use F7 to search for DirectoryIndex or just keep using the arrow down key. You want to add index.php to the DirectoryIndex list. I always make it first. Then after this line I use F9 bring up the menu and select Insert File and type /usr/ports/lang/php4/pkg-message.mod ... I delete the lines that would cause problems. This just saves me the typing. From this point on I won't address what editor to use. I will simply tell what files to edit. Whatever editor you choose is your choice to choose from your choice.

In order to test the PHP installation, key in the following:

	cd /usr/local/etc
	cp php.ini-recommended php.ini
	cd /usr/local/www/data-dist
	echo "<? phpinfo(); ?>" >test.php
	apachectl start
	php -v

We change to the directory where the files are served for the web. We create a test.php file with the one simple statement in it. If you point a browser to this machine's IP address or domain name, you should see the Apache2 default page. Tack on "/test.php" to the end of the URL, you should see that PHP is working on the web server. The last command above "php -v" should echo out to your console the CLI version of PHP (which is ultimately the installed version for all applications to use).

To have Apache2 start whenever your machine is started, edit /etc/rc.conf and add:

	apache2_enable="YES"

— Compiling the Custom Kernel

Edit /usr/src/sys/i386/conf/GENERIC:

	#options    INET6      #Around line 36 comment out if you're not using IPV6
	#                       (will cause PF not to auto generate IPV6 rules
	
	#
	# Then move to bottom of file and add:
	#
	
	# Bridge support
	device          if_bridge

	# PF support
	device          pf
	device          pflog
	device          pfsync

	# ALTQ support
	options         ALTQ
	options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
	options         ALTQ_RED        # Random Early Detection (RED)
	options         ALTQ_RIO        # RED In/Out
	options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
	options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
	#options         ALTQ_NOPCC      # Required for SMP build

	# other stuff
	options         IPSTEALTH
	options         HZ=1000

NOTE: Pick ONE of the following methods. If multiprocessor, use SMP. The SMP file pulls in the changes you've made to GENERIC above.

Build the new kernel for a single processor:

	cd /usr/src/sys/i386/conf
	config GENERIC
	cd ../compile/GENERIC
	make depend ; make ; make install
	reboot

Build the new kernel for a multi-processor:

	cd /usr/src/sys/i386/conf
	config SMP
	cd ../compile/SMP
	make depend ; make ; make install
	reboot

— Editing files for PF

Edit /etc/rc.conf and add (replace with your interface names):

	cloned_interfaces="bridge0"                 # create a bridge
	ifconfig_bridge0="addm fxp0 addm fxp1 up"   # set bridge to use particular NICs
	
	ifconfig_fxp0="up"                          #  Turn the bridge NICs up
	ifconfig_fxp1="up"                          #  Thanks to Bruce A. Mah <bmah@freebsd.org>
	
	pf_enable="YES"                             # Enable PF (load module if required)
	pf_rules="/etc/pf.conf"                     # rules definition file for pf
	pf_flags=""                                 # additional flags for pfctl startup
	
	pflog_enable="YES"                          # start pflogd(8)
	pflog_logfile="/var/log/pflog"              # where pflogd should store the logfile
	pflog_flags=""                              # additional flags for pflogd startup

Edit /etc/sysctl.conf and add:

	net.link.bridge.pfil_bridge=1    # enables packet filtering on bridge
	net.link.bridge.pfil_member=1    # enables packet filtering on in and out interfaces

Edit /etc/pf.conf:

	# Macros: define common values, so they can be referenced and changed easily.
	ext_if="fxp0"	# replace with actual external interface name i.e., dc0
	int_if="fxp1"	# replace with actual internal interface name i.e., dc1
	mgt_if="fxp2"	# replace with actual internal interface name i.e., dc2
	
	# Normalization: reassemble fragments and resolve or reduce traffic ambiguities.
	scrub in all

	# Filtering: the implicit first two rules are
	#pass in all
	#pass out all

	# Loopback interface
	pass  in  quick on lo0 all
	
	# Block everything and log it
	block log on $mgt_if all
	block log on $ext_if all
	block log on $int_if all
	
	# Internal Interface
	pass  out on $mgt_if all keep state
	pass  in  on $mgt_if proto tcp from any to $mgt_if port 80 keep state

This temporary pf.conf will allow the management interface NIC to access the Internet while the bridge will be blocking everything. If I were to activate the bridge at this point (connecting the cables in between my router and switch) I would not be able to get anywhere. With this configuration enabled, I will see all the traffic that exists on the network being blocked and can determine what rules I need to setup. Even that small sampling of blocked data will probably not show everything I will need to cover, but it will surely be a good starting point.

— Working on the PF Rules

Something to consider (reposted without permission) from here:

List:       openbsd-tech
Subject:    Re: pf and statesfull filtering on a bridge
From:       Daniel Hartmeier <daniel () benzedrine ! cx>
Date:       2001-10-04 15:29:24

Several people have asked about stateful filtering with pf for bridges, and I wasn't sure myself about what's
possible until I actually tried it today :)  Assume we want to build a stealth ethernet bridge that does stateful
packet filtering using pf. First set up the bridge and verify it's working correctly:

  /etc/hostname.rl0
  up

  /etc/hostname.rl1
  up

  /etc/bridgename.bridge0
  add rl0 add rl1 up

Now enable pf with an empty rule set (passing all packets by default), and observe how everything is still working.

To better understand how pf sees the packets, we add these four rules:

  pass in  log on rl0 inet proto icmp
  pass out log on rl0 inet proto icmp
  pass in  log on rl1 inet proto icmp
  pass out log on rl1 inet proto icmp

We send an echo-request (ping) from a host on the rl0 side of the bridge to a host on the rl1 side, and an echo-reply
comes back. We see the following log entries:

  rule 0, pass in on rl0, icmp echo-request
  rule 3, pass out on rl1, icmp echo-request
  rule 2, pass in on rl1, icmp echo-reply
  rule 1, pass out on rl0, icmp echo-reply

As you can see, each packet goes through pf twice. It comes in on one interface and goes out through the other.

With pf (unlike ipf), we can actually address each of these cases, both directions on both interfaces.

If we filter statefully, there's one thing to keep in mind: the states in the state table are sorted by a key which
consists of the two address and port pairs of the connection. The order of these two pairs is relevant. If an outgoing
packet from A to B creates state, pf will let pass outgoing packets from A to B and incoming packets from B to A.
But it will still block outgoing packets from B to A and incoming packets from A to B. In a non-bridging case this
is perfectly clear and obvious.

But in our bridging case, the same packet goes through pf twice, once through each interface. And the order of the
key pairs is different in both cases.

There are two solutions. The first one is to create two states per connection, one for each direction, using two
'keep state' rules. I don't think anyone will want to do that. Especially since the second solution is very simple
and elegant:

From pf perspective, packets go through our bridge twice. If you look at either interface, you see exactly the same
traffic, only the direction is reversed. Hence, we can ignore one interface and do all the filtering on the other one.

Here's an example:

  # we want to filter on rl0, hence just pass anything on rl1.
  pass in  quick on rl1 all
  pass out quick on rl1 all

  # as an example, we block eveything by default, and let pass only
  # icmp echo requests related replies (both directions, statefully).
  block in  on rl0 all
  block out on rl0 all
  pass in  on rl0 inet proto icmp all icmp-type echoreq keep state
  pass out on rl0 inet proto icmp all icmp-type echoreq keep state

This works equally well for more complex rules. The general recipe is to pick one interface you want to filter on.
Start with two rules that let pass anything on the other interface (quick). After that, filter only on the
interface you picked.

Regards,
Daniel

— Getting FTP servers to work behind PF

According to one of the documents linked below, there is a solution for routable addresses to have FTP work active and passive via a product called ftpsesame ... It is available here: http://www.sentia.org/projects/ftpsesame/.

— PF Commands

Enable PF: pfctl -e
Flush all rules and states: pfctl -F all
Load ruleset from file: pfctl -f /etc/pf.conf
Combine the two commands: pfctl -F all -f /etc/pf.conf
Show all information: pfctl -sa
Show current ruleset: pfctl -sr
Show current ruleset with stats: pfctl -sr -v

For a complete pfctl reference, check out the man page online here: FreeBSD.org pfctl

— PF Monitoring

To monitor logfile live: tcpdump -e -tttt -i pflog0
To view log file: tcpdump -n -e -tttt -r /var/log/pflog

For a complete tcpdump reference, check out the man page online here: FreeBSD.org tcpdump

— Install pftop

pftop is like top for PF. It will show active connections and states. The current version in ports is 0.4. It doesn't seem to display packet activity of ALTQ HFSC queues so you'll have to use pfctl -vsq or symon to see HFSC queue activity.

	cd /usr/ports/sysutils/pftop
	make install clean
	rehash

— Usage Graphs

I will be looking into pfstat and symon ... I believe symon provides all that pfstat does. Symon uses RRDtool where pfstat just uses GD.

— Reference Links

The OpenBSD PF user's guide
Firewalling with OpenBSD's PF packet filter
FreeBSD Handbook on Firewalls
FreeBSD Handbook on OpenBSD PF and ALTQ
Homepage of the FreeBSD packet filter (pf) ported by Pyun YongHyeon and Max Laier derived from OpenBSD.

This work is licensed under a Creative Commons License.