pfSense

Setting a virtual pfSense router at OVH.

Alejandro Olivan Alvarez

Dec 6, 2017 • 9 min read

Hi again world!
It is almost four months since my last article... way too long time... I sincerely didn't expected the return to my studies at IOC, toghether with the after-summer starting projects at work would be so overwheelming to me.... but that's exactly the case.
The good point on all that is that I'm collecting a lot of interesting IT experiences to share as articles here.

Right now, at my job, due to end-of-life / end-of-support of many technologies, I'm leading a whole migration from our current (both at OVH and at on-premises) infrastructue based on PROXMOX 3.4, pfSense 2.2 and using ELK 1.5 for monitoring, to a PROXMOX 5.x, pfSense 2.4.x and ELK 5.x new one.
Many other aspects will also change, such as the way we use VPN, our overall adoption of OSPF as routing protocol, moving to ZFS where possible, OpenVZ to LXC, server operating systems upgrading to Debian 9, new Check_MK services... the list goes on and on, so the TODO gets awesome...

Any remains of brain-energy left as I return home is quickly drained while learning, doing my practices and preparing the awaiting January exams... this means Java web / Spring MVC Back-end, JavaScript Front-end, and endless pages of theory about design guidelines, privacy laws, SEO, CDAG, and so on...

Fortunately, as Christmas time closes, preassure is relaxing, so I can get some time to transform all the notes I have being writing during this months into some technical articles someone may find useful!

So, the very first one, is one I've been doing for long time: using pfSense virtual routers, runing as KVM instances under PROXMOX hypervisors hosted at OVH.
We use them all the time with incredible results, and although we're migrating the main firewalling task to the PROXMOX integrated firewall, still pfSense is key doing inter-VLAN routing and ACL/control, VPN, dynamic routing, DHCP and so on.
But, the very first step on all that is getting your pfSense KVM instance integrated and inter connected between your virtual PROXMOX cluster world, and the Internet, which in this case, is OVH layer3 switches/routers.... so here we go!

The scenario

Our scenario involves several assumptions regarding OVH services involved in this setup. Due to the specific of the matter, it's better to put them clear first, so anyone would early identify this guide as useful... or not!

The physical server, the hypervisor, being part of a PROXMOX cluster, or a standalone PROXMOX hypervisor, is an OVH 'dedicated server'
We use 'infrastructure' type servers, which do include two NICs, one connected to the Internet, and one connected to a virtual Rack internal, private, broadcast domain (OVH vRack). Although not a requisite having those two phisical NICs, it is required that the hypervisor has at least two bridges: one facing the Internet, with the Internet NIC as port member (typically named vmbr0) and a second one, the internal one (typically called vmbr1, and with the internal NIC set as a port) where private/internal VMs are connected.
There's an extra IP address available for the server, or we can assign to it what OVH calls a 'FailOver IP'. that IP would be assigned to the pfSense router, and that would become its public IP address on its WAN facing interface.
The VM would therefore be able to route traffic from its public IP at the Internet side to the VMs bridged at the internal bridge (vmbr1), do NAT, PAT, allow VPN access, HAproxy, Snort/Suricata, and so on...

If that's your wanted scenario (or close) the following may be of your interest!

The OVH Manager part.

OK, first things first... Let's start at OVH Manager.

Guessing the local gateway address

First we need to identify (and eventually note down) the common IP address of the Internet gateway where our dedicated server is connected.
So use your OVH account to login into OVH Manager, and get to the the 'dedicated' section. There, get into the dedicated server (running PROXMOX) that will host the virtual router.
Take note of its public IP address. let's say it is 1.2.3.4
OK, OVH organizes its huge address space into a kind of /24 subnets, so, in this example, our server will be at kind of 1.2.3.0/24 subnet, where the last usable address (always a .254 for /24 subnets) is the gateway IP address responsible for that group of servers.
This would mean, following this example, that if our dedicated server address is 1.2.3.4, then there's a gateway listening at 1.2.3.254 address (I guess it is a virtual address shared by redundant layer3 switches/routers)...
Note down that address: xxx.yyy.zzz.254

OVH Virtual MAC

Next step is to get control of our spare IP address / FailOver IP address, so we can 'assign' it to a Virtual Machine.
So, let's say we do have IP address 5.6.7.8
What we need is to create a 'Virtual MAC'
By creating a virtual MAC on the OVH Manager, the single-address subnet 5.6.7.8/32 is created and propagated along all OVH routers. So any traffic going to 5.6.7.8 will end up at the gateway where your server is connected.
But two more factors have to be solved: one is that 1.2.3.4 and 5.6.7.8 are not at the same subnet, and the other is that both your dedicated server's NIC and the virtual router NIC, although bridged together, do in fact have different MAC addresses.
Again, by creating a 'Virtual MAC', a kind of static ARP entry is automatically generated at the layer3 switch where your dedicated server is connected. So, as a packet arrives at 1.2.3.254 with destination 5.6.7.8, although in a different subnet, the switch finds a suitable ARP entry and forwards the packet down the link towards your server, encapsulated within a layer2 frame which has the virtual MAC at its destination MAC.
Put anything accepting traffic for that MAC address on your server connection end, and it would get the frame!

So, still at the 'dedicated' section of the OVH Manager, look at the navigation side menu, and there's an item called 'IP'... go there!
You'll be presented with a list of all your servers and their assigned / available IP addresses, grouped by server.
Following the example, let's imagine the server with IP 1.2.3.4 has an unused 5.6.7.8 IP address.
A row for IP 5.6.7.8 should be at the list for our dedicated server, and at the right end of the row there's a gear icon, click on it, and a menu pops... then click 'create a Virtual MAC'

A new floating/pop up window opens with a form:

Asked to select on either using an already existing Virtual MAC or generating a new one: Choose generate a new one.
Asked for Virtual MAC type: select type to 'ovh'.
Asked for VM ID: write the VMID of the VM (I'm skeptikal wheter that field trully does anyting)
Then submit the form.

The VirtualMAC generation process may be slow if there's a lot of reconfiguration stress on the OVH systems, I've seing cases of up to 45 minutes... but normally, it lasts less than 10 minutes, or even less.
Once the virtual MAC is generated, it appears on the row of 5.6.7.8 address.
Note down the assigned virtual MAC for your floating IP xx:xx:xx:xx:xx:xx

The PROXMOX part.

Get into PROXMOX GUI and create suitable KVM VM for pfSense and install it onto it... Details on how to do it are beyond the scope of this article, and there's plenty of guidelines on how to do this.

Regarding the interesting points for this scenario, let's say that our KVM instance will have to have (at least) 2 NICs (In my experience, specially when dealing with VLANs, I have never ever had a single trouble using e1000 virtual NICs, while I cannot say the same for virtio devices; that said, in theory virtio interfaces would be better...).

At the first run of pfSense, a CLI wizzard will perform an early setup... To ease things, we need 2 active NICs (WAN and LAN).
If you have more, it is OK, but it is easier at the beginning to set them all to link-down state but just two: the WAN one and the LAN one... afterwards, just add/enable more as needed (Otherwise the first-run wizzard becomes awckward to use, and the objective is to get a minimal setup to get into GUI without loosing time).

Since I'm using e1000 virtual NICs, I would refer to them as em0 and em1 devices (as identified by the BSD kernel)... if you're using virtio NICs, you'll have vtnet0 and vtnet1.

The interesting one is the Internet connected on, the WAN one, to me it is usually the em0 (vtnet0 for virtio) NIC of the VM.
It MUST be bridged to the Internet-facing bridge (vmbr0)... DO NOT enable PROXMOX firewall on the interface (pfSense is a firewall itself!).
Now comes the magic: replace the PROXMOX random-generated MAC address by the OVH Manager generated Virtual MAC address!

Now the PROXMOX part is over... open a console for the virtual router on the PROXMOX GUI and turn the VM on, so it boots pfSense for the first time after pfSense Installation.

pfSense first boot at PROXMOX console

OK, now you just booted pfSense, and a first run wizzard appears to configure the router, deal with it as needed, but when it comes to IP addressing, which is the interesting part, here are the steps that have to be followed.
If for some reason the wizzard is already run, you need to reconfigure an existing setup or whatever, on the CLI menu, select the option to reconfigure IP addresses, and it is almost the same.
Here we go:

What do we really need, is to properly configure the WAN address. afterwards, from the pfSense GUI, we may reconfigure LAN or anything else.
So, ensure that pfSense detects em0 (vtnet0) as WAN, DO NOT use dhcp to obtain IP address on this interface, and when asked to enter the IP address, enter the OVH floating IP address (in this example 5.6.7.8)
Then, when asked for the address mask, choose the highest possible, /31. The correct values should be /32, but its not available on the CLI wizzard. /31 'just works' enough and the correct mask will be set afterwards.
Now, if you're at the wizzard, answer the remaining questions as needed, and upon getting to the CLI menu, select option 8 to enter the console.
This is time to get an early, temporary internet connection on the router, that would allow us to connect first time into the GUI. So, execute the following two commands (change values of the example with yours!)
```
  route add -net 1.2.3.254/32 -iface em0
  route add default 1.2.3.254
```

OK, now YOU SHOULD have gained internet access!!! ping some host to test it. If you have not Internet connection, then something is wrong, either PROXMOX side, or OVH side, or both... if properly done I can assure it works!
Now on, the remainig setup has to be done on the pfSense GUI, but, since this early steps are done from loging into GUI from the WAN address, which is firewalled by default, you'll need to keep this console opened and disable firewall everytime a config save reactivates it (and gets you out the GUI)... the command is:
```
  pfctl -d
```

Finishing setup at pfSense GUI

So, disable firewall at the CLI, and login into pfsense using its WAN IP address (following the example http://5.6.7.8).
Default username is admin and default password is pfsense.

WARN: Don't waste a minute while having the router at this state... it is very vulnerable now: firewall down, and default login credentials... it is imperative that you end up the remaining connectivity config ASAP, and just afterwards inmediatelly change password, and setup a minimal set of firewall rules so it can be left at normal firewall up state.

So Upon login, Skip the GUI first login wizzard by just navigate clicking System -> Routing -> Gateways: and click add new gateway:

On the config page that opens for the new gateway, go to the bottom, look, find and click 'Show advanced'.
Check the last option 'Use non-local gateway through interface specific route.'
Now fill the IP address data of the OVH gateway (in this example 1.2.3.254)
Check the option 'This will select the above gateway as the default gateway.'
Fill the fields 'Name' and 'Description' accordingly...
Save and Apply
The firewall will reactivate... disable it again on the console!

Now let's correct the IP address: navigate to Interfaces -> WAN

Adjust the IP address (in this example would have been 5.6.7.8)
Set the correct subnet mask to /32
Now select the gateway to monitor to the one we have just created.
Save/Apply
Disable firewall as needed from the CLI

Now, go to firewall -> NAT, and check in 'Outbound NAT' tab, that Automatic rules for outbound NAT are generated! if so... network setup is OK... Secure the router now!

Go to system -> Users and change/create credentials...Now you're quite more safe, but you'll still be again and again locked as firewall reactivates, just disable it for a last time...
Go to Firewall -> rules and at the WAN tab, add necessary exceptions so you can access the GUI from WAN at least temporarely (allow just acces from your current IP, for example) until you get VPN or whatever, then, save/Apply.

Now, you can close the PROXMOX CLI console, and you can keep setting your router....so, well... enjoy your virtual router!!!!