Photo By: Peter H Source: Pixabay
Panic in the morning
About two weeks ago I woke to the wonderful news that my pfSense instance that ran on an old core 2 duo decided to finally kick the bucket. The biggest issue was that both my kids needed to get online for school (thanks COVID). Thankfully I had a Ubquiti Security Gateway (USG3) so I grabbed it off the shelf, moved the cables from my pfSense box to the USG3, and powered it on.In this post my goal is to do the following:
- Outline my network setup before the switch,
- Steps taken to get the USG3 working and my network back up and running and how to do it
- Setting up services like DDNS, VPN, DynDNS, etc
- Creating a zero-trust network
As a warning this is a long post. If you want to skip the setup and get to my over all thoughts then click here.
My Network before the switch
I had the following hardware before I switched to the USG3.- Ubquiti CloudKey gen1
- Ubquiti UAC-Lite x 2
- Ubquiti Switch x 2
- HP Pro Curve Switch (manage)
- pfSense running on a custom built desktop
Network Overview
I have simple network, at least to me. My hone network is segmented into 6 networks, trusted client, iot, kids, lab, VPN, and guest. The client, iot, and kids network are also configured as a WLAN and no I do not hide the SSIDs. Each of these networks are also tagged VLANs.Each L3 network is of course a RFC 1918 compliant network. Yay. Each of these networks used the pfSense box as their gateway. In order to get my Wi-Fi clients running I setup each network as a vlan only network in the UniFi Network controller. These networks were then used in the WLAN configuration. Despite how disjointed it sounds I have to say that Ubquiti did a great job with making a decent configuration management tool that is actually pretty flexible.
pfSense:
I used a couple of different packages for pfSense, and since my pfSense box is down I will not be going to into details on how I had each item configured. You know, if I had only documented my configuration I could have been able to do that. Or linked to my GitHub repo.IPS: SNORT
Threat Feeds: pfBlockerNg using feeds from from Binary Defense and Steve Black.
AD Blocking: Pi-Hole ADBLs
VPN: OpenVPN
DNS/DHCP: unbound w/DNSSEC, dynamic host name updating
The Rapid Transition to USG3
- Create a new network configuration
- Update your Wi-Fi configuration to use that new network.
- Enable dynamic DNS for internal clients
- Create static IP assignments
- Setup Dynamic DNS with Google Synthetic Records
- Setup VPN (RADIUS, user accounts, etc)
- Setup Threat Management (if you can)
- Create a zero-trust network, as much as you can
I have to say this was by far a lot easier than I expected switch thanks to Ubiquiti's UniFi Network controller.
Creating networks!
You must create a new network since you cannot change a VLAN only network configuration to a layer 3 configuration or as Ubquiti calls it "Corporate". While annoying this does make sense and is pretty common for vendors to do.If you, like me, have several switches with VLANs already configured on the ports then I suggest you modify your networks one at a time starting with the least important network first. For example, if your VLAN only IoT network is VLAN 3 and called IoT then change it to VLAN 1313 and change the name iot-vlan-only and then create the new layer 3 network using VLAN 3 with the name iot. Honestly, I would just call it iot-l3. This way you do not have to modify your switch configuration.
You could also just assign a new VLAN ID to the new network and name then <net>-l3 and then swap the VLAN IDs after all networks have been created. But then you would have to modify your switch ports and honestly, who wants more work?
Unicorn Injection: Do not do what I did and delete your old VLAN only networks without at least documenting their setups. Thankfully, I saved one of them, so I know what my configuration looked like for when I move off of the USG3.
So lets create some networks:
| Classic Interface | ||||
|---|---|---|---|---|
|
| New Interface (beta) | ||||
|---|---|---|---|---|
|
Updating Wi-Fi for new networks
Now that the networks have been created the Wi-Fi networks need to have their settings updated and configured to use the new networks. This where taking an object orientated approach to settings really comes in handy. It is annoying as hell to script for (I am looking at you Palo, yes you and your damn XML configuration and where you put settings) but the modularity of it makes management freaking easy. The USG does a little better with than pfSense, but only marginally.
| Classic Interface |
|---|
|
| New Interface (beta) |
|---|
|
That is it. The configuration will get pushed to the APs causing them to take all the Wi-Fi networks down for a few minutes. Once everything is back up you should be able to log into the Wi-Fi network and get to the outside world.
Keep in mind that some clients might need to be rebooted after the change. I found that it was mostly my IoT devices that did not like the changes and I had to go around rebooting them.
So now that our network is back up and running we just need to setup few addition things to make it complete.
Static IPs
If you run a separate DHCP server then you can skip this part.
| Known Client |
|---|
|
| Unknown Client |
|---|
|
Either bring the device online, reboot it, or refresh the network configuration to have it get its static IP.
Enabling Dynamic DNS for local clients
I found that this setting was automagicly enabled and did not need to do anything but if you already have a DNS server running on your network then you will want to disable this.
Since my USG3 is also my DHCP server I left it enabled. As for how it works, it does but not for all devices. Strangely my CloudKey does not show up with the right name and neither does my USG3. For example, my CloudKey is called Klefki (yes, as in the Pokémon) and I cannot ping the name and I cannot access the web GUI using the FQDN. I have to use the IP address to access the device. My USG3 still has the stupid default name even though I changed it.
Something to note that if you are using the new interface the settings for this were not available. As in they do not exist. I searched and searched but could not find the settings. Either Ubiquiti just made it enabled by default that can only be changed by manually modifying the configuration by hand...or they forgot about it. I have not combed through the configuration file to determine which is the case.
While DDNS is fun and great and helps make the devices in the Clients page easier to determine who they are, devices like Amazon Alexa will show up with their name as "amazon-<mac>". To get around this I just use the alias feature to rename the clients in the GUI. This does not override what is registered in DNS just what you see in the UI.
pfSense just got things right with updating the records for the clients. The one annoying thing about pfSense was that tended to keep records for old clients for a long time but the USG went the other extreme end where clients that have not been active on the network in the past hour are removed from the client list. While I still prefer pfSense's approach I wish they had an alias feature so I could easily identify my stupid Alexa devices more easily without having to give them static IPs.
Setup Dynamic DNS with Google Synthetic Records
I use this to have the USG3 send Google my public IP and update a DNS record for a custom domain that is hosted by Google. This allows me to connect to my VPN from anywhere in the world, provided I am not in Russia, China, Brazil, North Korea, or Texas (kidding about Texas) as I block these countries.I first had a mild bout of a temper tantrum when I went to configure this and did not see an option for Google. However, after searching I found that that Google supports DynDNS v2 which is also used by DynDNS, so I used the DynDNS option.
Before you start make sure you have a synthetic record and you have access to the username and password.
Unicorn Injection: If you make a mistake with the password and find out after you click Save you will not be able to fix the password and be able to attempt to save again because it will fail. I am guessing it is a bug but if this happens to you just need click cancel and fill everything out again.
| Classic Interface | ||
|---|---|---|
You should have something that looks like this (minus the blocked out text):
|
| New Interface (beta) | ||
|---|---|---|
You should have something that looks like this (minus the blocked out text):
|
RADIUS, VPN, and Users OH MY!
Setting up the VPN configuration is just as simple as most of the other services and there is a default RADIUS profile that already exists so there is no need to do anything to get that working other than adding a user. It is possible to setup a second VPN, however it is not possible to have two L2TP VPNs and you will be limited to a PPTP server.Before we setup the VPN we will want to make sure that RADIUS is enabled.
Verifying RADIUS is setup
| Classic Interface | ||
|---|---|---|
|
| New Interface (beta) | ||||
|---|---|---|---|---|
|
Setup VPN Users
While we are on the RADIUS setup pages we should add a user or two. In the directions below I configured a VLAN for the users and used the 3rd octet from the VPN client IP range for the VLAN ID. Ubquiti's directions have the VLAN ID field left blank. From browsing the community forum I found a thread asking how to block traffic from the VPN. Turns out that you can browse any network from the VPN network by default and most solutions I found to stop this required the users to be assigned a static IP and that IP blocked via a firewall rule.
From messing around with this I found that you can use that it is possible to block traffic to and from the VPN network. All you need to do is create a group with the VPN IP address range and then use that group in your firewall rules.
| Classic Interface |
|---|
|
| New Interface (beta) |
|---|
|
VPN Setup
Now that we made sure that the RADIUS server was enabled and setup and we created a user or two we now need to create the VPN network for users to actually use.The only difference with this network is that we will not configure the VPN with a VLAN as that was setup with user accounts.
| Classic Interface | ||
|---|---|---|
Unicorn Injection: Please note that the image below does say Edit Network, this is because I already had my VPN setup before I thought about writing this post and I did not want to delete my setup.
|
| New Interface (beta) | ||
|---|---|---|
|
Setting up clients
Due to the number of different clients that can use this VPN I am not going to cover how to setup each client. Just make sure you select the L2TP/IPSec option when setting up your client.
I am sorry but did you say L2TP?
Yup. I had the same reaction when I was setting this up because when I see the acronym of L2TP encryption is not the first thing that comes to mind. Yes, the control messages that it uses is encrypted but beyond that the actual data that is in flight is not encrypted. Trust me, I double checked myself on this because I was hoping that I was wrong. But I was not.
This is where IPSec typically comes in to save the day and provide all that lovely confidentially that we like. For Windows and OS X when you setup the client you select L2TP/IPSec but I am not entirely sure if that is the case for all clients.
Until I can prove that the data is encrypted I am not going to use this unless I absolutely have to.
I am just going to say that pfSense outshines the USG in every category for the VPN, user management, and authentication.
Setup Threat Management
At the time of this writing Threat Management on the USG3 is so severely flawed that it would wet the bed and take the network down when enabled. There are only two ways to fix it: 1) unplug the device and plug it back in or 2) wait and see if comes back up. I was out of town the 2nd time it went pissed itself and it took 20 hours for the network to recover, but thankfully it went back down an hour later. This is not just a Sean issue but an USG3 and UDM (Ubiquiti Dream Machine) issue. When I started writing this post there was a bug tracker that Ubiquiti developers opened for this issue with a time stamp of two months before I started writing this. Some people have found that disabling DPI resolves this issue but then you loose a lot of data about your network, and it did not seem to work with me.
The issue: no one really knows and Ubiquiti has been silent about this since they created the bug tracker. All we know is that with this feature enabled your USG3 or UDM tends to lose its mind forgets to pass network packets.
So what are you suppose to be able to do with the USG and threat management? Great question. In theory you should be able to:
- Protect your network from Virus & Malware (i.e. BOTCC, shell code, etc.)
- Use a built in IDS/IPS
- block malicious IP addresses
- Use IP reputation
- block traffic based on geolocation
- Block traffic with a firewall
Thankfully you can always use the firewall and there are no limitations making you choose the firewall or something else.
I am listing the firewall under Threat Management because in the new UI Ubiquiti has moved it from its own section in classic interface to a section under threat management. Honestly, I can understand why they thought this was a good idea; firewalls "stop" threats. Regardless, I do not believe it should still be placed under threat management and it should maintain it's dedicated section away from threat management.
Speaking of the firewall
When comparing the firewall management provided with the EdgeOS running on the USG3 with pfSense there is a lot to be desired. I mean a lot. For starters it defines the term basic to the point of being frustratingly limited. There is also a default ANY ANY allow for traffic between corporate networks.
When setting up firewall rules you will at first think that you have a whole lot of options but quickly realize that it is all an illusion. Again, more later. Also keep in mind that you are not creating the firewall rules on the device directly but are creating them in the UniFi network configuration software.
So what can you do, well, you can:
- Give it a name (classic interface only)
- Give it a nice description (new interface)
- Accept, drop, or reject traffic
- Specify a source and destination but not a single IP for a single port
- Permit/block traffic from/to a defined network, except the VPN
- Permit traffic to a single IP for ANY port
- Target the rule for specific IPv4 protocols
The advanced options are not exactly advanced as they are checking if a packet matches a state for a session. For example, you can have the rule match based on the state of the packet being new or invalid. Invalid is neat but really outside of IPTABLES this is not a common setting that you can set with other vendors and from my experience seeing an invalid state normally indicated an issue. I guess if you have an network that is so horribly unstable or your clients are using jacked up network stacks (stop using Windows 3.1 for Workgroups) and would rather just toggle a switch and not fix the problem, then yeah, I can see the need.
What I do think is cool is the option to match existing sessions and not having to wait for the session to end and then be recreated. To do this you just toggle the setting Match State Established.
You can also create the rule to match sessions for new packets for an existing session by toggling Match State Related. Which is also pretty cool, but still not what I would consider an advanced setting.
Unicorn Injection: Ubiquiti is not the only firewall to match states like related or established. This has been part of iptables: --match state --state RELATED,ESTABLISHED. It would not surprise me if the EdgeOS was using iptables.
Also, if you want to block traffic between networks, excuse me, corporate networks, you will need to select LAN In or LAN Out. Even if you have several networks they will all be created with the type LAN In, LAN Out, LAN Local under the LAN section. Almost as if they are really just one giant flat network. This is very different from how pfSense organized how rules were displayed with each network having their own tab.
As mentioned above, there is a default deny but only for traffic that ingresses from the public. Traffic can travel between corporate networks including the user VPN networks freely. By default. This means that even though you have different networks that are segmented this is not a zero-trust configuration. To get to that point you need to create your own rules to drop packets on the floor.
Let's go over an example of blocking the devices on my IoT network from being able to communicate with my trusted client network.
Let The Packets Hit The Floor
In this example we are going to protect our trusted networks from our IoT devices. I am sticking to the new interface for the example.
- Click on Settings
- Click on Security (red shield)
- Click Internet Threat Management
- Expand the section titled Firewall
- Click on Create New Rule
- In the Create New Rule page set the following:
- Type: LAN In
- Description: <enter custom name>
- Enabled: Leave toggled on
- Rule Applied: Before
- Action: Drop
- Expand source and set the following options:
- Source Type: Network
- Network: <source network traffic is coming from, iot for me>
- Network Type: IPv4 Subnet
- Mac Address: <leave blank>
- Expand destination:
- Destination Type: Network
- Network: <destination network, for me it was client>
- Network Type: IPv4 Subnet
- Expand Advanced
- Enable any of the settings you want, I like to log so I enabled it
- Click Apply Changes
That is it. You now have a firewall rule blocking traffic from one network going to another network. Now you just have to do this for each network you want to block traffic for since you cannot add more than one item per rule. Or can you?
The image below is my setup:
 |
| All the settings to block traffic |
For a second example I am going to go use my Kid's network which, like the IoT network, is also blocked from talking to my trusted client network. Why? Because kids are like sys-admins who use servers to check their email and browse the game. The issue with this is that the USG does not allow you to select more than one network when creating a rule for either source or destination. Instead of creating multiple rules you just need to create a group for the untrusted networks and then use that group in the rule. You can create the group before the rule or after the rule.
 |
| Blocking traffic from more than one network |
While this is great this does cause a problem for my kids. This setup prevents them from using my network printer to print from their devices. To allow this I have a rule that sits above the rule that blocks all of their traffic that permit traffic from the kids network to just the printer on 9500/tcp. Here is what that rule looks like:
 |
| Allow only printing from kids network |
Unicorn Injection: After creating this rule I found that clients on the kids/IoT network could not get to my clients on my 192.168.1.1/24 network they could still get to the gateway address of 192.168.1.1/24 via a browser. So....yeah.
Here is what my rough rule set looks like right after the transition:
 |
| Rule list after setup |
Unicorn Injection: Spelling is not my forte.
What is this In, Out, Local stuff?
Each type of rule is listed as <type> In, <type> Out, or <type> Local. Each of these affects how a rule is processed so make sure you chose the right one. If you are trying to block traffic from one network to another network and choose Local then your rule is wrong. From Ubiquiti's website they explain what each of these are:
- Local: The USG itself
- In: Rules for traffic that will ingress a network
- Out: Rules for traffic that will egress a network
In terms of when you would use In over Out or Out over In is enterally dependent on what you are trying to achieve. If you want to stop traffic from coming into a network or permit traffic to come into a network you would want to chose In. If you want to stop traffic from going out or you want to permit traffic out of a network you would use Out.
If you are use to IPTABLES then this is no different then INPUT or OUTPUT.
Unicorn Injection: When might you want to use LOCAL? Whenever you want to block traffic being able to get to the USG itself, at least theoretically. Be warned, there are redefined rules for LAN IN that will trump any LAN LOCAL rules you create. I tried to block my pesky 7 year-old daughter from hitting the management interface...guess what she can do from the kids network. Access the USG's management web interface.
Final thoughts on this switch
 |
| Mr. Chan from here ^ |
If you want a device that provides basic firewall protection, a built in VPN solution, built in RADIUS auth and user management, and nothing else then this is the system for you. I have found over the past 3 weeks that it does those things very well. The a small form factor is a plus as I can mount it on the wall right next to my lead switch in my data closet. The lack of a fan means the USG is stupid quiet and surprisingly without the trade off of it running incredibly hot. The ability to manage the device from a separate tool has its ups and downs but as expected it integrates flawlessly with Ubiquiti's other network technology. Firewall management is simple, for me it just feels like a GUI wrapper for IPTABLES but that might be what some are looking for.
Their VPN is nice and has worked without fail for my Windows system and for my iPhone. I wish there was an option for TLS. Yes I know, it is flawed, but provided they are not implementing a flawed version of TLS then why not? L2TP has known issues itself.
I really like how I can very easily add networks, the simplify of it really does encourage the use of multi-networks. Even for those that are not into networking or fully understand it. It just works.
Now, if you are like me and want to segment the crap out of your network, setting up IDS/IPS to better protect your network, have logs viewable in a GUI for quick troubleshooting and have it all wrapped up in something easy to manage, then this is not the device for you. Why?
The reason this might not be the device for you is because there are so many things that are just wrong with the USG.
There are two areas that I have a lot of issues with and find myself asking "what" and "but why" more often than not. These are: Firewall and Threat Management. I would like to say documentation but that is pretty much an issue with all of their devices.
Firewall
I am baffled at how Ubiquiti can continue to market this device as a router with "advanced security." Nothing about this device is advanced not their firewall and definitely not their threat management. Their data sheet even claims that their firewall has advanced features but I am not seeing it and I do not consider a setting that allows a rule to match a state as being advanced. IPTABLES has had that ability for a long time and yes it might be a new concept for some it is not an advanced function.
Yes I know I hear some of you saying, "Sean you are not protecting a data center" and to that I say, well in a way I am. I am protecting not just my privacy but my family's. So in a way, it is a data center.
Three are three issues I found with the firewall that really frustrated me.
The first issue is with the firewall rules, starting with the fact that I cannot specify a port when creating a rule that uses an IP address. Maybe I do not want to have an IP address accessible on all 65,535 ports so why do I only have the option of ANY? Is there ever a need for ANY for ports? Yes, but not for every case. One of the older security practices for firewalls is to avoid using ANY. Give me the ability to limit on ports. You let me create rules with network groups and port groups so why not allow the use of a port when creating a rule for a single host.
The second is with the firewall's the lack of logs in the GUI. If you want to see the traffic that is getting blocked or permitted you have to drop to the shell on the device and tail the logs. Or use grep to find past traffic. Maybe I am in the wrong year, *looks at the calendar*, nope, my calendar says 2020 not 1995. Yes it is possible to export the logs via syslog and from there import them into Splunk, or an ELK stack and the community has done a fantastic job with providing each other queries to use for dashboards and searches (see here). But that should be an option, not the only way to get the logs for your firewall. My old TrendNet firewall had this and then my old Belkin Wi-Fi router even had a place in their GUI that I could view logs (and filter! I felt l337). If the issue is with the use of needing their network controller then I am still not sure why this would prevent them forward the logs from the USG to the network controller running on a PC, MAC, or Linux box or even the CloudKey. They clearly have the ability to export. The saddest part of this is that there has been a request submitted by users for this feature. Worst yet Ubiquiti even acknowledged the request and said that they are working on it. The request was submitted 4 years ago. Not 4 months but 4 years. In that time they have completely redesigned their GUI twice, added additional features, and bells and whistles.
The last issue I have is the way rules are displayed in their new interface. The important details are missing, such as source, destination, ports, etc. I can get over how the rules are organized but not letting me view the important details is a real pain. It was there in the classic interface but it is now gone.
Oh, I know I said 3 but not being able to block traffic to the management interfere is just stupid. Seriously. WTF.
Threat Management
I am going to be honest, threat management on the USG3 is a complete joke. It is not even in the same galaxy let alone the same solar system as pfSense.
For starters if you enable IDS/IPS it disables hardware offloading and on top of that you are able to use GEO blocking. Now this affects the USG only, at least at the time of this writing, but still. It does not take much processing to check what region an IP is registered in. I like to block egress and ingress traffic from counties in the event my kids bring home an infected device.
The most grievous of the issues I have with the USG is the fact that I cannot even use the threat management features without bringing my network down. is the lack of a patch to correct issue with threat management causing the device to crash. The lack of information from Ubiquiti just makes this issue even worse. If you are going to make a market a product as being focused on security then you should make sure that your priority is on getting patches released to address issues that break security features. Further more, you should at least tell people you are working on it and actively developing a solution. Some have reported that disabling DPI can help while others have reported that it did not help. This remains an issue even with the latest release of 4.4.51.
In Closing (about time)
While I find a few things that I do like about the USG I do not plan on keeping this as my edge device. It does not provide the features or stability I need or want, let alone are updates provided in a timely manner to resolve major issues for all USG customers. For me, the USG is overall a $130 piece of junk.
Do not get me wrong, I think Ubiquiti provide a lot of good equipment and I plan on continuing to be a customer for now. I just believe they missed the mark on this device and are continuing to fail with communicating with their customers.