How LAN problems can impact on your firewalls
Last week seems to have been a firewall issue week for me. I worked on incidents in two large networks where problems on the LAN managed to bring down firewalls at the edge of the network. Firewalls have come a long way since the original stateless varieties, which were little more than a collection of access control lists. These were superseded by stateful firewalls which kept track of the state of network connections. Modern firewalls are now application aware, you can do things like block access to certain applications or prevent users from using applications like Bittorrent.
However, these firewalls don’t come without their problems. They require lots of computing power to analyze and filter applications as they pass through. On some networks it can be a challenge getting the right balance of application awareness and traffic throughput. The more features you enable, the more stressed your firewall becomes. A lot of firewalls are priced on throughput: the greater the throughput, the greater the price. As a result of this, I see some networks with firewalls that are just about coping due to the costs involved of upgrading to the next model.
One incident I worked on last week was with an Internet service provider (ISP). A couple of their corporate firewalls, which linked their staff networks to the Internet, were resetting themselves. They suspected a resource issue on the firewalls and installed extra memory, but this seemed to make the problem worse. We discovered the source of the problem by looking at the network traffic going from the network core to the firewalls.
We found that large volumes of syslog traffic was being routed to an external IP address. This was unusual as syslog is normally used for managing local networks. We looked at the external IP addresses and saw that they were from an IP range that was once owned by the ISP. They sold it off and updated routing tables. However, nobody checked if any remaining systems were exporting data to this old subnet. Once routing tables were updated the syslog traffic was now sent out via their firewall. This caused an overload and the firewalls reset themselves.
It’s an interesting problem caused by the lack of IPV4 address space. A block of addresses is very valuable. For example, earlier this year Microsoft paid $7.5 million to purchase a block of 666,624 IPv4 addresses. If you are looking to change IP addressing on your network, make sure you have monitoring in place so that you can identify what is using the IP addresses and for what reason.
The second incident I looked at involved a large university in Scotland. The connection rate limit on their main firewalls was peaking. This resulted in complaints from network users that connections were dropping or that downloading and uploading was slow.
Networks within educational campuses are typically more open than they would be in a corporate environment. This means that even things like what’s on the TV schedules can affect network performance. Applications like Bittorrent can generate large numbers of connections as it downloads small pieces of data from lots of other clients on the Bittorrent network. In the case of this university network, users on their WiFi networks were tunnelling Bittorrent over TCP port 80. This resulted in large amounts of traffic and connections. The firewall rules were updated to only allow 80 parallel connections per IP address and the firewalls stabilized.
I do like the way application aware firewalls are evolving, more awareness of traffic coupled with clever management interfaces. However, no matter what firewall you choose you should be able to see what is going in and out of your network perimeter. Try and do this independently of the firewall, no point in trying to switch on logging or extra diagnostics if it’s already stressed out. Knowing what is going in and out of your network can also help prevent network breaches.
Comments are closed.