A few months ago during the first stage of our 1-to-1 iPad deployment, I ran into an issue where I could connect to our wireless network just fine and send and receive traffic to my heart’s content, but sometimes when I strayed to another location within the school I would lose connectivity for several minutes. I checked out the configuration on the Wireless Access Points and it looked fine–the SSIDs were all set up appropriately, the radios were beaconing and everything looked great from the management interface. Why couldn’t I roam? We had this same style of configuration at the previous 7 schools and this issue was not present. To make things a little more confusing, the issue wasn’t 100% reproducible. Sometimes it happened, sometimes it didn’t.

Almost luckily, we hadn’t finished properly documenting and naming the access points at this facility, and before attempting to begin the troubleshooting of what I thought was a wireless issue, I was physically locating and documenting each AP via their MAC addresses by bouncing the switch interfaces they were connected to (a crude method for sure, but these APs lacked any location features, and it was after-hours). I noticed that a few of the access points had “Static” as their learning method in the MAC address table, where I expected to see “Dynamic”. I checked the configuration on the switches to make sure no static entries were actually entered, then by chance ran across some port-security settings in some of the interface configurations. After a bit of reading, I remembered that port-security settings can result in a static entry in the MAC address table–it doesn’t have to result from a manual entry typed by hand. Aha!

Unfortunately I didn’t grab the configuration before making changes to it, and this was months ago, but if memory serves here’s what the port-security configurations looked like, with my comments following the double slashes:

S1#show run
[output omitted]
interface FastEthernet0/1
switchport mode access //configure the port as an access port, meaning only allow the port to be a member of one VLAN–required for port-security settings
switchport port-security //enable port security on this interface
switchport port-security maximum 50 //allow a maximum of 50 MAC addresses to be learned on this interface
switchport port-security violation protect //if more than the maximum specified number of MAC addresses are seen on this port, don’t allow them to be added to the MAC address table but leave the interface up rather than shutting it down
switchport port-security aging static //allow the MAC addresses to age, otherwise the first 50 will stay in the MAC address table forever and no new addresses will be learned
switchport port-security aging time 5 //remove the MAC addresses from the MAC address table after 5 minutes when…
switchport port-security aging type inactivity //…no frames from that MAC address are seen on the port during that time

After conversing with the Director of Technology, it seems that these settings were an attempt to limit the number of clients associating with each access point, as they were rated from the vendor as capable of supporting up to about 50 devices at a time. Unfortunately, this implementation was a bit ill-fated. Any time a client would connect to an AP that was connected to a port with the settings above, that client would have problems when attempting to roam. The association with the next access point would go off without a hitch, but the switch itself wouldn’t allow the client’s MAC address to be learned on that new port for 5 minutes. Some ports had the configuration and some didn’t, which led to the inconsistencies.

So as it turns out, the problem that I originally thought was a wireless issue had nothing to do with wireless. It also turns out that our brand of access points (AeroHive) allows you to configure a maximum number of connections in the access point configuration itself, which I think is a much better way to go. I think this scenario presents solid evidence of the need for troubleshooting skills. I had nothing to do with entering those commands, and had no idea they were there, but nevertheless I found myself troubleshooting the issue. This also prompted a new phrase of which I need to constantly remind myself: “The wireless network is connected to the wired network!”

Hello again Internet!

You can use the “tee” utility on Linux to record an entire SSH or Telnet session. This method will allow you to capture all of the session output including commands that were typed. This is helpful for logging your own actions and configurations for later browsing or for letting a customer see what you were doing on their device.  It also allows you to parse the session output using grep, cut, sed and other tools after the session is over. Two examples follow:

ssh user@10.10.10.10 | tee logfile

This will start an SSH session to IP address 10.10.10.10 with the username “user”. The pipe is used to send standard output to the tee command, which writes the session to the file named “logfile” while simultaneously reprinting the information back to standard output, allowing it to show in your terminal.

telnet 10.10.10.10 | tee -a logfile

This will start a telnet session to IP address 10.10.10.10. The pipe is used to send standard output to the tee command, and the -a argument is used to denote that all information should be appended to the file “logfile”, rather than overwriting it. Let’s look at one more example of what we can accomplish using this command.

ssh user@10.10.10.10 | tee sshlog
…authentication…
switch#show run
…show command output…
switch#exit

linux:~ #cat sshlog | egrep ‘description|Ethernet’
interface GigabitEthernet2/1
description Trunk to IDF300
interface GigabitEthernet2/2
description Trunk to IDF301
interface GigabitEthernet2/3
description VLAN 10 Access to Toshiba Copier
interface GigabitEthernet2/4
description VLAN 50 Access to Video Server

Using what we’ve learned, we start an SSH session to the switch at 10.10.10.10 from our Linux machine and pipe the output to the tee command, saving it to the file “sshlog”. We perform a show run command, then exit. Back on our Linux machine, we cat the “sshlog” file, and use an egrep command to print only lines that have the string “description” or “Ethernet”. This gives us a very readable output with only the interface names and descriptions.

The tee utility is also great to use any time you are doing something that will generate a fair deal of output to standard out, like installations. I’m sure you will find other uses too! If you’re using Putty on Windows, check out the “Logging” section under the “Session” tab for a variety of logging options.