Cisco Port-Security Settings and Wireless Access Points

A few months ago during the first stage of our 1-to-1 iPad deployment, I ran into an issue where I could connect to our wireless network just fine and send and receive traffic to my heart’s content, but sometimes when I strayed to another location within the school I would lose connectivity for several minutes. I checked out the configuration on the Wireless Access Points and it looked fine–the SSIDs were all set up appropriately, the radios were beaconing and everything looked great from the management interface. Why couldn’t I roam? We had this same style of configuration at the previous 7 schools and this issue was not present. To make things a little more confusing, the issue wasn’t 100% reproducible. Sometimes it happened, sometimes it didn’t.

Almost luckily, we hadn’t finished properly documenting and naming the access points at this facility, and before attempting to begin the troubleshooting of what I thought was a wireless issue, I was physically locating and documenting each AP via their MAC addresses by bouncing the switch interfaces they were connected to (a crude method for sure, but these APs lacked any location features, and it was after-hours). I noticed that a few of the access points had “Static” as their learning method in the MAC address table, where I expected to see “Dynamic”. I checked the configuration on the switches to make sure no static entries were actually entered, then by chance ran across some port-security settings in some of the interface configurations. After a bit of reading, I remembered that port-security settings can result in a static entry in the MAC address table–it doesn’t have to result from a manual entry typed by hand. Aha!

Unfortunately I didn’t grab the configuration before making changes to it, and this was months ago, but if memory serves here’s what the port-security configurations looked like, with my comments following the double slashes:

S1#show run
[output omitted]
interface FastEthernet0/1
switchport mode access //configure the port as an access port, meaning only allow the port to be a member of one VLAN–required for port-security settings
switchport port-security //enable port security on this interface
switchport port-security maximum 50 //allow a maximum of 50 MAC addresses to be learned on this interface
switchport port-security violation protect //if more than the maximum specified number of MAC addresses are seen on this port, don’t allow them to be added to the MAC address table but leave the interface up rather than shutting it down
switchport port-security aging static //allow the MAC addresses to age, otherwise the first 50 will stay in the MAC address table forever and no new addresses will be learned
switchport port-security aging time 5 //remove the MAC addresses from the MAC address table after 5 minutes when…
switchport port-security aging type inactivity //…no frames from that MAC address are seen on the port during that time

After conversing with the Director of Technology, it seems that these settings were an attempt to limit the number of clients associating with each access point, as they were rated from the vendor as capable of supporting up to about 50 devices at a time. Unfortunately, this implementation was a bit ill-fated. Any time a client would connect to an AP that was connected to a port with the settings above, that client would have problems when attempting to roam. The association with the next access point would go off without a hitch, but the switch itself wouldn’t allow the client’s MAC address to be learned on that new port for 5 minutes. Some ports had the configuration and some didn’t, which led to the inconsistencies.

So as it turns out, the problem that I originally thought was a wireless issue had nothing to do with wireless. It also turns out that our brand of access points (AeroHive) allows you to configure a maximum number of connections in the access point configuration itself, which I think is a much better way to go. I think this scenario presents solid evidence of the need for troubleshooting skills. I had nothing to do with entering those commands, and had no idea they were there, but nevertheless I found myself troubleshooting the issue. This also prompted a new phrase of which I need to constantly remind myself: “The wireless network is connected to the wired network!”

How To Fix A Missing Or Blank App Icon On An iPad

I use Casper to manage about 5,000 iPads in a K-12 environment. A few weeks ago, I started noticing an issue where some shortcuts I pushed would show the Apple icon development grid instead of the appropriate icon. If you’re not a graphic artist or App developer (I’m not), you have probably never seen this graphic before. This is what it looks like:


I had one user describe it to me as a spider web and another user describe it as a bull’s eye, whereas I referred to it as a blank or default icon. It’s used to provide a system to help align the elements of an App icon from a design perspective. Technically the shortcut was still functional, it just wasn’t very aesthetic. But who cares–we want it gone from our iPad, right?

I found that removing the icons and pushing them back down via Casper solved the problem, but didn’t feel like this was appropriate to do for every iPad considering the issue was very sparsely presented.

Another fix is to force the icons to refresh on each iPad where the problem is present. This is an immediate fix that the user can do on their side and the steps are very simple. From the home screen, choose Settings > Display & Brightness, then toggle the “Bold Text” option to its alternate setting and restart the iPad when prompted. This should force the icons to update and assuming the correct artwork got stored in the right place, the afflicted icon should now appear correctly on the home screen. You should be able to toggle the “Bold Text” option back to its previous setting now. Good luck!

How To Record an SSH or Telnet Session to a Text File

Hello again Internet!

You can use the “tee” utility on Linux to record an entire SSH or Telnet session. This method will allow you to capture all of the session output including commands that were typed. This is helpful for logging your own actions and configurations for later browsing or for letting a customer see what you were doing on their device.  It also allows you to parse the session output using grep, cut, sed and other tools after the session is over. Two examples follow:

ssh user@ | tee logfile

This will start an SSH session to IP address with the username “user”. The pipe is used to send standard output to the tee command, which writes the session to the file named “logfile” while simultaneously reprinting the information back to standard output, allowing it to show in your terminal.

telnet | tee -a logfile

This will start a telnet session to IP address The pipe is used to send standard output to the tee command, and the -a argument is used to denote that all information should be appended to the file “logfile”, rather than overwriting it. Let’s look at one more example of what we can accomplish using this command.

ssh user@ | tee sshlog
switch#show run
…show command output…

linux:~ #cat sshlog | egrep ‘description|Ethernet’
interface GigabitEthernet2/1
description Trunk to IDF300
interface GigabitEthernet2/2
description Trunk to IDF301
interface GigabitEthernet2/3
description VLAN 10 Access to Toshiba Copier
interface GigabitEthernet2/4
description VLAN 50 Access to Video Server

Using what we’ve learned, we start an SSH session to the switch at from our Linux machine and pipe the output to the tee command, saving it to the file “sshlog”. We perform a show run command, then exit. Back on our Linux machine, we cat the “sshlog” file, and use an egrep command to print only lines that have the string “description” or “Ethernet”. This gives us a very readable output with only the interface names and descriptions.

The tee utility is also great to use any time you are doing something that will generate a fair deal of output to standard out, like installations. I’m sure you will find other uses too! If you’re using Putty on Windows, check out the “Logging” section under the “Session” tab for a variety of logging options.


How To Use Wireshark To View tcpdump Captures

Hello again world! This week I am relaxing with my family in the beautiful Blue Ridge Mountains of North Carolina! However, I still found time to play with tcpdump a little bit and wanted to share how to open tcpdump captures using Wireshark.

tcpdump is a Linux/Unix command line utility that allows you to capture network packets, similar to Wireshark’s command line capture utility named tshark. Running tcpdump without any parameters will simply cause it to print out a basic description of the captured packets using one line per packet in the current shell. Redirecting the output of the command to a file isn’t very useful since it’s not really packet data that’s being captured, just the descriptions. If we want to capture the entire contents of each packet for later analysis, we will need to use the -w parameter and supply a filename. In our example, we’ll go ahead and append a file extension of “.pcap” to the filename so that Wireshark will recognize the filetype in our GUI, whether we are opening the file on the current system or transferring it to a Windows system. And depending on our OS, we might need to run tcpdump as root so we’ll use sudo:

sudo tcpdump -w capture.pcap

This will cause tcpdump to write all captured packets to capture.pcap in the current working directory. If you don’t specify any limiting factors like a maximum packet capture count or a maximum file size, the capture session will run until you press Ctrl+C. After tcpdump is stopped, you can view the trace file using Wireshark on a system of your choice. Best of luck!

Inconsistent Timestamps Over Long Wireshark Captures

I’ve recently been running Wireshark around the clock on a couple of servers at work to troubleshoot a problem that pops up whenever it feels inclined. One late night while perusing trace files with sleepy eyes, I happened to realize that packets were arriving at server B two minutes and forty seconds after they were sent from server A, according to the timestamps on the packets. I thought that was very odd, of course. It seemed rather impossible that packets were taking almost three minutes to reach their destination on our high capacity network–and where were the errors? I checked the system time on both servers, and they were both correct and within a second or two of each other. I restarted both captures and double checked the system times and the timestamps on the packet and they were correct at that time. A few days later, I see the same issue arise. Then I notice that the current packets that are coming in are stamped a few minutes behind the system time. What the heck?

Well it turns out that Wireshark’s reliance on the WinPcap library in Windows causes this issue. When a capture is started, WinPcap checks the system time for a reference point and then tracks the time on its own by referencing the KeQueryPerformanceCounter routine (that’s processor mumbo-jumbo to me, maybe you are more well-versed). It looks like there are two options to handling this problem, and number one is to deal with the issue with an understanding that the time is going to lag more and more behind the system time the longer the capture runs. You might be able to alleviate the problem by restarting the capture session frequently, although you will probably lose packets during that process. You can use Wireshark’s timestamp offsetting feature to help get things on the same track if you are referencing two trace files taken at the same time, but there might still be some discrepancies.

Option two is to change a registry key setting in Windows which will force WinPcap to use the system time for its packet timestamps. The drawback with this option is that it is less precise compared to the KeQueryPerformanceCounter value. Your packets will be stamped with a time in 10-15 millisecond increments, rather than microseconds. That means you are likely to have packets with the same timestamps pretty often, like in the picture below.


According to a Wireshark mailing list exchange, the registry value to change is:


And the possible values are:

“0 (default) -> Timestamps generated through KeQueryPerformanceCounter, less reliable on SMP/HyperThreading machines, precision = some microseconds
2 -> Timestamps generated through KeQuerySystemTime, more reliable on SMP/HyperThreading machines, precision = scheduling quantum (10/15 ms)
3 -> Timestamps generated through the i386 instruction RDTSC, less reliable on SMP/HyperThreading/SpeedStep machines, precision = some microseconds”

And you’ll need to restart the NetGroup Packet Filter Driver service with these commands before kicking off your capture:

net stop npf
net start npf

I hope this is useful information for you! Good luck!