Cisco Port-Security Settings and Wireless Access Points

/in /by

A few months ago during the first stage of our 1-to-1 iPad deployment, I ran into an issue where I could connect to our wireless network just fine and send and receive traffic to my heart’s content, but sometimes when I strayed to another location within the school I would lose connectivity for several minutes. I checked out the configuration on the Wireless Access Points and it looked fine–the SSIDs were all set up appropriately, the radios were beaconing and everything looked great from the management interface. Why couldn’t I roam? We had this same style of configuration at the previous 7 schools and this issue was not present. To make things a little more confusing, the issue wasn’t 100% reproducible. Sometimes it happened, sometimes it didn’t.

Almost luckily, we hadn’t finished properly documenting and naming the access points at this facility, and before attempting to begin the troubleshooting of what I thought was a wireless issue, I was physically locating and documenting each AP via their MAC addresses by bouncing the switch interfaces they were connected to (a crude method for sure, but these APs lacked any location features, and it was after-hours). I noticed that a few of the access points had “Static” as their learning method in the MAC address table, where I expected to see “Dynamic”. I checked the configuration on the switches to make sure no static entries were actually entered, then by chance ran across some port-security settings in some of the interface configurations. After a bit of reading, I remembered that port-security settings can result in a static entry in the MAC address table–it doesn’t have to result from a manual entry typed by hand. Aha!

Unfortunately I didn’t grab the configuration before making changes to it, and this was months ago, but if memory serves here’s what the port-security configurations looked like, with my comments following the double slashes:

S1#show run
[output omitted]
interface FastEthernet0/1
switchport mode access //configure the port as an access port, meaning only allow the port to be a member of one VLAN–required for port-security settings
switchport port-security //enable port security on this interface
switchport port-security maximum 50 //allow a maximum of 50 MAC addresses to be learned on this interface
switchport port-security violation protect //if more than the maximum specified number of MAC addresses are seen on this port, don’t allow them to be added to the MAC address table but leave the interface up rather than shutting it down
switchport port-security aging static //allow the MAC addresses to age, otherwise the first 50 will stay in the MAC address table forever and no new addresses will be learned
switchport port-security aging time 5 //remove the MAC addresses from the MAC address table after 5 minutes when…
switchport port-security aging type inactivity //…no frames from that MAC address are seen on the port during that time

After conversing with the Director of Technology, it seems that these settings were an attempt to limit the number of clients associating with each access point, as they were rated from the vendor as capable of supporting up to about 50 devices at a time. Unfortunately, this implementation was a bit ill-fated. Any time a client would connect to an AP that was connected to a port with the settings above, that client would have problems when attempting to roam. The association with the next access point would go off without a hitch, but the switch itself wouldn’t allow the client’s MAC address to be learned on that new port for 5 minutes. Some ports had the configuration and some didn’t, which led to the inconsistencies.

So as it turns out, the problem that I originally thought was a wireless issue had nothing to do with wireless. It also turns out that our brand of access points (AeroHive) allows you to configure a maximum number of connections in the access point configuration itself, which I think is a much better way to go. I think this scenario presents solid evidence of the need for troubleshooting skills. I had nothing to do with entering those commands, and had no idea they were there, but nevertheless I found myself troubleshooting the issue. This also prompted a new phrase of which I need to constantly remind myself: “The wireless network is connected to the wired network!”

How To Record an SSH or Telnet Session to a Text File

/in /by

Hello again Internet!

You can use the “tee” utility on Linux to record an entire SSH or Telnet session. This method will allow you to capture all of the session output including commands that were typed. This is helpful for logging your own actions and configurations for later browsing or for letting a customer see what you were doing on their device.  It also allows you to parse the session output using grep, cut, sed and other tools after the session is over. Two examples follow:

ssh user@10.10.10.10 | tee logfile

This will start an SSH session to IP address 10.10.10.10 with the username “user”. The pipe is used to send standard output to the tee command, which writes the session to the file named “logfile” while simultaneously reprinting the information back to standard output, allowing it to show in your terminal.

telnet 10.10.10.10 | tee -a logfile

This will start a telnet session to IP address 10.10.10.10. The pipe is used to send standard output to the tee command, and the -a argument is used to denote that all information should be appended to the file “logfile”, rather than overwriting it. Let’s look at one more example of what we can accomplish using this command.

ssh user@10.10.10.10 | tee sshlog
…authentication…
switch#show run
…show command output…
switch#exit

linux:~ #cat sshlog | egrep ‘description|Ethernet’
interface GigabitEthernet2/1
description Trunk to IDF300
interface GigabitEthernet2/2
description Trunk to IDF301
interface GigabitEthernet2/3
description VLAN 10 Access to Toshiba Copier
interface GigabitEthernet2/4
description VLAN 50 Access to Video Server

Using what we’ve learned, we start an SSH session to the switch at 10.10.10.10 from our Linux machine and pipe the output to the tee command, saving it to the file “sshlog”. We perform a show run command, then exit. Back on our Linux machine, we cat the “sshlog” file, and use an egrep command to print only lines that have the string “description” or “Ethernet”. This gives us a very readable output with only the interface names and descriptions.

The tee utility is also great to use any time you are doing something that will generate a fair deal of output to standard out, like installations. I’m sure you will find other uses too! If you’re using Putty on Windows, check out the “Logging” section under the “Session” tab for a variety of logging options.

 

How To Use Wireshark To View tcpdump Captures

/in /by

Hello again world! This week I am relaxing with my family in the beautiful Blue Ridge Mountains of North Carolina! However, I still found time to play with tcpdump a little bit and wanted to share how to open tcpdump captures using Wireshark.

tcpdump is a Linux/Unix command line utility that allows you to capture network packets, similar to Wireshark’s command line capture utility named tshark. Running tcpdump without any parameters will simply cause it to print out a basic description of the captured packets using one line per packet in the current shell. Redirecting the output of the command to a file isn’t very useful since it’s not really packet data that’s being captured, just the descriptions. If we want to capture the entire contents of each packet for later analysis, we will need to use the -w parameter and supply a filename. In our example, we’ll go ahead and append a file extension of “.pcap” to the filename so that Wireshark will recognize the filetype in our GUI, whether we are opening the file on the current system or transferring it to a Windows system. And depending on our OS, we might need to run tcpdump as root so we’ll use sudo:

sudo tcpdump -w capture.pcap

This will cause tcpdump to write all captured packets to capture.pcap in the current working directory. If you don’t specify any limiting factors like a maximum packet capture count or a maximum file size, the capture session will run until you press Ctrl+C. After tcpdump is stopped, you can view the trace file using Wireshark on a system of your choice. Best of luck!

Inconsistent Timestamps Over Long Wireshark Captures

/in /by

I’ve recently been running Wireshark around the clock on a couple of servers at work to troubleshoot a problem that pops up whenever it feels inclined. One late night while perusing trace files with sleepy eyes, I happened to realize that packets were arriving at server B two minutes and forty seconds after they were sent from server A, according to the timestamps on the packets. I thought that was very odd, of course. It seemed rather impossible that packets were taking almost three minutes to reach their destination on our high capacity network–and where were the errors? I checked the system time on both servers, and they were both correct and within a second or two of each other. I restarted both captures and double checked the system times and the timestamps on the packet and they were correct at that time. A few days later, I see the same issue arise. Then I notice that the current packets that are coming in are stamped a few minutes behind the system time. What the heck?

Well it turns out that Wireshark’s reliance on the WinPcap library in Windows causes this issue. When a capture is started, WinPcap checks the system time for a reference point and then tracks the time on its own by referencing the KeQueryPerformanceCounter routine (that’s processor mumbo-jumbo to me, maybe you are more well-versed). It looks like there are two options to handling this problem, and number one is to deal with the issue with an understanding that the time is going to lag more and more behind the system time the longer the capture runs. You might be able to alleviate the problem by restarting the capture session frequently, although you will probably lose packets during that process. You can use Wireshark’s timestamp offsetting feature to help get things on the same track if you are referencing two trace files taken at the same time, but there might still be some discrepancies.

Option two is to change a registry key setting in Windows which will force WinPcap to use the system time for its packet timestamps. The drawback with this option is that it is less precise compared to the KeQueryPerformanceCounter value. Your packets will be stamped with a time in 10-15 millisecond increments, rather than microseconds. That means you are likely to have packets with the same timestamps pretty often, like in the picture below.

timestamps

According to a Wireshark mailing list exchange, the registry value to change is:

HKLM\System\CurrentControlSet\Services\NPF\TimestampMode

And the possible values are:

“0 (default) -> Timestamps generated through KeQueryPerformanceCounter, less reliable on SMP/HyperThreading machines, precision = some microseconds
2 -> Timestamps generated through KeQuerySystemTime, more reliable on SMP/HyperThreading machines, precision = scheduling quantum (10/15 ms)
3 -> Timestamps generated through the i386 instruction RDTSC, less reliable on SMP/HyperThreading/SpeedStep machines, precision = some microseconds”

And you’ll need to restart the NetGroup Packet Filter Driver service with these commands before kicking off your capture:

net stop npf
net start npf

I hope this is useful information for you! Good luck!

Wireshark 101 – Sending And Analyzing An ICMP Ping, Part 1

/in /by

Welcome, readers! In the last Wireshark 101 article, we installed Wireshark and got introduced to the interface. This time we are diving in and analyzing our first packets! I’m using Windows 7 in this tutorial, so following along will be easiest if you are using Windows yourself. Also noteworthy is the fact that I’m using a computer plugged into an Ethernet II network. If you are using wireless or a different layer 2 wired technology other than Ethernet II (unlikely, but possible), you’ll have a different experience than what is depicted, but feel free to follow along anyway! Go ahead and fire up Wireshark and start a new live capture by choosing your capture interface and clicking the green shark icon. Once the capture’s going, open up cmd.exe by pressing Windows Key+R, typing “cmd.exe” without the quotes, and pressing enter. Now type the following command to send 4 ICMP (Internet Control Message Protocol) packets to 8.8.8.8, Google’s public DNS server:

ping 8.8.8.8

This command will send 4 ICMP Echo Request (“ping”) packets to the IP address 8.8.8.8. Your screen should look pretty much like the window below.

pinggoogle

After the pings have been sent, minimize cmd.exe, pull Wireshark back up and stop the capture by pressing the red stop button just beside the green shark fin. Now let’s go back to the cmd.exe window so we can discuss our results. A ping is a low level network packet that, among other things, is often used to test basic connectivity to a network device. Each echo request ping packet is a humble request that the receiving device respond back. The fact that you received a response to your pings is a confirmation that we have a path from your computer to 8.8.8.8, a path from 8.8.8.8 back to your computer, and that the device that responded is being nice and not ignoring or blocking our pings. The ping does not tell us who 8.8.8.8 is or where it is located or what services it offers. We just happen to know in advance that this is Google’s Public DNS server (with that in mind, please don’t spam the server by sending more pings than are necessary for the purposes of this tutorial).

In the cmd.exe window, we are given some information about the response back from our pings. The first line after our command tells us which IP address we are pinging and how many “bytes of data” are being sent in each packet. The next four lines detail the responses we received from each ping packet. The address where the reply originated, the amount of data bytes that were received, the amount of time it took for the response to return in milliseconds, and the TTL (Time to Live) value for each packet are seen in each line, from left to right. At the bottom of our window, we see some summary statistics.

Let’s switch back over to the Wireshark window. If any applications on your computer were sending network traffic while you performed your packet capture, or if there were noisy broadcasting devices on your network, you’ve likely got extra packets besides the 8 we are looking for in your trace file. The filter field is going to help us solve this “needle in the haystack” issue. Simply type “icmp” without quotes into the filter field and press enter. As long as there were no other ICMP packets sent during your capture, you should now have a window very similar to this:

4pings

Wow, this is much easier to deal with! Notice that the filter field turns green since we have typed an appropriate display filter pattern. Alright, so what are we looking at here? Each row in the packet list pane corresponds to a single packet. Currently, and for reasons we’ll explore later on, each packet is colored pink (the top packet is blue since it is selected). Follow along by referencing the first packet in the list (No. 583, the blue one). The columns in the packet list pane denote the following pieces of information:

No.: The number of the packet within the capture file. Wireshark counts packets in the order that they were received, starting with number 1. Packet 583 would be the 583rd packet that Wireshark saw since beginning its capture. The packet numbers in this picture aren’t sequential because we have applied a display filter to only show ICMP traffic. It just so happens that other packets were received between the pings, and the pings began after 582 packets were already captured.
Time: The time that the packet was received in seconds after the capture was started. A time value of 8.246 indicates that Wireshark received this packet 8.246 seconds after the capture began.
Source: The source IP address of the packet. This first packet originated from 192.168.1.100, which happens to be the internal IP address of my home computer. Wireshark captured this packet as it left the computer.
Destination: The destination IP address of the packet. This first packet was addressed to be delivered to 8.8.8.8.
Length: The length of the packet in bytes. Each of these packets is 74 bytes in length.
Info: Some relevant information about the packet depending on what protocol is being used. In this instance, Wireshark has determined that these are ICMP ping packets, and has displayed whether each packet is an ICMP echo request or reply, the id and sequence values of each packet, the TTL value for each packet, and the packet number of the accompanying reply (if it’s a request) or the packet number of the accompanying response is (if it’s a reply).

ethernetframe

Let’s dive further into this first packet. With the first packet highlighted in the packet list pane, we’ll move our attention to the packet details pane in the middle of the screen. Each row corresponds to a layer of the packet and essentially, a particular protocol. The first layer (Frame xxx…) corresponds to the physical layer of the OSI model. The only important thing we need to know about the layer 1 information is that Wireshark received a string of bytes called a preamble indicating that an Ethernet II frame header was coming up. Feel free to investigate this layer, but we won’t be discussing it–we’re going straight to layer 2! Click on the the little plus sign to the left of “Ethernet II…” to expand the Ethernet II header. What we see is a destination and source address, and a “Type” field. These destination and source addresses are known as Media Access Control (MAC) addresses, and they are used to identify devices at layer 2 of the OSI model. Each MAC address is 6 bytes or 48 bits in length and corresponds to a particular hardware interface. The first 3 bytes (the first half) of the MAC address is known as the Organizational Unique Identifier (OUI) and identifies the name of the organization which manufactured the network interface card (NIC). There’s an option in Wireshark to check OUI’s against a database and display the name of the company that manufactured the NIC–that’s why in my capture file we can see “Asiarock” and “Cisco-Li” (Cisco-Linksys). The non-interpreted MAC address is shown in full in parenthesis to the right of the interpreted addresses. The Type field designates what type of protocol will be coming up after the Ethernet II header. In my case, it is IP (Internet Protocol version 4), which you can confirm by looking at the layer directly beneath the Ethernet II header.

If you select a line from the packet details pane, the corresponding bytes will also be highlighted in the the packet bytes pane. I’ve just realized that the default setting for the packet bytes pane is to display the bytes in hexadecimal, and I’m viewing mine in binary. To switch between the two views, right click anywhere in the packet bytes pane and choose “Bits View”. In the picture above, I have selected the entire Ethernet II header by clicking the text on the line rather than the plus symbol (see the blue line in the packet details pane in the picture above). By doing this, we can see every byte and bit that comprises the Ethernet II header and it’s ASCII interpretation (which isn’t that interesting for this part) in the packet bytes pane highlighted in blue. As expected, the Ethernet II header is 14 bytes in length (14 sets of 8 bits, or one hundred and twelve 1’s and 0’s–go ahead, count them yourself!). It all adds up considering we know that there is a destination MAC address (6 bytes), a source MAC address (6 bytes) and a Type field (2 bytes). You could also select any of these fields to see their corresponding bytes in the packet bytes pane. In the picture below, I’ve selected the Destination MAC address and the corresponding 6 bytes are highlighted:

destinationmacbytes

If you understand how to convert binary to hexadecimal, you’ll see that the binary and hexadecimal do indeed match up. And even if you don’t know how to convert binary to hexadecimal, you can switch to hexadecimal view in the packet bytes pane. Keep in mind that the bits we see in the packet bytes pane are the result of the actual electrical signals sent across the wire which correspond to 1’s and 0’s. Wireshark applies dissectors to these bit streams in order to know which bits and bytes correspond to particular pieces of information. For instance, when we evaluate the first six bytes of the Ethernet II header, we see that we can convert it to hexadecimal as 00:22:6b:80:56:0d. There is nothing in these bytes that translates into the text “Destination” or “Cisco-Li_”. Wireshark knows that this is a destination MAC address because it has applied a dissector which specifies that the first 6 bytes of the Ethernet II header are to be interpreted as a destination MAC address. This same group of bytes would be interpreted differently if it were located in a different place in the packet. Likewise, “Cisco-Li_” is the OUI which is supplied by Wireshark after checking an OUI dictionary. There is no information located within the bytes themselves that would provide this knowledge. Additionally, Wireshark knew to apply it’s Ethernet II dissector to this section of the packet due to the preamble it received indicating that an Ethernet II header was coming up. This is one of the beautiful things about Wireshark. You are able to quickly access a ton of information without having to know the protocols inside and out and without having to manually compute what each bit and byte means.

Whew! This is shaping up to be a two-parter. Go ahead and save your Wireshark capture by choosing File > Save As… and choosing a place to save the file. Next week we’ll use this same file to examine the rest of the packet. Thanks for reading!