Archive for July, 2010

show interface history command

July 31, 2010 2 comments

Today I want to show a command that was introduced with IOS 15.1T. Using this command will upgrade your troubleshooting capabilities as you can (finally) look back and have some decent historical data on the interface level.

My home lab uses old equipment that is not supported by this IOS version but fortunately enough I had access to a lab router (2800 series) that was loaded with this latest and greatest version.

So here are few notes on the show interface history command:
To begin with, this command allow you to collect utilization history and show it in a (Cisco kind of) graphical representation. If you’re familiar with the show processes cpu history command you know what I’m talking about. Another similarity to the cpu command is the options: last 60 seconds, last 60 minutes and last 72 hours but since it’s an interface command, you get 2 graphs per time frame: Input and Output
The data that you get can be packets per second (pps) or bit per second (bps), both are useful when you try to get some better understanding of what’s going on with your interface.

Check this example (followed by explanation):

Lab65R2# show interface gigabitethernet 1/1 history 60min


9    *
8   **  *                  *
7   *#  #*        **       #*
6  *##  ##    #   ## #* ** ##*  *
5  #### #### *#   ## ##### ###* *
4  ######### ##  *#############**
3 ############## ###############*
2 ############## ################
1 ###############################
.          0    5   0   5    0    5   0    5   0   5    0

Mlcst 556555555565555555555565535555700000000000000000000000000000
22322111111     121221211211
57149774766867 133175814422022
iDrop 425727636317619265454496840996600000000000000000000000000000
GigabitEthernet1/1 input rate(mbits/sec)  (last 60 minutes)
* = maximum   # = average

As you can see, it’s not so pretty but it does offer some useful troubleshooting information.
I show here one example of Input history for 60 minutes time frame. You can see the number of multicast (Mlcst) packets and drops, both show per minute data. To get the total number you’ll have to work a bit and add up all the numbers, one by one. You can also see the rate per minute in the graph with average (#) and max (*)  points marked.

To summarize, this is not one of those commands that you’ll use on a daily basis but when time comes and you have a problem, it will come handy and might shed some light on traditionally dark corners.

Invitation: My new Exchange blog

July 28, 2010 Leave a comment

I started a new ‘study from scratch’ Exchange blog.
Read all about it here: and bring your friends 😉

Linux – a post about nothing

July 27, 2010 2 comments

Warning: This post is not healthy for penguins

As I told you I have to learn few new things at my new job, one of them is Linux.
Today I took the first step, installing Red Hat 5.1 Server. After a simple installation I wanted to configure the network settings:
IP address, subnet mask, default gateway and DNS server – the basic network interface parameters.

Using the local console (SSH will only be available with a valid IP) and some help from nixCraft and UnixGuide I found it easier than expected and very Windows like 🙂

So, the first thing I did is a typo error, after all it is a Windows world:

Fixing the typo, it looked better:

Even the output is Cisco like – both in the format and content 🙂
Next, I had to change the IP, look how simple it is:

At this point I had the correct settings, I could ping out but I couldn’t SSH in or connect with my application. Like in Windows servers, the firewall was the reason and had to be disabled. Once again a Windows driven typo:

Everything was working at this point but I wanted to check how close it is to the commands I know. This is just one example of a similar command:

So similar to Windows, so what is the big fuss out there???

CDP and some sniffer results

July 25, 2010 1 comment

One of the benefits on a Cisco based network is Cisco’s proprietary discovery protocol: CDP

Over the years I read opinions for and against using it:
Those in favor think its a very useful feature that help getting data about other devices, including router type, versions and IPs. If you have a remote router that you have to identify, CDP is your friend.
On the other side of this discussion you’ll find the security experts that want to turn off every feature just because they can. They claim that it allows neighboring routers (and their admins) to put their hands on too much information.

Who’s right and who’s wrong? I’m somewhere in the middle, leaning toward disabling it unless you have a very good reason to keep it on.
In today’s post I’ll show what type of info CDP can find and show some Wireshark captured packets to prove my point.

I’ll use my switch to show some of the output options. First a list of neighbors:

Switch#show cdp neighbors
Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge
S – Switch, H – Host, I – IGMP, r – Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
ITDualism2.ITDUALFas 0/15           138          R        2611      Eth 0/1
ITDualism1.ITDUALFas 0/14           175          R        2621      Fas 0/1
ITDualism3       Fas 0/16           133          R        2611      Eth 0/1

You can see any Cisco device that has direct connection and has CDP enabled. In this case its my 3 routers and for each you can see to which port they connect (Local Intrfce), the platform and port on the remote device.

If you want to get more details on a specific router you have few options:

Switch#show cdp entry ITDualism3 version

Version information for ITDualism3 :
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.2(46a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Wed 11-Jul-07 20:22 by pwade

Information on a specific entry (you can use ‘*’ for all) which gets the router platform and IOS version or if you want even more, you can use the following:

Switch#show cdp neighbors detail
Device ID: ITDualism2.ITDUALISM
Entry address(es):
IP address:
Platform: cisco 2611,  Capabilities: Router
Interface: FastEthernet0/15,  Port ID (outgoing port): Ethernet0/1
Holdtime : 160 sec

Version :
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.2(46a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Wed 11-Jul-07 20:22 by pwade

advertisement version: 2
Duplex: full

Additional information such as IP address, interfaces on both ends and the configured duplex. In this output I cut the rest of the devices to make it short but obviously you’ll get the same information for each of your devices.

Another option is getting the status of each port on the local device and its CDP status (again, just a portion of the output):

Switch#show cdp interface
FastEthernet0/1 is down, line protocol is down
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/13 is down, line protocol is down
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/14 is up, line protocol is up
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/15 is up, line protocol is up
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds

I’ll jump to one of the router to show one last point. When one of your neighbors is a switch there is another piece of information that might come useful: VTP domain name

ITDualism2#show cdp entry Switch
Device ID: Switch
Entry address(es):
IP address:
Platform: cisco WS-C2924-XL,  Capabilities: Trans-Bridge Switch
Interface: Ethernet0/1,  Port ID (outgoing port): FastEthernet0/15
Holdtime : 147 sec

Version :
Cisco Internetwork Operating System Software
IOS ™ C2900XL Software (C2900XL-C3H2S-M), Version 12.0(5.3)WC(1), MAINTENANCE INTERIM SOFTWARE
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Mon 30-Apr-01 07:34 by devgoyal

advertisement version: 2
Protocol Hello:  OUI=0x00000C, Protocol ID=0x0112; payload len=27, value=00000000FFFFFFFF010121FF0000000000000002FDE4D540FF0001
VTP Management Domain: ‘ITDUALISM’

You have enough information on CDP and you can start making your pros\cons list. One more thing you should know is that CDP doesn’t stay exclusively within the Cisco domain.
A simple network sniffing allow you to get some interesting details:

So what do you think about CDP, should it be enabled?

NORTEL switch – a different experience

July 24, 2010 2 comments

I’m a spoiled network admin – throughout my career I used only Cisco equipment.

Using only one vendor has a huge advantage as you have to learn one system, one syntax (yes, I know different devices or even IOS versions use different syntax but the major part never change). Using Cisco as your only vendor is expensive but if you can afford it the benefits are obvious: you get high quality hardware, great support (TAC), online documentation that is both easy to find and accessible and Googling any problem is easy, you always find a dozen of answers and useful information.

Now at my new job I had a different experience. We have two Nortel switches, BES1020 and I was assigned with the simple task: turn on monitoring on the switch (thinking of Cisco’s SPAN).

Before I continue two quick notes:

  1. As it was one of my first assignments I was anxious to prove my skills which mean make it work and do it fast.
  2. I’ll be more than happy to find a comment that would say something like:
    “dude, you’re wrong! this is how you can do it”

The first step was finding the IPs of the 2 switches. As we didn’t know it I had to use Nortel’s Business Element Manager – their switch management tool. Couple of minutes later I found one of the switches and had no problem changing the IP. This is a screen shot of BEM with the 3 switches we have:

If you wonder, BEM’s scanning found another switch in a different site – a switch I wasn’t aware of. The one big problem with this tool is that it finds the switches by IP and since both local switches used the factory default IP –, I could only find one of them, had to change the IP and remove all previous data from BEM before I could find the second switch.

Victory was never closer. I opened PuTTY and tried to telnet the IPs I just assigned but…
Yes, every good story has its but 🙂
Telnet failed to connect and googling a bit I found that people mention ‘enable telnet-access’ command, not for my switch but I assume that Nortel’s default configuration disable telnet. I had to connect using console to make those changes.

When we found the right DB9 connectors and finally got into console mode (using Hyper Terminal for Win7) I had another surprise. While I expected a command line to show up (like a good Cisco device), this is what I saw:

Clicking CTRL+Y to get in, this is the next screen:

Very limited, very old school and most important – no word on telnet or port monitoring.
It was time to find the Business Ethernet Switch 1000 Series guide and figure out what’s going on here. Reading the BES quick install guide (check this doc for the default password – it is kind of funny) just to confirm my finding resulted in the amazing sad conclusion: BES 1020 does not support telnet or port monitoring 😦

Yes, I checked the documentation few times (conig by BEM and config by Web – these are the 2 options), looked at the web configuration which has the same options as BEM and telnet does not exist. Oh, I miss my Cisco…

If anyone can correct me here I’ll be the happiest guy in the blog-sphere. If anyone has another idea on How to port-monitor this switch – stand up and HELP ME!!!

Home Lab – NTP and logging

July 18, 2010 Leave a comment

I got few emails asking for basic troubleshooting steps & tips. I’ll dedicate few posts over the next weeks to some simple basic (yet very important) steps.

Today I’m working on some little things that at some point make a big difference: Date & Time.

What? Who cares about it? !
Right? Wrong!
Setting the time on all your devices is critical when troubleshooting. It is important to have the same method across the organization to allow troubleshooting effectiveness. I’ll start with the ‘how’:

ITDualism1(config)#ntp server prefer
ITDualism1(config)#ntp server
ITDualism1(config)#ntp server

The basic command set an IP or Hostname as the time-server. I used my PC, which I always keep on as the prefered source and added a couple of outside sources (I types in the names, the router translated to IP). In your production environment you should use any device that provide NTP services or an outside source. Make sure you allow UDP port 123 traffic between the routers and the NTP server.
Use the ntp command help for many more options, it is a small feature that allow great flexibility which you’ll need in a complex environment.

The next thing we’ll do is verify that we actually sync the time. In a router that was just installed (or turned on) you can use show clock and verify the time (unless you’re in 1993 :)). On a production router you can use either show ntp status or show ntp associations:

ITDualism1#show ntp associations

address            ref clock       st      when    poll   reach   delay   offset    disp
~       5       29      1024   377     4.2     -8.59     1.6
+~     3       69      128    377     4.1     3.48      2.3
*~     3       32      128    377     7.9     11.18     3.6
~          16        –      64     0       0.0     0.00      16000.
* master (synced), # master (unsynced), + selected, – candidate, ~ configured

One last note on the time and date methods. If your organization is multi time-zone make sure you set the correct time-zone per site or use UTC for all the routers. I set my clock to NYC time:

ITDualism1(config)#clock timezone EDT -5

Now that the time is synced I’ll configure logging. As I mentioned, time and have a major role when you troubleshoot your routers. When you log events and try to analyze the data your goal is to capture events that happen at the same time across the network and understand what was happening.

The best way to start is installing a Syslog server. I’ve used KIWI’s Syslog server for years but any other server would be just as good. Make sure you have the IP address of the server and go back to your console.

If you never used the logging command check the help option and take a look at my configuration:

ITDualism1(config)#logging trap
ITDualism1(config)#service timestamps log datetime
ITDualism1(config)#snmp-server enable traps config

The commands I used configured (my PC) as logging destination, allow trap and add timestamp that include date & time. The last command is a specific log for a selected protocol or in my case, configuration changes. This is just a sample of the different options and since there are many different parameters that should be considered when you build your routers (and switches) you should spend few minutes and ask yourself what is your goal, how much data you want to log and fine tune as you go.

Home Lab – login configuration

July 15, 2010 2 comments

Now that my lab is up and all the pieces connect, it is time to get off the console cable and the tight room at the lab area and relax on my sofa with great view to the AC 😉

In a production environment you always strive on max security and when it comes to routers remote access it is called ssh (if you know nothing about it, start here). Since my lab uses some older routers I have to compromise but lucky enough I enjoy both worlds and can show you how to configure ssh. Keep in mind that (like many other commands) syntax might be a bit different on different IOS versions (but the key components will always be the same).

Out of the 3 routers only my 2621 support ssh. I picked this order of commands to create some of the messages you might see on the way. I’ll explain each of them following the output.

ITDualism1(config)#ip ssh time-out 120
Please create RSA keys to enable SSH.

ITDualism1(config)#ip ssh authentication-retries 3
Please create RSA keys to enable SSH.

The first commands set the values for time-out (0-120) and authentication-retries (0-5). Though it prompted to create RSA keys, the commands did go through and the values I set are in the system. Now it’s time to create the key:

ITDualism1(config)#crypto key generate rsa usage-keys
% Please define a domain-name first.

As you can see, domain-name is prerequisite and the router couldn’t create the key.

ITDualism1(config)#ip domain-name ITDUALISM
ITDualism1(config)#crypto key generate rsa usage-keys
The name for the keys will be: ITDualism1.ITDUALISM
Choose the size of the key modulus in the range of 360 to 2048 for your
Signature Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]: 768
Generating RSA keys …
Choose the size of the key modulus in the range of 360 to 2048 for your
Encryption Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]:
Generating RSA keys …

Now that the domain-name is configured, I could create the key.  The reason I choose 768bit for the key is ssh2’s min requirement. At this point I don’t know if the router does support it but want to keep all the options. Generally speaking you should remember that higher bits offer better security but (like everything in life) it come with a cost: performance. Make your decision carefully and consider the hardware specs, expected load on the router and your security requirements. This is a good size selection document.

To complete this step I’ve used the show ip ssh command to verify both version and configuration:

ITDualism1#show ip ssh
SSH Enabled – version 1.5
Authentication timeout: 120 secs; Authentication retries: 3

As you can see, this router does not support ssh2 and the parameters I’ve set does show, though I got the messages when I typed them.

The next step is configuring remote access using ssh. Like access-list (and other features), configuring the parameters doesn’t mean we use the feature and we have to configure it on the required lines:

ITDualism1(config)#line vty 0 4
ITDualism1(config-line)#transport input all

In this case I allow access into vty 0 4 using all protocols, including ssh and telnet. I’ll show the other options with my 2611 routers. At this point ssh is ready and I was able to connect:

This is the standard message you expect to see when you connect via ssh the first time from a new computer. Clicking ‘Yes’ get you to the login screen and you’re good to go.

Older routers do not support ssh. Few ways to check it are shown below:

ITDualism2(config)#ip s?
sap  security  source-route  subnet-zero

The ssh command that we used on the 2621 router is not available.

ITDualism2(config-line)#transport input ?
all     All protocols
none    No protocols
pad     X.3 PAD
rlogin  Unix rlogin protocol
telnet  TCP/IP Telnet protocol
udptn   UDPTN async via UDP protocol
v120    Async over ISDN

Once again, you can see the list of all the available protocols for remote connection and ssh is not one of them.
The one last piece I want to show is how to configure the protocol for the line:

ITDualism2(config-line)#transport input all {all | none | pad | rlogin | | telnet | udptn |v120}
ITDualism2(config-line)#transport preferred telnet

In this example I allow using all protocols but configured telnet as the preferred protocol. The transport preferred setting specifies a search order when attempting to resolve names that might be valid for multiple protocols. If the address or service does not match the preferred protocol, all other valid output protocols are searched to find a valid match.

You can find some good reading from Cisco here, here and here. You might also want to look at this guide.