Archive for the ‘LAB’ Category

show interface history command

July 31, 2010 2 comments

Today I want to show a command that was introduced with IOS 15.1T. Using this command will upgrade your troubleshooting capabilities as you can (finally) look back and have some decent historical data on the interface level.

My home lab uses old equipment that is not supported by this IOS version but fortunately enough I had access to a lab router (2800 series) that was loaded with this latest and greatest version.

So here are few notes on the show interface history command:
To begin with, this command allow you to collect utilization history and show it in a (Cisco kind of) graphical representation. If you’re familiar with the show processes cpu history command you know what I’m talking about. Another similarity to the cpu command is the options: last 60 seconds, last 60 minutes and last 72 hours but since it’s an interface command, you get 2 graphs per time frame: Input and Output
The data that you get can be packets per second (pps) or bit per second (bps), both are useful when you try to get some better understanding of what’s going on with your interface.

Check this example (followed by explanation):

Lab65R2# show interface gigabitethernet 1/1 history 60min


9    *
8   **  *                  *
7   *#  #*        **       #*
6  *##  ##    #   ## #* ** ##*  *
5  #### #### *#   ## ##### ###* *
4  ######### ##  *#############**
3 ############## ###############*
2 ############## ################
1 ###############################
.          0    5   0   5    0    5   0    5   0   5    0

Mlcst 556555555565555555555565535555700000000000000000000000000000
22322111111     121221211211
57149774766867 133175814422022
iDrop 425727636317619265454496840996600000000000000000000000000000
GigabitEthernet1/1 input rate(mbits/sec)  (last 60 minutes)
* = maximum   # = average

As you can see, it’s not so pretty but it does offer some useful troubleshooting information.
I show here one example of Input history for 60 minutes time frame. You can see the number of multicast (Mlcst) packets and drops, both show per minute data. To get the total number you’ll have to work a bit and add up all the numbers, one by one. You can also see the rate per minute in the graph with average (#) and max (*)  points marked.

To summarize, this is not one of those commands that you’ll use on a daily basis but when time comes and you have a problem, it will come handy and might shed some light on traditionally dark corners.


CDP and some sniffer results

July 25, 2010 1 comment

One of the benefits on a Cisco based network is Cisco’s proprietary discovery protocol: CDP

Over the years I read opinions for and against using it:
Those in favor think its a very useful feature that help getting data about other devices, including router type, versions and IPs. If you have a remote router that you have to identify, CDP is your friend.
On the other side of this discussion you’ll find the security experts that want to turn off every feature just because they can. They claim that it allows neighboring routers (and their admins) to put their hands on too much information.

Who’s right and who’s wrong? I’m somewhere in the middle, leaning toward disabling it unless you have a very good reason to keep it on.
In today’s post I’ll show what type of info CDP can find and show some Wireshark captured packets to prove my point.

I’ll use my switch to show some of the output options. First a list of neighbors:

Switch#show cdp neighbors
Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge
S – Switch, H – Host, I – IGMP, r – Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
ITDualism2.ITDUALFas 0/15           138          R        2611      Eth 0/1
ITDualism1.ITDUALFas 0/14           175          R        2621      Fas 0/1
ITDualism3       Fas 0/16           133          R        2611      Eth 0/1

You can see any Cisco device that has direct connection and has CDP enabled. In this case its my 3 routers and for each you can see to which port they connect (Local Intrfce), the platform and port on the remote device.

If you want to get more details on a specific router you have few options:

Switch#show cdp entry ITDualism3 version

Version information for ITDualism3 :
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.2(46a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Wed 11-Jul-07 20:22 by pwade

Information on a specific entry (you can use ‘*’ for all) which gets the router platform and IOS version or if you want even more, you can use the following:

Switch#show cdp neighbors detail
Device ID: ITDualism2.ITDUALISM
Entry address(es):
IP address:
Platform: cisco 2611,  Capabilities: Router
Interface: FastEthernet0/15,  Port ID (outgoing port): Ethernet0/1
Holdtime : 160 sec

Version :
Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.2(46a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Wed 11-Jul-07 20:22 by pwade

advertisement version: 2
Duplex: full

Additional information such as IP address, interfaces on both ends and the configured duplex. In this output I cut the rest of the devices to make it short but obviously you’ll get the same information for each of your devices.

Another option is getting the status of each port on the local device and its CDP status (again, just a portion of the output):

Switch#show cdp interface
FastEthernet0/1 is down, line protocol is down
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/13 is down, line protocol is down
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/14 is up, line protocol is up
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds
FastEthernet0/15 is up, line protocol is up
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds

I’ll jump to one of the router to show one last point. When one of your neighbors is a switch there is another piece of information that might come useful: VTP domain name

ITDualism2#show cdp entry Switch
Device ID: Switch
Entry address(es):
IP address:
Platform: cisco WS-C2924-XL,  Capabilities: Trans-Bridge Switch
Interface: Ethernet0/1,  Port ID (outgoing port): FastEthernet0/15
Holdtime : 147 sec

Version :
Cisco Internetwork Operating System Software
IOS ™ C2900XL Software (C2900XL-C3H2S-M), Version 12.0(5.3)WC(1), MAINTENANCE INTERIM SOFTWARE
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Mon 30-Apr-01 07:34 by devgoyal

advertisement version: 2
Protocol Hello:  OUI=0x00000C, Protocol ID=0x0112; payload len=27, value=00000000FFFFFFFF010121FF0000000000000002FDE4D540FF0001
VTP Management Domain: ‘ITDUALISM’

You have enough information on CDP and you can start making your pros\cons list. One more thing you should know is that CDP doesn’t stay exclusively within the Cisco domain.
A simple network sniffing allow you to get some interesting details:

So what do you think about CDP, should it be enabled?

Home Lab – NTP and logging

July 18, 2010 Leave a comment

I got few emails asking for basic troubleshooting steps & tips. I’ll dedicate few posts over the next weeks to some simple basic (yet very important) steps.

Today I’m working on some little things that at some point make a big difference: Date & Time.

What? Who cares about it? !
Right? Wrong!
Setting the time on all your devices is critical when troubleshooting. It is important to have the same method across the organization to allow troubleshooting effectiveness. I’ll start with the ‘how’:

ITDualism1(config)#ntp server prefer
ITDualism1(config)#ntp server
ITDualism1(config)#ntp server

The basic command set an IP or Hostname as the time-server. I used my PC, which I always keep on as the prefered source and added a couple of outside sources (I types in the names, the router translated to IP). In your production environment you should use any device that provide NTP services or an outside source. Make sure you allow UDP port 123 traffic between the routers and the NTP server.
Use the ntp command help for many more options, it is a small feature that allow great flexibility which you’ll need in a complex environment.

The next thing we’ll do is verify that we actually sync the time. In a router that was just installed (or turned on) you can use show clock and verify the time (unless you’re in 1993 :)). On a production router you can use either show ntp status or show ntp associations:

ITDualism1#show ntp associations

address            ref clock       st      when    poll   reach   delay   offset    disp
~       5       29      1024   377     4.2     -8.59     1.6
+~     3       69      128    377     4.1     3.48      2.3
*~     3       32      128    377     7.9     11.18     3.6
~          16        –      64     0       0.0     0.00      16000.
* master (synced), # master (unsynced), + selected, – candidate, ~ configured

One last note on the time and date methods. If your organization is multi time-zone make sure you set the correct time-zone per site or use UTC for all the routers. I set my clock to NYC time:

ITDualism1(config)#clock timezone EDT -5

Now that the time is synced I’ll configure logging. As I mentioned, time and have a major role when you troubleshoot your routers. When you log events and try to analyze the data your goal is to capture events that happen at the same time across the network and understand what was happening.

The best way to start is installing a Syslog server. I’ve used KIWI’s Syslog server for years but any other server would be just as good. Make sure you have the IP address of the server and go back to your console.

If you never used the logging command check the help option and take a look at my configuration:

ITDualism1(config)#logging trap
ITDualism1(config)#service timestamps log datetime
ITDualism1(config)#snmp-server enable traps config

The commands I used configured (my PC) as logging destination, allow trap and add timestamp that include date & time. The last command is a specific log for a selected protocol or in my case, configuration changes. This is just a sample of the different options and since there are many different parameters that should be considered when you build your routers (and switches) you should spend few minutes and ask yourself what is your goal, how much data you want to log and fine tune as you go.

Home Lab – login configuration

July 15, 2010 2 comments

Now that my lab is up and all the pieces connect, it is time to get off the console cable and the tight room at the lab area and relax on my sofa with great view to the AC 😉

In a production environment you always strive on max security and when it comes to routers remote access it is called ssh (if you know nothing about it, start here). Since my lab uses some older routers I have to compromise but lucky enough I enjoy both worlds and can show you how to configure ssh. Keep in mind that (like many other commands) syntax might be a bit different on different IOS versions (but the key components will always be the same).

Out of the 3 routers only my 2621 support ssh. I picked this order of commands to create some of the messages you might see on the way. I’ll explain each of them following the output.

ITDualism1(config)#ip ssh time-out 120
Please create RSA keys to enable SSH.

ITDualism1(config)#ip ssh authentication-retries 3
Please create RSA keys to enable SSH.

The first commands set the values for time-out (0-120) and authentication-retries (0-5). Though it prompted to create RSA keys, the commands did go through and the values I set are in the system. Now it’s time to create the key:

ITDualism1(config)#crypto key generate rsa usage-keys
% Please define a domain-name first.

As you can see, domain-name is prerequisite and the router couldn’t create the key.

ITDualism1(config)#ip domain-name ITDUALISM
ITDualism1(config)#crypto key generate rsa usage-keys
The name for the keys will be: ITDualism1.ITDUALISM
Choose the size of the key modulus in the range of 360 to 2048 for your
Signature Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]: 768
Generating RSA keys …
Choose the size of the key modulus in the range of 360 to 2048 for your
Encryption Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]:
Generating RSA keys …

Now that the domain-name is configured, I could create the key.  The reason I choose 768bit for the key is ssh2’s min requirement. At this point I don’t know if the router does support it but want to keep all the options. Generally speaking you should remember that higher bits offer better security but (like everything in life) it come with a cost: performance. Make your decision carefully and consider the hardware specs, expected load on the router and your security requirements. This is a good size selection document.

To complete this step I’ve used the show ip ssh command to verify both version and configuration:

ITDualism1#show ip ssh
SSH Enabled – version 1.5
Authentication timeout: 120 secs; Authentication retries: 3

As you can see, this router does not support ssh2 and the parameters I’ve set does show, though I got the messages when I typed them.

The next step is configuring remote access using ssh. Like access-list (and other features), configuring the parameters doesn’t mean we use the feature and we have to configure it on the required lines:

ITDualism1(config)#line vty 0 4
ITDualism1(config-line)#transport input all

In this case I allow access into vty 0 4 using all protocols, including ssh and telnet. I’ll show the other options with my 2611 routers. At this point ssh is ready and I was able to connect:

This is the standard message you expect to see when you connect via ssh the first time from a new computer. Clicking ‘Yes’ get you to the login screen and you’re good to go.

Older routers do not support ssh. Few ways to check it are shown below:

ITDualism2(config)#ip s?
sap  security  source-route  subnet-zero

The ssh command that we used on the 2621 router is not available.

ITDualism2(config-line)#transport input ?
all     All protocols
none    No protocols
pad     X.3 PAD
rlogin  Unix rlogin protocol
telnet  TCP/IP Telnet protocol
udptn   UDPTN async via UDP protocol
v120    Async over ISDN

Once again, you can see the list of all the available protocols for remote connection and ssh is not one of them.
The one last piece I want to show is how to configure the protocol for the line:

ITDualism2(config-line)#transport input all {all | none | pad | rlogin | | telnet | udptn |v120}
ITDualism2(config-line)#transport preferred telnet

In this example I allow using all protocols but configured telnet as the preferred protocol. The transport preferred setting specifies a search order when attempting to resolve names that might be valid for multiple protocols. If the address or service does not match the preferred protocol, all other valid output protocols are searched to find a valid match.

You can find some good reading from Cisco here, here and here. You might also want to look at this guide.

Home Lab – crashed router troubleshooting

July 10, 2010 3 comments

The spirit came sooner than expected and I spent this evening working on my crashed router.
If you missed my previous post, I started building my new home lab and one of the routers crashed. It’s a Cisco 2611 with 24MB of RAM and it got itself into a booting loop that look like this:

Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support:
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:23 by dchih


%Software-forced reload

00:00:06 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 700, PC = 8049A76C
-Traceback= 8049A76C 8049A71C 80476E5C 804A3890 8048B3C8 8048B67C 80581C00 80494954 804949F4 803AB260 803AB44C 8049C09C 8049F79C
File flash:crashinfo_19930301-000006 Device Error :No memory

=== Flushing messages (00:00:06 UTC Mon Mar 1 1993) ===

Queued messages:
*Mar  1 00:00:06.991: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:06.891: %SYS-2-MALLOCFAIL: Memory allocation of 12000 bytes failed from 0x804A388C, alignment 0
Pool: Processor  Free: 0  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= “Init”, ipl= 3, pid= 3
-Traceback= 80476224 80476E5C 804A3890 8048B3C8 8048B67C 80581C00 80494954 804949F4 803AB260 803AB44C 8049C09C 8049F79C
*** System received a Software forced crash ***
signal= 0x17, code= 0x700, context= 0x81446a98
PC = 0x8049a76c, Vector = 0x700, SP = 0x814b4938

System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.
TAC:Home:SW:IOS:Specials for info
C2600 platform with 24576 Kbytes of main memory

PC = 0xfff0a530, Vector = 0x500, SP = 0x80004374

monitor: command “boot” aborted due to user interrupt

Isn’t it nice? I can tell you it is very frustrating 😦
Using Ctrl+BREAK  I stopped the loop cycle and got into rommon. As I mentioned before that would let you reload the IOS file and get some details on your system. Since all the indications pointed at the memory I started with the meminfo command:

rommon 25 > meminfo

Main memory size: 24 MB.
Available main memory starts at 0x10000, size 24512KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

I wanted to take a look at the crash dump file. Using my TFTP server I loaded the file and looking at it I found no clues. Using Cisco’s Retrieving Information from the Crashinfo File as a quick reference I found no quick answers. There are some dump analyzing sites and a Cisco service if you have a TAC account. This is another good link from Cisco.

The next step was unscrew the box ->  remove the cover -> take a look at the memory sticks. I have 2 sticks: 16MB and 8MB
I’ve decided to remove one at a time and try to find which one cause the problem. I also removed the BRI module that was in the router, just incase.
Starting with the 16MB stick which I guess is newer, I removed the 8MB out and turned back the power. I’ve loaded rommon and using the set options loaded a new IOS file (taken off ITDualism2, a working Cisco 2611). this is how my console looked:

Receiving c2600-i-mz.122-46a.bin from !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!
File reception completed.
Copying file c2600-i-mz.122-46a.bin to flash.
Erasing flash at 0x607c0000
program flash location 0x60550000

So far so good, right?
Before I booted the router I wanted to make sure the memory and flash details:

rommon 24 > meminfo
Main memory size: 16 MB.
Available main memory starts at 0x10000, size 16320KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

rommon 27 > dir flash:
File size           Checksum   File name
5582584 bytes (0x552ef8)   0xbd0f    c2600-i-mz.122-46a.bin

When the router loaded the new IOS it got into an endless loop with the following error:

Unexpected exception to CPUvector 1100, PC = 0
-Traceback= 0 803EFB20 803EF57C 803EF958 803F0B7C

This is obviously not good and time to try the 2nd stick. Before I did that I tried to move the 16MB stick to the other slot, to make sure it is not a slot issue. The result was similar.
Now it was time to remove the memory stick and place in the 8MB stick. Same drill with rommon and copying the IOS, this is how it looked when the transfer completed:

rommon 23 > meminfo

Main memory size: 8 MB.
Available main memory starts at 0x10000, size 8128KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

rommon 24 > dir flash:
File size           Checksum   File name
5582584 bytes (0x552ef8)   0xbd0f    c2600-i-mz.122-46a.bin

When the router started I received a message indicating that it requires more memory to load the IOS. I’ve added the 16MB stick (now they where both in but in opposite slots) and restarted. Finally the router started, I was able to login and even the old config file loaded.
My lab is ready and I can start configuring different scenarios.

Home Lab

July 9, 2010 1 comment

Amazing as it sound it’s been a month since I started my new job and while I learn new technologies and deepen my knowledge, my Cisco knowledge is not being used (yet?). I’m happy and know I’m in a good place and did the right move. BUT (every good story have one) since I enjoy Ciscoing and want to keep my skills at a good level, I’ll try spending few hours every weekend on my new home lab.

Unlike the lab I used for my CCNP exam, this is a smaller basic lab built of the following:

1. ITDswitch @ VLAN1
cisco WS-C2924-XL
using IOS ™ C2900XL Software (C2900XL-C3H2S-M), Version 12.0(5.3)WC(1)

2. ITDualism1 @
cisco 2621
using IOS ™ C2600 Software (C2600-IK9S-M), Version 12.2(6)

3. ITDualism2 @
cisco 2611
using IOS ™ C2600 Software (C2600-I-M), Version 12.2(46a)

4. ITDualism3 @
cisco 2611
using IOS ™ C2600 Software (C2600-I-M), Version 12.3(26)
This is the router I mentioned in my last post. This router keep crashing and will get its own troubleshooting post when I’ll have the time and spirit.

The switch and routers will join my “production” home network. I’m using my Netgear wireless router as default gateway and Internet access point. In addition I have a PC, 2 laptops, 2 iPhones and a Wii console – all of which I’ll use to test\play\drive my wife nuts 😉

As I said in my opener, the goal here is to practice my skill and keep the rust away. I do not have tons of time for this as I work longer hours and spend additional time going over work related materials at night (plus the fact that my brain is blasted at work, it’s a matter of time but I’ll get used to being busy and thinking ALL day).

Feel free to through in lab ideas and let me know if you’re interested in non-networking topics (I have some interesting production stories) as an extra. Reading the comments and personal messages I get the feeling most of you are into networking, so this is the time for the rest of you to raise your voices 🙂

Xmodem – do you remember how slow it is?

July 5, 2010 3 comments

After a long time I had a chance to use my home lab. I have a 2600 router that was waiting for a while to be added to the lab setup. The end of this long weekend was a great opportunity to reconnect to the Cisco in me and get some work done.

I’ve connected to the router using a console cable and this is what I got on-screen:

System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.
TAC:Home:SW:IOS:Specials for info
C2600 platform with 24576 Kbytes of main memory

boot: cannot determine first file name on device “flash:”

I found that the IOS file was deleted and had to load a new IOS file. Since I was already on console I used the Xmodem command, using the following syntax:

rommon 1 > xmodem -c c2600-i-mz.123-26.bin

This is the output that follow my command (asking for my approval):

Do not start the sending program yet…
File size           Checksum   File name
4487708 bytes (0x447a1c)   0x8f40    c2600-i-mz.121-12.bin (deleted)

WARNING: All existing data in bootflash will be lost!
Invoke this application only for disaster recovery.
Do you wish to continue? y/n  [n]:  y
Ready to receive file c2600-i-mz.123-26.bin …

Now I had to send the file to the router and wait. The one thing I forgot is how slow this copy process run. Check the screen shot:

Yes, unbelievable 810 cps which translate to (about) 2:40 hours for a 7.5mb file!

Since it is totally insane I switched to the IP system. I used my Netgear wireless router and plugged the router to one of the 4 ports on the back. Since my lab PC is on the same network all I had to do is pick up an unused IP and assign it to the router.
This is a good reminder for anyone who’s got to use tftp via rommon:

rommon 6 > set
rommon 7 > IP_ADDRESS=
rommon 8 > IP_SUBNET_MASK=
rommon 10 > TFTP_SERVER=
rommon 11 > TFTP_FILE=c2600-i-mz.123-26.bin
rommon 12 > tftpdnld

TFTP_FILE: c2600-i-mz.123-26.bin

Invoke this command for disaster recovery only.
WARNING: all existing data in all partitions on flash will be lost!
Do you wish to continue? y/n:  [n]:  y

Receiving c2600-i-mz.123-26.bin from !!!!!.!!!!!!!!!!!!!!.!!!!
File reception completed.
Copying file c2600-i-mz.123-26.bin to flash.
Erasing flash at 0x607c0000
program flash location 0x60440000
rommon 13 >

You can imagine how fast this process completed…