Home > LAB > Home Lab – crashed router troubleshooting

Home Lab – crashed router troubleshooting

The spirit came sooner than expected and I spent this evening working on my crashed router.
If you missed my previous post, I started building my new home lab and one of the routers crashed. It’s a Cisco 2611 with 24MB of RAM and it got itself into a booting loop that look like this:

Cisco Internetwork Operating System Software
IOS ™ C2600 Software (C2600-I-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:23 by dchih

SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!

%Software-forced reload

00:00:06 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 700, PC = 8049A76C
-Traceback= 8049A76C 8049A71C 80476E5C 804A3890 8048B3C8 8048B67C 80581C00 80494954 804949F4 803AB260 803AB44C 8049C09C 8049F79C
File flash:crashinfo_19930301-000006 Device Error :No memory

=== Flushing messages (00:00:06 UTC Mon Mar 1 1993) ===

Queued messages:
*Mar  1 00:00:06.991: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:06.891: %SYS-2-MALLOCFAIL: Memory allocation of 12000 bytes failed from 0x804A388C, alignment 0
Pool: Processor  Free: 0  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= “Init”, ipl= 3, pid= 3
-Traceback= 80476224 80476E5C 804A3890 8048B3C8 8048B67C 80581C00 80494954 804949F4 803AB260 803AB44C 8049C09C 8049F79C
*** System received a Software forced crash ***
signal= 0×17, code= 0×700, context= 0x81446a98
PC = 0x8049a76c, Vector = 0×700, SP = 0x814b4938

System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.
TAC:Home:SW:IOS:Specials for info
C2600 platform with 24576 Kbytes of main memory

PC = 0xfff0a530, Vector = 0×500, SP = 0×80004374

monitor: command “boot” aborted due to user interrupt

Isn’t it nice? I can tell you it is very frustrating :(
Using Ctrl+BREAK  I stopped the loop cycle and got into rommon. As I mentioned before that would let you reload the IOS file and get some details on your system. Since all the indications pointed at the memory I started with the meminfo command:

rommon 25 > meminfo

Main memory size: 24 MB.
Available main memory starts at 0×10000, size 24512KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

I wanted to take a look at the crash dump file. Using my TFTP server I loaded the file and looking at it I found no clues. Using Cisco’s Retrieving Information from the Crashinfo File as a quick reference I found no quick answers. There are some dump analyzing sites and a Cisco service if you have a TAC account. This is another good link from Cisco.

The next step was unscrew the box ->  remove the cover -> take a look at the memory sticks. I have 2 sticks: 16MB and 8MB
I’ve decided to remove one at a time and try to find which one cause the problem. I also removed the BRI module that was in the router, just incase.
Starting with the 16MB stick which I guess is newer, I removed the 8MB out and turned back the power. I’ve loaded rommon and using the set options loaded a new IOS file (taken off ITDualism2, a working Cisco 2611). this is how my console looked:

Receiving c2600-i-mz.122-46a.bin from 192.168.1.6 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!
File reception completed.
Copying file c2600-i-mz.122-46a.bin to flash.
Erasing flash at 0x607c0000
program flash location 0×60550000

So far so good, right?
Before I booted the router I wanted to make sure the memory and flash details:

rommon 24 > meminfo
Main memory size: 16 MB.
Available main memory starts at 0×10000, size 16320KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

rommon 27 > dir flash:
File size           Checksum   File name
5582584 bytes (0x552ef8)   0xbd0f    c2600-i-mz.122-46a.bin

When the router loaded the new IOS it got into an endless loop with the following error:

Unexpected exception to CPUvector 1100, PC = 0
-Traceback= 0 803EFB20 803EF57C 803EF958 803F0B7C

This is obviously not good and time to try the 2nd stick. Before I did that I tried to move the 16MB stick to the other slot, to make sure it is not a slot issue. The result was similar.
Now it was time to remove the memory stick and place in the 8MB stick. Same drill with rommon and copying the IOS, this is how it looked when the transfer completed:

rommon 23 > meminfo

Main memory size: 8 MB.
Available main memory starts at 0×10000, size 8128KB
IO (packet) memory size: 10 percent of main memory.
NVRAM size: 32KB

rommon 24 > dir flash:
File size           Checksum   File name
5582584 bytes (0x552ef8)   0xbd0f    c2600-i-mz.122-46a.bin

When the router started I received a message indicating that it requires more memory to load the IOS. I’ve added the 16MB stick (now they where both in but in opposite slots) and restarted. Finally the router started, I was able to login and even the old config file loaded.
My lab is ready and I can start configuring different scenarios.

About these ads
  1. July 12, 2010 at 8:30 pm | #1

    Well I’m glad that your router is back to life! Errors like that are always really weird. I’ve dealt with a lot of strange errors like that for a number of people, they always ask me how I fixed it, and I just have to shrug my shoulders and say, “I don’t know”. But it always feels good to have things working again.

  1. July 10, 2010 at 10:51 pm | #1
  2. July 13, 2010 at 10:45 am | #2

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 44 other followers

%d bloggers like this: