It was my task to replace old Cisco 2811 Routers with a new generation. The WAN connection at one branch office had been upgraded to 50 Mbit, but the crypto performance of the old router was too low for the bandwidth. We decided to use the Cisco 4331 with the SM-X EtherSwitch Service Module (SM-X-ES3-16-P ).
Until I had a running system, there were some major challenges I had to deal with. Since there was not that much documentation available and not so many public support cases that had been solved, I will present my experiences here to help others to avoid the problems.
The first problem I had, was to find the right image. In the download section on the cisco.com page there is no section for the SM-X-ES3-16-P. Instead I had to take an image from the 3560-X series.
Unfortunately the image I downloaded (c3560e-universalk9-mz.152-4.E1) lead to my first bug. After every reboot of the switch, interface vlan 1 was in shutdown state. The switch commented this with following output:
SETUP: new interface vlan 1 placed in „shutdown“ state.
This seems to be a bug that was fixed in version c3560e-universalk9-mz.152-4.E2 . After I installed this version, the output was still there but vlan 1 went directly to the up state. However before I used this release, I tried the suggested release c3560e-universalk9-mz.150-2.SE9 . Unfortunately the switch does not boot with this release and keeps rebooting. The only way to interrupt the boot process is to force a password recovery from the router!:
hw-module subslot 1/0 error-recovery password_reset
The switch then gets to the rommon and you can decide which image to boot.
The next goal I wanted to achieve was the communication between the switch and the router. You can use the Ethernet-Internal interfaces for this. In the documentation of the switch module you can only find a difficult configuration using BVI interfaces. With recent images Cisco implemented a new command so that you can create SVIs on the router. The command is:
ethernet-internal subslot 1/0
platform switchport svi
After entering this command you have to reboot the router. When the router is up again, you are able to create SVIs with the command “Interface vlan 1” and use the internal ports as trunk ports. This has been described in this post: supportforums.cisco.com
Then I encountered a problem caused by our configuration. We use dhcp snooping in combination with ip source guard. For this combination it is necessary to use dhcp snooping with option 82. All our other switches are connected to catalyst switches, which act as layer 3 switches. On them you can define the trunk ports as trusted (“ip dhcp snooping trust”). On the router however this command is not available. So my dhcp requests were not redirected by the “ip helper address” command because of the option 82 information. Turning off option 82 allowed the dhcp requests to reach the dhcp server, but the dhcp replies then got stuck at the switch because without the option 82 information it does not know to which interface it should forward the packet. To solve this I activated option 82 again and on the router I entered following command:
ip dhcp relay information trust-all
This is the router equivalent to the “ip dhcp snooping trust”.
But still my setup was not working. No traffic was going through the internal ports after I reentered my full configuration. The reason for this seems to be another bug. I had activated rapid spanning tree “spanning-tree mode rapid-pvst” (while using default STP the problem did not occur). With the command “show spanning-tree” on the switch I found that the Gi0/18 was in “P2p Dispute” state. Comparing the output from switch and router I saw that both claimed to be the root bridge and had the same priority. However the mac address of the switch was lower so the behavior of the switch was totally correct. I fixed this by lowering the priority of the router and making it root bridge:
spanning-tree vlan 1-4094 root primary
Finally everything was working, but the debugging and searching for information had cost me several hours. I hope I can help others avoid having the same problems and getting stuck.