At a customer I’ve recently had to commit a grave operational sin; to connect a small switch at the end of a floor patch. These things are normally operational nightmares as they have a tendency to quickly bring an entire LAN environment down to its knees when such a ‘switch’ is connected to the network twice. Always by accident but having management kick you for something someone else did is not anyone’s idea of fun. I won’t go into the underlying principles here as I’m assuming most who frequent my blog will know about broadcast storms, their causes and the tools and solutions available to mitigate the risks.

Our justification to operations was that we wanted a few more local LAN ports to test VoIP devices on than we had available through floor patches. As such I reasoned with Operations that this was a calculated choice to segregate our testing from the rest of the LAN but still make it as realistic as possible. Using the means available meant that I had to make do with a Cisco 1801. Single routed and 8 switched interfaces. Think of it as a router with one Ethernet interface and an 8 port HWIC-ESW nailed to it. Didn’t need the ATM or WiFi it has.

So I set out, disabling IP routing, admin down all non-Ethernet ports. set up the vlan database -old style, remember?-; I did not want this baby to participate in VTP, in fact I don’t think it even can! It’s limited to 8 vlans. Pulled two cables to it. One switched port as trunked with some data and voice vlans and configured the routed interface for management access.

All sweet and dandy, tested the BPDU-guard functionality prior to installation by connecting an access-port to the LAN. Clunk! it went down as desired, result I thought… Then when installing the LAN wouldn’t bring up the LAN port. Doh! I’d missed that the 1801 doesn’t send BPDU’s until a VLAN becomes active. I’d checked if spanning-tree was operational, and it wasn’t until I brought an interface up. So I disabled STP for all vlans in the VLAN database. Now my laptop received an IP address and the data VLANs all worked.

So, time to connect a Mitel phone. No dice, it received it’s first DHCP response with VLAn information, then it would just sit ennuncing it was waiting for a DHCP response. Dang, I’d configured the voice vlan so why did the switch not detect the phone, enable trunking so that the phone could send it’s DHCP request on the voice VLAN?

It was only when I started reading up on HWIC-ESW voice-VLAN config I noticed that Cisco hasn’t implemented the auto enable of dot1q trunking when a phone is detected… The solution is to add two lines of code; “switchport truck native vlan xyz” and “switchport mode trunk”. The crux is that this platform is at heart a router, not a native switch…

Cisco documentation

However, it’s common knowledge that many providers use Alcatel and they seem to do pretty well in the ‘booming’ broadband market. Hence I thought I’d share a little snippet of an annoyance I recently encountered.

When using an Alcatel 7210 to sniff traffic and interconnect different media; 1Gbps copper and 10Gbps fibre. I found that sniffing is counter intuitive to people only trained on Cisco. A few pointers:

1. Port mirror destinations are defined in configuration
2. Port mirror sources are set through debug commands
3. When mirroring VPLS ports (I needed an e-pipe/Layer-2 tunnel) I found that egress sources did not work, only ingress did and only one ingress port can be set per mirror session. It did not matter if I use the port or the SAP as source.

I was left to sniff in two places to capture both up- & down-stream traffic. YMMV as a 7750 will be different, but I don’t have one available to me to test on…

Commands used:

#--------------------------------------------------
echo "Mirror Configuration"
#--------------------------------------------------
  mirror
    mirror-dest 4 create
      sap 1/1/4 create
      exit
      no shutdown
    exit
    mirror-dest 11 create
      sap 1/1/11 create
      exit
      no shutdown
    exit
  exit

And the debug command:

*A:<hostname># debug mirror-source 4 port ?
- no port ...
- port <port-id> egress ingress
- port <port-id> egress
- port <port-id> ingress
- port lag ...

*A:<hostname># debug mirror-source 4 sap ?
- no sap <sap-id> [ingress]
- sap <sap-id> {[ingress] }

As can be seen above capturing by SAP is only supported at ingress. Using port and SAP yielded the same result, only ingress packets were ever sent to the destination port. Despite show mirror stating both Egr & Ing.

*A:<hostname># show mirror mirror-dest 11

===============================================================================
Mirror Service
===============================================================================
Service Id       : 11                   Type          : Ether
Description      : (Not Specified)
Admin State      : Up                   Oper State    : Up
Forwarding Class : be                   Remote Sources: No
Slice            : 0
Destination SAP  : 1/1/11               Egr QoS Policy: 1

-------------------------------------------------------------------------------
Local Sources
-------------------------------------------------------------------------------
Admin State      : Up

-Port                                   1/1/26                          Egr Ing
===============================================================================

A quick refresh for all:

1. PE physical link(s), ties one VRF to another using one subnet for each VRF. Commonly deployed using vlans across a DOT1q trunk (anyone still using ISL? ok, ok a tagged trunk but don’t tell me I’m not allowed to say “trunk”). Possible to use routing protocols between each individual VRF, most commonly staticly routed as it’s the safest model, the other side isn’t trusted…
2. PE MP-eBGP, ties the two MPLS clouds together exchanging VPN labels. BGP can be used to exchange the vpn labels, no need for tag-switching of IP traffic. Requires the inter- link to be known in both IGPs as the next-hop changes, only the vpn label is preserved end-to-end.
3. P tag-switching, joins the two domains together at the lowest level. The inter-link can be between P devices all labels are preserved end-to-end. Even the IGPs must have ‘full’ connectivity, of course I mean PE-to-PE…

Type-2 seemed to make most sense due to the amount of vrfs involved, no need to go for type-3 and type-1 would add way too much complexity to be able to support a multi month long migration.

Now for the MTU issue. The old MPLS cloud uses 3600′s as PE’s in the relevant sites and the new network uses 7600′s. Admittedly a bit slow on the ball -mind you I was not the original designer- we implemented an mtu of 1536 on the sup720 GE based infrastructure links on the new network. However the vpn label on the 4 FE inter-AS/MP-eBGP links was forgotten untill a service using TCP keepalives was partially migrated. Sessions flapped due to the keepalives being 1500 bytes in size and having their df-bit set. I wonder why they used TCP but as of yet I’m left guessing.

The moment we noticed the error of our ways we found that IOS threw us a few curve balls:

1. A 3600 running 12.3 doesn’t accept anything over 1500 as mtu on a FastEthernet interface
2. A 7600 with 6748-GE (LAN interfaces) doesn’t allow a larger than 1500 mtu size either (routed interface)

The short answers are the following:

1. On the 3600: tag switching mtu 1508
2. On the 7600: mpls mtu 1508

fyi, 1504 would have done fine as an mpls label is just 4 bytes we just threw in another 4 bytes for luck

Some links to back things up:

• Cisco MTU config for 7600 12.2SR & layer3 ports
• Cisco IOS hints and tricks: The tale of three MTUs
• You’ll have to take my word for this quote from Cisco: “I can confirm that using tag-switching mtu at 1508 on the 3600 should resolve the issue from that side” (despite not being able to set the interface mtu higher than 1500)

Command to verity mpls mtu:

sh mpls [intf] det

My first google tonight led me to an article on the excellent ios hints and tricks site (ioshints.info). Though this is for a full public DNS server it was a little overkill for me, please read this article as well if you do plan to go that route.

My goal was to have my DSL router serve local IP’s for a few public and local domain names. For example this allows me to use sip.djerk.nl as my proxy address in my soft-phone both at home and away.

The following code is what I ended up with.

# These hostnames will be served locally
# As such they overrule their public dns
# clients do need to use this router as a dns server
ip host view VoIP sip.djerk.nl 192.168.100.130
ip host sip.djerk.nl 192.168.100.130
ip host c877 192.168.100.1

interface Vlan10
description *** VoIP LAN ***
ip dns view-group VoIP

# For my Voice vlan
ip dns view VoIP
domain name djerk.nl
# search list for this view
domain list djerk.nl
domain name-server x.x.x.x
domain name-server y.y.y.y
domain round-robin

# Default dns handling
ip dns view default
domain name geurtscass.com
domain list geurtscass.com

ip dns view-list VoIP
view VoIP 1
view default 100

# Enable local dns server (proxy dns if no domains are configured here)
# I do not allow inbound dns requests (intf ACL) so no need to worry about abuse
ip dns server

Most interesting is the usage of views and a view-list. According to CCO the view-list is needed to bind a view to an interface, views can’t be assigned directly. Also noteworthy is that I’ve not found a way to reduce the sip.djerk.nl host mapping to a single line, as adding the default view to the VoIP view-list was not enough. Maybe I should add the VoIP view to the default view-list.

Well just tried that and it didn’t work either. Anyone out there who knows how to do the same in IOS but with less lines?

Well my working config turned out to be:

crypto keyring Daniel
pre-shared-key address 1.2.3.4 key s3cr3t
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp client configuration group Daniel-grp
key s3cr3t
crypto isakmp profile Daniel
keyring Daniel
match identity address 1.2.3.4 255.255.255.255
crypto isakmp profile Daniel-ez
match identity group Daniel-grp
!
crypto ipsec transform-set tset1 esp-3des esp-sha-hmac
!
crypto map Daniel 10 ipsec-isakmp
set peer 1.2.3.4
set transform-set tset1
set isakmp-profile Daniel
match address 102
!
Interface Dialer 1
crypto map Daniel
!
access-list 102 permit ip 192.168.100.0 0.0.0.255 192.168.1.0 0.0.0.255
!
ip route 192.168.1.0 255.255.255.0 1.2.3.4 name Daniel

Also I had to prevent traffic destined for the ipsec tunnel being natted, this meant adding a simple deny line in the NAT ACL. 1.2.3.4 is the remote site IP address, 192.168.100.0/24 my local LAN, 192.168.1.0/24 the remote LAN. As opposed to the ezvpn tunnel I tried earlier this tunnel is dynamic and a static route is required rather than routes being added dynamically. Personally I think ezvpn can be quite cool for remote access (read client access) but I have sslvpn working fine for that, which was much easier to set up to boot.

It appears that Debian Lenny by default only binds to 127.0.0.1 (localhost). I had to add the following two lines to snmpd.conf to get snmpd to listen to external requests.

interface eth0
agentaddress 217.195.248.251:161

I eventually found the solution at debianhelp.org.

I debugged and pinged, even installed hping3 to do UDP pings. I don’t want to run cacti as root, especially not on a vhost. So the UDP ping had to work. The pings arrived but still no replies.

Getting sidetracked I noticed that the one router that did work was being hit by SSH login attempts and it’s cpu was spiking. An ACL took care of the break-in attempts but then I noticed that directed broadcasts were made to my server’s segment. So I nailed that down plus proxy-arps when I noticed that the router which had worked before now was causing errors in Cacti as well.

Tracking back I noticed that the UDP ping ‘replies’ were unreachables rather than ICMP replies (doh, how obvious!) . I enabled IP unreachables on both routers again and I was done. It’s amazing how blind one can be at times to the blatantly obvious…

ip route prefix mask {ip-address | interface-type interface-number [ip-address]} [dhcp] [distance] [name next-hop-name] [permanent | track number] [tag tag]

Static routes that point to an interface on a connected router will be advertised by way of Routing Information Protocol (RIP) and EIGRP regardless of whether redistribute static commands are specified for those routing protocols.

This situation occurs because static routes that point to an interface are considered in the routing table to be connected and hence lose their static nature. Also, the target of the static route should be included in the network (DHCP) command. If this condition is not met, no dynamic routing protocol will advertise the route unless a redistribute static command is specified for these protocols

The following is also good to know, I’ve noticed it’s source of common misunderstanding:

Specifying a numerical next hop that is on a directly connected interface will prevent the router from using proxy ARP. However, if the interface with the next hop goes down and the numerical next hop can be reached through a recursive route, you may specify both the next hop and interface (for example, ip route0.0.0.0 0.0.0.0 ethernet 1/2 10.1.2.3) with a static route to prevent routes from passing through an unintended interface.

Source  (UniverCD)

I’m counting on being there, though untill my manager gives approval and it’s been booked I will not know fur sure…

If you’re going as well and want to meet me then drop me a line, with a suggestion of when you’re free. I don’t think I’ll be able to intentionally run into anyone there, my schedule is way too busy for that.

Suggestions for surviving Networkers are welcome as I’ve never been before.