Archive for the 'Work' category

Cisco LACP config for Aruba AP

 | 8 Jul 2015 17:30

Aruba LogoDon’t we all love it when we find that a standard requirement states one thing and what to date is implemented elsewhere doesn’t comply? Dual active uplinks for a premium office standard is one of those requirements I found. Now I haven’t seen the standard Cisco wireless deployment for premium sites, but in light of vendor ‘diversity’ Aruba is deployed instead of Cisco.

Motivation aside, the dual uplink raises an interesting question for lightweight access points (LWAPs). Aruba (by default) GRE tunnels all client traffic to the wireless LAN controller (WLC) for processing, filtering and forwarding there, like Cisco and common in corporate environments. The alternatives are split tunneling or no tunneling, which normally comes at the cost of losing corporate controls. The QoS trade-off and headaches of tunneling WLAN traffic to WLCs is food for another post entirely.


Using AP225 APs, I found I had LACP at my disposal. Cheaper models (< AP220) don’t do LACP and only have STP for redundancy. Some of my first concerns:

  • Standard Cisco LACP is mostly configured unconditional, which means the ports don’t come up if LACP isn’t detected on the link. How is an AP meant to get its profile from a WLC if it can’t get there. Remember I don’t want to reconfigure the switch ports after an AP has connected and obtained its profile (configuration) from the WLC.
  • Aruba documentation and forums (Airheads) didn’t list much configuration about Cisco switch port configuration. What I did find was that LACP is supported and needs switch configuration for it to work.
  • A single GRE tunnel using 2 etherchannel members?! LACP uses an IP hash table to select which member link to forward packets on. An AP only has a single IP address and without LACP the WLC also only has a single IP address for termination of LWAP GRE tunnels. Surely all GRE tunnels would only use a single LACP bundle-member, restricting maximum throughput to 1 Gbps. If so, what’s the point?

Reading up I found the following helpful information:

  • Aruba solves the LACP IP hash table problem by using a second WLC IP address to terminate a second GRE tunnel. This second tunnel uses the 2nd member-link. Each GRE tunnel serves a radio, 2.4GHz and 5GHz, this does not enable more than 1 Gbps for 5GHz but at least 2.4GHz traffic won’t eat into the uplink speed available to 5GHz traffic. The Aruba config for LACP centres around “AP LACP GRE striping IP” (see Google for more info).
  • “no port-channel standalone-disable”, this port-channel configuration gem permits link members to come up as individual links. This allows a LWAP to connect to the network, get an IP via DHCP, find the WLC and pull its configuration. Once provisioned by the WLC LACP kicks in.


Beware of the LACP hash algorithm, Cisco switch default is src-mac. In an edge-routed design the source-mac will be the mac of the switch SVI towards the WLC. The Switch terminating the LWAPs is the same as the one terminating the WLC and the WLC also uses LACP to connect to the LAN. For my deployment the solution was src-ip as the GRE sessions towards the LWAPs have a distinct WLC IP address (must be odd/even). Traffic destined for the WLC is also src-ip based, which is good as the load-balancing will then be based on the targets of the clients whether internet or LAN based it works as long as corporate clients don’t all hit the same target at the same time. I think is most situations the resulting total bandwidth restriction of a single LAN source towards wireless clients at 1 Gbps is beneficial to the fair sharing of bandwidth between LAN based services.

The AP225 only pulls PoE over a single link. If the link providing PoE goes down it will reboot and come up one the remaining link.

Though the dual links provide extra bandwidth, if the your NOC doesn’t monitor these links either via WLC management or switch trap/port monitoring, a single link failure won’t be noticed. I think this is no different to the issue of APs losing their physical link and continuing in mesh connectivity, which is great as a last resort but not when the situation isn’t resolved before things get really bad.

Cisco config

This is the LWAP switch port config that worked for me:

WLAN-SW01(config)#int range g1/0/1,g2/0/1
 description WLAN-AP01
 switchport access vlan 4
 switchport mode access
 channel-group 1 mode active
WLAN-SW01(config)#int po1
 description WLAN-AP01
 switchport access vlan 4
 switchport mode access
 no port-channel standalone-disable
WLAN-SW01#sh eth 1 sum
Flags: D - down P - bundled in port-channel
       I - stand-alone s - suspended
       H - Hot-standby (LACP only)
       R - Layer3 S - Layer2
       U - in use f - failed to allocate aggregator

       M - not in use, minimum links not met
       u - unsuitable for bundling
       w - waiting to be aggregated
       d - default port
Group  Port-channel  Protocol    Ports
1      Po1(SU)         LACP      Gi1/0/1(P) Gi2/0/1(P)

When the LWAP hasn’t fetched it’s configuration the Flags show either (D) for down or (I) when the port is up but LACP is inactive. As long as LACP is inactive the APs MAC address will hop between the two ports and a MAC flap warning is reported by the switch.

Jul  8 2015 08:33:59.259 UTC: %SW_MATM-4-MACFLAP_NOTIF: Host 94b4.0f50.47f0 in vlan 4 is flapping between port Gi2/0/1 and port Gi1/0/1

Another error I’ve seen is about PoE. What happens is that both member ports offer PoE but the AP only signals acceptance on a single port. The switch doesn’t seem to understand the lack of response, calls the AP rude, turns off PoE on that port and logs the ‘error’.

Jul 8 2015 17:08:39.030 UTC: %ILPOWER-7-DETECT: Interface Gi1/0/2: Power Device detected: IEEE PD
Jul 8 2015 17:08:41.202 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi2/0/2: PD removed
Jul 8 2015 17:08:41.203 UTC: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi2/0/2: Power given, but Power Controller does not report Power Good
Jul 8 2015 17:08:41.885 UTC: %ILPOWER-7-DETECT: Interface Gi2/0/2: Power Device detected: IEEE PD
Jul 8 2015 17:08:42.995 UTC: %ILPOWER-5-POWER_GRANTED: Interface Gi2/0/2: Power granted
Jul 8 2015 17:08:50.035 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Jul 8 2015 17:08:50.187 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Jul 8 2015 17:08:55.025 UTC: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/2: PD removed

WLAN-SW01#sh power inline
Module Available Used Remaining
 (Watts) (Watts) (Watts)
------ --------- -------- ---------
1 1110.0 200.2 909.8
Interface Admin  Oper       Power   Device              Class Max
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/1   auto   on         15.4    Ieee PD             4     30.0
Gi2/0/1   auto   off        0.0     n/a                 n/a   30.0

Check LACP from the WLC

Some great LACP related WLC CLI tools I found on Airheads:

Check if GRE striping IP has been set: “show ap system-profile ”

(WLAN-WLC01) #show ap system-profile LACP

AP system profile "LACP"
Parameter Value
--------- -----
RF Band g
RF Band for AM mode scanning all
Backup LMS IP N/A
Backup LMS IPv6 N/A
LMS Preemption Disabled
LMS Hold-down Period 600 sec
LMS ping interval 20
GRE Striping IP

Check the if an APs LACP has come up: “show ap debug lacp ap-name ”

(WLAN-WLC01) #show ap debug lacp ap-name WLAN-AP01

AP LACP Status
Link Status  LACP Rate  Num Ports  Actor Key  Partner Key  Partner MAC
-----------  ---------  ---------  ---------  -----------  -----------
Up           slow       2          17         1            88:90:8d:d9:b8:00
Slave Interface Status
Slave I/f Name  Permanent MAC Addr  Link Status  Member of LAG  Link Fail Count
--------------  ------------------  -----------  -------------  ---------------
eth0            94:b4:0f:c2:83:b2   Up           Yes            0
eth1            94:b4:0f:c2:83:b3   Up           Yes            0

Check if GRE tunnels are being created to both the switch IP address and the GRE stripping IP address configured in the AP system profile: “show datapath session | include ”

(WLAN-WLC01) #show datapath session | include "30.29"
...   17   4500  4500   0/0  0    0   0   pc1         119  70         71872      FC     47   0     0      0/0  0    0   1   pc1         c    0          0          FC           

There you have it, LACP between an Aruba AP and a Cisco switch. Kudos to Abi’s over at Airheads for this article about LACP on the Aruba AP225 and AirOS 6.3. I was working on 6.4, ymmv with different versions.

Temporary London CCIE LABs

 | 25 Jun 2014 15:23

Just received an email about Cisco’s mobile CCIE LAB coming to London:

Mobile CCIE Lab Available in London, United Kingdom, from October 6 to October 14, 2014

To address the urgent need for certified IT professionals, and to offer more convenient testing, Cisco has developed the Mobile CCIE Lab for qualified candidates who are ready to take their CCIE Routing and Switching exam or CCIE Security exam.

We encourage you to take advantage of the mobile lab scheduled in London, United Kingdom, from October 6 to October 14, 2014. The CCIE mobile testing lab will allow qualified candidates to more easily and quickly take the exam, reducing the waiting time, effort, and costs accrued by having to travel to take the exam. The eight-hour lab exam tests your ability to configure actual equipment and get the network running in a timed-test situation. There will be 42 seats available for the CCIE Routing and Switching exam and 7 seats available for the CCIE Security exam.

Apart from the Cisco Certified Architect (CCAr) certification, the Cisco CCIE certifications are the highest level of achievement for network professionals. Less than 3 percent of all Cisco certified professionals earn their CCIE certification.

Click here to register for the CCIE Routing and Switching exam or CCIE Security exam in London.

For information on registering for a Mobile CCIE Lab event or for additional information about the Mobile CCIE Lab program, visit the Cisco Learning Network.

SSL Intercept headaches

 | 17 Jun 2014 22:50

BlueCoat Logo

A recent proxy upgrade, has seen me working many hours – fixing things that weren’t broken before. It was intended to be a drop-in replacement, but somebody couldn’t resist the opportunity to specify ‘a few minor’ new requirements.

  • 1 year log retention of all traffic
  • SSL interception to enable data leakage protection for all traffic types

The first doesn’t sound like a big issue, however it turned our we underestimated the logging volume for 8000 concurrent users. Additionally the reseller hadn’t flagged the issue either, I’m ‘sure’ they’ll pay more attention next time… As for SSL interception. It broke a host of things. Some lessons learned:

  • Bluecoat ProxySG devices come with root CA certificates installed. Many site-admins using SSl fail to install the intermediate certs which slows down session set-up but also means we had to install many intermediates as the proxy does not go looking for them. This means manually finding and installing certs based on users calling the help-desk because they weren’t allowed to access sites with untrusted certificates.
  • Commercial sites using self signed certificates. Bad practise, but sadly it’s not always up to engineers/consultants whether or not such a site should be honoured with a business critical status or not.
  • Applications tunneling proprietary protocols over TCP:443. Some encrypted, some not so much. The ProxySG was configured to detect the protocol and to deny all unrecognised traffic. This breaks Adobe Creative Cloud for example. Skype is another hot-potato.

Skype in particular proved to be a big time-waster. As you may well know Skype uses proprietary protocols and tries very hard to remain hidden from prying eyes. As Skype was an application that was in use before the migration and the ‘as-is’ rule lingered, there was some pressure to get Skype working. The short is that I got it working without globally turning off SSL Intercept, well – to a degree anyway…

Show me more… »

Cisco Voice-VLAN (VVLAN) inconsistencies

 | 12 Nov 2012 12:41

First off I’d like to say that this is just a minor issue, more relating for routers versus switch, I’m still a lot happier at how Cisco implements config and features as opposed to most if not all of their competitors…

At a customer I’ve recently had to commit a grave operational sin; to connect a small switch at the end of a floor patch. These things are normally operational nightmares as they have a tendency to quickly bring an entire LAN environment down to its knees when such a ‘switch’ is connected to the network twice. Always by accident but having management kick you for something someone else did is not anyone’s idea of fun. I won’t go into the underlying principles here as I’m assuming most who frequent my blog will know about broadcast storms, their causes and the tools and solutions available to mitigate the risks.

Our justification to operations was that we wanted a few more local LAN ports to test VoIP devices on than we had available through floor patches. As such I reasoned with Operations that this was a calculated choice to segregate our testing from the rest of the LAN but still make it as realistic as possible. Using the means available meant that I had to make do with a Cisco 1801. Single routed and 8 switched interfaces. Think of it as a router with one Ethernet interface and an 8 port HWIC-ESW nailed to it. Didn’t need the ATM or WiFi it has.

So I set out, disabling IP routing, admin down all non-Ethernet ports. set up the vlan database -old style, remember?-; I did not want this baby to participate in VTP, in fact I don’t think it even can! It’s limited to 8 vlans. Pulled two cables to it. One switched port as trunked with some data and voice vlans and configured the routed interface for management access.

All sweet and dandy, tested the BPDU-guard functionality prior to installation by connecting an access-port to the LAN. Clunk! it went down as desired, result I thought… Then when installing the LAN wouldn’t bring up the LAN port. Doh! I’d missed that the 1801 doesn’t send BPDU’s until a VLAN becomes active. I’d checked if spanning-tree was operational, and it wasn’t until I brought an interface up. So I disabled STP for all vlans in the VLAN database. Now my laptop received an IP address and the data VLANs all worked.

So, time to connect a Mitel phone. No dice, it received it’s first DHCP response with VLAn information, then it would just sit ennuncing it was waiting for a DHCP response. Dang, I’d configured the voice vlan so why did the switch not detect the phone, enable trunking so that the phone could send it’s DHCP request on the voice VLAN?

It was only when I started reading up on HWIC-ESW voice-VLAN config I noticed that Cisco hasn’t implemented the auto enable of dot1q trunking when a phone is detected… The solution is to add two lines of code; “switchport truck native vlan xyz” and “switchport mode trunk”. The crux is that this platform is at heart a router, not a native switch…

Cisco documentation

Twittertools is dead

 | 14 Jun 2012 16:12

Long live Social

Testing automatic posting to Twitter and Facebook from my blog, sorry if you consider this spam.

To CCIE or not to CCIE?

 | 15:30

This one has been coming for a while… Last time I went for my CCIE was Q3 2007 and it’s been pretty quiet here since.

In the mean time I’ve moved country, changed jobs and moved again, though not as far. So how is my CCIE doing? Well to be honest, it’s not. You see, I came from a sponsored certification track -which incidentally I forged with my employer at the time- and thus had to sort out a transfer sum from my new employer. No problem for the Dutch employment market, at least at the time. But here in the UK I had no such luck, suffice to say that financially it was not a good idea to move country. My new employer expressed that they would be happy to support attaining CCIE status. Little did I know that we had completely different concepts of ‘support’.

Loving a new challenge I quickly settled into my new position, only to find two years had gone before the question of attaining CCIE popped up again. No wonder really when you have a 100% billable target and there’s plenty of over time involved in the projects at hand. Then when things settled down and I moved to another customer I only found myself further and further away from (CCIE related) technology.

Yes, I still design and do have a consulting role but I’m in no way challenged and kept on my toes protocol wise. It took a few technical interviews to fully realise the impact of this. Which brought me to this point: Either, I work and retrain myself towards CCIE in my own time, hope my employer will pay for the lab and favour me with a mere week off for full-time study. Or do I let go of the desire to attain CCIE and trust that there will be employers out there who are smart enough not to be blind sided by the highly praised numbers of my peers.

My choice has been to no longer pursue CCIE. I can’t ask my family for a long and hefty sacrifice once again, we’ve been there done that. I’m now one kid and two cancers ( /@maizymoo tweet) further and value my own time more than my career, if this means UK employers don’t like me any more than so be it. I know what I’m capable of, just wish I was better at convincing prospective employers…

[Above edited, below added - Jan 27th 2014]

Sad thing is that the CCIE we find ourselves hiring. often do not have the consultancy skills needed to satisfy our customer’s needs. To the HR managers out there: Hiring a CCIE does not mean, you get good communication skills, customer facing skills or even basic networking experience for that matter. Trust me I know, I’ve had to replace a number of (single, dual and a quad) CCIEs on various projects where things had gone so bad we were about to lose all future business.

Don’t get me wrong, I value certifications and see them as a good means for career progression. I do have issue though with the UK market putting so much pressure on individuals to develop themselves, alienating them completely from a corporate drive to improve through a shared responsibility. UK employer – employee relationships have become too one sided… For more on this see a recent article of mine.

Alcatel 7210 port mirroring

 | 12:52

Recently I’ve been doing more on Alcatel as I’m working in O2′s test-bed down in Slough, slaving away at testing aspects of their new LLU broadband core and new BT 21CN wholesale connectivity. Although I’ve not been able to write a lot in recent years due working for an integrator rather than an ISP; I’m mostly not allowed or it’s unwise for me to divulge what I’m working on…

However, it’s common knowledge that many providers use Alcatel and they seem to do pretty well in the ‘booming’ broadband market. Hence I thought I’d share a little snippet of an annoyance I recently encountered.

When using an Alcatel 7210 to sniff traffic and interconnect different media; 1Gbps copper and 10Gbps fibre. I found that sniffing is counter intuitive to people only trained on Cisco. A few pointers:

  1. Port mirror destinations are defined in configuration
  2. Port mirror sources are set through debug commands
  3. When mirroring VPLS ports (I needed an e-pipe/Layer-2 tunnel) I found that egress sources did not work, only ingress did and only one ingress port can be set per mirror session. It did not matter if I use the port or the SAP as source.

I was left to sniff in two places to capture both up- & down-stream traffic. YMMV as a 7750 will be different, but I don’t have one available to me to test on…

Commands used:

echo "Mirror Configuration"
    mirror-dest 4 create
      sap 1/1/4 create
      no shutdown
    mirror-dest 11 create
      sap 1/1/11 create
      no shutdown

And the debug command:

*A:<hostname># debug mirror-source 4 port ?
- no port ...
- port <port-id> egress ingress
- port <port-id> egress
- port <port-id> ingress
- port lag ...

*A:<hostname># debug mirror-source 4 sap ?
- no sap <sap-id> [ingress]
- sap <sap-id> {[ingress] }

As can be seen above capturing by SAP is only supported at ingress. Using port and SAP yielded the same result, only ingress packets were ever sent to the destination port. Despite show mirror stating both Egr & Ing.

*A:<hostname># show mirror mirror-dest 11
Mirror Service
Service Id       : 11                   Type          : Ether
Description      : (Not Specified)
Admin State      : Up                   Oper State    : Up
Forwarding Class : be                   Remote Sources: No
Slice            : 0
Destination SAP  : 1/1/11               Egr QoS Policy: 1
Local Sources
Admin State      : Up
-Port                                   1/1/26                          Egr Ing

Python rocks

 | 2 Mar 2009 11:21

Doing a network upgrade of 28 sites, 60 services and roughly 160 old devices to just 60 devices. Python has become my friend for reading configs, creating csv files and verification. Will be posting my scripts later. I’m sure there are clever people out there who can tell me where I went wrong or what I could be doing better. However time fails me to post them right now.

F5 certification

 | 11 Feb 2008 11:32

Not posted much recently, it’s time to add some more meat…

Last week I passed the F5 BIG-IP LTM v9.0 essentials exam. Was easier than I thought, however tomorrow I may be humbled further as I go for the advanced version of the F5 BIG-IP LTM v9.0 test. Studying with the training books from 4 days worth of training but without further hands on is not what I call fun. Admittedly it’s a lot easier than CCIE so what really am I moaning about. I suppose it the fact that I’m not studying for my next lab…

Humbling experience

 | 11:27

Ouch. Humility is a painful process…

I’ve just been taught a lesson in humility and authorship. My post about Cisco’s NTP authentication implementation received a comment from a certain Frank. I’ve added my comment and need to verify my statment as soon as I can manage to get some hands-on in a lab again (not got Dynamips set-up yet)…

Changing jobs

 | 26 Nov 2007 21:19

Finally some news to write. Not that I haven’t been busy but just not with CCIE. Have had too much on my mind and finally I can write about it.

Today I’ve resigned from my position at Easynet and signed with nscglobal as a Consultant. First thing while writing here is to thank Easynet for the time there, the opportunities and trust given to me. Despite moving my desk from Rotterdam to Amsterdam within six months of employment, I have thoroughly enjoyed working for Easynet. Seeing it mature from a internationally fragmented company into an upcoming global enterprise player has been both challenging and inspiring.

For those who don’t know nscglobal, they’re a UK integrator and I’ll be working in one of their London offices. This means moving from The Netherlands to the United Kingdom and we’ll be doing so physically in January. We’re looking for a rental house for multiple reasons but for this site the relevant one is my CCIE. Once in the UK I need to start focussing on my CCIE again and I hope to be able to do so with the support of some of the CCIE’s nscglobal has. So be ready for some new CCIE updates starting January.

Till then I’ll probably blog about whatever technology comes my way during the move. For example the UK mobile operator 3 has an offer with free Skype calls so I might be looking into UMTS coverage in London, Hitchin and the rest of the extended North of London. Plus I’ll be looking for a hosting location for my two servers, offers are welcome. Offers for dedicated hosting too, might help the uptime of things… My VoIP setup will have to change as well, only slightly as I’ll only be adding an FXO port at home.

Anyway just keep reading and you’ll find out, the great thing is that I don’t have to keep silent any more about what is going on and that we’re looking forward to new things.

No IP unreachables (and Cacti)

 | 11 Oct 2007 15:49

*Sigh* Took me an hour or two to figure this one out. Cacti now does a ping before actually polling a device for stats. I’m running a small cacti site which had been neglected for a long time. After updating cacti and cleaning up some mess I was confused why one router did get polled and the other’s graphs remained a dumb “nan”.

I debugged and pinged, even installed hping3 to do UDP pings. I don’t want to run cacti as root, especially not on a vhost. So the UDP ping had to work. The pings arrived but still no replies.

Getting sidetracked I noticed that the one router that did work was being hit by SSH login attempts and it’s cpu was spiking. An ACL took care of the break-in attempts but then I noticed that directed broadcasts were made to my server’s segment. So I nailed that down plus proxy-arps when I noticed that the router which had worked before now was causing errors in Cacti as well.

Tracking back I noticed that the UDP ping ‘replies’ were unreachables rather than ICMP replies (doh, how obvious!) . I enabled IP unreachables on both routers again and I was done. It’s amazing how blind one can be at times to the blatantly obvious…

Exam price increase

 | 10 Oct 2007 10:33

Only just noticed but in effect since September 7th. The CCIE lab exam has gone up from $1250 to $1400, which results in $1694,- / €1199,30 including the 21% VAT for the Brussels lab location.

Even though it’s a 12% price increase it’s been 8 years since the previous increase. It’s been argued on Groupstudy that a 2% annual increase is not bad as it’s relatively on par with the inflation we’ve seen over the last 8 years. Never the less for those of us forced to pay for our own labs it’s a bitter pill to take.

Also note that all exams from CCNA to the CCIE written and lab have increased in price.


 | 4 Oct 2007 20:20

A recent question prompted this post. I’ve noticed that my site is getting more popular, despite my silence over the past weeks. I hope to propperly break the silence soon. All I can say now is that I’m busy working out an action plan towards my next lab attempt. By no means am I giving up!

Since my last post I’ve been very busy with work related issues and as such have not had the time or the right frame of mind to study. Please bear with me as I work to get back into the game. Feel free to comment on my previous posts, or this one if you like. It really helps me to know that other find this stuff useful.

Static routes since 12.3

 | 31 Aug 2007 10:56

Since 12.3 (T?), static routes pointing to interfaces will be advertised by RIP and EIGRP as these static subnets are assumed to be part of the interfaces on which RIP and/or EIGRP is activated.

ip route prefix mask {ip-address | interface-type interface-number [ip-address]} [dhcp] [distance] [name next-hop-name] [permanent | track number] [tag tag]

Show me more… »

%d bloggers like this: