At a customer I’ve recently had to commit a grave operational sin; to connect a small switch at the end of a floor patch. These things are normally operational nightmares as they have a tendency to quickly bring an entire LAN environment down to its knees when such a ‘switch’ is connected to the network twice. Always by accident but having management kick you for something someone else did is not anyone’s idea of fun. I won’t go into the underlying principles here as I’m assuming most who frequent my blog will know about broadcast storms, their causes and the tools and solutions available to mitigate the risks.

Our justification to operations was that we wanted a few more local LAN ports to test VoIP devices on than we had available through floor patches. As such I reasoned with Operations that this was a calculated choice to segregate our testing from the rest of the LAN but still make it as realistic as possible. Using the means available meant that I had to make do with a Cisco 1801. Single routed and 8 switched interfaces. Think of it as a router with one Ethernet interface and an 8 port HWIC-ESW nailed to it. Didn’t need the ATM or WiFi it has.

So I set out, disabling IP routing, admin down all non-Ethernet ports. set up the vlan database -old style, remember?-; I did not want this baby to participate in VTP, in fact I don’t think it even can! It’s limited to 8 vlans. Pulled two cables to it. One switched port as trunked with some data and voice vlans and configured the routed interface for management access.

All sweet and dandy, tested the BPDU-guard functionality prior to installation by connecting an access-port to the LAN. Clunk! it went down as desired, result I thought… Then when installing the LAN wouldn’t bring up the LAN port. Doh! I’d missed that the 1801 doesn’t send BPDU’s until a VLAN becomes active. I’d checked if spanning-tree was operational, and it wasn’t until I brought an interface up. So I disabled STP for all vlans in the VLAN database. Now my laptop received an IP address and the data VLANs all worked.

So, time to connect a Mitel phone. No dice, it received it’s first DHCP response with VLAn information, then it would just sit ennuncing it was waiting for a DHCP response. Dang, I’d configured the voice vlan so why did the switch not detect the phone, enable trunking so that the phone could send it’s DHCP request on the voice VLAN?

It was only when I started reading up on HWIC-ESW voice-VLAN config I noticed that Cisco hasn’t implemented the auto enable of dot1q trunking when a phone is detected… The solution is to add two lines of code; “switchport truck native vlan xyz” and “switchport mode trunk”. The crux is that this platform is at heart a router, not a native switch…

Cisco documentation

Testing automatic posting to Twitter and Facebook from my blog, sorry if you consider this spam.

You see, I came from a sponsored certification track -which incidentally I forged with my employer at the time- and thus had to sort out a transfer sum from my new employer. No problem for the Dutch employment market, at least at the time. But here in the UK I had no such luck, suffice to say that financially it was not a good idea to move country. My new employer expressed that they would be happy to support attaining CCIE status. Little did I know that we had completely different concepts of ‘support’.

Loving a new challenge I quickly settled into my new position, only to find two years had gone before the question of attaining CCIE popped up again. No wonder really when you have a 100% billable target and there’s plenty of over time involved in the projects at hand. Then when things settled down and I moved to another customer I only found myself further and further away from technology. Yes I still design and have a consulting role but I’m in no way challenged and kept on my toes protocol wise. It took a few technical interviews to fully realise the impact of this.

Which brought me to this point: Either, I work and retrain myself towards CCIE in my own time, hope my employer will pay for the lab and favour me with a week off for full-time study. Or do I let go of the desire to attain CCIE and trust that there will be employers out there who are smart enough not to be blind sided by a the highly praised numbers of my peers.

In conclusion I refuse to ask my family for a long and hefty sacrifice once again, been there done that. I’m now one kid and two cancers (promisestoday.com/@maizymoo tweet) further and value my own time more than my career, if this means UK employers don’t like me any more than so be it. I know what I’m capable of, just wish I was better at convincing prospective employers…

However, it’s common knowledge that many providers use Alcatel and they seem to do pretty well in the ‘booming’ broadband market. Hence I thought I’d share a little snippet of an annoyance I recently encountered.

When using an Alcatel 7210 to sniff traffic and interconnect different media; 1Gbps copper and 10Gbps fibre. I found that sniffing is counter intuitive to people only trained on Cisco. A few pointers:

1. Port mirror destinations are defined in configuration
2. Port mirror sources are set through debug commands
3. When mirroring VPLS ports (I needed an e-pipe/Layer-2 tunnel) I found that egress sources did not work, only ingress did and only one ingress port can be set per mirror session. It did not matter if I use the port or the SAP as source.

I was left to sniff in two places to capture both up- & down-stream traffic. YMMV as a 7750 will be different, but I don’t have one available to me to test on…

Commands used:

#--------------------------------------------------
echo "Mirror Configuration"
#--------------------------------------------------
  mirror
    mirror-dest 4 create
      sap 1/1/4 create
      exit
      no shutdown
    exit
    mirror-dest 11 create
      sap 1/1/11 create
      exit
      no shutdown
    exit
  exit

And the debug command:

*A:<hostname># debug mirror-source 4 port ?
- no port ...
- port <port-id> egress ingress
- port <port-id> egress
- port <port-id> ingress
- port lag ...

*A:<hostname># debug mirror-source 4 sap ?
- no sap <sap-id> [ingress]
- sap <sap-id> {[ingress] }

As can be seen above capturing by SAP is only supported at ingress. Using port and SAP yielded the same result, only ingress packets were ever sent to the destination port. Despite show mirror stating both Egr & Ing.

*A:<hostname># show mirror mirror-dest 11

===============================================================================
Mirror Service
===============================================================================
Service Id       : 11                   Type          : Ether
Description      : (Not Specified)
Admin State      : Up                   Oper State    : Up
Forwarding Class : be                   Remote Sources: No
Slice            : 0
Destination SAP  : 1/1/11               Egr QoS Policy: 1

-------------------------------------------------------------------------------
Local Sources
-------------------------------------------------------------------------------
Admin State      : Up

-Port                                   1/1/26                          Egr Ing
===============================================================================

As far as I’m aware no sensitive data was access nor was any serious damage done other than a few index.html files over written. The only real loss is the joomla site that was broken into as the owner is on holiday and hadn’t updated it for a while, suffice to say it’s down for the time being.

The defacement was the work of the hacker group 1923turk-grup. The following are some of my findings.

Analysis

[First activity seen]

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2639||||78.175.34.28 – - [20/Oct/2009:20:03:30 +0100] “GET /index.php?option=com_user&view=remind HTTP/1.1″ 200 2639 “http://www.google.com.br/search?hl=pt-BR&q=inurl:%3Foption%3Dcom_user++intitle:.nl&start=20&sa=N”

The attacker uses google to find vulnerable site, runs Windows XP uses IE7 and every version of .net known to mankind. That is if the string wasn’t spoofed…

LESSON #1 : Protect admin pages from being spidered, use robots.txt and .htaccess

[Probing starts]

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2767||||78.175.34.28 – - [20/Oct/2009:20:03:50 +0100] “GET //index.php?option=com_user&view=reset&layout=confirm HTTP/1.1″ 200 2767 “-”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “POST //index.php?option=com_user&task=confirmreset HTTP/1.1″ 301 – “http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm”

# Another 301

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2731||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “GET /index.php?option=com_user&view=reset&layout=complete HTTP/1.1″ 200 2731 “http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||696||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “GET /templates/avant1/js/ffont-config.js.php?pfad=%2Ftemplates%2Favant1&color1=%236699CC&color2=%2399CCFF&font=font6 HTTP/1.1″ 200 696 “http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:04:58 +0100] “POST /index.php?option=com_user&task=completereset HTTP/1.1″ 301 – “http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete”

# There are a few more attempts

[20/Oct/2009:20:04:30]    POST //index.php?option=com_user&task=confirmreset HTTP/1.1    301    -    http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm
[20/Oct/2009:20:04:58]    POST /index.php?option=com_user&task=completereset HTTP/1.1    301    -    http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete
[20/Oct/2009:20:06:36]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/
[20/Oct/2009:20:07:23]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/index.php?option=com_users&view=user&task=edit&cid[]=62
[20/Oct/2009:20:07:33]    POST /administrator/index.php HTTP/1.1    200    3766    http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0
[20/Oct/2009:20:07:45]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/index.php
[20/Oct/2009:20:08:26]    POST /templates/beez/index.php HTTP/1.1    200    64506    http://www.hackedsite.nl/templates/beez/index.php

“beez” seems to be the vulnerability. Failing at administrator/index.php, the attacker directs it’s attention directly at beez/index.php and succeeds in changing this file resulting in an r57shell being available to the hacker via beez/index.php. It’s an ingenious file as it contains a gz compressed/encrypted data which executes each time it is accessed. One weakness being all activity shows up in the log files, hence I’m able to track exactly what was parsed to the script and what data was pulled off the system.

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||10721||||78.175.34.28 – - [20/Oct/2009:20:07:30 +0100] “GET //templates/beez/template_thumbnail.png HTTP/1.1″ 200 10721 “http://www.hackedsite.nl/administrator/index.php?option=com_templates”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3097||||78.175.34.28 – - [20/Oct/2009:20:07:31 +0100] “GET /administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0 HTTP/1.1″ 200 3097 “http://www.hackedsite.nl/administrator/index.php?option=com_templates”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||1330||||78.175.34.28 – - [20/Oct/2009:20:07:31 +0100] “GET /administrator/templates/khepri/images/toolbar/icon-32-preview.png HTTP/1.1″ 200 1330 “http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0″
…
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3766||||78.175.34.28 – - [20/Oct/2009:20:07:33 +0100] “POST /administrator/index.php HTTP/1.1″ 200 3766 “http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0″
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:07:45 +0100] “POST /administrator/index.php HTTP/1.1″ 301 – “http://www.hackedsite.nl/administrator/index.php”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3202||||78.175.34.28 – - [20/Oct/2009:20:07:47 +0100] “GET /administrator/index.php?option=com_templates&client=0&task=edit&cid[]=beez HTTP/1.1″ 200 3202 “http://www.hackedsite.nl/administrator/index.php”

# A quick google for “joomla beez template hack” turned up the following fro july/09: http://news.maiahost.com/warning-beez-joomla-template-found-to-have-hack-inside

In short :
template/beez/index.php contained a compressed version of r57shell 1.40
r57shell was used to create a new file template/beez/10.php which contained a compressed version of a shell written by 1923turk-grup member Enes_60 and called “c100″.

The second shell offers to the hacker single button access to “multi-site defame” scripts and other tools, each of these trying to exploit vulnerabilities in the servers code or sysadmin misconfiguration of said server.

The whole hack did not take long and must have been fairly easy to do given the fact that the joomla site hadn’t been looked after for a prolonged time. The purpose seems to primarily have been to post Turkish nationalistic propaganda. I couldn’t find any proof of any serious root kit attempts or attempts to find sensitive data.

Lessons learned

1. Ensure that customers who run php script regularly check for vulnerabilities. Packages like joomla, drupal and wordpress make this very easy these days
2. Always use .htaccess files. For example forward requests to index.html to index.php even if there is no index.html, a hacker might somehow manage to upload one
3. Ensure that permissions are appropriately set
4. Raise the priority of upgrading my server to a chroot enabled setup, this will ensure that next time a site gets hacked only that site will suffer
5. Even themes can be vulnerable to attack, remove all unused plugins and themes from dynamic sites
6. Put dummy index.html & index.php files in place with 444 permissions making it harder to upload a new file

None of this will give 100% security but every extra hurdle helps.

The thing that makes me laugh most is that this was not a distributed attack, no apparent attempt was made to hide the hackers identity. The used IP address is owned by TurkTelekom and belongs to a dynamic pool for TTnet ADSL users in Istanbul. I’m not sure if this is the actual idiot himself or if some poor Turkish soul got abused for this. I wrote an email to TurkTelekom but am not holding my breath. Will update this post if I ever hear back from them, like I said don’t stick around waiting…

Strangely no attempt has been made to connect to either shell from any other IP address. Does it need mentioning that my server now dislikes the IP address used in the attack? All altered files have been removed. The scripts have been moved to a safe location for further inspection and possibly personal use. Further safety measures have been taken and all possible fixes & patches done and dusted.

about 1923turk-grup

I really do not see the point these guys are trying to make. Do they honestly believe they are going to reach their target group in this way? I read the page they posted and it’s a nationalistic rant about how Turkey should be defended from goodness knows who. They’d have more success writing a nice website somewhere where it won’t be taken down by disgruntled sysadmins. Then they can work on getting to the top in Google searches for something like “Turkish national front”. Do these guys honestly believe they’re achieving anything by comparing sizes of private parts -erm, defame list size- with other hacker groups?

Related links:

http://ddanchev.blogspot.com/2007/11/overperforming-turkish-hacktivists.html (mentions Turks vs Swedes online war)

http://ip-address-lookup-v4.com/lookup.php?ip=78.175.34.28 (Source address of the attacker, as seen by my server)

inetnum:        78.175.0.0 - 78.175.239.255
netname:        TurkTelekom
descr:          TT ADSL-TTnet Alc dynamic_aci

Country	Turkey
Country Code	TR
Region	Istanbul

A quick refresh for all:

1. PE physical link(s), ties one VRF to another using one subnet for each VRF. Commonly deployed using vlans across a DOT1q trunk (anyone still using ISL? ok, ok a tagged trunk but don’t tell me I’m not allowed to say “trunk”). Possible to use routing protocols between each individual VRF, most commonly staticly routed as it’s the safest model, the other side isn’t trusted…
2. PE MP-eBGP, ties the two MPLS clouds together exchanging VPN labels. BGP can be used to exchange the vpn labels, no need for tag-switching of IP traffic. Requires the inter- link to be known in both IGPs as the next-hop changes, only the vpn label is preserved end-to-end.
3. P tag-switching, joins the two domains together at the lowest level. The inter-link can be between P devices all labels are preserved end-to-end. Even the IGPs must have ‘full’ connectivity, of course I mean PE-to-PE…

Type-2 seemed to make most sense due to the amount of vrfs involved, no need to go for type-3 and type-1 would add way too much complexity to be able to support a multi month long migration.

Now for the MTU issue. The old MPLS cloud uses 3600′s as PE’s in the relevant sites and the new network uses 7600′s. Admittedly a bit slow on the ball -mind you I was not the original designer- we implemented an mtu of 1536 on the sup720 GE based infrastructure links on the new network. However the vpn label on the 4 FE inter-AS/MP-eBGP links was forgotten untill a service using TCP keepalives was partially migrated. Sessions flapped due to the keepalives being 1500 bytes in size and having their df-bit set. I wonder why they used TCP but as of yet I’m left guessing.

The moment we noticed the error of our ways we found that IOS threw us a few curve balls:

1. A 3600 running 12.3 doesn’t accept anything over 1500 as mtu on a FastEthernet interface
2. A 7600 with 6748-GE (LAN interfaces) doesn’t allow a larger than 1500 mtu size either (routed interface)

The short answers are the following:

1. On the 3600: tag switching mtu 1508
2. On the 7600: mpls mtu 1508

fyi, 1504 would have done fine as an mpls label is just 4 bytes we just threw in another 4 bytes for luck

Some links to back things up:

• Cisco MTU config for 7600 12.2SR & layer3 ports
• Cisco IOS hints and tricks: The tale of three MTUs
• You’ll have to take my word for this quote from Cisco: “I can confirm that using tag-switching mtu at 1508 on the 3600 should resolve the issue from that side” (despite not being able to set the interface mtu higher than 1500)

Command to verity mpls mtu:

sh mpls [intf] det

Today is Ada Lovelace Day, an international celebration of women in technology that centres around the use of blogs.

Launched by Suw Charman-Anderson, a freelance software consultant, Ada Lovelace Day is a day of blogging designed to draw attention to women who are “excelling in technology”.

Seeing this initiative made me think of my wife who has spent the last year carving out her own piece of the IT industry. Some may call me biased, true. Call it what you like, I’m taking this opportunity to put my wife in the limelight and honour her for her efforts and achievements after we relocated to the UK.The entrepreneur

After being a hard working employee for a number of years, my wife made the choice to be self employed to strike a better balance between work and being a mother of two. Also the state of UK childcare wasn’t what we were used to in NL. After a few different ventures she discovered that writing websites was what gave a healthy mix between personal contacts and exercising creativity. My wife’s strength lies in her knack for figuring out what someone needs through effective communication and then delivering as required.

The driving force

Knowing how motivated and driven my wife can be did not stop me from being surprised at how fast she mastered new technology and made things her own. Let me list some of her achievements.

Mastered CSS within a matter of weeks, I know she’ll be modest and say she knows the beginnings but she knows a lot more than she lets people believe.

Managed to pull a company through a restart and did not hesitate to re-engineer the internal processes in such a way that they are now ready for growth as well as have the ability to run at a much lower cost and with much less overhead (time and money).

The above involved a lot more than just a little bit of IT. She worked with the systems supplier to iron out kinks and smooth the processes. She introduced net-books with mobile internet for instant access to the on-line database of the systems supplier. She managed advertising and the phone system, and I could go on.

Co-rewrote the website for her father’s primary school, now a very pretty and smooth website.

Became a member of the Parents and Teachers Association.

Became a school governor.

Authored a few other websites and has a number of requests in the pipeline.

Technology

As the initial root of this is to blog about women in IT, I can state that my wife has gone from an excellent computer user to a self made IT Consultant and Web Designer. She’s still finding her way, as we all are. But she’s doing so at great speed and completely under her own strength. I want to commend her for her devotion and desire to help people make the most of available resources and her courage to venture into a world that is both riddled with people out for a cheap buck and others who have been around for so long we often look down upon those just wandering in.

Honestly commands me to say I’m probably one of the latter which doesn’t always make it easy for others, especially my wife. For my wife and all others who have a sincere desire to use IT for the better and to help people get the most out of work and life I say, “Thank you. You are desperately needed.”

Note to self: DO NOT alter permalinks for pages! my about page was linked as who-am-i and this caused major issues. I also turned off the automatic XHTML corrections in the write-options page, not sure what it does so I’d rather get an error when writing a page than have my site go down. Priorities they say…

next to that my gallery now runs mod_rewrite in safe mode which works fine and I’ve re-linked an image on my about page. I recon the link was old style from WP 2.3 and older versions of WPG2 and Gallery2.

Please let me know if you find any issues, particularly with missing images. I hope all is well now and my server won’t die another terrible dead due to rewite/permalink hell.

Finally I have found the real culprit: /.htaccess was messing with things. What made it hard to find was that this file has been in place for a few years now. Upgrading WordPress to v2.7 as well as WPG2/Gallery2 tipped the balance.

What worked?
RewriteRule ^$ /wordpress/ [R=301,L]

Instead of:
RewriteCond %{REQUEST_URI} !^/.+
RewriteRule ^(.*)$ /wordpress/ [R=301,L]