As far as I’m aware no sensitive data was access nor was any serious damage done other than a few index.html files over written. The only real loss is the joomla site that was broken into as the owner is on holiday and hadn’t updated it for a while, suffice to say it’s down for the time being.

The defacement was the work of the hacker group 1923turk-grup. The following are some of my findings.

Analysis

[First activity seen]

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2639||||78.175.34.28 – - [20/Oct/2009:20:03:30 +0100] “GET /index.php?option=com_user&view=remind HTTP/1.1″ 200 2639 “http://www.google.com.br/search?hl=pt-BR&q=inurl:%3Foption%3Dcom_user++intitle:.nl&start=20&sa=N”

The attacker uses google to find vulnerable site, runs Windows XP uses IE7 and every version of .net known to mankind. That is if the string wasn’t spoofed…

LESSON #1 : Protect admin pages from being spidered, use robots.txt and .htaccess

[Probing starts]

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2767||||78.175.34.28 – - [20/Oct/2009:20:03:50 +0100] “GET //index.php?option=com_user&view=reset&layout=confirm HTTP/1.1″ 200 2767 “-”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “POST //index.php?option=com_user&task=confirmreset HTTP/1.1″ 301 – “http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm”

# Another 301

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||2731||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “GET /index.php?option=com_user&view=reset&layout=complete HTTP/1.1″ 200 2731 “http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||696||||78.175.34.28 – - [20/Oct/2009:20:04:30 +0100] “GET /templates/avant1/js/ffont-config.js.php?pfad=%2Ftemplates%2Favant1&color1=%236699CC&color2=%2399CCFF&font=font6 HTTP/1.1″ 200 696 “http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:04:58 +0100] “POST /index.php?option=com_user&task=completereset HTTP/1.1″ 301 – “http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete”

# There are a few more attempts

[20/Oct/2009:20:04:30]    POST //index.php?option=com_user&task=confirmreset HTTP/1.1    301    -    http://www.hackedsite.nl//index.php?option=com_user&view=reset&layout=confirm
[20/Oct/2009:20:04:58]    POST /index.php?option=com_user&task=completereset HTTP/1.1    301    -    http://www.hackedsite.nl/index.php?option=com_user&view=reset&layout=complete
[20/Oct/2009:20:06:36]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/
[20/Oct/2009:20:07:23]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/index.php?option=com_users&view=user&task=edit&cid[]=62
[20/Oct/2009:20:07:33]    POST /administrator/index.php HTTP/1.1    200    3766    http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0
[20/Oct/2009:20:07:45]    POST /administrator/index.php HTTP/1.1    301    -    http://www.hackedsite.nl/administrator/index.php
[20/Oct/2009:20:08:26]    POST /templates/beez/index.php HTTP/1.1    200    64506    http://www.hackedsite.nl/templates/beez/index.php

“beez” seems to be the vulnerability. Failing at administrator/index.php, the attacker directs it’s attention directly at beez/index.php and succeeds in changing this file resulting in an r57shell being available to the hacker via beez/index.php. It’s an ingenious file as it contains a gz compressed/encrypted data which executes each time it is accessed. One weakness being all activity shows up in the log files, hence I’m able to track exactly what was parsed to the script and what data was pulled off the system.

/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||10721||||78.175.34.28 – - [20/Oct/2009:20:07:30 +0100] “GET //templates/beez/template_thumbnail.png HTTP/1.1″ 200 10721 “http://www.hackedsite.nl/administrator/index.php?option=com_templates”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3097||||78.175.34.28 – - [20/Oct/2009:20:07:31 +0100] “GET /administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0 HTTP/1.1″ 200 3097 “http://www.hackedsite.nl/administrator/index.php?option=com_templates”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||1330||||78.175.34.28 – - [20/Oct/2009:20:07:31 +0100] “GET /administrator/templates/khepri/images/toolbar/icon-32-preview.png HTTP/1.1″ 200 1330 “http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0″
…
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3766||||78.175.34.28 – - [20/Oct/2009:20:07:33 +0100] “POST /administrator/index.php HTTP/1.1″ 200 3766 “http://www.hackedsite.nl/administrator/index.php?option=com_templates&task=edit&cid[]=beez&client=0″
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||-||||78.175.34.28 – - [20/Oct/2009:20:07:45 +0100] “POST /administrator/index.php HTTP/1.1″ 301 – “http://www.hackedsite.nl/administrator/index.php”
/var/log/httpd/ispconfig_access_log_2009_10_20:www.hackedsite.nl||||3202||||78.175.34.28 – - [20/Oct/2009:20:07:47 +0100] “GET /administrator/index.php?option=com_templates&client=0&task=edit&cid[]=beez HTTP/1.1″ 200 3202 “http://www.hackedsite.nl/administrator/index.php”

# A quick google for “joomla beez template hack” turned up the following fro july/09: http://news.maiahost.com/warning-beez-joomla-template-found-to-have-hack-inside

In short :
template/beez/index.php contained a compressed version of r57shell 1.40
r57shell was used to create a new file template/beez/10.php which contained a compressed version of a shell written by 1923turk-grup member Enes_60 and called “c100″.

The second shell offers to the hacker single button access to “multi-site defame” scripts and other tools, each of these trying to exploit vulnerabilities in the servers code or sysadmin misconfiguration of said server.

The whole hack did not take long and must have been fairly easy to do given the fact that the joomla site hadn’t been looked after for a prolonged time. The purpose seems to primarily have been to post Turkish nationalistic propaganda. I couldn’t find any proof of any serious root kit attempts or attempts to find sensitive data.

Lessons learned

1. Ensure that customers who run php script regularly check for vulnerabilities. Packages like joomla, drupal and wordpress make this very easy these days
2. Always use .htaccess files. For example forward requests to index.html to index.php even if there is no index.html, a hacker might somehow manage to upload one
3. Ensure that permissions are appropriately set
4. Raise the priority of upgrading my server to a chroot enabled setup, this will ensure that next time a site gets hacked only that site will suffer
5. Even themes can be vulnerable to attack, remove all unused plugins and themes from dynamic sites
6. Put dummy index.html & index.php files in place with 444 permissions making it harder to upload a new file

None of this will give 100% security but every extra hurdle helps.

The thing that makes me laugh most is that this was not a distributed attack, no apparent attempt was made to hide the hackers identity. The used IP address is owned by TurkTelekom and belongs to a dynamic pool for TTnet ADSL users in Istanbul. I’m not sure if this is the actual idiot himself or if some poor Turkish soul got abused for this. I wrote an email to TurkTelekom but am not holding my breath. Will update this post if I ever hear back from them, like I said don’t stick around waiting…

Strangely no attempt has been made to connect to either shell from any other IP address. Does it need mentioning that my server now dislikes the IP address used in the attack? All altered files have been removed. The scripts have been moved to a safe location for further inspection and possibly personal use. Further safety measures have been taken and all possible fixes & patches done and dusted.

about 1923turk-grup

I really do not see the point these guys are trying to make. Do they honestly believe they are going to reach their target group in this way? I read the page they posted and it’s a nationalistic rant about how Turkey should be defended from goodness knows who. They’d have more success writing a nice website somewhere where it won’t be taken down by disgruntled sysadmins. Then they can work on getting to the top in Google searches for something like “Turkish national front”. Do these guys honestly believe they’re achieving anything by comparing sizes of private parts -erm, defame list size- with other hacker groups?

Related links:

http://ddanchev.blogspot.com/2007/11/overperforming-turkish-hacktivists.html (mentions Turks vs Swedes online war)

http://ip-address-lookup-v4.com/lookup.php?ip=78.175.34.28 (Source address of the attacker, as seen by my server)

inetnum:        78.175.0.0 - 78.175.239.255
netname:        TurkTelekom
descr:          TT ADSL-TTnet Alc dynamic_aci

Country	Turkey
Country Code	TR
Region	Istanbul

A quick refresh for all:

1. PE physical link(s), ties one VRF to another using one subnet for each VRF. Commonly deployed using vlans across a DOT1q trunk (anyone still using ISL? ok, ok a tagged trunk but don’t tell me I’m not allowed to say “trunk”). Possible to use routing protocols between each individual VRF, most commonly staticly routed as it’s the safest model, the other side isn’t trusted…
2. PE MP-eBGP, ties the two MPLS clouds together exchanging VPN labels. BGP can be used to exchange the vpn labels, no need for tag-switching of IP traffic. Requires the inter- link to be known in both IGPs as the next-hop changes, only the vpn label is preserved end-to-end.
3. P tag-switching, joins the two domains together at the lowest level. The inter-link can be between P devices all labels are preserved end-to-end. Even the IGPs must have ‘full’ connectivity, of course I mean PE-to-PE…

Type-2 seemed to make most sense due to the amount of vrfs involved, no need to go for type-3 and type-1 would add way too much complexity to be able to support a multi month long migration.

Now for the MTU issue. The old MPLS cloud uses 3600′s as PE’s in the relevant sites and the new network uses 7600′s. Admittedly a bit slow on the ball -mind you I was not the original designer- we implemented an mtu of 1536 on the sup720 GE based infrastructure links on the new network. However the vpn label on the 4 FE inter-AS/MP-eBGP links was forgotten untill a service using TCP keepalives was partially migrated. Sessions flapped due to the keepalives being 1500 bytes in size and having their df-bit set. I wonder why they used TCP but as of yet I’m left guessing.

The moment we noticed the error of our ways we found that IOS threw us a few curve balls:

1. A 3600 running 12.3 doesn’t accept anything over 1500 as mtu on a FastEthernet interface
2. A 7600 with 6748-GE (LAN interfaces) doesn’t allow a larger than 1500 mtu size either (routed interface)

The short answers are the following:

1. On the 3600: tag switching mtu 1508
2. On the 7600: mpls mtu 1508

fyi, 1504 would have done fine as an mpls label is just 4 bytes we just threw in another 4 bytes for luck

Some links to back things up:

• Cisco MTU config for 7600 12.2SR & layer3 ports
• Cisco IOS hints and tricks: The tale of three MTUs
• You’ll have to take my word for this quote from Cisco: “I can confirm that using tag-switching mtu at 1508 on the 3600 should resolve the issue from that side” (despite not being able to set the interface mtu higher than 1500)

Command to verity mpls mtu:

sh mpls [intf] det

Today is Ada Lovelace Day, an international celebration of women in technology that centres around the use of blogs.

Launched by Suw Charman-Anderson, a freelance software consultant, Ada Lovelace Day is a day of blogging designed to draw attention to women who are “excelling in technology”.

Seeing this initiative made me think of my wife who has spent the last year carving out her own piece of the IT industry. Some may call me biased, true. Call it what you like, I’m taking this opportunity to put my wife in the limelight and honour her for her efforts and achievements after we relocated to the UK.The entrepreneur

After being a hard working employee for a number of years, my wife made the choice to be self employed to strike a better balance between work and being a mother of two. Also the state of UK childcare wasn’t what we were used to in NL. After a few different ventures she discovered that writing websites was what gave a healthy mix between personal contacts and exercising creativity. My wife’s strength lies in her knack for figuring out what someone needs through effective communication and then delivering as required.

The driving force

Knowing how motivated and driven my wife can be did not stop me from being surprised at how fast she mastered new technology and made things her own. Let me list some of her achievements.

Mastered CSS within a matter of weeks, I know she’ll be modest and say she knows the beginnings but she knows a lot more than she lets people believe.

Managed to pull a company through a restart and did not hesitate to re-engineer the internal processes in such a way that they are now ready for growth as well as have the ability to run at a much lower cost and with much less overhead (time and money).

The above involved a lot more than just a little bit of IT. She worked with the systems supplier to iron out kinks and smooth the processes. She introduced net-books with mobile internet for instant access to the on-line database of the systems supplier. She managed advertising and the phone system, and I could go on.

Co-rewrote the website for her father’s primary school, now a very pretty and smooth website.

Became a member of the Parents and Teachers Association.

Became a school governor.

Authored a few other websites and has a number of requests in the pipeline.

Technology

As the initial root of this is to blog about women in IT, I can state that my wife has gone from an excellent computer user to a self made IT Consultant and Web Designer. She’s still finding her way, as we all are. But she’s doing so at great speed and completely under her own strength. I want to commend her for her devotion and desire to help people make the most of available resources and her courage to venture into a world that is both riddled with people out for a cheap buck and others who have been around for so long we often look down upon those just wandering in.

Honestly commands me to say I’m probably one of the latter which doesn’t always make it easy for others, especially my wife. For my wife and all others who have a sincere desire to use IT for the better and to help people get the most out of work and life I say, “Thank you. You are desperately needed.”

Note to self: DO NOT alter permalinks for pages! my about page was linked as who-am-i and this caused major issues. I also turned off the automatic XHTML corrections in the write-options page, not sure what it does so I’d rather get an error when writing a page than have my site go down. Priorities they say…

next to that my gallery now runs mod_rewrite in safe mode which works fine and I’ve re-linked an image on my about page. I recon the link was old style from WP 2.3 and older versions of WPG2 and Gallery2.

Please let me know if you find any issues, particularly with missing images. I hope all is well now and my server won’t die another terrible dead due to rewite/permalink hell.

Finally I have found the real culprit: /.htaccess was messing with things. What made it hard to find was that this file has been in place for a few years now. Upgrading WordPress to v2.7 as well as WPG2/Gallery2 tipped the balance.

What worked?
RewriteRule ^$ /wordpress/ [R=301,L]

Instead of:
RewriteCond %{REQUEST_URI} !^/.+
RewriteRule ^(.*)$ /wordpress/ [R=301,L]

My first google tonight led me to an article on the excellent ios hints and tricks site (ioshints.info). Though this is for a full public DNS server it was a little overkill for me, please read this article as well if you do plan to go that route.

My goal was to have my DSL router serve local IP’s for a few public and local domain names. For example this allows me to use sip.djerk.nl as my proxy address in my soft-phone both at home and away.

The following code is what I ended up with.

# These hostnames will be served locally
# As such they overrule their public dns
# clients do need to use this router as a dns server
ip host view VoIP sip.djerk.nl 192.168.100.130
ip host sip.djerk.nl 192.168.100.130
ip host c877 192.168.100.1

interface Vlan10
description *** VoIP LAN ***
ip dns view-group VoIP

# For my Voice vlan
ip dns view VoIP
domain name djerk.nl
# search list for this view
domain list djerk.nl
domain name-server x.x.x.x
domain name-server y.y.y.y
domain round-robin

# Default dns handling
ip dns view default
domain name geurtscass.com
domain list geurtscass.com

ip dns view-list VoIP
view VoIP 1
view default 100

# Enable local dns server (proxy dns if no domains are configured here)
# I do not allow inbound dns requests (intf ACL) so no need to worry about abuse
ip dns server

Most interesting is the usage of views and a view-list. According to CCO the view-list is needed to bind a view to an interface, views can’t be assigned directly. Also noteworthy is that I’ve not found a way to reduce the sip.djerk.nl host mapping to a single line, as adding the default view to the VoIP view-list was not enough. Maybe I should add the VoIP view to the default view-list.

Well just tried that and it didn’t work either. Anyone out there who knows how to do the same in IOS but with less lines?