Yesterday Exherbo’s Quote database experienced a very primitive attack. The purpose was to turn all the rank values on our quotes into negative numbers.
In this post we are going to analyse the log files from the attack and see “who did this”.
By simply reading the log files I can see that the attack was done via the program called wget which is a fairly shiny little Unix tool for fetching websites and wget is ideal for this kind of attacks.
The first thing we did when Ciaran discovered that all of our quotes had negative values was to fetch our access log file for further analysis.
Anyways, lets get started:
First thing we want to do is to remove all the legal traffic and only get the wget clients. And then, the only really interesting stuff for us is their IP-addresses. We fire off a simple Unix command:
grep -i 'wget' access_log | cut -s -d' ' -f1 > wget_attack
The wget_attack file is available here.
Now we have a file called wget_attack that contains only the IP-address of these requests to our webserver.
I wanted to get an idea about who was behind this attack so it would be fairly nice to do a simple reverse DNS lookup of these IP’s to get an idea about where these attacks were launched from:
for host in $(cat wget_attack | sort -u) ; do
host_info=$(host "${host}")
amount=$(grep "${host}" access_log | wc -l)
echo -e ">>> ${host}: \t${amount} \t${host_info} ...";
done
Running this script really gave me a shock:
>>> 140.211.166.168: 18 168.166.211.140.in-addr.arpa domain name pointer osprey.gentoo.osuosl.org. ...
>>> 140.211.166.183: 322 183.166.211.140.in-addr.arpa domain name pointer smtp.gentoo.org. ...
>>> 64.127.104.142: 27 142.104.127.64.in-addr.arpa domain name pointer miranda.amd64.dev.gentoo.org. ...
>>> 86.10.230.114: 554 114.230.10.86.in-addr.arpa domain name pointer cpc3-york1-0-0-cust625.leed.cable.ntl.com. ...
>>> 89.16.163.100: 348 Host 100.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.163.108: 345 Host 108.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.163.110: 349 Host 110.163.16.89.in-addr.arpa not found: 3(NXDOMAIN) ...
>>> 89.16.176.11: 338 11.176.16.89.in-addr.arpa domain name pointer albatross.gentoo.org. ...
(the integer after the IP-address is the amount of requests in the access_log. The attack started at 18:53:26 UTC and ended at 19:39:46 UTC.)
Yup, that is right. 4 of these machines are Gentoo controlled.
If I recall correctly then smtp.gentoo.org is the main developer machines where Gentoo developers are able to run their IRC clients from, fetch e-mail, etc. so every Gentoo developer has access to this machine. So basically, we now know that a Gentoo developer is behind this.
If we look at Gentoo Infrastructure Server Specifications page, we will see that miranda.amd64.dev.gentoo.org is not listed, but that is probably because it is a machine available to the AMD64 developers in Gentoo (you can see the full list of available developer machines here), but osprey.gentoo.osuosl.org is listed on the infrastructure page about non-developer machines, but albatross.gentoo.org isn't.
The funny thing about the osprey machine is that it is listed as being Gentoo’s master mirror which probably means that not many people have access to this machine.
Doing a quick whois on the IP-addresses reveals that both osprey.gentoo.osuosl.org and smtp.gentoo.org are hosted by OSUOSL which is a very nice company that hosts various hardware for FOSS projects.
Doing a whois on the miranda.amd64.dev.gentoo.org machines shows that it is hosted on Gentoo’s own netrange at a company called Global Netoptex — a company also knowns as GNi which is one of Gentoo’s sponsors and a company that has two Gentoo developers as employees.
The last interesting machine is albatross.gentoo.org one. Doing a whois on that reveals that it is located on the network of the company called Bytemark Computer Consulting Ltd which among other things also hosts one of Freenode’s IRC servers.
But, uh, the 3 IP-addresses without a reverse DNS entry seems to appear on the same network as the albatross.gentoo.org machine, which is also in Bytemarks IP-range.
Oh, but the fun doesn’t end here. The last IP which looks like a normal “personal” IP-address has something interesting in it. If we just look at it we will see that: cpc3-york1-0-0-cust625.leed.cable.ntl.com. Apparently this IP looks like coming from the city of York in England (I am not sure about that though — and it is not really important either).
Normally I would just fire up Google now, but I thought that the network service called LinkedIn might be useful for this.
Spending two minutes on the site shows that the only person in my network that works for Bytemark is Alex Howells who lists himself as being System Administrator at Bytemark Hosting and Developer at Gentoo Linux. According to his LinkedIn profile, Alex Howells lives in York, United Kingdom.
This begins to look like it is one person who is behind all of this, but a person in the secret IRC channel on Freenode known as #gentoo-infra posted this after we discovered the attack:
[ ... ]
18:55:51 * astinus has a funny idea
18:56:56 astinus: for i in $(seq 1 25); do wget -O /dev/null http://quotes.exherbo.org/?ratingminus\&id=$i; sleep 6; done
18:58:55 astinus: additional brownie points if you use -q with wget and CFengine it for a one-time run on 500+ machines
18:59:13 astinus: "Hey guys! All your quotes suck!"
18:59:21 * astinus sighs
[ ... ]
(Timestamps are in UTC.)
In case you wonder: astinus is Alex Howells’s IRC nickname. Doing a simple /whois on IRC shows that as well:
hostmask : astinus!n=alex@gentoo/developer/astinus
ircname : Alex Howells <alex.howells@0wn3d.us>
channels : @#gentoo-dev #gentoo-userrel #gentoo-server @#gentoo-ops @#gentoo
info : is identified to services
info : is signed on as account astinus
idle : 0d 1h 13m 22s [signon: Sun Jun 8 23:57:52 2008]
From the integer value after the IP-address in the scripts output, we can see that 1380 requsts came from the network of Bytemark and 1934 requests if what we believe is his home machine is included. The other 367 other requests came from either OSUOSL or GNi.
I must admit I am very disappointed with Gentoo these days and this just tops the list. I used to like working for the Gentoo project, but these days this kind of stuff appears to be accepted which is really silly. Gentoo should rather spend some time kicking people like Alex Howells out.
Conclusion
Gentoo should avoid giving people like Alex Howells access to their infrastructure to avoid these very embarrassing issues.
What now?
I have filed a bug to Gentoo’s Bugzilla about this (bug #229895) plus I am going to write an e-mail to the abuse contacts from the various networks where these machines are located.
I will keep this post updated with information when I get it and maybe even write a follow up post.
