Beginning on Sunday, October 2nd and continuing through Friday, October 14th, our website was the target of a persistent yet innocuous cyberattack. Before we delve into the details of this issue, we want to assure our visitors that:

  • The matter has been successfully resolved
  • Unlike in some major recent corporate-scale attacks, our data was not compromised
  • We are in the process of conducting our own investigations into the incident and will turn over any pertinent and related information to appropriate authorities
  • We have taken steps to mitigate similar incidents in the future

Impact

During this attack, services were occasionally unresponsive and typically had to be manually restarted until we figured out the issue and its source. Frequent visitors to our site during this timeframe probably noticed that attempts to reach us at sipadcg.org occassionally resulted in one of two issues. Either the URL was simply unresolvable, or you received a message stating “Error establishing a database connection”.

As sipadcg.org is run by volunteer sysadmins, who also happen to be full-time graduate students, services were responded to in the timeliest fashion possible. However, we were able to respond to in a relatively swift manner to ensure maximum uptime. Although the attacks were consistent, the longest time that DCG was offline was a little over an hour on Friday the 14th.

Cause

The cause of the attack has been definitively identified as a low level XML-RPC attack. XML-RPC (Remote Procedure Call) functionality is built into WordPress to enable its clients to build-in remote accessibility functions to their sites and blogs. Unfortunately, as XML-RPC by its very nature is designed to allow devices to run procedures remotely in distant networked address spaces, this functionality provides malicious agents a potential attack vector to unguarded sites.

Initially, we believed that this problem was due to scalability problems. At inception, we designed and built sipadcg.org to sustain fair yet relatively low traffic, but reality far surpassed our expectations. After consistent scaling up of our site’s AWS instance without a resolution to the issue, scalability was very clearly ruled out as the cause of the problem.

A look at the Apache Access Logs

During a more in-depth analysis, we took a look at our Apache logs to look for any irregularities that could be the basis for consistent failures for a resolvable connection state and for MySQL consistently becoming overwhelmed.

Upon review of the Apache logs, it was very apparent that we were getting hit by thousands XML-RPC requests to our site. /var/log/apache2/access.log contained many thousands of lines similar to [Attacker IP] - - [13/Oct/2016:17:18:31 +0000] "POST /xmlrpc.php HTTP/1.0" 200 596 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)".

The interesting anomaly is that they were all coming from one of a few different and consecutively-numbered IP addresses. Whois queries of these IPs point them as a block registered to a company located in the Seychelles Islands named DMZHost.co (interestingly, whose own whois record is obscured by WhoIsGuard).

After running geographical ip lookups however, it appears that the connections actually route back to Russia. geoiplookup returns the following:

$ geoiplookup 191.96.249.53
GeoIP Country Edition: RU, Russian Federation
GeoIP City Edition, Rev 1: RU, 48, Moscow City, Moscow, 101194, 55.752201, 37.615601, 0, 0
GeoIP ASNum Edition: AS29073 QUASINETWORKS

$ geoiplookup 191.96.249.54
GeoIP Country Edition: RU, Russian Federation
GeoIP City Edition, Rev 1: RU, 48, Moscow City, Moscow, 101194, 55.752201, 37.615601, 0, 0
GeoIP ASNum Edition: AS29073 QUASINETWORKS

$ geoiplookup 191.96.249.80
GeoIP Country Edition: RU, Russian Federation
GeoIP City Edition, Rev 1: RU, 48, Moscow City, Moscow, 101194, 55.752201, 37.615601, 0, 0
GeoIP ASNum Edition: AS29073 QUASINETWORKS

These lat/long combinations point to the Red Square in Moscow.

Further analysis is to be conducted on the origin of this attack.  Due to the unsophisticated nature of this attack and the low value of the target (a Columbia SIPA student group website) we have little confidence that this attack was actually perpetrated from actors residing in Moscow.  A more likely scenario is that the perpetrators were simply using servers in Moscow to route their traffic.   

Remedy

Once it was discovered that this was an attack and not a hardware/software issue, we took appropriate measures to respond.

  • Block IP Addresses. We blocked all IP addresses that we found in /var/log/apache2/access.log that were found to have malicious intent, to include the three IP addresses listed above. Iptables has been configured to drop packets from any attempted connection from these IP addresses.
  • Firewall check. We ensured that all ports are appropriately blocked except for approved devices.
  • Ensure that all software is patched. We ensured that all updates on our server had been applied to guard against all publicly-known vulnerabilities.
  • Verify and improve authentication mediums. Access to the site was checked to ensure that only authorized users had access, and that this access is as secure as possible.
  • Installed additional protection. We installed additional software to help automatically blacklist IPs that are causing excessive connection requests to our server.

Final Ideas, Thoughts, and Observations

The Good News

As stated above, we are not a high value target, unlike major corporations and government networks. This server simply exists to host our website and provide a platform for cross-dialogue between academia, enterprise, and the public sector. As we rely on alternative services for our own critical infrastructure, we are pleased to announce that data has not been inappropriately accessed, damaged, or deleted.

Our Hopes

SIPA DCG hopes that this attack (and our observations) serve as a technical reference for community members doing research on ways to harden their webserver and wordpress installations. For future reference, if your server is experiencing very high load which is not attributable to your own services, and/or if your MySQL/WP installation continually experiences “Error establishing a database connection”, then there is high likelihood that this is an XML-RPC attack. Further, please blacklist the following IPs to ensure that this problem does not replicate on other devices.

  • 191.96.249.53
  • 191.96.249.54
  • 191.96.249.80

Likelihood of a Blanket Attack

In consulting with an outside sysadmin, we arrived at a possible follow-on conclusion: that sipadcg.org was not the target in this attack and we were simply caught in the cross-fire of a larger operation. As was mentioned earlier, we run our website off an AWS instance which has an associated elastic IP. Amazon owns an almost unimaginably large block of IP addresses (52.0.0.0 - 52.31.255.255 to be exact), a significant portion of which are dedicated to AWS/EC2 and other Elastic IP allocations. Although unlikely, it’s possible that there was a larger scale attack against many other servers in this range, and we were just caught in the crossfire as one of the IPs were allocated to us. Regardless, it’s impossible to know more about this possibility without knowing who also maintains servers with IPs in this range.

We greatly appreciate your time and attention, and we’re excited to continue to offer sipadcg.org in the future.

Should you have any questions about the foregoing, please contact DCG Sysadmin Christian Van de Werken at dcg@sipadcg.org.