Hello,

One of my server had the whole syslog full of lines like this:

And it was happening for a long time. It wasn’t a big deal because the request is denied anyway until I had to do some serious modification on this server and discovered that syslog was nearly unusable, thanks to this amazing flood:

It seems to be impossible to have fine-grained logging with bind9, so I decided to try something else: let’s use shorewall (iptables frontend) to drop all pattern matching “x99moyu.net” (all requests are against this specific domain).

Let’s give iptables a try:

Yeah! Syslog stopped complaining. However, I’m not really happy with solution:

• TCP is not handled as well
• IPV6 isn’t either
• It matches x99moyu instead of x99moyu.net
• It’s not integrated into the system
• It’s not self-documenting

Let’s try to figure out how to match the whole domain first:

Won’t work. In fact, the DNS request in constructed a different way:
http://stackoverflow.com/questions/14096966/can-iptables-allow-dns-queries-only-for-a-certain-domain-name?answertab=votes#tab-top

If you look at the contents of the DNS request packet in wireshark or similar you will find that the dot character is not used. Each part of the domain name is a counted string, so the actual bytes of the request for google.com will be:

06 67 6f 6f 67 6c 65 03 63 6f 6d
The first byte (06) is the length of google, followed by the 6 ASCII characters, then a count byte (03) for the length of com followed by… you get the idea.

Yep, I got it. We’ll also need to do a “hex” match instead of a simple string:

Here we go, here’s the proper iptable line to use, now we can integrate it into our /etc/shorewall/rules and /etc/shorewall6/rules above the “DNS/ACCEPT” line.

# With logging (x99moy is a "tag" displayed in the log lines, limited to 6 chars)

#DNS/DROP:info:x99moy	loc	fw ; -m string --algo bm --hex-string "|07|x99moyu|03|net"

# Without logging

DNS/DROP	loc		fw ; -m string --algo bm --hex-string "|07|x99moyu|03|net"

					

We had some issues today, at work, with a PHP-based CMS (hello |*@#-?! joomla) being used as a spam gateway.

 

• The root cause (Joomla)

I fixed the issue by figuring out what was the broken PHP file using findbot.pl tool from abuseat.org. But my main concerns is that there’s no way to prevent this to happen again. PHP is broken by design, especially while being used for a CMS.

Abuseat’s script helped me to find suspicious code, then confirmed by the apache logs:

In the meanwhile, Joomla has been updated an hopefully the security issue has been fixed.
After removing the bad file, the owner of my turned-into-a-spambox-cms looks being annoyed and seemed to try break-in again:

No thanks, really. It’s been a pleasure but it’s time for me to move on:

 

• Preventing this from happening again ?

So how could you care about this ? First thing, be sure to not mess your main SMTP IP address with it. Be sure to relay the CMS emails throught a specific dedicated SMTP server that’s not hidden being the same NAT as your main SMTP server. Otherwise, you will get blacklisted as soon as any flows open in Joomla.

To ensure you’re fine, you can use one the multi-rbl checks online like anti-abuse.org or senderbase.org by Cisco. If you’re not listed here, you’re probably fine. Otherwise it’s time to ask for removal on any blacklist and be patient. Your SMTP server won’t be trusted again until at least a couple of hours, probably couple of days to be un-blacklisted on the whole Internet.

Of course, you may consider upgrading Joomla, changing password and avoid having thousands of useless plugins, but I guess you’re not in charge of this Joomla website, right ?

Another thing that may help is to enable some PHP hardening tool called “suhosin“. It wasn’t ready while Debian Jessie has been released, so we’ll use the official upstream repository to get it.

Here’s an extract of my docker file to enable this extension:

 

• Treat the symptoms, as well as the cause

So now, you’re using a different SMTP to relay emails coming from the insecure website… To avoid spaming the world and/or overloading the internet connection, we’ll setup rate-limiting on the postfix server.

We’ll use postfwd for this.

If using Debian Wheezy, make sure to get the one from backports, the default one is completly broken.

Then, we set-up a rule limiting enforcing each client_address (IP connecting this SMTP server) to not send more than 5 emails every 5 minutes.

Create new /etc/postfix/postfwd.cf configuration file containing the following:

Then set STARTUP=1 in /etc/default/postfwd.

Then, edit your postfix configuration in /etc/postfix/main.cf to add a new smtpd_recipient_restrictions setting like this:

The check_policy_service will check postfwd running on port 10040 which will return either permit or deny. Postfwd will reply with a 450 temporary error if the rate has been exceeded.

Beware of the order, in this example, even hosts being allowed to relay emails with this SMTP server, listed in $mynetworks, have been rate-limited.
The reason is that this SMTP server is outside main corporate network and I don’t trust any of the hosts using it.

Here’s another snippet from a production server:

If you don’t have this setting yet, you can get the default value on your system by running

I suggest to always add “permit” as the last action, even if it’s implicit it’s way more easy to understand the workflow by adding it.

You can now restart both service and check the log files:

Of course, postfwd has many more feature, check its online documentation !