A Engineers Ranting

If you have been the target of an NDR attack* attempt or made an error when configuring your Exchange server and have left yourself an open relay, then you may find that your queues on the Exchange server have a large number of invalid email messages.
* An NDR Attack is where messages are sent to your server with an invalid email address on purpose. Your server then attempts to bounce them back to the sender. The only problem is that the sender has been spoofed and it is that address that is the intended target of the message. These attacks can be avoided with Exchange 2003 and Windows 2003 Service Pack 1 and higher using a new option.Other symptoms include hard disk space is dropping rapidly and the server has become unresponsive. The Exchange logs are much larger than normal.

This article is based on the MS KB 324958 which was written for Small Business Server and MSKB 909005 which is for the full version of Exchange. Some of the techniques have been adjusted based on our experience with following the guides. Original articles:
http://support.microsoft.com/default.aspx?kbid=324958
http://support.microsoft.com/default.aspx?kbid=909005

---

Find the Problem

Before you start cleaning up the server, you need to find the source of the problem and deal with it.

Check Whether the Exchange Server is an Open SMTP Relay using a Telnet Test

A Telnet test involves establishing a Telnet session from a computer that is not located on the local network to the external (public) IP address of the Exchange server. You need to carry out the test from a machine at home, or from another office. Doing the test from a machine on your own network will produce useless results.

1. Start a command prompt.
   Either click start, run and type CMD
   or Choose Command Prompt from Start, Programs, Accessories, Command Prompt
2. Type “telnet” (minus quotes) and press enter.
3. At the Telnet prompt, type

   set localecho
   
   (minus quotes) and press enter. This lets you see what is going on.

4. Still in the telnet prompt, enter the following command and then press enter

   open 111.222.333.444 25

   where 111.222.333.444 is your Exchange server’s external IP address
   

5. You should get a response back similar to the following:

   220 mail.server.domain Microsoft ESMTP MAIL Service, Version: 6.0.2790.0 Ready at
   

6. Type the following command in to the telnet windows:

   ehlo testdomain.com
   
   and press enter (note “testdomain.com” can be anything that isn’t a domain that the Exchange server is responsible for.

7. After pressing OK you should get a response back

   250 OK
   

8. Type the following command in to the telnet window:

   mail from:address@testdomain.com
   
   and press enter (again where address@testdomain is an email address that is not on the Exchange server. Note the lack of space between from and the first part of the address).

9. After pressing OK you should get a response back:

   250 2.1.0 address@testdomain.com….Sender OK
   

10. Type the following command in to the telnet window:

    rcpt to:address@anotherdomain.com
    
    and then press enter (where address@anotherdomain.com is not either an address you use internally or the address you entered earlier as the from. Once again note the lack of space between to and the first part of the e-mail address).

11. After pressing enter you will get one of two responses.
    If you get

    550 5.7.1 Unable to relay for address@anotherdomain.com
    
    then you are relay secure.
    However if you get

    250 2.1.5 address@anotherdomain.com
    
    Then you are an open relay.

What now?

There are two parts of the Exchange that can make your Exchange server an open relay, the Default SMTP Virtual Server and SMTP connectors. You need to check both to ensure that you haven’t configured them wrongly and turned your machine in to a spammers target.

Default SMTP Virtual Server

To check or correct the configuration of the Default SMTP Virtual Server:

1. Start Exchange System manager (ESM)
2. Expand Servers, <your server>, Protocols, SMTP.
3. Right click on “Default SMTP Virtual Server” and choose Properties.
4. Click on the “Access” Tab.
5. There are four buttons, click on “Relay…” at the bottom.
6. Ensure that “Only the list below” is enabled and the list is empty.
7. If you don’t have users sending email through your email server with Outlook Express or another POP3 client then you can disable “Allow all users that successfully authenticate to relay regardless of the list above”.
8. Apply/OK until all windows are closed.

SMTP Connections

1. Start ESM, then open Connectors.
2. Right click on each SMTP Connector in turn and choose Properties.
3. Click on the “Address Space” tab.
4. If you have a “*” in the address list, check that “Allow messages to be routed to these domains” is not enabled.
5. Apply/OK until all windows are closed.

Once you have made the changes, restart the SMTP server service and then repeat the telnet test above to ensure that you have closed everything.

---

Check Whether an Authenticated User is Relaying

This technique requires the Windows Event Viewer to determine whether a user is trying to use the SMTP service in Exchange to send email. If you have disabled the authenticated user option already then this isn’t an issue.

1. Start ESM.
2. Expand Servers and then right click on your server and choose Properties.
3. Click on the “Diagnostic Logging” tab.
4. In the list of “Services” on the right, find “MSExchangeTransport”.
5. In the resulting list choose “SMTP Protocol”.
6. Below the list, change the “Logging Level” to Maximum and click Apply.
7. Repeat for “Authentication”
8. Press Apply/OK to close Server Properties.

You now need to watch the Event Logs on the Exchange server. In the application log you will see something similar to the following which can indicate that a user is trying to send email through the SMTP interface.

Event Type: Information
Event Source: MSExchangeTransport
Event Category: SMTP Protocol
Event ID: 1708
Date: 30/08/2004
Time: 15:45:08
User: N/A
Computer: EXCH-SRV1
Description: SMTP Authentication was performed successfully with client test-pc1. The authentication method was LOGIN and the username was domain\username.

If the account being used is “Guest” then you need to disable the Guest account.
If it is another account then you need to either change the password or disable the account.

Ideally you do not want any kind of relaying going on. The best option if this is happening is to disable the feature altogether. If this isn’t practical for business reasons, then you need to secure it as best you can.

The Administrator Account is the most common target

Note that the most common account that is used for this type of attack is the Administrator account. Therefore if you suspect that that the administrator account is being abused, then change its password and restart the SMTP Server Service to ensure that the new credentials are used. The administrator account is attacked because it doesn’t lock out.

---

Check whether you are under an NDR Attack

If you are under an NDR attack, then you will find lots of messages in the queues of the server. These messages have special characteristics which make them easy to spot.

1. Start Exchange System Manager.
2. Go to Servers, <your server>, Queues in Exchange 2003, or down to Protocols, SMTP in Exchange 2000.
3. Select a queue that contains many messages, click Find messages, and then click Find Now.
4. In the Sender field of the messages will be an address. If it is postmaster@ your domain then the message is an NDR. You can view the recipient of the NDR by double clicking on the message.

Note: If you are using an SMTP Connector to route email through your ISP using a smart host, then you cannot detect this type of attack. The messages are sent straight out to the ISP by your server. If your ISP has alerted you that there may be a problem, you will need to use message tracking and the SMTP log to detect the cause of the attack.

If you are on Exchange 2003 with Windows 2003 then you can stop an NDR attack by using recipient filtering and the tar pit option in Windows 2003. You will still need to clean the queues using the techniques outlined, but it will stop further traffic.

If you are on Exchange 2003 on Windows 2000 then you should NOT enable recipient filtering as this exposes your server to a directory harvest attack.
Exchange 2000 users do not have any kind of recipient filtering options available to them.
Therefore you should look at a third party tool that can do the filtering for you, often referred to as an LDAP lookup. Vamsoft ORF has Active Directory filtering and has a 30 day trial version.

---

Cleaning up the Server

Once you have found out the cause of the problem and dealt with it, then you need to clean up the server.

Block port 25 on your firewall/router so that SMTP traffic is not coming in while you cleaning the server. This stops new messages, both spam and valid messages and also ensures that nothing you want is lost while you clean up the server. As long as you do not leave the port closed for longer than 48 hours then genuine inbound email will still be delivered.

When following this procedure you should note that it can take NUMEROUS ATTEMPTS OVER MANY HOURS before the queues are clear. Exchange System Manager is notorious for being unable to show the true extent of the queues when it has been abused in this way, so messages can continue to appear even after you think you have cleaned the queues.

Cleaning Up the Exchange Server’s SMTP Queues

Warning: This process will delete all email that is due to go to external recipients. Internal messages are not affected, neither are new inbound messages from the Internet unless they are from the spammer continuing to try and abuse your server.

Capturing the Messages Into a Single Queue

This process requires an SMTP connector for all addresses. If you don’t already have one (with a * on the namespace tab) then you need to create one using the instructions below.
If you already have an SMTP Connector with a * on the namespace tab, then you can use it for this process. You will need to adjust the settings as appropriate. You may wish to just make a note of the settings, delete the connector and create a new one for this process. When complete recreate your live connector.

1. In ESM, Connectors.
2. Right click on connectors and choose New, SMTP Connector.
3. On the “General Tab” type a name for the connector. “Spam Cleanup” or similar.
4. Click the “Add” button under “Local Bridgeheads” and choose your Exchange server.
5. Click on the “Address Space” tab.
6. Click “Add” and choose SMTP. Leave each setting (* and cost of 1) and press ok.
   If all the spam is to one domain, then you could remove the * and enter the domain that the messages are being sent to. This should leave legitimate messages in the queue.
7. Click on the General tab again. Change the option in the centre from DNS to “Forward all mail through this connector to the following smart hosts”.
8. Enter an invalid IP address in square brackets:  [99.99.99.99].
9. Click on the “Delivery Options” tab and ensure that “Specify when messages are sent through this connector” is selected.
10. Change the option to 11pm. (If it is close to 11pm when you are doing this, use a much earlier time – 6am or similar. The time doesn’t matter as long as it is not close).
11. Press Apply/OK to close the SMTP Connector dialogue.
12. Restart SMTP Virtual Server.
    1. Expand Servers, <your server>, Protocols, SMTP.
    2. Right click on the “Default SMTP Virtual Server”
    3. Choose “Stop”. This may take a few minutes.
    4. Once it has stopped, right click again and choose “Start”.

The Exchange SMTP virtual server is now processing all the messages and placing them in to a single queue for your SMTP connector. This can take some time. You may want to wait until the number of messages in the queue stays constant before attempting the next stage.

Exchange 2000: The queues can be found in Servers, <your server>, Protocols, SMTP.

Exchange 2003: The queues can be found in Servers, <your server>, Queues.

You may also find the Servers listed under an administrative group.

To locate the required queue, look for a small red clock on the yellow icon. This indicates that it is on a timed delivery.

Deleting the Messages

Now that the messages are in one queue, it is quite easy to delete them

Exchange 2003

1. Right click on this connector and choose “Find Messages”.
2. In the drop down box select the number of messages to be listed in the search.
3. Click “Find Now”.
4. Once the search is complete, select all of the messages (use the shift-page down key combination)
5. Then click “Delete all Messages (No NDR).

Exchange 2000

1. Right click on this connector and choose “Delete All Message (No NDR)”
2. Select Yes when asked if you want to delete all the messages in the queue.

Once the messages have been deleted, which could take some time, refresh the queues to ensure that they don’t continue to build. If they do then Exchange is still processing the messages. You will need to repeat the procedure to delete more messages until the queues are completely clear and stay at zero.

Once you have flushed out the messages, undo the changes that you have made.

If it was a new SMTP connector, delete it.
If you adjusted an existing connector, put the settings back how they were. Don’t forget the time on the “Delivery Options” tab. it should be “Always Run”.

Finally restart SMTP virtual server to get Exchange to start using the new settings.

If you closed port 25 during this process, then remember to open it up again.

Alternative Queue Clean Up Method

If you have a very large number of messages, then there is a command line tool that you can get from Microsoft.

ftp://ftp.microsoft.com/pss/Tools/

Then go in to the folders: Exchange Support Tools / Aqadmcli
(Due to the use of spaces in the folder names, a direct link isn’t possible)

After downloading the utility use the following command to clear all the queues.

aqadmcli delmsg flags=all

---

Clear up “Bad Mail” (Exchange 2000 or Exchange 2003 without SP1)

Messages that have been stuck in the queue but cannot deliver will usually end up in the “badmail” folder. This folder can take up a lot of space. You should remove the content of this folder to free up some valuable space.

Exchange 2003 SP1 and higher does not use the Badmail folder unless you specifically enable it via registry hack.

1. Locate the Badmail folder
   The default location is C:\Program Files\Exchsrvr\Mailroot\Vsi 1
2. DO NOT OPEN THE BADMAIL FOLDER. It may contain a large amount of messages which could make the server unresponsive.
3. Right click on the folder “Badmail” and choose “Rename”. Give it a new name.
4. While holding down the shift key, press the “Delete” key on your keyboard. Click yes to delete the contents immediately. Holding down shift bypasses the recycle bin, so make sure that you have selected the right folder.
5. Exchange will recreate the “Badmail” folder when it needs to.

There are various techniques for dealing with the badmail folder. This blog posting outlines a useful script that you can use to do it for you:http://hellomate.typepad.com/exchange/2003/07/dealing_with_ba.html

---

Email Blacklists

If you were an open relay then you may have ended up on some of the blacklists. When the message bounces back you will get a reason code which should include which blacklist has rejected you.
To get off the blacklist you will have to find their web site and follow their procedure.

As a short term measure, setup an SMTP connector to send all your email via your ISPs SMTP server.

Using Email Blacklists (Exchange 2003 ONLY)

If you want to use an Email Blacklist yourself, then you will need to setup filtering. This article at MS tells you how:
http://support.microsoft.com/default.aspx?kbid=823866

Those of you using an older version of Exchange will have to use a third party tool – whether this is commercial or open source. Vamsoft’s ORF has blacklist support.