Stop Helping Spammers!

(or "How come my email bounced and what do I do about it?")

Hi there! We've put this list of questions and answers together to have a single place with information about our email-handling policies and procedures.

If you are a member of our service and you were referred to this page as a result of contacting us, it's because you referenced a problem with receiving email. It's either not something we can help with for a reason listed here or you did not provide enough information.

If you aren't a member of our service and you were referred to this page, somebody (maybe us) thinks you've got a problem with an email server you use or run, and they'd like you to look into it.

What is the NearlyFreeSpeech.NET policy on accepting email?

NearlyFreeSpeech.NET mail servers accept mail from properly configured servers administered in accordance with Internet standards and best current practices for mail and DNS that are not listed in one of the realtime blocklists we use. Email is accepted only for our personnel and systems and for our members who have domains with our email forwarding enabled.

Accepting email from broken servers:

  • Makes it more difficult to tell spam forgeries from an otherwise-legitimate message sent by a misconfigured server.
  • Makes it easier for viruses to propagate rapidly.
  • Encourages system administrators to neglect email server maintenance.
  • Shifts the burden created by a broken server from whoever is (supposed to be) responsible for that server to every conscientious mail server operator in the world.

In short, accepting email from broken servers harms the Internet, and NearlyFreeSpeech.NET does not do things that harm the Internet. Therefore, we cannot accept email from broken servers.

Why is this page called "Stop Helping Spammers?"

Would you open this email?

From: Evil Spammer <liar@criminal-enterprise.com>
To: You <you@youraddress.com>
Subject: I would like to rip you off.

Of course not. You'd delete it on sight, but only if your anti-spam software didn't get to it first.

Spammers know that. They may be duplicitous, lying cheats, but they aren't (all) stupid.

They know that nobody wants to read the junk they pump out.

They know that they have to trick you into reading it.

They know they have to trick your anti-spam software into letting you read it.

They know that they have to hide their tracks, because once people figure out where they're sending it from, they'll be kicked off and have to start over somewhere else.

That's why the lies in spam emails start long before the sales pitch in the body of the message.

When a spammer connects to the email server that handles your mail, they lie about who they are and where they are. They pick legitimate domains at random and pretend to be part of them. They connect through networks of virus-infected PCs to mask their origin.

But that doesn't make sense, does it? Surely we can figure that out, right? All this fancy Internet security and nobody's servers can figure out if an incoming message is a fake?

No, many servers can't. It's not that the means to do so do not exist; they do. There are detailed checks that can be used to determine if a particular message is legitimate or not to a very high degree of accuracy. It's just that they've been turned off.

You see, there are many misconfigured email servers out there on the Internet. For one reason or another, their email doesn't "look right" when it gets where it's going. If their emails get checked, they won't pass.

That means that when there's a problem, there are two ways to make it go away. One is to fix it. The other is to try to get everyone to ignore the problem. For a long time, plan B was the more popular, simply because it's a little easier. The result is a big gray area that spammers exploit to conceal their identity. How do you tell a message that's not legitimate by accident from one that's fake on purpose? You can't.

Together, creating an environment where checks on the legitimacy of email are unpopular and allowing broken servers to flourish are directly responsible for empowering spammers to choke all of our Inboxes with unwanted garbage.

That's why we do use every legitimacy check at our disposal on every email message we process. That's why we insist that the email we process be demonstrably legitimate. And that's why we exhort everyone who sends or accepts illegitimate email, "Stop helping spammers!"

I'm a member of your service, and I really need to get mail from somebody, but it got blocked. What do I do?

First, please understand that such messages never arrive at our servers. Consequently, there is nothing we can do to recover them. Nor do we make exceptions to our email acceptance policies.

If you are a subscription member, you can contact us with the email address of the person who sent you a message that bounced and the approximate time it was sent. Please include the time zone! If you aren't a subscription member, you can try asking on our forum, providing the same information.

The more info you can give us, the better. It's best if you can provide a copy of the bounced message, which usually explains the exact problem, or have the affected person forward it to us at support@NearlyFreeSpeech.NET.

Also, the sooner you let us know about the problem, the better; we only keep the relevant logs for about a week, so we won't be able to look into anything older than that.

We won't fix the problem; we can't. But in some cases we can do some basic investigation, then figure out who to contact at the organization responsible and let them know. About 9 out of 10 times, they fix the problem, often within an hour or two. The tenth place doesn't care if you get their email or not, so in those cases you'll have to make some other arrangement to get mail from them.

How come the person emailing me claims that this never happens to them when they email someone else?

It does. It probably happens all the time, and they don't realize it. Many servers silently discard such email or refile it into Spam folders alongside hundreds of other junk messages. The only thing we do that is unusual is to immediately and "loudly" refuse the message. That lets people know when there is an otherwise-hidden problem making their email service unreliable.

What error messages indicate that this happened to me?

Here is a list of the most common errors and a brief explanation of each:

  • 450 <your.example.com>: Helo command rejected: Host not found

    The name your mail server gives in its EHLO/HELO must exist and be valid.

  • 450 <user@example.com>: Sender address rejected: Domain not found

    You can't send email through our server from a domain that doesn't exist.

  • 504 <10.11.12.13>: Helo command rejected: need fully-qualified hostname

    You sent your IP address in an EHLO/HELO command. Viruses and spammers do this all day long, thousands of times per server per hour. Half the time they even lie and try to tell us they're on our own network. (Won't work.) They even try to tell the server they're connected to that it's talking to itself! (Still won't work.) Although using an IP address in the HELO field is technically allowed by the SMTP standard, it's intended for use between email clients and email servers only, not between two email servers.

  • 504 <something>: Helo command rejected: need fully-qualified hostname

    Just like it says, the server needs to send a fully-qualified hostname, not whatever thing you sent. RFC 821 says that a server should send its exact hostname so that if subsequent delivery attempts fail, the server can be precisely identified and/or notification can be sent back to the appropriate party.

    This can also happen if you point your Windows email client directly at our mail server, thinking you'll use it to relay mail to all your friends. Ain't happening.

  • 550 Client host rejected: cannot find your hostname, [10.11.12.13]

    Your mail server does not have working reverse DNS.

  • 554 Service unavailable; Client host [10.11.12.13] blocked using ...

    Your server is listed in one of the real-time blocklists we consult. Contact the list operator for further assistance. Come on, they gave you a URL and everything.

Note that the 4XX errors mean "try again later" and the 5XX codes mean that the failure is permanent. We use 4XX errors when we can't tell whether the problem is a temporary DNS glitch or a real DNS problem. The 4XX code tells your server to try again a little later in case the glitch has gone away or the DNS problem has been fixed. (Hint, hint.)

Why don't you send back easy-to-read error messages?

There are two primary reasons why the error messages generated by our policies are not clearer and easier to read.

First, the email software we use does not currently allow the errors it generates to be customized. The error text it creates is accurate but somewhat terse.

Second, we do not generate the message containing the error report. That is the responsibility of the sending server. It determines the format and content of the non-delivery report it sends, and apart from one line that tends to appear around the middle of that report, that content is totally beyond our control. Some particularly ill-behaved mail servers even throw away our one line and replace it with their own!

As if that wasn't enough, there is an additional limitation arising from the age of the SMTP standards that apply to email delivery. It was created at a time when there was no spam, only universities were on the Internet, and email servers were run by experts who trusted each other implicitly. Email servers assumed that if there was a problem with the message, the sending server would have refused to send it. Consequently, there's no error code for "you have a problem." This means that emails are refused using a numeric code that incorrectly indicates a problem with the recipient. Those ill-behaved servers mentioned above unfailingly identify this as "recipient not found" no matter what the text of the error actually says. This is most unfortunate, and the source of a great deal of confusion.

Although we cause error messages to be generated, and although we attempt to make the messages as informative as possible in the space allowed, the obscure mutterings of MAILER-DAEMON in response to a problem aren't going anywhere anytime soon. This is certainly the single most significant drawback to our approach.

At such time as our options become more flexible in this area, we will certainly take advantage of them. We would very much like to send a simple, clear message to people with problems that makes it clear what the problem is and who should be contacted to fix it.

What do the standards require for my server's HELO/EHLO information?

From RFC 2821:

A domain name that is not in FQDN form is no more than a local alias. Local aliases MUST NOT appear in any SMTP transaction. [2.3.5]

[...]

The domain name given in the EHLO command MUST BE either a primary host name (a domain name that resolves to an A RR) or, if the host has no name, an address literal as described in section 4.1.1.1. [3.6]

[...]

The argument field contains the fully-qualified domain name of the SMTP client if one is available. [4.1.1.1]

[...]

The SMTP client MUST, if possible, ensure that the domain parameter to the EHLO command is a valid principal host name (not a CNAME or MX name) for its host. [4.1.4]

(An "address literal" is an IP address in brackets, like [192.0.2.51], instead of a hostname like mail.example.com.)

The RFC allows an alternate address literal form for machines with no hostname, but running a mail server with no hostname doesn't make sense. (Unless you're spamming.) In practice, address literals are only legitimately used when email clients (like those using DHCP) send mail to their local servers. Address literals from the open Internet typically indicate compromised machines that aren't mail servers that are trying to send spam.

We will accept an EHLO/HELO if the FQDN provided is deliverable. That means it must have either an A record, as the RFC requires, or an MX record. So we are actually less strict than the standard would allow us to be. This helps when the mail server's hostname is problematic, but its software can be configured to EHLO with the domain it handles email for. It also helps when the sending host is behind NAT and its apparent name isn't externally visible.

We do not verify the EHLO name against the reverse DNS for the server. Rejecting email based on a mismatch there is specifically prohibited by RFC 2821. This check, which some overzealous people do perform, causes bogus errors in NAT environments and where more than one physical server may be assigned to handle mail for a single IP address (for example, by using Layer 4 switching).

We do not accept address literals. They are only valid from local email clients, and our local mail clients are configured not to send them. If you have an email server using an address literal, you should either fix it or configure it to forward all outbound mail to your ISP's mail server. Your ISP should be more inclined to regard your server as a "client" and allow an address literal.

Why does my email server's DNS matter?

DNS has two components, forward and reverse. The presence and validity of both components is the only way to be reasonably sure that a mail server belongs to whom it says it does. Either forward or reverse DNS on their own can be easily subverted by spammers to forge email to appear as though it came from your domain. If you've ever gotten large numbers of bounce messages and complaints about email you never sent, this is how that happened.

Spammers can (and do) buy service from unsuspecting ISPs and set up their reverse DNS to say things like mail.example.com. Next thing you know, they're spamming, and it looks like you—the real owner of example.com—did it. But if we check the DNS for mail.example.com to see that it points somewhere else entirely, their plan is foiled. Only you control example.com, so the DNS for mail.example.com won't match their fake reverse DNS. So we have to check the forward DNS.

Without reverse DNS checks, spammers can use any valid name from any IP address they want. All forward DNS checks do on their own is ensure that they're forging a legitimate domain name (yours, for instance). That makes the spam much harder to identify. And, once again, it makes you look like the culprit. So we have to check the reverse DNS as well.

While not perfect, matching up forward and reverse DNS is a powerful anti-forgery measure. It is easy for you to set up DNS properly and very hard for evildoers to fake. By having valid forward and reverse DNS for your mail server, you're making it harder for spammers to pretend to be you, which saves you time and frustration.

When we check this, we're also making it harder for spammers to pretend to be you. You're welcome.

What does NearlyFreeSpeech.NET do differently from other places that follow these rules?

We try to make as much noise about this problem as possible so people can see and understand how serious the issue is.

We refuse messages much earlier in the delivery process. This keeps the junk off of our servers entirely. And when a message is not spam but there's a problem, it causes the resulting error messages to make it back to the sender much more often.

Most places filter such messages later on, tagging them as junk, moving them to a Spam folder, or silently discarding them. That's easier and less confrontational, but not really helpful in the long run.

Another nice but minor benefit is that certain spam software that connects directly to target mail servers will remove an address from the list if it gets a permanent delivery error. This is a little rare, but every little bit helps, and it does mean that our members wind up on a below-average number of spam lists.

How effective is your email policy?

For every 1,000,000 messages blocked, all but approximately 50–100 appear to be either spam or viruses. That's more than 99.99% accurate. Also, the volume of spam and viruses being blocked is constantly increasing. The number of broken email servers being blocked is trending downward, suggesting that this is the only email-handling policy that not only works but actually improves the Internet with the passage of time.

On top of that, as a result of our policy, we've been able to help people find and fix dozens of broken email servers. Every server that gets fixed makes spam harder to send and easier to identify, making the Internet a better place. That's the kind of "effective" that really counts.

Wouldn't it be better just to run a spam content filter?

A strong email policy and a good spam filter are orthogonal and mutually reinforcing.

Overall, our email policy is tremendously effective at stopping the vast supermajority of the virus propagation attempts that reach us. Viruses tend to have horrible SMTP implementations and generally can't hide the fact that they're on a machine that isn't supposed to be a mail server. (In fact, it's so effective that the volume of viruses getting forwarded from person to person has dropped precipitously over the years.)

Although the vast bulk of the rest of the mail we block is spam, a fair amount of spam does get through. In that sense, our email policy is not very effective at blocking spam. But that is OK, because that is not our intent. Our intent is to provide a clear audit trail that assists subsequent determinations based on the content of a message.

For a spam filter to be good, it really needs to be specific to the recipient—one person's spam is another person's ham. That means spam filtering is about content, and it's really a form of censorship. Censoring content isn't really what we do here; while we definitely believe people ought not to have to see spam, we think they ought to have the freedom to decide for themselves what is and isn't spam, rather than have a third party take the decision out of their hands. So we resisted doing any spam filtering at all for a long time. However, what we found was that relatively few of the people who use our email-forwarding service handle their own email. People who handle their own email tend not to need our email forwarding. So, the bulk of our forwarded domains send messages on to the shrinking number of very large email providers. All of those have their own spam filters, and they don't share our views on third-party censorship; they readily (and frequently automatically) block servers that send too much spam, each using their own slightly different definition of spam. So we were getting blocked fairly often for forwarding messages that were easily identifiable as spam. (Ironically because the only spam our email policy lets through is easily identifiable as spam.)

For that reason, we've essentially been pushed into performing some rudimentary spam/virus checks. We try to do the minimum necessary to keep us from getting blocked. We leave as much as we safely can to spam filters closer to the end user that can do a better job. Thus, our system diverts the most egregious spam into a special quarantine (checkable from our website). Everything else gets passed through.

Who do you think you are, telling me how to run my own mail server?

We are not telling anyone how to run their email servers. We are merely publishing clear details of our policy on accepting email. No one who does not wish to follow these simple, standards-based rules is under any obligation to send email through our servers.

I can't fix my mail server because I don't know how. What should I do?

Turn it off and send email through your ISP's mail server. They have a huge advantage. Their business depends on investing the time and resources to do it right.

If you have had problems delivering email to our members, you may send us email. If you ask nicely, and if we have the time available, may be able to point you in the right direction as far as identifying the problem. Fixing it, however, will be your own journey of discovery.

I can't fix my mail server for political/bureaucratic reasons. Will you make an exception for me?

No.

I really need to receive email from somebody who can't/won't fix their server. Will you make an exception for me?

Also no.

Because it's the right thing to do. Spam sucks. Too many places don't care or can't be bothered to get it right, and that allows spam to flourish. We're not going to be one of those places.

You know you lose business over this, right?

Yes.

We have no urge to be the biggest web hosting company on the Internet. Particularly not if the only way to get there involves pandering to the lowest-common-denominator mentality and making the Internet we love a haven for spammers and their ilk.

We want to be the best web hosting company on the Internet, and if you want to be the best, you start by doing good. Fixing this problem is good, and in the long run, it will hurt a lot less than the alternative.

We are aware that not everyone shares our philosophy, and that some choose other email services because of it. That is their prerogative. Our commitment to a safer, healthier Internet is not for sale.

What do you do to people who try to send spam using your service?

Sorry, but our privacy policy prevents us from telling you where their remains are buried.