Stop Helping Spammers!

(or "How come my email bounced and what do I do about it?")

Hi there! We've put this list of questions and answers together to have a single place with information about our email-handling policies and procedures.

If you are a member of our service and you were referred to this page as a result of contacting us, it's because you referenced a problem with receiving email, but you did not provide enough information for us to help out.

If you aren't a member of our service and you were referred to this page, somebody (maybe us) thinks you've got a problem with an email server you use or run, and they'd like you to look into it.

Questions

  1. What is the NearlyFreeSpeech.NET policy on accepting email?
  2. Why is this page called "Stop Helping Spammers!?"
  3. I'm a member of your service, and I really need to get mail from somebody, but it got blocked. What do I do?
  4. How come the person emailing me claims that this never happens to them when they email someone else?
  5. What error messages indicate that this happened to me?
  6. Why don't you send back easy-to-read error messages?
  7. What do the standards require for my server's HELO/EHLO information?
  8. Why does my email server's DNS matter?
  9. What does NearlyFreeSpeech.NET do differently from other places that follow these rules?
  10. How effective is your email policy?
  11. Wouldn't it be better just to run a spam content filter?
  12. How should I configure my mail server to avoid problems?
  13. What mail server software is responsible for most problems of this nature?
  14. Who do you think you are, telling me how to run my own mail server?
  15. I can't fix my mail server because I don't know how. What should I do?
  16. I can't fix my mail server for political/bureaucratic reasons. Will you make an exception for me?
  17. I need to receive email from somebody who can't/won't fix their server. Will you make an exception for me?
  18. Why make such a big deal about it?
  19. Do you lose business over this?
  20. What do you do to people who try to spam using your service?

Answers

  1. What is the NearlyFreeSpeech.NET policy on accepting email?

    NearlyFreeSpeech.NET mail servers accept mail from properly-configured servers administered in accordance with Internet standards and best current practices for mail and DNS that are not listed in one of the blackhole lists we use. Email is accepted only for NearlyFreeSpeech.NET personnel and systems, and for NearlyFreeSpeech.NET members who have domains with email forwarding enabled.

    Accepting email from broken servers facilitates spam by making it more difficult to tell a spam forgery from an otherwise-legitimate message sent by a misconfigured server. It makes it easier for viruses to propagate rapidly.

    Accepting email from broken servers also encourages system administrators to neglect email server maintenance. Maintaining a "whitelist" of "broken servers run by people we mustn't impose upon to fix them" is even worse; it serves to shift the burden created by a broken server from the person who is (supposed to be) responsible for that server to every conscientious mail server operator in the world.

    In short, accepting email from broken servers harms the Internet, and NearlyFreeSpeech.NET does not do things that harm the Internet. Therefore we cannot accept email from broken servers.

  2. Why is this page called "Stop Helping Spammers!?"

    Would you open this email?

    From: Evil Spammer <liar@criminal-enterprise.com>
    To: You <you@youraddress.com>
    Subject: I would like to rip you off.

    Of course not. You'd delete it on sight, but only if your anti-spam software didn't get to it first.

    Spammers know that. They may be duplicitous, lying cheats, but they aren't (all) stupid.

    They know that nobody wants to read the junk they pump out.

    They know that they have to trick you into reading it.

    They know they have to trick your anti-spam software into letting you read it.

    They know that they have to hide their tracks, because once people figure out where they're sending it from, they'll be kicked off and have to start over somewhere else.

    That's why the lies in spam emails start long before the sales pitch in the body of the message. When a spammer connects to the email server that handles your mail, they lie about who they are and where they are. They pick legitimate domains at random and pretend to be part of them. They connect through networks of virus-infected PCs to mask their origin.

    But that doesn't make sense, does it? Surely we can figure that out, right? All this fancy Internet security and nobody's servers can figure out if an incoming message is a fake?

    No, many servers can't. It's not that the means to do so do not exist; they do. There are detailed checks that can be used to determine if a particular message is legitimate or not to a very high degree of accuracy. It's just that they've been turned off.

    You see, there are many misconfigured email servers out there on the Internet. For one reason or another, their email doesn't "look right" when it gets where it's going. If their emails get checked, they won't pass.

    That means that when there's a problem, there are two ways to make it go away. One is to fix it. The other is to try to get everyone to ignore the problem. For a long time, plan B was the more popular, simply because it's a little easier. The result is a big gray area that spammers exploit to conceal their identity. How do you tell a message that's not legitimate by accident from one that's fake on purpose? You can't.

    Together, creating an environment where checks on the legitimacy of email are unpopular and allowing broken servers to flourish are directly responsible for empowering spammers to choke all of our Inboxes with unwanted garbage.

    That's why we do use every legitimacy check at our disposal on every email message we process. That's why we insist that the email we process be demonstrably legitimate. And that's why we exhort everyone who sends or accepts illegitimate email, "Stop helping spammers!"

  3. I'm a member of your service, and I really need to get mail from somebody, but it got blocked. What do I do?

    First, please understand that such messages never arrive at our servers and consequently there is nothing we can do to recover them.

    If you are a subscription member, you can contact us with the approximate time a message that bounced was sent (and please don't forget to include the time zone) and the email address of the person who sent it to you. If you aren't a subscription member, you can try asking on our forum, providing the same information.

    The more info you can give us, the better. It's best if you can provide a copy of the bounced message, which usually explains the exact problem, or have the affected person forward it to us at support@NearlyFreeSpeech.NET.

    Also, the sooner you let us know about the problem, the better; we only keep the relevant logs for about a week, so we won't be able to look into anything older than that.

    We won't fix the problem; we can't. But in some cases we can do some basic investigation, then figure out who to contact at the organization responsible and let them know. About 9 out of 10 times, they fix the problem, often within an hour or two. The tenth place doesn't care if you get their email or not, so in those cases you'll have to make some other arrangement to get mail from them.

  4. How come the person emailing me claims that this never happens to them when they email someone else?

    It does; it happens all the time and they don't know it. Many servers just silently discard such email or refile it into "Spam" folders alongside hundreds of other junk messages. That forces them to constantly wonder what happened to any given message. The only thing we do that is unusual is to immediately and "loudly" refuse the message so they can see there is a hidden problem making their email service unreliable.

  5. What error messages indicate that this happened to me?

    Here is a list of the most common errors and a brief explanation of each:

    • 450 <your.example.com>: Helo command rejected: Host not found

      The name your mail server gives in its EHLO/HELO must exist and be valid.

    • 450 <user@example.com>: Sender address rejected: Domain not found

      You can't send email through our server from a domain that doesn't exist.

    • 504 <10.11.12.13>: Helo command rejected: need fully-qualified hostname

      You sent your IP address in an EHLO/HELO command. Viruses and spammers do this all day long, thousands of times per server per hour. Half the time they even lie and try to tell us they're on our own network. (Won't work.) They even try to tell the server they're connected to that it's talking to itself! (Still won't work.) Although using an IP address in the HELO field is technically allowed by the SMTP standard, it's intended for use between email clients and email servers only, not between two email servers.

    • 504 <something>: Helo command rejected: need fully-qualified hostname

      Just like it says, the server needs to send a fully-qualified hostname, not whatever thing you sent. RFC 821 says that a server should send its exact hostname so that if subsequent delivery attempts fail, the server can be precisely identified and/or notification can be sent back to the appropriate party.

      This can also happen if you point your Windows email client directly at our mail server, thinking you'll use it to relay mail to all your friends. Ain't happening.

    • 550 Client host rejected: cannot find your hostname, [10.11.12.13]

      Your mail server does not have working reverse DNS.

    • 554 Service unavailable; Client host [10.11.12.13] blocked using ...

      Your server is listed in one of the real-time blackhole lists we consult. Contact the list operator for further assistance. Come on, they gave you a URL and everything.

    Note that the 4xx errors mean "try again later" and the 5xx codes mean that the failure is permanent. We use 4xx errors when we can't tell whether the problem is a temporary DNS glitch or a real DNS problem. The 4xx code tells your server to try again a little later in case the glitch has gone away or the DNS problem has been fixed. (Hint, hint.)

  6. Why don't you send back easy-to-read error messages?

    There are two primary reasons why the error messages generated by our policies are not clearer and easier to read.

    First, the email software we use does not currently allow the errors it generates to be customized. Therefore, the error text it creates is accurate, but somewhat terse.

    Second, we do not generate the message containing the error report. That is the responsibility of the sending server. It determines the format and content of the non-delivery report it sends, and apart from one line that tends to appear around the middle of that report, that content is totally beyond our control. Some particularly ill-behaved mail servers even throw away our one line and replace it with their own!

    As if that wasn't enough, there is an additional limitation arising from the age of the SMTP standards that apply to email delivery. It was created at a time when there was no spam, only universities were on the Internet, and email servers were run by experts who trusted each other implicitly. Email servers assumed that if there was a problem with the message, the sending server would have refused to send it. Consequently, there's no error code for "you have a problem." This means that emails are refused using a numeric code that incorrectly indicates a problem with the recipient. Those ill-behaved servers mentioned above unfailingly identify this as "recipient not found" no matter what the text of the error actually says. This is most unfortunate, and the source of a great deal of confusion.

    The result of these factors is that although we cause error messages to be generated, and although we attempt to make the messages as informative as possible in the space allowed, the obscure mutterings of MAILER-DAEMON in response to a problem aren't going anywhere anytime soon. This is certainly the single most significant drawback to our approach.

    At such time as our options become more flexible in this area, we will certainly take advantage of them, as we would very much like to send a simple, clear message to people with problems that makes it clear to them what the problem is and who should be contacted to fix it.

  7. What do the standards require for my server's HELO/EHLO information?

    From RFC 2821:

      A domain name that is not in FQDN form is no more than a local alias. Local aliases MUST NOT appear in any SMTP transaction. [2.3.5]

    [...]

    The domain name given in the EHLO command MUST BE either a primary host name (a domain name that resolves to an A RR) or, if the host has no name, an address literal as described in section 4.1.1.1. [3.6]

    [...]

    The argument field contains the fully-qualified domain name of the SMTP client if one is available. [4.1.1.1]

    [...]

    The SMTP client MUST, if possible, ensure that the domain parameter to the EHLO command is a valid principal host name (not a CNAME or MX name) for its host. [4.1.4]

     

    The RFC also allows an alternate "address literal" form for machines with no hostname, but running a mail server with no hostname doesn't make sense. Address literals are only used when email clients (like those using DHCP) send mail to their local servers. In practice, address literals from other mail "servers" indicate virus-infected clients trying to spread outside their local network.

    We will accept an EHLO/HELO if the FQDN provided is deliverable. That means, it must have either an A record, as the RFC requires, or an MX record. So we are actually less strict than the standard would allow us to be. This helps many cases where the mail server's hostname is problematic, but it can be configured to EHLO with the domain it handles email for, and also some NAT-related cases where the apparent name isn't externally visible.

    We do not verify the EHLO name against the reverse DNS for the server. Rejecting email based on a mismatch there is specifically prohibited by RFC 2821. This check, which some overzealous people do perform, causes bogus errors in NAT environments and where more than one physical server may be assigned to handle mail for a single IP address (for example by using Layer 4 switching).

    We do not accept address literals since all of our local mail clients are configured not to send them. If you have an email server using an address literal, you should either fix it or configure it to forward all outbound mail to your ISP's mail server. It should be more inclined to regard your server as a "client" and allow an address literal.

  8. Why does my email server's DNS matter?

    DNS has two components, forward and reverse. The presence and validity of both of these components is the only way to be reasonably sure that a mail server belongs to who it says it does. Either forward or reverse DNS on their own can be easily subverted by spammers to forge email to appear as though it came from your domain, leading to large numbers of bounce messages and complaints about email you never sent.

    If we just check reverse DNS, spammers can (and do) buy service from unsuspecting ISPs and set up their reverse DNS to say things like mail.example.com. Next thing you know, they're spamming, and it looks like you did it. But if we check mail.example.com, their plan is foiled because only you control example.com and mail.example.com won't match what they claimed.

    If we just check forward DNS, spammers can use any valid name from any IP address they want, and all we've done is ensure that they're forging a legitimate domain name (yours for instance). That makes the spam much harder to identify, and once again it looks like you are the culprit.

    While not perfect, matching up forward and reverse DNS is a powerful anti-forgery measure. It is easy for you to set up DNS properly and very hard for evildoers to fake. By having valid forward and reverse DNS for your mail server, you're making it harder for spammers to pretend to be you, which saves you time and frustration. By checking for it, we're also making it harder for spammers to pretend to be you.

  9. What does NearlyFreeSpeech.NET do differently from other places that follow these rules?

    We refuse messages much earlier in the delivery process. This not only keeps the junk off of our servers entirely, but it causes the resulting error messages to make it back to the sender much more often. Most places filter such messages later on, either tagging them as junk or silently discarding them. That's easier and less confrontational, but not really helpful in the long run.

    We contact the people who run servers with problems like these and try to help them fix things.

    We try to make as much noise about this problem as possible, so people can see and understand how serious the issue is.

    Another nice but minor benefit is that certain spam software that connects directly to target mail servers will remove an address from the list if it gets a permanent delivery error. This is a little rare, but every little bit helps, and it does mean that our members are on a below-average number of spam lists.

  10. How effective is your email policy?

    For every 1,000,000 messages blocked, all but approximately 50-100 appear to be either spam or viruses. That's more than 99.99% accurate. Also, the number of spam and viruses being blocked is constantly increasing, and the number of broken email servers being blocked is trending downward, suggesting that this is the only email-handling policy that not only works, but actually gets better with the passage of time.

    That makes perfect sense when you realize that this is the email policy that actively improves the email network, rather than trying to work around increasing breakage.

    On top of that, as a result of our policy, we've been able to help find and fix dozens of broken email servers. Every server that gets fixed makes spam harder to send and easier to identify, making the Internet a better place. That's the kind of "effective" that really counts.

  11. Wouldn't it be better just to run a spam content filter?

    A strong email policy and a good spam filter are orthogonal and mutually reinforcing.

    Overall, our email policy is tremendously effective at stopping the vast supermajority of the virus propagation attempts that reach us, because viruses tend to have horrible SMTP implementations, and generally can't hide the fact that they're on a machine that isn't supposed to be a mail server anyway.

    Although the vast bulk of the rest of the mail we block is spam, a fair amount of spam does get through. In that sense, our email policy is not very effective at blocking spam. But that is OK, because that is not its intent. Its intent is to provide a clear audit trail that assists subsequent determinations based on the content of a message.

    For a spam filter to be good, it really needs to be specific to the recipient -- one person's spam is another person's ham. That means spam filtering is about content, and it's really a form of censorship. Censoring content isn't really what we do here; while we definitely believe people ought not to have to see spam, we think they ought to have the freedom to decide for themselves what is and isn't spam, rather than have a third party take the decision out of their hands. So we resisted spam filtering ourselves for a long time. However, what we found was that relatively few of the people who use our email forwarding service handle their own email; people who do that tend not to need our email forwarding. So, the bulk of our email forwarding domains get forwarded to other email providers (gmail, hotmail, ISPs, etc). All of those have their own spam filters, and they don't share our views on third-party censorship; they readily (and frequently automatically) block servers that send too much spam, each using their own slightly different definition of spam. So we were getting blocked fairly often for forwarding messages that were easily identifiable as spam. (Ironically because the only spam our email policy lets through is easily identifiable as spam.)

    For that reason, we've essentially been pushed into performing some rudimentary spam/virus checks designed to do the minimum necessary to keep us off blacklists and let spam filters closer to the end user do a better job. Thus, our system will divert the most egregious spam into a special quarantine (checkable from our website). Everything else gets passed through. The goal is to block just enough of the worst spam to stay off of blacklists, and let anything even remotely questionable through so the recipient or a recipient-specific spam filter can hopefully make the final decision.

  12. How should I configure my mail server to avoid problems?

    Here are the surefire ways to be a good Netizen and stay out of trouble:

    • The name you present in EHLO/HELO must be valid for mail delivery.

      That means it must have either an A or an MX record in the DNS. It does not need to match the apparent IP/hostname of your mail server, but it doesn't hurt anything if it does.

    • Your IP address must have valid reverse DNS in the form of a valid PTR record and a matching A record.

      If your reverse DNS exists, but it's 13.12.11.10.yourcompany.com (or worse, 13.12.11.10.yourcompanysprovider.com) and a forward lookup for that name fails, that's not valid.

      It is a very common spammer trick to set up reverse DNS to announce that their spam-sending server belongs to some respectable (but misconfigured) company; it's trivial for them to fake the PTR record. But it's nearly impossible for them to fake the matching A record, so to distinguish yourself, you need both. Similarly, it's also not valid to have only an MX record that matches the PTR because then spammers can use domains with wildcard MX's to forge spam.

    • Use Sender Policy Framework (SPF) if the option is available to you.

      A lot of otherwise lethal misconfigurations can be overlooked if you have and use SPF records for your domain. Matching SPF will override most of the other checks (except the blackhole lists). However, SPF isn't suitable for every situation; if email from your domain doesn't always come from the same server(s) that you control, SPF is not for you. But it's quick, easy, and fun, so use it if you can. (Note: Because our members' email handling is asymmetric, they are generally in the category of people not well served by SPF.)

    That's it; that's all it takes to avoid an entire class of delivery problems that is only going to get worse with time. Note that each of the above can be accomplished by a proficient system administrator in a matter of minutes. That's why we don't have much sympathy for people who can't get it done.

  13. What mail server software is responsible for most problems of this nature?

    By an incredibly wide margin, servers with standards-compliance problems are running Microsoft Exchange. We've even run into several Exchange servers that refuse mail for "postmaster" at the domain they are handling, and it's hard to find an easier rule to follow than that one.

    Unfortunately, these are the servers that tend not to get fixed when problems are reported. This is in part because Microsoft sold that company's IT department a bill of goods about how great Exchange is and how easy it is to set up. Consequently, there appear to be a few people who stick in the CD, click "Install" and hope for the best. Bad idea.

    For our recommendation to people in this situation, please see the "I can't fix my mail server," question below.

  14. Who do you think you are, telling me how to run my own mail server?

    We are not telling anyone how to run their email servers. We are merely publishing the details of our policy on accepting email. No one who does not wish to follow these simple, standards-based rules is under any obligation to send email through our servers.

  15. I can't fix my mail server because I don't know how. What should I do?

    Turn it off and send email through your ISP's mail server. They have a huge advantage, because their business depends on investing the manpower and resources to do it right.

    If you have had problems delivering email to our members, you may send us email and ask nicely, and if we have the time available, we can try to point you in the right direction as far as identifying the problem. Fixing it will be your own journey of discovery, however.

  16. I can't fix my mail server for political/bureaucratic reasons. Will you make an exception for me?

    There are no exceptions to our email policy.*

    *Well, there's one. No matter how screwed up your server is, you can always send email to our issue tracking system (including support@NearlyFreeSpeech.NET, abuse@NearlyFreeSpeech.NET, and postmaster@NearlyFreeSpeech.NET) so we can give you feedback on your efforts to fix it.

  17. I need to receive email from somebody who can't/won't fix their server. Will you make an exception for me?

    No. See above.

  18. Why make such a big deal about it?

    In the beginning (once the Internet spread past ivy-wreathed walls into the public eye), this was the attitude. The mad grab for cash was on, and ISPs had no incentive to force their mail-server-running customers to do things right, either because they didn't know enough to care, or because they were worried those customers might get pissed off and quit. After all, who cares about dusty old standards as long as the money flows and the email gets delivered?

    Then the spam came. And came. And came. Now, the majority of all email is spam and CNN runs bits about people who are going back to writing letters because sorting the crap out of their inbox is too much work.

    To handle it all, ISPs and other large companies pile on mail server after mail server, firewalls and filtering tools, to try to deal with all the crap. Anti-spam is a thriving cottage industry, and spam itself is big business.

    At the same time, mail servers all over the place break (second law of thermodynamics) and don't get fixed because overworked sysadmins have more urgent problems; and as long as email isn't actually bouncing, who cares if the Received: header has a typo in it?

    All this to keep from risking that revenue from a few big corporate clients who might threaten to quit if they were forced to do it right.

    But at some point, someone at an ISP sat down and figured out that the constantly escalating cost of dealing with all this junk is now more than the revenue that would be lost if all the customers who threaten to quit if they have to play by the rules actually quit because they had to play by the rules. A lot more.

    Meanwhile, accountants at Gmail and Hotmail and MSN and Yahoo! figured out that they didn't have any corporate customers with misconfigured email servers who would quit, but that they were spending tens of millions of dollars on spam and virus management anyway. There's no way to characterize that other than paying to clean up someone else's mess. And, thanks to that second law, the mess just gets bigger and bigger.

    And overnight, the worm turned.

    Now we have SPF, Domainkeys, Sender ID, and a host of other technologies advanced by big email providers. We have solutions like "Goodmail" which doesn't care how screwed up your email is as long as you're willing to pay extra to have it delivered. And they all, to varying degrees, have implemented some or all of the same checks that we perform, and they've all made noise about implementing the rest.

    In other words, if the people affected by our policies don't like it, those are the very same people who may soon be paying a "misconfigured server" tax to the likes of America Online to get their email delivered.

    So, to make a long story short, fixing your email to talk to us may keep you from having to pay to talk to them.

  19. Do you lose business over this?

    Yes.

    We have no urge to be the biggest web hosting company on the Internet, particularly if the only way to get there is pander to the lowest-common-denominator mentality and make the Internet we love a haven for spammers and their ilk. We just want to be the best web hosting company on the Internet, and if you want to be the best, you start by doing good. Fixing this problem is good, and in the long run, it will hurt a lot less than the alternative.

    We are aware that not everyone shares our philosophy, and that some choose other email services because of it. That is their prerogative.

  20. What do you do to people who try to spam using your service?

    Sorry, but our privacy policy prevents us from telling you where their remains are buried.