Results 1 to 4 of 4

Thread: Whitelisting - How Does Mail Work? - IPod Touch and E-mail

  1. #1
    PowWeb Staff
    Join Date
    May 2006
    Planet Earth - for now!
    Rep Power

    Whitelisting - How Does Mail Work? - IPod Touch and E-mail

    First: White, Grey and Blacklists only apply to mail entering from outside the system (external mail).

    Second: After a systemwide RBL and Disallow list check, the account Whitelist is checked next -- if the address and IP combination are present, the mail message is delivered directly to the intended mailbox bypassing greylisting and Spam Threshold.

    Third: If a an email message address is not on the whitelist, the blacklist would be checked next (so that the message could be discarded), then if Greylisting is on, it is checked against the greylist and autowhitelist before being sent on to the Spam Threshold Filters...
    If Greylist is off, the mail flows to the Spam Threshold filters (SA) and then to the mailbox.

    By using a whitelist entry for an inbound email address, you circumvent/prevent the rest of the scanning process from being followed... this should make the delivery more immedate and reduce any risk that the mesage will be "trapped" somehow in the filter.

    As you note, it is necessary to Whitelist (or blacklist) the X-ORIG-SENDER value from the header to have this work correctly -- which, I concur is a pain -- one would think in this day and age the code to check mail headers would be advanced enough to to pick this value up and add the correct entry to the table... who knows, perhaps that will eventually be available/implemented.

    And, I concur; with Greylisting Off and Spam Threshold off, one would assume that whitelisting shouldn't matter ... but, I have seen situations in which it did...

    Aside from the above, which I believe to be accurate for the platform --- I'm not involved with the Powweb mail system directly, so the following is just guesses based on what I've seen in other mail systems I've been exposed to.

    I'm not privy to the exact way the system works, but having looked at a number of these implementations, I suspect the Spam Threshold "Off" is not "turned off" but rather "scoring off" -- so that the mail might still be scanned and scored, but the results are not applied to filter the mail or trigger the alternative delivery (e.g. prefixed by a word like SPAM, or sent to a SPAM Folder, etc.). In any event, the mail does "flow through" the filter unless its on the Whitelist.

    So, the whitelist entry, which causes the mail scanning "not" to occur for a specific message -- since the system is "going to deliver the mail anyway", I think actually provides the greater assurance of message delivery.
    Well, that's the way I see it -- for the moment!

  2. #2
    PowWeb Staff
    Join Date
    Mar 2006
    Near Boston, MA
    Rep Power

    How Does Mail Work?

    There's been a bunch of questions about how our mail system works. What happens to incoming and outgoing mail, where the greylisting interaction occurs, how spam scanning works. I (with the help of one of our brilliant technical specialists) have tried to break it down and give as much info as we can without being overwhelming. It still might be overwhelming. But hopefully it's useful.

    We have 3 basic layers in our mail system.
    1. Incoming - Accepts mail from remote servers
    2. Internal - Accepts mail from the Incoming layer, as well as other internal sources such as CGI, AuthSMTP, Web mail, etc.
    3. Outgoing - Accepts mail from Internal layer and sends to remote hosts

    We have 4 basic mail services.
    1. AuthSMTP - accepts mail from users with a mailbox/password on our system
    2. POP/IMAP - accepts connections from clients and allows you to get your mail
    3. Web mail - basically IMAP over the web
    4. CGI - takes email from our CGI pools, add some headers to track abusers, and logs usage (so we can turn off people who are sending thousands of spam messages, hopefully preventing exploited scripts from landing our servers on blacklists)

    For an approximate understanding of the scale of our mail system:
    Incoming servers - about 30
    Scanning servers - about 60
    Outbound servers - about 20
    SMTP - about 10
    POP - about 25
    Web mail - about 15

    We run various versions of Exim as our mail transfer agent (MTA, what handles the mail) and Courier as our POP/IMAP service. We don't run native sendmail anywhere (it's just a pointed to Exim).

    Incoming Layer
    Our incoming MTA handles many millions of messages a day. 90+% of that mail is spam. Greylisting, our anti-virus/malware service (ClamV), and our internal rule checks are in this layer.

    When a message gets rejected at this layer, it generally gets a 4XX or 5XX response code (temporary or permanent rejection).

    The greylisting service is checked first. The "triplet" (Sending IP, Sender Email, Recipient Address) is compared to the receiving domain's whitelist, then the blacklist. If whitelisted, it's sent to the inbox. If blacklisted, it's rejected. Next, the triplet is compared to the auto-whitelist (addresses you have mailed). If that fails, it checks the greylist. If it's the first time we've seen the message, it's temporarily rejected (that's greylisting, in a nutshell). If it's not the first time we've seen this triplet, the message is accepted.

    Now, because greylisting occurs at the Incoming layer, it does not work on internal domains (i.e. mail that goes from user to user in our network). This can be problematic, but it's not generally proven to be an issue.

    The rules are checked next. Basically, this says "is the mail on any of the spam blacklists, or from a null sender, or from a non-resolving domain, or not using SPF encoding, or including a nefarious extension, or with a really really really spammy subject". If the mail triggers any rules, it's bounced with the 4XX/5XX message.

    Once all that's happened the mail is sent to the ...

    Internal Layer
    These boxes are only accessible on our private network. When the message is received, if the destination mailbox has spam scanning turned on, the message is routed through SpamAssassin (today, that could be some other spam scanning application in the future). We add SpamAssassin headers to the messages at this stage.

    This is also when the mail quota is checked and messages are bounced if the mailbox is over quota.

    The message is delived to the mailbox.

    Now let's say you're sending an outgoing message. Your mail would be handled by the Internal Layer then the ...

    Outgoing Layer
    First, a bit more on the Internal Layer:
    Any message going outbound has to go through through the Outbound SpamAssassin scan. Messages need to stay under a score of 6.0 (most ISPs use 5.0 as the threshold, but we're bumping it up a tiny bit to allow more borderline mail through). Once it passes that, the address is rewritten using the SRS/SPF encoding scheme. The MX record for the recipient server is checked here and all of it is passed along to the Outgoing layer.

    Obviously, our outbound servers send mail to remote MTAs (AOL, Verizon, your local ISP or college). One problem that most shared hosts have is that there's no way to prevent spammers (or well-intentioned people who are inadvertently acting like spammers) from mucking up the works for everyone else. We have a myriad of things we do to attempt to keep our outbound servers from ending up on blacklists.

    First, we've got whitelist relationships with most of the big mail folks (AOL, Verizon, etc.). This doesn't mean that we'll never get blacklisted, it just means it's less likely, assuming we do certain things.

    Second, we have a number of IP addresses that can be assigned to each of our outbound mail servers. When one is blocked, we can rotate that IP address to a new one. Since most blocks are temporary, by the time it rolls back around again, the block is gone. This means that, in general, we're only blocked on individual servers for a few hours at most.

    Third, we have a limit on the number of messages that can be sent. This has been pretty widely discussed, but we have a rolling limit on the messages you can send in an hour or in any 24 hour period. Quite frankly, there are very few customers who ever run afoul of this. It is in place to simply limit the damage any individual person can do.

    (As an aside, if your business relies on sending thousands of messages a day, you should have a dedicated mailing system of your own, where you can ensure that you're never blacklisted and are abiding by the CAN-SPAM act. You shouldn't be relying on a shared web host to handle your thousands of messages a day. It's simply too risky for your buisiness.)

    Finally, your message goes out. If it isn't accepted, we'll retry it every 15 minutes for 2 hours. Then for the next 14 hours, we'll up the retry interval 1.5 times. At 16 hours, we'll retry it again, then fail.

    That's our mail system. It handles millions of messages a day (as high as 120 million over the holidays), with the vast majority inbound spam.

    Some suggestions
    1. Don't use a catch all: Catch-alls are kind of antiquated. They're also spam magnets. If you're going to use a catch-all, certainly do not forward its contents anywhere. That's just a recipe to get our servers blocked for spam.
    2. Make sure your mail has text content: The most common spam these days is spam that has an image rather than text. Lots of hosts have started blocking mail that has no (or limited) text content. Make sure your message has some text content. Ideally, it would have as much text as it does HTML/image content.
    3. Don't set up a forward to forward your mail externally: If you can avoid doing it, don't forward your mail to your AOL or Comcast address. That will forward your spam, and then when you're being a good citizen, you'll mark the message as spam, and thus ding our servers as the spam sender!

    Hopefully that's helpful in giving a little insight into our systems, why they're setup the way they are, what we're doing to combat spamming, scamming, and keep ourselves off of blacklists. We know it's not perfect, which is why we're constantly updating and refining things. We're currently (and have been for a few months) evaluating new spam scanning technology (really really expensive stuff!) to help reduce the false positives on incoming and outgoing mail.

  3. #3
    PowWeb Staff
    Join Date
    May 2006
    Planet Earth - for now!
    Rep Power

    More details

    The precedence for inbound mail from an external source:
    1. Commercial RBL (SURBL, etc.) check at Server level
    2. Valid recipient address exists on Powweb system
    3. Check against Whitelist entries -- if listed, send immediately to recipient mailbox
    4. If not White, then is it Black? -- if listed, discard message immediately
    5. If not White, and not Black -- is Greylisting enabled?
    6. If greylist enabled -- a) log the entry as hitting the greylist
    b) Is this an entry in the list we have sent to?
    (an address is added/auto-whitelisted for 7 days -- if it is already in the list
    it is not updated, it expires at the end of 7 days and then gets re-added the
    next time you send -- the "sent to" list works differently than the greylist which renovates the date each time it is seen)
    If Yes to "sent in 7" then accept the inbound message
    c) If not "sent in 7", but already in the log (matching address + IP)
    then accept this email and forward to the Spam Threshold Filter
    ( if it is active (YES)) - otherwise deliver to the recipient mailbox
    d) Otherwise, we have not sent in 7, or seen the message sender combo before ---
    return a a 452 GL/GL message to the sending server advising that the mailbox
    is temporarily unavailable; the sending server should back off 10-15 minutes at minimum, then re-present the email from the same server (IP and address combo), [it should always retry within 4 hours, if it doesn't, the server is not following the RFC rules)
    This second send should hit rule/item (c) - logged as already seen, so the message is accepted the 2nd time it is seen.

    Remember that if the Spam Threshold filter is on (YES) -- even after a message clears the greylist, it is subject to additional filtering level based on the setting of the recipient mailbox.

    How a Greylist Entry is Formed:
    There are 3 elements to a greylist entry: the recipient address, the sending IP, and the sending email address. Once this triplet is greylisted, emails from the same triplet will be "ok" for the next 7 days. If any one of the triplet changes, the address is greylisted (the sender is given the GL/GL bounce message - "mailbox temporarily unavailable") and must wait for a retry. An "ok" greylisting entry is valid for 7 days since the last time that triplet was seen. In other words, each time an email is received from that triplet, the 7 day clock is reset (the datestamp is "renovated") and the triplet is ok for another 7 days.
    Well, that's the way I see it -- for the moment!

  4. #4

    Join Date
    Jun 2005
    Brooklyn NY
    Rep Power

    Ipod Touch and E-mail

    Setting up the ipod touch to access my web mail account, I have it working now.

    Here is the setting i used:

    POP Account Information:
    Name: your Name
    Address: email add
    Description: My Name or email

    Incoming Mail Server

    Host Name:
    user Name: My user Name
    Password: Password

    Outgoing Mail Server (SMTP)

    Host Name: smtp:
    Username: username

    Click on advanced and turn on Use SSL for both Incoming and outgoing

    Authentication: password
    Incoming setting
    Server port: 995

    Outgoing Setting
    Server Port: 993

    Save and it should work
    Last edited by Croc Hunter; 3-16-08 at 07:06 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts