Index

NAME
STATUS
SYNOPSIS
- METHOD OF PROTECTION
- - 1
  - - Local_check_relay
    - 2
    - - Local_check_rcpt
      - {Bounce}
        
        *
      - 3
      - check_data
      - 4
      - screen_header
        
        From:
        
        Reply-to:
        
        Received:
        
        by
        
        with HTTP
- GENERAL
AUTHOR
USES
EXPORTS
HISTORY
- 0.1
- 0.2
- - 22 Sept 2007
- 0.21
- - 23 September 2007
- 0.22
- - 23 Sept 2007
- 0.23
- - 24 Sept 2007
- 0.24
- - 24 Sept 2007
- 0.25
- - 25 Sept 2007
  - 30 Sept 2007
  - - 1
    - 2
    - 3
  - 0.26
  - - 30 Sept 2007
  - 0.27
  - - 01 Oct 2007
  - 0.28
  - - 03 Oct 2007
  - 0.29
  - - 03 Oct 2007
  - 0.30
  - - 05 Oct 2007
  - 0.31
  - - 08 Oct 2007
  - 0.32
  - - 10 Oct 2007
    - 12 Oct 2007,
  - 0.33
  - - 14 Oct 2007
  - 0.34

NAME

Sendmail::M4::Mail8 - Stop fake MX and most spammers, sendmail M4 hack file

STATUS

Version 0.33 (Beta)

Now running at mail.celmorlauren.com our own mail server, and has been doing so since 0.23

SYNOPSIS

SPAM consitutes the bulk of e-mail on the internet, many methods exist to fight this scurge, some better than others. However we think that this module is the simplest, quickest and most efective, relying as it does on the basic power of sendmail macros for most of its methods.

METHOD OF PROTECTION

1
2
3
4

GENERAL

As all systems have an IP address and most have some sought of domain-name, it is possible to base the protection on wether the IP ties up with whom they claim to be at the <helo> stage. You can set sendmail to be picky about this. But many peoples IP address does not resolve to what they would like, its easy to setup domain to IP via people like network solutions, but the other way round needs a friendy ISP. So as this is a common problem, base the protection on the helo resolving to their IP.

Next check that their domain does not contain their IP encoded somehow, people who are not real MXs tend to have numeric user addresses, this has tuning to control how strict this is.

Keep a record of whom the system has sent mail to, so that we have a chance of spotting spammers using fake bounces to fill up a users email box, at the most paranoid this refuses all bounces. That causes some problems with some systems who use fake bounces to check wether you are an MX, some even come from a completly different domain to the one being talked to at the time!??! Stupid or what?

Next check that the From address is not pretending to be one of your own hosted domains, ie the IP is external and is not known to you as an outside user.

After that noraml sendmail DB files will do the rest, use the cookbook, all you need to know is there.

Sendmail::M4::Utils does most of the work for this module, all this does is format the rules and supply a default name for the hack. Various tuning methods exist, and of course you can add your own rules to this.

This module is non OO, and exports the methods descriped under EXPORTS.

AUTHOR

Ian McNulty, celmorlauren limited (registered in England & Wales 5418604).

email <development@celmorlauren.com>

USES

 Sendmail::M4::Utils    the module created to make testing this easier

EXPORTS

HASH REF = mail8_setup(@_) Sendmail::M4::Utils::setup HASH REF

This configures this module, and is allways required first.

Expected/Allowed values allways as a (hash value pairing), see Sendmail::M4::Utils for hash=>value pairings it expects, the list bellow are either default values or additional for use by this.

    file    SCALAR with default value of "mail8-stop-fake-mx.m4",
    build   SCALAR with default value of 1
    install SCALAR with default value of 1
    test    SCALAR with default value of 1

    paranoid
            SCALAR see heading below for values

0

not paranoid at all, has local users and is content to accept bounces and "callback verify" sudo bounces.

Standard sendmail rules and databases will handle user and bounce requests. This just verifys that the sending host appears to be legimate. Assuming that the hit rate on the system is not too great, use sendmail "milters" as well to take care of "spam" and "viruses"

1

slightly paranoid, has local users and is content to accept "callback verify" sudo bounces, but will refuse any bounce request that really is a bounce, that is a bounce with data.

2

mildly paranoid, is a relay host with no local users, will say OK to all "callback verify" requests that refer to hosted domains, regardless of wether the user exits or not! Refuses all real bounces.

3

paranoid, is a hassled relay host, will just say OK to any "callback verify" request, regardless of wether it relays for that domain or not! Refuses all real bounces.

4

fairly paranoid, is a really hasseled relay host, and has no time for any type of bounce, all refused. Most e-mail and even more bounces are bogus.

copyright(@_)

this should live in /usr/share/sendmail/hack/

some settings that are advised FEATURE(`access_db', `hash -T<TMPF> -o /etc/mail/access.db') FEATURE(`greet_pause', `2000') define(`confPRIVACY_FLAGS', `goaway')

version_id

This is really a reminder to use VERSIONID with your own value, or just use this to use the default

VERSIONID "ANTI SPAM & FAKE MX"

local_config

Required statement, this inserts required statements into the hack file.

This inserts required statements before and after LOCAL_CONFIG, you may add more statements that belong here.

Main items "-" added to confOPERATORS KRlookup for DNS check on HELO host name H*: $>+ScreenHeader to check received headers KMath arith to join the IP address together into a single token KCleanFrom regex to enable checking of Header line "From:"

    KZombie program  /etc/mail/mail8/mail8_zombie
                        this is included in the script regardless 
                        of wether it is installed or not.
                        Included as part of the distro, install it
                        to get the full benifits.

    dnl white list      no other way to to let these past
    Kmail4db hash -o -a.FOUND /etc/mail/mail8/mail4.db

    Standard black list, checked at Reply-to: Header
    dnl standard black list
    Kstdaccessdb hash -o -a.FOUND /etc/mail/access.db

PerlHelpers

This enables the use of the additional Perl scripts to identify and block bogus e-mail hosts, especialy when the site is being bombed by an abusive system.

None of the scripts are currently available on CPAN, and there is no current intention of releasing them at this time, this is mostly due to the extra system setup required, such as interfaces to the iptables firewall script bring used!

If you would like to use these, contact celmorlauren for help.

mail8_db(SCALAR test)

These are configured automatically by the above PerlHelpers, and are only useable with PerlHelpers, if the above has not been defined then this returns without doing anything.

However there is one exception to this, mail4 is always required as there is no other way of allowing "broken mail systems" that you want to accept mail from!

To keep sendmail error messages down (/var/log/mail) ensure you create all the required database's by hand, mail4 at the very least!

For manual creation of files, use vi ###; makemap hash ###.db <### where ### is the database source.

    /etc/mail/mail8/mail9.db        ip (address port 25) to TarPit
                                    in firewall rules
    /etc/mail/mail8/mail8.db        refuse connect to SPAMMER
                                    access.db also does this and more
    /etc/mail/mail8/mail4.db        allow, OK this host would fail tests
    /etc/mail/mail8/mail3.db        single shot, allow, like mail4
    /etc/mail/mail8/mail2.db        relays hosted domains
                                    $=R, $=w, & ${VirtHost} also does this
    /etc/mail/mail8/mail1.db        relays internal hosts by IP
                                    192.168.#.#     assummed local
                                    172.16.#.#      assummed local
                                    10.#.#.#        assummed local

NOTE This files are all optional, so this can be specified even if none of these exist.

The single useable argument if SCALAR will place the DataBases in /var/tmp/mail8, which enables you to test with alternate files to the running version.

local_rulesets

Required statement, this inserts required statements into the hack file.

This inserts required statements before and after LOCAL_RULESETS, you may add more statements that belong here.

Main items D{Paranoid}"%setup{paranoid}" paranoid level set above D{mail8yhabr}"YOU HAVE ALREADY BEEN REFUSED!" D{mail8ctboood}"SPAMMER CLAIMED TO BE ONE OF OUR DOMAINS!" D{mail3tt}"ONLY MAIL TO SUPPLIED Trouble Ticket ACCEPTED"

screen_domain GLOBAL B

HELO DOMAIN NAME CHECKING

Most SPAMMERS use ZOMBIE PC's to send their spam, most if not all have completly numeric DNS names.

Most have their IP address in simple dotted or dashed notation, often all 4 parts of their IP address, we will not let any who have 2 or more parts of their address as their name through.

We are considering a tuning element to vary this, maybe to just one, however a lot of real senders with several servers use the last part of the IP address in their name.
Some unhelpfull ISP's string the IP's together as a single number.
Some very unhelpfull ISP's encode the numbers in HEX.
And finally totally random strings of numbers and letters, which have led us in the past to completly block the entire domain in the standard access.db file, this is often the best thing to do with hard to otherwise stop SPAMMER domains.

As much as possible of the code is INLINED to reduce the "named rulesets" total, as inlined code must be defined before use, macros are in reverse order.

[Pad,Hex,PadHex]Number convert single digits to what the name sugests.

(Pad,Hex,PadHex)IpNumber

convert four digit IP address to what the name sugests.

ScreenMash

The worker, matchs supplied patten with $s (HELLO).

However Hello must be clearly tokenised.

Splice used by ScreenIP

There must be a better way to do this, however as far as decimal numerics goes this works, nothing todate (lots of time spent trying) works for HEX.

celmorlauren will continue to use the original Perl helpers for now.

ScreenIpPatten used by above, trys patten dotted then dashed

ScreenIP

used by above, trims IP address from 4 then 3 then 2, also trys re-arranging and all 4 parts spliced together as a single token

ScreenDomainIP small often used macro, re-arranges IP to check

local_check_relay GLOBAL A

CONTACT

This bit arrived at on first contact, and so permissions based on IP can be set

Local_check_relay standard rule, to check incoming connection against mail8 databases and of course standard local ip addresses, further rules are based on what happens here.

uses Macro Screem_bad_relay (GLOBAL B) to do the main checking

{GoodRelay} and {BadRelay} both contain result of check, such values as (where # is checked value).

    #.Local.FOUND       $w                                  {GoodRelay}
    #.VirtHost.FOUND    ${VirtHost}                         {GoodRelay}
    #.RelayDomain.FOUND $R                                  {GoodRelay}
    #.mail1.FOUND       mail1.db                            {GoodRelay}
    #.Private.FOUND     192.168.#.# 172.16.#.# 10.#.#.#     {GoodRelay}
    #.mail4.FOUND
    #.mail3.FOUND

mail8 amd mail9 checks result im $#error

Being found does not mean that the host is a BadRelay, just that it will need handling differently to other hosts. Hosts recorded as being GoodRelay.

yahoo and hotmail hosts are recorded as {RestrictedHosts}, to help to ensure that e-mail purporting to come from these domains does infact come from these domains.

local_check_mail GLOBAL A

HELO & FROM

After intial HELO and every FROM following

This insists that the HELO host name must either be the same as the {client_name} or resolve to an address that is the same as the {client_name}.

This insists that mail purporting to come from hotmail or yahoo does come from the relevant domain. The mail addres is recorded by {RestrictedHost}

This also handles empty FROM's which are normally bounces of some kind, or the un-helpfull callback verify sudo bounce, which often originates from poorly configured e-mail systems that blindly bounce back to Forged FROM addresses.

{Bounce} records that a empty FROM has been recieved, these are accepted according to the value of {Paranoid}.

{Refused} and {RefusedAgain} record that the connection has been refused, only spammers will cause {RefusedAgain} to be generated, also if the Perl Helpers are installed these will attempt to ammend both sendmail databases and the firewall rules.

Refers to

ScreenMail8blocker GLOBAL B: this is called regardless of wether the PerlHelpers have been installed.
ScreenMail9blocker GLOBAL B: this is called regardless of wether the PerlHelpers have been installed.
ScreenDomain GLOBAL B: this checks the HELO host for being highly numeric, and having its IP encoded in the name.

local_check_rcpt GLOBAL A

RCPT

standard sendmail rules do most of the work, however depending on the value {Paranoid} responses will vary in responce to {Bounce} requests.

check_data GLOBAL A

DATA

This bit is for checking for callback verify requests which are a fake bounce with no DATA, action depends on setting of {Paranoid} and {Bounce}

For all values of {Paranoid} other than 0, this will not accept any bounces from any unknown systems.

{Bounce} will not be defined if mail is from permitted systems.

screen_header(@_) GLOBAL A

HEADER LINES

All this does at the moment is check the following header statements

Received:

with HTTP

HTTP webmail is so insecure and open to abuse, that we have taken the position that we will no longer accept mail from systems that have received mail from a system that received mail vi HTTP.

Those systems already in the mail4 database (relays who fail one or more tests, and who you want to recieve mail from) are excempt from this, and so even if they have received mail with HTTP it will be accepted.

Otherwise this is controlled by the level of {Paranoid} 0 or 1 will accept, otherwise not.

by DOMAIN

some spammers pass other tests but show themselves by pretending to send from one of our domains!

From:

ALL users must conform, they must not FAKE their "From:" header to show something other than the FROM used in the mail discussion with the mail host. No exceptions! A considerable amount of SPAM comes from mailers that allow their users to send SPAM. It is in your own intrest to stop your users sending SPAM as you are very likly to be registered as a SPAMMER and be blocked by mail servers the world over.

Reply-to:

This uses the standard Sendmail access database to check that the reply is not to to an address that you would not accept mail from.

The entry in the access file would be

From:##### ERROR:"550 We do not accept mail from you"

Where ##### is tha banned server address. Many spammers are using yahoo addresses, from systems that have nothing todo with yahoo! So From: is not really the best way to stop these.

Where e-mail has been received from a {RestrictedHost} such as yahoo or hotmail, the Reply-to: must be the same as f.

Where a Reply-to: of hotmail or yahoo is used by anyone else it is refused!

Other tests are possible. But on balance we think that these should be left to a 2nd level e-mail system that can take a closer look with both anti virus and something like SpamAssassin, it should be noted that these systems tend to be rather slow, so should never be run on a busy front line e-mail system under constant attack. If you wish additional rules may be supplied, these will be tacked on the end of the SScreenHeader definition.

HISTORY

Versions

0.1

Nov 2006 1st version, pure sendmail M4 hack, using plug-in Perl programs.

0.2

25 August, this 1st CPAN M4 hack script, original script split into, this Mail8 and Utils for creation and testing.

21 Sept 2007 Released onto CPAN.

Amendments to release version

Most changes are recorded in Utils as this really was a messy hack conversion, with many revisions required to get something that worked.

22 Sept 2007

added this HISTORY

mail8 DataBase file refrences (apart from mail4) have been removed, and will now only added if the {Perl_Helpers} have been installed, currently these scripts are not available, their future will be decided later.

mail8_zombie now included in distro, install it to get the full benifits

0.21

22 September 2007 CPAN Release

Amendments to release version

23 September 2007

sendmail works fine in command line test mode. But complains of "too many long names" when this is installed and run, "sendmail" assigns on the "fly" macro names!????!!!!?.

A limit of 96 (8 Bit limits?) is really too constrictive.

1st try now using {MashTemp} instead of {MashStack}, saves some, so sendmail falls over a little later!
OPTION NOMASH used where possible, also "OPTION MASH 1" to conserve name space in other places, this and changes to Utils have removed 20 names

0.22

23 September 2007 CPAN Release

Amendments to release version

23 Sept 2007

on SuSE 10.1 (remastered) sendmail version 8.13.6, complained of an unbalanced >, not the fault of the sender but a macro? why did sendmail then decide to 553???, had no problem with the same code on SuSE 9.3 sendmail 8.13.3?

So a single line added and hopefully all will be well, and this system can finally protect celmorlauren, we suffer quite a lot of spam, so we should be able to spot any other weakness in the code.

0.23

23 September 2007 CPAN Release

Amendments to release version

24 Sept 2007: it appears we have caused a problem with the changes above, HELO's are failing when they should not. added failed hosts to test data, to try and ensure it does not happen again.

0.24

24 September 2007 CPAN Release

Amendments to release version

24 Sept 2007: Patch required, spammer whose domain matched who they said they were, was not put through the numeric zombie check as expected, a Fix broke the intended logic, test data added to mail8.

0.25

24 September 2007 CPAN Patch Release

Amendments to release version

25 Sept 2007

HELO checking for {daemon_addr} added, should have there already (whoops sorry), however numeric addresses will not resolve anyway, so CPAN patch release delayed untill more serious problems encountered. Develelpment on test_cgi will cause further amendments, so will wait for that before release.

30 Sept 2007

ref to "network associates" changed to "network solutions", just been amending my "domains", so spotted the error, as I tend to look for them using google. Other typo's will be corrected when found.

Noted rather more spam than normal has been arriving at my mail system, all fake MX's have been stopped as expected, but a few from Open Relays get through this (the 1st level filter), only to fail at the 2nd level email system. In an attempt to allow the 2nd level email system to sleep more, and only have to deal with real e-mail.

1: Header From: checked is the same as f or at least contains f. Failures are {Refused}, no exceptions all users are checked!
2: Header Recieved: check now sets {Refused} for failures. {GoodRelay} hosts are exempted from this check.
3: Found during testing, that mail only worked once only! corrections made to ensure essential macros are set up again, and others use sendmail macros such as f instead of supplied values, which may be the from address 1st time round, 2nd and more times it is (conneted as HOST & IP)?!

0.26

30 September 2007 CPAN Patch Release

Amendments to release version

30 Sept 2007: noted that the Pause upload server falls foul of the just uploaded system, so the mail4.db will now permit those listed to skip all checks, however local users will not be allowed to skip mail FROM checks.

0.27

30 September 2007 CPAN Patch Release

Amendments to release version

01 Oct 2007

sendmail needs more free "long names" for its own use, this had been using only 21 "long names", Sendmail::M4::Utils has been modified to provide routines that can use just one "long name" {MashFound}, so this will be recoded as required as the {macros} used before can no longer be direcltly accessed. Total saving minus 5, leaving 16.

Also found 2 other names, that did not need persistance, these now MashTempC & D

0.28

02 October 2007 CPAN Patch Release

Amendments to release version

03 Oct 2007: noticed that header line "From:" where email address not included in < > brackets, will not match when should.

0.29

03 October 2007 CPAN Patch Release

Amendments to release version

03 Oct 2007

the Received: header checking does not work, and currently nothing tried makes it work, fine in test mode. But as headers are not tokenised with sendmail 8.13, can not do anthing usefull with the supplied tools.

So currently does nothing, just returns.

Fix must wait for Sendmail::M4::mail8_daemon and its worker module Sendmail::M4::Mail8_daemon.pm, which will now do more than originally designed to do.

Patch required to remove useless error messages from the logs.

0.30

03 October 2007 CPAN Patch Release

Amendments to release version

05 Oct 2007: noted that dots and dashs did not work in From: header checker for names, fixed. However underscores will still not work

0.31

05 October 2007 CPAN Patch Release

Amendments to release version

08 Oct 2007: Noted that lots of SPAM has Reply-to: yahoo or hotmail, system will now check sendmails standard access database for From:#### where #### is system we do not accept mail from. SPAMMERS are now concentrating on finding open-relays or other poorly setup and vunerable systems, many of which seem to be Exim servers. These are of course blocked on their third SPAM or sometimes on their first if they are an open-relay. SPAM detection being done by the second level e-mail system, the CPAN version of the interface between the two systems is currently being re-writen, and we will make it available soon.

0.32

08 October 2007 CPAN Patch Release

Amendments to release version

10 Oct 2007

Noted that the bulk of spam that has got through to the second level filter, has originated from WEB mail, which will now be refused with a {Paranoid} setting of 2 or more.

Received: DOMAIN name checking for phoney received from # by # now working, at least for us at the 1st level.

12 Oct 2007,

yahoo and hotmail domains are the Reply-to: and From: names of choice for spammers. ALL sending from domains that have nothing to do with yahoo or hotmail claiming to relay their domains, will be now stopped. Hard coded as these are so well known!

Documentation clean-up. HISTORY moved to end of document.

0.33

12 October 2007 CPAN Patch Release

Amendments to release version

14 Oct 2007

Whoops, a hotmail mail was refused too early due to a typo, strung several tests together at the end with the unlucky users connection details, problem fixed.

Corrected version will run on the mail server, before it is uploaded.

0.34

14 October 2007 CPAN Patch Release

Amendments to release version