https://wiki.asrg.sp.am/api.php?action=feedcontributions&user=Randy&feedformat=atomASRG - User contributions [en]2024-03-29T01:14:15ZUser contributionsMediaWiki 1.41.0https://wiki.asrg.sp.am/index.php?title=HELO/EHLO_Pattern_Matching&diff=1480HELO/EHLO Pattern Matching2008-03-15T13:43:08Z<p>Randy: New page: {{ast |date= |difficult=Low |popular=Low |effective=High |harm=Low to Medium |where=MTA }} SMTP requires the client to send either its fully-qualified domain name or its IP address in squa...</p>
<hr />
<div>{{ast<br />
|date=<br />
|difficult=Low<br />
|popular=Low<br />
|effective=High<br />
|harm=Low to Medium<br />
|where=MTA<br />
}}<br />
SMTP requires the client to send either its fully-qualified domain name or its IP address in square brackets as its EHLO/HELO name. Some spambots use fixed strings, and some use dynamic strings that in some cases uses the actual domain name or the IP address of its computer. Since spambots are often running on compromised home systems, the resulting EHLO/HELO names may contain "localhost", IP addresses, "ADSL", "dynamic", etc. Some servers attempt to pattern match, looking for strings with a high likelihood of being a compromised home user and a low risk of being a legitimate organization. The risk is that many organizations have improperly configured mail servers and as a result send poorly-chosen EHLO names. Smaller organizations especially might be using DSL lines and not have good PTR records. Such techniques therefore require careful monitoring to detect when legitimate organizations are blocked.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Taxonomy_of_anti-spam_techniques&diff=1479Taxonomy of anti-spam techniques2008-03-15T13:32:56Z<p>Randy: /* Network techniques */</p>
<hr />
<div>Many people have invented many anti-spam techniques over the past decade. And a lot of the techniques keep being reinvented. Our goal here is to list all the anti-spam techniques we know, both the good ones and the bad ones.<br />
----<br />
==Message content techniques==<br />
<br />
* [[Static filters]]<br />
<br />
* [[Adaptive filters]]<br />
<br />
==SMTP techniques==<br />
<br />
* [[Timing and protocol defects]]<br />
<br />
* [[Greylisting]]<br />
<br />
* [[Callbacks]]<br />
<br />
==Address management==<br />
<br />
* [[Tagged addresses]]<br />
<br />
* [[Code words]]<br />
<br />
* [[Disposable addresses]]<br />
<br />
==Network techniques==<br />
<br />
* [[DNS blacklists]]<br />
<br />
* [[DNS validation]]<br />
<br />
* [[HELO/EHLO Pattern Matching]]<br />
<br />
* [[Tarpitting and other timing techniques]]<br />
<br />
==Whitelist techniques==<br />
<br />
* [[Recognizing known correspondents]]<br />
<br />
* [[Path validation]]<br />
<br />
* [[Signatures]]<br />
<br />
* [[Subject keywords]]<br />
<br />
==Introduction techniques==<br />
<br />
* [[Challenges]]<br />
<br />
* [[Proof of work]]<br />
<br />
* [[Postage]]</div>Randyhttps://wiki.asrg.sp.am/index.php?title=DNS_validation&diff=1478DNS validation2008-03-15T13:30:06Z<p>Randy: New page: {{ast |date= |difficult=Low |popular=Medium |effective=High to Low |harm=Low to High |where=MTA }} DNS Validation techniques attempt to verify that the SMTP client system is a legitimate n...</p>
<hr />
<div>{{ast<br />
|date=<br />
|difficult=Low<br />
|popular=Medium<br />
|effective=High to Low<br />
|harm=Low to High<br />
|where=MTA<br />
}}<br />
DNS Validation techniques attempt to verify that the SMTP client system is a legitimate nail server with proper DNS records related to it.<br />
<br />
There are a variety of such techniques, some more effective than others, and some more harmful than others.<br />
<br />
The techniques include:<br />
*Looking for an MX or A record for the HELO/EHLO name used by the SMTP client system<br />
*Looking for a PTR record for the IP address of the SMTP client system side of the connection<br />
*Looking for an MX or A record for the HELO/EHLO name used by the SMTP client system and then matching the IP addresses from the DNS lookup to that of the SMTP client side of the connection<br />
*Looking for a PTR record for the IP address of the SMTP client system side of the connection and then doing an A record lookup on that name to see if it matches the IP address of the SMTP client system side<br />
<br />
There are several points to keep in mind when considering such techniques:<br />
*The DNS is prone to transient errors (as well as cache poisoning, which is less likely to be used here)<br />
*The reverse zones (where PTR records live) is generally set up with a very different delegation mechanism than the forward zones, and so PTR records for other than large organizations tend to have entirely separate administration from the forward zone; thus, PTR records are often wrong and hard to maintain<br />
*Legitimate mail servers may use EHLO domains whose MX records point to different servers (for example, separate inbound and outbound servers)<br />
*Legitimate mail servers may serve multiple domains<br />
<br />
Therefore, techniques involving PTR records generally have lower effectiveness and higher harm. Techniques which merely look for the presence of MX or A records for the domain sent in the EHLO/HELO command generally have higher effectiveness and lower harm (since they only check that the server claims to represent a valid domain). Techniques which are multi-step, using the results of one DNS query to feed the next query, or which attempt to match the IP address of the SMTP client system, tend to have higher harm.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Timing_and_protocol_defects&diff=1477Timing and protocol defects2008-03-15T13:08:03Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=<br />
|difficult=Low<br />
|popular=Medium<br />
|effective=High<br />
|harm=Low<br />
|where=MTA<br />
}}<br />
<br />
SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check'' pr ''early talking', detects extra data in the input buffer prior to the server sending the HELO/EHLO response, or prior to any command if the server hasn't offered the PIPELINEING extension.<br />
<br />
Other protocol defects include:<br />
*Sending MAIL FROM without first sending HELO or EHLO<br />
*Omitting the angle brackets required in MAIL FROM and RCPT TO<br />
*Adding a space after the colon in MAIL FROM or RCPT TO<br />
*Syntactically invalid HELO/EHLO name<br />
<br />
If such data is found, the client has failed the test. The server might then reject the message, close the connection, blacklist the client, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Callbacks&diff=1474Callbacks2008-03-14T14:46:12Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=early 2000s<br />
|difficult=Low<br />
|popular=Low<br />
|effective=Low<br />
|harm=High<br />
|where=MTA<br />
}}<br />
<br />
SMTP server connects to SMTP client on port 25 during transaction. <br />
<br />
Usually, following MAIL FROM and at least one RCPT TO, the SMTP server opens a connection to the SMTP client system on port 25 and uses the RCPT TO value in a MAIL FROM commands, and the MAIL FROM value in a RCPT TO command, in an attempt to see if the client system is an SMTP server and accepts mail.<br />
<br />
This is a deceptively attractive approach, since it appears to validate the client system as a proper server. However, it quickly runs into numerous problems. The most severe is that if both systems try this, they both deadlock and mail cannot be exchanged. Other problems include temporary unavailability of the mail system, sites which operate separate systems for sending and receiving mail, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Greylisting&diff=1473Greylisting2008-03-14T14:43:19Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=early 2000s<br />
|difficult=medium<br />
|popular=Low, increasing<br />
|effective=High<br />
|where=MTA<br />
|harm=Medium<br />
}}<br />
<br />
Greylisting (AKA graylisting) is an extremely effective method against illegitimate spammers who use cracked PCs (botnets, for example) to send the spam. It is utterly ineffective against spamers sending from conforming conventional MTAs.<br />
<br />
SMTP requires clients to maintain a queue of outgoing messages. During a delivery attempt, a 4xx response code indicates a temporary failure. The client system must then retry the message. See RFC 2821 Section 4.5.4. Clients can immediately try another host listed in an MX record for the target domain, or re-queue the message and try again later. When the message is re-queued, the client SHOULD wait at least 30 minutes before the next delivery attempt, and generally should use progressively larger delays for each subsequent retry attempt.<br />
<br />
Greylisting attempts to detect SMTP client systems that are not true SMTP servers and do not maintain a message queue, instead using a "blast and dump" approach.<br />
<br />
Greylisting operates by initially deferring incoming messages by giving a 4xx (temporary failure) response during the smtp protocol dialogue. The receiving MTA stores information about the attempted delivery, at minimum the client SMTP system's IP address, but in some implementations also the [[envelope sender address]] and [[envelope recipient address]]). When all three items are stored, it is called the triplet.<br />
<br />
If a new delivery attempt matching the stored client information (that is, the same IP address or the same triplet) arrives within the minimum delay time enforced by the receiving MTA, the retry is also rejected and the minimum delay time reset. If the retry arrives after the minimum delay time, the message is accepted and the saved data is marked so that subsequent attempts which match the saved data are immediately accepted with no delay. Depending on the implementation, subsequent attempts need only match the IP address of the client SMTP system, or the entire triplet.<br />
<br />
While effective against certain kinds of spam, the technique has its drawbacks:<br />
* greylisted messages are initially delayed, possibly for hours or days<br />
* problems with outbound server pools that retry with different IPs<br />
* problems with inbound server pools that accept connections on multiple IPs<br />
* problems when the retry time of the sending system is smaller than the minimum retry time enforced by the receiving system (note that RFC 2821 says that the 30 minute delay is a SHOULD, not a MUST)<br />
* unwanted interactions with servers doing callbacks<br />
* extra burden on the sending MTAs<br />
<br />
Because of the large number of situations and implementations in which problems occur, sites using greylisting need to monitor the greylisting state to detect legitimate sites whose mail is not being received. Often, whitelist tables of such hosts need to be maintained.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Timing_and_protocol_defects&diff=1470Timing and protocol defects2008-03-14T14:27:53Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=<br />
|difficult=Low<br />
|popular=Medium<br />
|effective=High<br />
|Harm=Low<br />
|where=MTA<br />
}}<br />
<br />
SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check'', detects extra data in the input buffer prior to the server sending the HELO/EHLO response. If such data is found, the client has failed the test. The server might then reject the message, close the connection, blacklist the client, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Callbacks&diff=1467Callbacks2008-03-14T14:12:35Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=early 2000s<br />
|difficult=medium<br />
|popular=Low<br />
|effective=Low<br />
|Harm=High<br />
|where=MTA<br />
}}<br />
<br />
SMTP server connects to SMTP client on port 25 during transaction. <br />
<br />
Usually, following MAIL FROM and at least one RCPT TO, the SMTP server opens a connection to the SMTP client system on port 25 and uses the RCPT TO value in a MAIL FROM commands, and the MAIL FROM value in a RCPT TO command, in an attempt to see if the client system is an SMTP server and accepts mail.<br />
<br />
This is a deceptively attractive approach, since it appears to validate the client system as a proper server. However, it quickly runs into numerous problems. The most severe is that if both systems try this, they both deadlock and mail cannot be exchanged. Other problems include temporary unavailability of the mail system, sites which operate separate systems for sending and receiving mail, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Greylisting&diff=1466Greylisting2008-03-14T14:09:34Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=early 2000s<br />
|difficult=medium<br />
|popular=Low, increasing<br />
|effective=High<br />
|where=MTA<br />
}}<br />
<br />
Greylisting (AKA graylisting) is an extremely effective method against illegitimate spammers who use cracked PCs (botnets, for example) to send the spam. It is utterly ineffective against spamers sending from conforming conventional MTAs.<br />
<br />
SMTP requires clients to maintain a queue of outgoing messages. During a delivery attempt, a 4xx response code indicates a temporary failure. The client system must then retry the message. See RFC 2821 Section 4.5.4. Clients can immediately try another host listed in an MX record for the target domain, or re-queue the message and try again later. When the message is re-queued, the client SHOULD wait at least 30 minutes before the next delivery attempt, and generally should use progressively larger delays for each subsequent retry attempt.<br />
<br />
Greylisting attempts to detect SMTP client systems that are not true SMTP servers and do not maintain a message queue, instead using a "blast and dump" approach.<br />
<br />
Greylisting operates by initially deferring incoming messages by giving a 4xx (temporary failure) response during the smtp protocol dialogue. The receiving MTA stores information about the attempted delivery, at minimum the client SMTP system's IP address, but in some implementations also the [[envelope sender address]] and [[envelope recipient address]]). When all three items are stored, it is called the triplet.<br />
<br />
If a new delivery attempt matching the stored client information (that is, the same IP address or the same triplet) arrives within the minimum delay time enforced by the receiving MTA, the retry is also rejected and the minimum delay time reset. If the retry arrives after the minimum delay time, the message is accepted and the saved data is marked so that subsequent attempts which match the saved data are immediately accepted with no delay. Depending on the implementation, subsequent attempts need only match the IP address of the client SMTP system, or the entire triplet.<br />
<br />
While effective against certain kinds of spam, the technique has its drawbacks:<br />
* greylisted messages are initially delayed, possibly for hours or days<br />
* problems with outbound server pools that retry with different IPs<br />
* problems with inbound server pools that accept connections on multiple IPs<br />
* problems when the retry time of the sending system is smaller than the minimum retry time enforced by the receiving system (note that RFC 2821 says that the 30 minute delay is a SHOULD, not a MUST)<br />
* unwanted interactions with servers doing callbacks<br />
* extra burden on the sending MTAs</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Greylisting&diff=1465Greylisting2008-03-14T14:08:39Z<p>Randy: </p>
<hr />
<div>{{ast<br />
|date=early 2000s<br />
|difficult=medium<br />
|popular=Low, increasing<br />
|effective=High<br />
|where=MTA<br />
}}<br />
<br />
Greylisting (AKA graylisting) is an extremely effective method against illegitimate spammers who use cracked PCs (botnets, for example) to send the spam. It is utterly ineffective against spamers sending from conforming conventional MTAs.<br />
<br />
SMTP requires clients to maintain a queue of outgoing messages. During a delivery attempt, a 4xx response code indicates a temporary failure. The client system must then retry the message. See RFC 2821 Section 4.5.4. Clients can immediately try another host listed in an MX record for the target domain, or re-queue the message and try again later. When the message is re-queued, the client SHOULD wait at least 30 minutes before the next delivery attempt, and generally should use progressively larger delays for each subsequent retry attempt.<br />
<br />
Greylisting attempts to detect SMTP client systems that are not true SMTP servers and do not maintain a message queue, instead using a "blast and dump" approach.<br />
<br />
Greylisting operates by initially deferring incoming messages by giving a 4xx (temporary failure) response during the smtp protocol dialogue. The receiving MTA stores information about the attempted delivery, at minimum the client SMTP system's IP address, but in some implementations also the [[envelope sender address]] and [[envelope recipient address]]). When all three items are stored, it is called the triplet.<br />
<br />
If a new delivery attempt matches the stored client information (that is, the same IP address or the same triplet) arrives within the minimum delay time enforced by the receiving MTA, the retry is also rejected and the minimum delay time reset. If the retry arrives after the minimum delay time, the message is accepted and the saved data is marked so that subsequent attempts which match the saved data are immediately accepted with no delay. Depending on the implementation, subsequent attempts need only match the IP address of the client SMTP system, or the entire triplet.<br />
<br />
While effective against certain kinds of spam, the technique has its drawbacks:<br />
* greylisted messages are initially delayed, possibly for hours or days<br />
* problems with outbound server pools that retry with different IPs<br />
* problems with inbound server pools that accept connections on multiple IPs<br />
* problems when the retry time of the sending system is smaller than the minimum retry time enforced by the receiving system (note that RFC 2821 says that the 30 minute delay is a SHOULD, not a MUST)<br />
* unwanted interactions with servers doing callbacks<br />
* extra burden on the sending MTAs</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Timing_and_protocol_defects&diff=1464Timing and protocol defects2008-03-14T13:53:10Z<p>Randy: </p>
<hr />
<div>SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check'', detects extra data in the input buffer prior to the server sending the HELO/EHLO response. If such data is found, the client has failed the test. The server might then reject the message, close the connection, blacklist the client, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Timing_and_protocol_defects&diff=1463Timing and protocol defects2008-03-14T13:50:35Z<p>Randy: </p>
<hr />
<div>SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check," detects extra data in the input buffer prior to the server sending the HELO/EHLO response.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Timing_and_protocol_defects&diff=1462Timing and protocol defects2008-03-14T13:46:51Z<p>Randy: New page: SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check", detects extra data in the...</p>
<hr />
<div>SMTP requires that clients wait for a 2xx response to the HELO or EHLO command before proceeding. One ant-spam technique, known as a ''premature pipeline check", detects extra data in the input buffer prior to the server sending the HELO/EHLO response.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Callbacks&diff=1461Callbacks2008-03-14T13:42:48Z<p>Randy: </p>
<hr />
<div>SMTP server connects to SMTP client on port 25 during transaction. <br />
<br />
Usually, following MAIL FROM and at least one RCPT TO, the SMTP server opens a connection to the SMTP client system on port 25 and uses the RCPT TO value in a MAIL FROM commands, and the MAIL FROM value in a RCPT TO command, in an attempt to see if the client system is an SMTP server and accepts mail.<br />
<br />
This is a deceptively attractive approach, since it appears to validate the client system as a proper server. However, it quickly runs into numerous problems. The most severe is that if both systems try this, they both deadlock and mail cannot be exchanged. Other problems include temporary unavailability of the mail system, sites which operate separate systems for sending and receiving mail, etc.</div>Randyhttps://wiki.asrg.sp.am/index.php?title=Callbacks&diff=1460Callbacks2008-03-14T13:41:58Z<p>Randy: New page: SMTP server connects to SMTP client on port 25 during transaction. Usually, following MAIL FROM and at least one RCPT TO, the SMTP server opens a connection to the SMTP client system on p...</p>
<hr />
<div>SMTP server connects to SMTP client on port 25 during transaction. Usually, following MAIL FROM and at least one RCPT TO, the SMTP server opens a connection to the SMTP client system on port 25 and uses the RCPT TO value in a MAIL FROM commands, and the MAIL FROM value in a RCPT TO command, in an attempt to see if the client system is an SMTP server and accepts mail.<br />
<br />
This is a deceptively attractive approach, since it appears to validate the client system as a proper server. However, it quickly runs into numerous problems. The most severe is that if both systems try this, they both deadlock and mail cannot be exchanged. Other problems include temporary unavailability of the mail system, sites which operate separate systems for sending and receiving mail, etc.</div>Randy