https://wiki.asrg.sp.am/api.php?action=feedcontributions&user=Johnl&feedformat=atomASRG - User contributions [en]2024-03-19T05:50:02ZUser contributionsMediaWiki 1.41.0https://wiki.asrg.sp.am/index.php?title=URL_filtering&diff=1681URL filtering2011-11-09T17:32:38Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=early 2000<br />
|difficult=Medium<br />
|popular=High<br />
|effective=Medium<br />
|harm=Low<br />
|where=MTA<br />
}}<br />
<br />
Most spams contain URLs to redirect people to a web site. Software can extract all URLs present in the body of the message and check them against a blacklist. Primitive filters can use static flat file blacklists, and the efficiency and drawbacks are the same of a static list of keywords. Most modern filters use URL blacklists stored in DNS zones, as this is an easier way to distribute these lists.<br />
<br />
As long as URLs found in spam change very frequently, maintaining of this kind of blacklist is a hard work and usually needs a lot of spam traps to collect spams.<br />
<br />
Efficiency of URL filtering is usually something between 50 % and 70 %. False positive rate can be as low as 0.1 %, but some lists are more aggressive, and can present a higher false positive rate.<br />
<br />
Popular URL blacklists include the Spamhaus DBL, SURBL, and URIBL.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Abuse_Reporting&diff=1680Abuse Reporting2011-11-09T17:11:41Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=Before 2003<br />
|difficult=?<br />
|popular=Medium<br />
|effective=?<br />
|harm=?<br />
|where=External<br />
}}<br />
<br />
'''Abuse Reporting''' is a cooperative technique whereby spam is suppressed at the origin by the administrators of an MSA. Mail domains' reputation can be preserved by limiting outgoing spam, thus operators are motivated to suppress spamming. Probably the earliest anti-spam defense, this technique relies on end users to submit abuse reports timely and correctly. It is supported by huge mailbox providers, who feature a ''This is Spam'' (TiS) button on their web-based mail clients.<br />
<br />
==Abuse Reporting Format (ARF)==<br />
The format for transmitting abuse reports is defined in [RFC 5965]. See ''abuse-feedback-report'' list. The IETF is looking at extenstions to it.<br />
<br />
<!-- Handy? easier integration with wikipedia is possible by adding a MySQL record, as exemplified in<br />
http://www.mediawiki.org/wiki/Manual:Interwiki#Single_line --><br />
See also the Wikipedia [http://en.wikipedia.org/wiki/Feedback_Loop_%28email%29 page].<br />
<br />
==The Feedback Loop (FBL) and other usage of abuse reports==<br />
Huge mailbox providers may provide a means for ESPs to get abuse reports issued for messages that had been sent by themselves. This tool can be used to automate opt-out. The sender should tag each message, setting an opaque token and possibly signing it, so as to be able to retrieve necessary submission details.<br />
<br />
Another use of abuse reports is to tune an MX server's spam filter. For Bayesian filters this requires a spam/ham indicator. Some hosts automatically convert reports into unsubscribe requests when the reported message contained a ''List-Unsubscribe'' header field.<br />
<br />
[[Image:Abuse-report-flow.png|thumb|right|350px|Hypothetical usage of an abuse report (green) submitted in response to a spam message (violet).]] <br />
While it is felt that proper use of abuse reports may result in reliable reputation recording, how abuse reports should be routed is still a debatable question. Rough consensus indicates that users should send abuse reports to their mailbox providers. However, a third party can manage the ''Reporting Service'' instead of (or in addition to) the ''Last MX''. For example, [http://www.spamcop.net/ SpamCop] offers such service. FBLs as operated, e.g., by [http://returnpath.net/ ReturnPath], are not scalable and may be questionable from a privacy point of view. The [http://abusix.org/ Abusix] approach provides for routing abuse reports hierarchically, through top level IP assignments. The image beside depicts possible use:<br />
<br />
# Tuning filters at the 1st MX, assuming it is on the ''recipient side''. The 1st MX can be identified as ''responsible for receiving'', e.g. by authentication results.<br />
# Let the spammer's MSA suppress further attempts; this step is useless and questionable if the MSA is directly operated by the spammer. The MSA can be identified as ''responsible for sending'', e.g. by author signatures.<br />
# Arguably, connection providers should terminate contracts with blatantly spamming MSAs. The connection provider can be identified via RIR whois; Abusix's database provides abuse-box lookup by IP address.<br />
# Reputation trackers should verify whether the MSA actually blocks the spammer.<br />
<br />
The ''Reputation tracker'', a.k.a. ''vouching service'', sometimes also referred to as ''The "Mythical" Global Reputation System'', would better operations and extend the functionality that is currently obtained by submitting mail through a well-reputed mailbox provider, as it decouples IP addresses from senders' reputation<ref>{{list archive|msg=15884|subject="Mythical" Global Reputation System|from=John Leslie|date=Sun, 20 Dec 2009 18:21:04 -0500}}</ref>.<br />
<br style="clear:both" /><br />
<br />
==Adding a spam button to MUAs==<br />
Difficulties stem from the fact that a user may have multiple mailbox providers, which rules out single-address configurations<ref name="start">{{list archive|msg=15780|subject=Adding a spam button to MUAs|from=John R. Levine|date=Wed, 9 Dec 2009 00:35:03 -0500}}</ref>.<br />
<br />
The possibility that reports are cascaded back to the spammer may be considered a potential hazard that exposes the reporters and allows list washing<ref name="bobotek">{{list archive|msg=15790||from=BOBOTEK, ALEX (ATTCINW)|date=Tue, 8 Dec 2009 23:08:56 -0800}}</ref>. However, it has been pointed out that spammers are not interested in feedback and don't bother<ref>{{list archive|msg=15802|from=der Mouse|date=Wed, 9 Dec 2009 21:47:12 -0500 (EST)}}</ref>. There is also concern that a button might cause a flood of unwanted messages<ref name="bobotek" />, specially when multiple targets can be determined.<br />
<br />
===Message's header based approach===<br />
An MX server may embed an header field indicating the address where abuse reports may be sent. If the MX doesn't support such field, MUAs may use the field provided by the sender, if any. Otherwise, MX operators may override that by removing the field, or by replacing it with their own FBL reporting address instead.<ref>{{list archive|msg=15782|from=Steve Atkins|date=Tue, 8 Dec 2009 22:00:50 -0800}}</ref><br />
<br />
Reusing existing field has also been proposed: the topmost ''Received''<ref name="bobotek" />, any ''Delivered-To'', ''VBR-Info'', or any trusted ''Authentication-Results''. The latter one already requires users to confirm that they trust the relevant ''authserv-id'', thereby clearing that hazard concern<ref>{{list archive|msg=15786|from=Alessandro Vesely|date=Wed, 09 Dec 2009 12:32:49 +0100}}</ref>. Fields with multiple values allow for multiple targets, which may be needed for forwarded messages in case abuse reports are used to tune spam filters. In addition, using a DNS RR to store one or more addresses allows even more flexibility: verifying that the service is actually supported, using addresses other than ''abuse@'', and directly sending copies to reputation providers seem valid demands that a DNS call can fulfill.<br />
<br />
===Client's protocol based approach===<br />
Setting up an IMAP folder for this purpose has the advantage of requiring no special software on clients. This is in facts the most common solution for servers that maintain per-user Bayesian data, although it may be difficult to interpret users' intent in a non-interactive environment, and notwithstanding possible confusion or conflict with junk folders maintained by the client. When the SUBMIT and IMAP servers coincide, sending the report can be optimized by using BURL or similar techniques.<br />
<br />
POP clients have no folders, hence they'd need special machinery, such as a new command to upload an abuse report or to retrieve the address where it should be sent<ref name="start" />.<br />
<br />
==UI problems and interpretation of spam reports==<br />
Difficulties with manually submitted reports stem from the scarce reliability that characterizes users and their computing equipments<ref>{{list archive|msg=15848|from=Rich Kulawiec|date=Wed, 16 Dec 2009 07:07:42 -0500}}</ref>. Users error rates have been reported to be around 5~10%<ref>{{list archive|msg=15881|from=Jose-Marcio Martins da Cruz|date=Sun, 20 Dec 2009 13:54:14 +0100}} Cormack, G. V. and Kolcz, A. 2009. [http://plg.uwaterloo.ca/~gvcormac/cormacksigir09-spam.pdf Spam filter evaluation with imprecise ground truth]. In ''Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in information Retrieval'' (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM, New York, NY, 604-611.</ref><ref>{{list archive|msg=15891|from=Matthias Leisi|date=Mon, 21 Dec 2009 19:24:06 +0100}}</ref>. In addition, users tend to categorize mail incoherently as ''wanted/not-wanted''<ref>{{list archive|msg=15888|from=Jose-Marcio Martins da Cruz|date=Mon, 21 Dec 2009 15:32:41 +0100}}</ref> rather than ''legitimate/abusive'' (never mind the ''ham/spam'' dichotomy, which is fuzzy by definition.) Accordingly, servers may resort to interpret users' intentions by extracting further info from reported messages, e.g., if the sender has sent multiple messages before and it has a List-Unsubscribe: header, it's probably an opt-out, not a spam complaint<ref>{{list archive|msg=15900|from=John Levine|date=23 Dec 2009 01:53:00 -0000}}</ref>.<br />
<br />
UI design includes issues such as the number of buttons that users can reach. That UI might be configurable, so as to suit users with different skills<ref>{{list archive|msg=15898|from=Nathaniel Borenstein|date=Tue, 22 Dec 2009 14:54:07 -0500}}</ref>. It seems clear that some experimentation is needed in order to understand users behavior<ref>{{list archive|msg=15894|from=Ian Eiloart|date=Tue, 22 Dec 2009 11:40:24 +0000}}</ref>. However, besides a user's skill and the UI design, a mailbox provider's stance<ref>{{list archive|msg=15899|from=Michael Thomas|date=Tue, 22 Dec 2009 12:56:34 -0800}}</ref> and any effects of the way it processes abuse reports will also play a role.<br />
<br />
==See also==<br />
[[Adding a junk button to MUAs]]<br />
==References==<br />
<References /></div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Abuse_Reporting&diff=1679Abuse Reporting2011-11-09T17:10:44Z<p>Johnl: /* Abuse Reporting Format (ARF) */</p>
<hr />
<div>{{ast<br />
|date=Before 2003<br />
|difficult=?<br />
|popular=Medium<br />
|effective=?<br />
|harm=?<br />
|where=External<br />
}}<br />
<br />
'''Abuse Reporting''' is a cooperative technique whereby spam is suppressed at the origin by the administrators of an MSA. Mail domains' reputation can be preserved by limiting outgoing spam, thus operators are motivated to suppress spamming. Probably the earliest anti-spam defense, this technique relies on end users to submit abuse reports timely and correctly. It is supported by huge mailbox providers, who feature a ''This is Spam'' (TiS) button on their web-based mail clients.<br />
<br />
==Abuse Reporting Format (ARF)==<br />
The format for transmitting abuse reports is defined in [RFC 5965]. See ''abuse-feedback-report'' list.<br />
<br />
<!-- Handy? easier integration with wikipedia is possible by adding a MySQL record, as exemplified in<br />
http://www.mediawiki.org/wiki/Manual:Interwiki#Single_line --><br />
See also the Wikipedia [http://en.wikipedia.org/wiki/Feedback_Loop_%28email%29 page].<br />
<br />
==The Feedback Loop (FBL) and other usage of abuse reports==<br />
Huge mailbox providers may provide a means for ESPs to get abuse reports issued for messages that had been sent by themselves. This tool can be used to automate opt-out. The sender should tag each message, setting an opaque token and possibly signing it, so as to be able to retrieve necessary submission details.<br />
<br />
Another use of abuse reports is to tune an MX server's spam filter. For Bayesian filters this requires a spam/ham indicator. Some hosts automatically convert reports into unsubscribe requests when the reported message contained a ''List-Unsubscribe'' header field.<br />
<br />
[[Image:Abuse-report-flow.png|thumb|right|350px|Hypothetical usage of an abuse report (green) submitted in response to a spam message (violet).]] <br />
While it is felt that proper use of abuse reports may result in reliable reputation recording, how abuse reports should be routed is still a debatable question. Rough consensus indicates that users should send abuse reports to their mailbox providers. However, a third party can manage the ''Reporting Service'' instead of (or in addition to) the ''Last MX''. For example, [http://www.spamcop.net/ SpamCop] offers such service. FBLs as operated, e.g., by [http://returnpath.net/ ReturnPath], are not scalable and may be questionable from a privacy point of view. The [http://abusix.org/ Abusix] approach provides for routing abuse reports hierarchically, through top level IP assignments. The image beside depicts possible use:<br />
<br />
# Tuning filters at the 1st MX, assuming it is on the ''recipient side''. The 1st MX can be identified as ''responsible for receiving'', e.g. by authentication results.<br />
# Let the spammer's MSA suppress further attempts; this step is useless and questionable if the MSA is directly operated by the spammer. The MSA can be identified as ''responsible for sending'', e.g. by author signatures.<br />
# Arguably, connection providers should terminate contracts with blatantly spamming MSAs. The connection provider can be identified via RIR whois; Abusix's database provides abuse-box lookup by IP address.<br />
# Reputation trackers should verify whether the MSA actually blocks the spammer.<br />
<br />
The ''Reputation tracker'', a.k.a. ''vouching service'', sometimes also referred to as ''The "Mythical" Global Reputation System'', would better operations and extend the functionality that is currently obtained by submitting mail through a well-reputed mailbox provider, as it decouples IP addresses from senders' reputation<ref>{{list archive|msg=15884|subject="Mythical" Global Reputation System|from=John Leslie|date=Sun, 20 Dec 2009 18:21:04 -0500}}</ref>.<br />
<br style="clear:both" /><br />
<br />
==Adding a spam button to MUAs==<br />
Difficulties stem from the fact that a user may have multiple mailbox providers, which rules out single-address configurations<ref name="start">{{list archive|msg=15780|subject=Adding a spam button to MUAs|from=John R. Levine|date=Wed, 9 Dec 2009 00:35:03 -0500}}</ref>.<br />
<br />
The possibility that reports are cascaded back to the spammer may be considered a potential hazard that exposes the reporters and allows list washing<ref name="bobotek">{{list archive|msg=15790||from=BOBOTEK, ALEX (ATTCINW)|date=Tue, 8 Dec 2009 23:08:56 -0800}}</ref>. However, it has been pointed out that spammers are not interested in feedback and don't bother<ref>{{list archive|msg=15802|from=der Mouse|date=Wed, 9 Dec 2009 21:47:12 -0500 (EST)}}</ref>. There is also concern that a button might cause a flood of unwanted messages<ref name="bobotek" />, specially when multiple targets can be determined.<br />
<br />
===Message's header based approach===<br />
An MX server may embed an header field indicating the address where abuse reports may be sent. If the MX doesn't support such field, MUAs may use the field provided by the sender, if any. Otherwise, MX operators may override that by removing the field, or by replacing it with their own FBL reporting address instead.<ref>{{list archive|msg=15782|from=Steve Atkins|date=Tue, 8 Dec 2009 22:00:50 -0800}}</ref><br />
<br />
Reusing existing field has also been proposed: the topmost ''Received''<ref name="bobotek" />, any ''Delivered-To'', ''VBR-Info'', or any trusted ''Authentication-Results''. The latter one already requires users to confirm that they trust the relevant ''authserv-id'', thereby clearing that hazard concern<ref>{{list archive|msg=15786|from=Alessandro Vesely|date=Wed, 09 Dec 2009 12:32:49 +0100}}</ref>. Fields with multiple values allow for multiple targets, which may be needed for forwarded messages in case abuse reports are used to tune spam filters. In addition, using a DNS RR to store one or more addresses allows even more flexibility: verifying that the service is actually supported, using addresses other than ''abuse@'', and directly sending copies to reputation providers seem valid demands that a DNS call can fulfill.<br />
<br />
===Client's protocol based approach===<br />
Setting up an IMAP folder for this purpose has the advantage of requiring no special software on clients. This is in facts the most common solution for servers that maintain per-user Bayesian data, although it may be difficult to interpret users' intent in a non-interactive environment, and notwithstanding possible confusion or conflict with junk folders maintained by the client. When the SUBMIT and IMAP servers coincide, sending the report can be optimized by using BURL or similar techniques.<br />
<br />
POP clients have no folders, hence they'd need special machinery, such as a new command to upload an abuse report or to retrieve the address where it should be sent<ref name="start" />.<br />
<br />
==UI problems and interpretation of spam reports==<br />
Difficulties with manually submitted reports stem from the scarce reliability that characterizes users and their computing equipments<ref>{{list archive|msg=15848|from=Rich Kulawiec|date=Wed, 16 Dec 2009 07:07:42 -0500}}</ref>. Users error rates have been reported to be around 5~10%<ref>{{list archive|msg=15881|from=Jose-Marcio Martins da Cruz|date=Sun, 20 Dec 2009 13:54:14 +0100}} Cormack, G. V. and Kolcz, A. 2009. [http://plg.uwaterloo.ca/~gvcormac/cormacksigir09-spam.pdf Spam filter evaluation with imprecise ground truth]. In ''Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in information Retrieval'' (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM, New York, NY, 604-611.</ref><ref>{{list archive|msg=15891|from=Matthias Leisi|date=Mon, 21 Dec 2009 19:24:06 +0100}}</ref>. In addition, users tend to categorize mail incoherently as ''wanted/not-wanted''<ref>{{list archive|msg=15888|from=Jose-Marcio Martins da Cruz|date=Mon, 21 Dec 2009 15:32:41 +0100}}</ref> rather than ''legitimate/abusive'' (never mind the ''ham/spam'' dichotomy, which is fuzzy by definition.) Accordingly, servers may resort to interpret users' intentions by extracting further info from reported messages, e.g., if the sender has sent multiple messages before and it has a List-Unsubscribe: header, it's probably an opt-out, not a spam complaint<ref>{{list archive|msg=15900|from=John Levine|date=23 Dec 2009 01:53:00 -0000}}</ref>.<br />
<br />
UI design includes issues such as the number of buttons that users can reach. That UI might be configurable, so as to suit users with different skills<ref>{{list archive|msg=15898|from=Nathaniel Borenstein|date=Tue, 22 Dec 2009 14:54:07 -0500}}</ref>. It seems clear that some experimentation is needed in order to understand users behavior<ref>{{list archive|msg=15894|from=Ian Eiloart|date=Tue, 22 Dec 2009 11:40:24 +0000}}</ref>. However, besides a user's skill and the UI design, a mailbox provider's stance<ref>{{list archive|msg=15899|from=Michael Thomas|date=Tue, 22 Dec 2009 12:56:34 -0800}}</ref> and any effects of the way it processes abuse reports will also play a role.<br />
<br />
==See also==<br />
[[Adding a junk button to MUAs]]<br />
==References==<br />
<References /></div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Description_of_DNSBLs&diff=1677Description of DNSBLs2011-03-23T02:31:07Z<p>Johnl: </p>
<hr />
<div>This draft is now [http://tools.ietf.org/html/rfc5782 RFC 5782].<br />
<br />
Primary author is John Levine, document shepherd was Alexey Melnikov.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Main_Page&diff=1676Main Page2011-03-23T02:25:00Z<p>Johnl: take out openid</p>
<hr />
<div>= The Anti Spam Research Group wiki =<br />
<br />
== Projects ==<br />
<br />
* [[Taxonomy of anti-spam techniques]]<br />
* [[Taxonomy of spamming techniques]]<br />
* [[Description of DNSBLs]]<br />
* [[BCP for DNSBLs]]<br />
* [[Assessing anti-spam techniques]]<br />
* [[Adding a junk button to MUAs]]<br />
<br />
== Events and Notes ==<br />
<br />
* [[ASRG at IETF 71]]<br />
<br />
* [[ASRG at IETF 72]]<br />
<br />
* [[Interesting Papers]] mentioned in ASRG discussions<br />
<br />
== Mailing List ==<br />
<br />
Like all IRTF groups, the ASRG operates primarily by e-mail.<br />
[[https://www.ietf.org/mailman/listinfo/asrg Subscriber page]]<br />
<br />
== Access ==<br />
<br />
ASRG members can edit this wiki. Send a request by e-mail to [[mailto:chair@asrg.sp.am chair@asrg.sp.am]] to get an account. <br />
<br />
If you want to edit this wiki, please also join the [[https://www.ietf.org/mailman/listinfo/asrg ASRG mailing list]].</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=SRS&diff=1675SRS2010-12-15T04:32:33Z<p>Johnl: Created page with "{{ast |date=2003 |difficult=Easy |popular=Low |effective=Low |harm=Low |where=MTA }} A Sender Rewriting Scheme (SRS) has been proposed for mail forwarders to use to make forward…"</p>
<hr />
<div>{{ast<br />
|date=2003<br />
|difficult=Easy<br />
|popular=Low<br />
|effective=Low<br />
|harm=Low<br />
|where=MTA<br />
}}<br />
<br />
A Sender Rewriting Scheme (SRS) has been proposed for mail forwarders to use to make forwarded mail more compatible with SPF, but it has seen little use. It embeds the original bounce address of a message in a new address in the domain of the forwarder which will pass SPF. if the message bounces, the bounce is intended to return to the forwarder, which can recover the original address and forward back the bounce.<br />
<br />
= References =<br />
* [http://en.wikipedia.org/wiki/Sender_Rewriting_Scheme Sender Rewriting Scheme]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=SPF&diff=1674SPF2010-12-15T04:20:41Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=2003<br />
|difficult=Easy<br />
|popular=Medium<br />
|effective=Medium<br />
|harm=High<br />
|where=MTA<br />
}}<br />
<br />
SPF (originally Sender Permitted From, now Sender Policy Framework) is a path management technique that assigns a set of permitted IP addresses to the domain in a message's bounce (MAIL FROM) address. The SPF record is a TXT record published in the DNS under the name of the domain, in a complex format. If the IP address from which a message is sent matches the domain's record, SPF succeeds. If not, there are various levels of hard and soft failure.<br />
<br />
Like all path validation schemes, SPF does not deal well with mail that does not all come from a fixed source.<br />
In practice, it works well for bulk mailers that send from a small set of servers, somewhat well for organizations where most users use the organization's own mail system, and not at all for courtesy forwarders, and "permanent" mail addresses at universities and professional societies.<br />
<br />
[[SRS]] is intended to make forwarded mail more compatible with SPF.<br />
<br />
= References =<br />
* [http://www.openspf.org/ SPF Project Overview]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=SPF&diff=1673SPF2010-12-15T04:17:54Z<p>Johnl: Created page with "{{ast |date=2003 |difficult=Easy |popular=Medium |effective=Medium |harm=High |where=MTA }} SPF (originally Sender Permitted From, now Sender Policy Framework) is a path manageme…"</p>
<hr />
<div>{{ast<br />
|date=2003<br />
|difficult=Easy<br />
|popular=Medium<br />
|effective=Medium<br />
|harm=High<br />
|where=MTA<br />
}}<br />
SPF (originally Sender Permitted From, now Sender Policy Framework) is a path management technique that assigns a set of permitted IP addresses to the domain in a message's bounce (MAIL FROM) address. The SPF record is a TXT record published in the DNS under the name of the domain, in a complex format. If the IP address from which a message is sent matches the domain's record, SPF succeeds. If not, there are various levels of hard and soft failure.<br />
<br />
Like all path validation schemes, SPF does not deal well with mail that does not all come from a fixed source.<br />
In practice, it works well for bulk mailers that send from a small set of servers, somewhat well for organizations where most users use the organization's own mail system, and not at all for courtesy forwarders, and "permanent" mail addresses at universities and professional societies.<br />
<br />
A Sender Rewriting Scheme (SRS) has been proposed for mail forwarders to use to make forwarded mail more compatible with SPF, but it has seen little use.<br />
<br />
= References =<br />
* [http://www.openspf.org/ SPF Project Overview]<br />
* [http://en.wikipedia.org/wiki/Sender_Rewriting_Scheme Sender Rewriting Scheme (SRS)]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Attention_bonds&diff=1672Attention bonds2010-10-02T19:06:53Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=1997?<br />
|difficult=High<br />
|popular=Very low<br />
|effective=Unknown<br />
|harm=Varies<br />
|where=MTA/MUA<br />
}}<br />
<br />
In an attention bond system, the sender of a message puts a sum of money in escrow as a bond warranting that the message is not spam. If the recipient decides that the message is, in fact, spam, the bonded money is transferred from escrow holder to recipient.<br />
<br />
Philip Raymond invented the idea in the late 1990s and holds [http://www.johnlevine.com/6697462.pdf a patent] on it, under the name Vanquish. Thede Loder and others, unaware of Raymond's work, reinvented it in about 2004 and called it attention bonds.<br />
<br />
== Advantages ==<br />
<br />
Attention bonds would impose an actual cost on spammers, both on 'pure' spammers and on otherwise legitimate organizations that occasionally spam.<br />
<br />
Users who do not wish to participate in the system do not have to, in that they can opt to continue to receive e-mail with no bond required.<br />
<br />
== Disadvantages ==<br />
<br />
* Requires a working digital cash micropayment system with universal deployment and low transaction overheads. No such system exists, and financial organizations are motivated to avoid deploying one.<br />
<br />
* Requires one or more trustworthy escrow organizations, as otherwise spammers will simply send e-mail with forged attention bonds.<br />
<br />
* Requires that users be able to exempt certain senders from bond requirements, so that e-mail newsletters can still exist.<br />
<br />
* If the system is poorly implemented with inadequate security, hijacked user accounts may result in financial cost to the victims.<br />
<br />
* Requires a new protocol so that the sender's e-mail software can determine the bond required, and the user can be prompted as to whether to accept it.<br />
<br />
* If users are allowed to set their own bond values, some people may set theirs so high that they are effectively cut off.<br />
<br />
* Creates a variety of new frauds, such as send mail and get a free trinket, actually to collect the bonds.<br />
<br />
* Because of extreme variations in wealth between nations, people in poorer nations might find themselves cut off from e-mailing anyone in wealthy nations.<br />
<br />
* Opinion polls suggest an extremely high level of resistance to any kind of financial cost potentially being involved in sending e-mail.<br />
<br />
So far no system of attention bonds has been widely deployed, so the effectiveness it would have is widely debated.<br />
<br />
== Examples ==<br />
<br />
* [http://www.vanquish.com Vanquish]<br />
<br />
== References ==<br />
<br />
* [http://www.informationweek.com/news/security/vulnerabilities/showArticle.jhtml?articleID=163104354 <i>Information Week</i> article] citing Raymond and Van Alstyne<br />
* [http://weblog.johnlevine.com/Email/attentionbond.html 2004 blog entry] on Vanquish and attention bonds</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Attention_bonds&diff=1671Attention bonds2010-10-02T19:02:49Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=1997?<br />
|difficult=High<br />
|popular=Very low<br />
|effective=Unknown<br />
|harm=Varies<br />
|where=MTA/MUA<br />
}}<br />
<br />
In an attention bond system, the sender of a message puts a sum of money in escrow as a bond warranting that the message is not spam. If the recipient decides that the message is, in fact, spam, the bonded money is transferred from escrow holder to recipient.<br />
<br />
Philip Raymond invented the idea in the late 1990s and holds [http://www.johnlevine.com/6697462.pdf a patent] on it, under the name Vanquish. Thede Loder and others, unaware of Raymond's work, reinvented it in about 2004 and called it attention bonds.<br />
<br />
== Advantages ==<br />
<br />
Attention bonds would impose an actual cost on spammers, both on 'pure' spammers and on otherwise legitimate organizations that occasionally spam.<br />
<br />
Users who do not wish to participate in the system do not have to, in that they can opt to continue to receive e-mail with no bond required.<br />
<br />
== Disadvantages ==<br />
<br />
* Requires a working digital cash micropayment system with universal deployment and low transaction overheads. No such system exists, and financial organizations are motivated to avoid deploying one.<br />
<br />
* Requires one or more trustworthy escrow organizations, as otherwise spammers will simply send e-mail with forged attention bonds.<br />
<br />
* Requires that users be able to exempt certain senders from bond requirements, so that e-mail newsletters can still exist.<br />
<br />
* If the system is poorly implemented with inadequate security, hijacked user accounts may result in financial cost to the victims.<br />
<br />
* Requires a new protocol so that the sender's e-mail software can determine the bond required, and the user can be prompted as to whether to accept it.<br />
<br />
* If users are allowed to set their own bond values, some people may set theirs so high that they are effectively cut off.<br />
<br />
* Creates a variety of new frauds, such as send mail and get a free trinket, actually to collect the bonds.<br />
<br />
* Because of extreme variations in wealth between nations, people in poorer nations might find themselves cut off from e-mailing anyone in wealthy nations.<br />
<br />
* Opinion polls suggest an extremely high level of resistance to any kind of financial cost potentially being involved in sending e-mail.<br />
<br />
So far no system of attention bonds has been widely deployed, so the effectiveness it would have is widely debated.<br />
<br />
== Examples ==<br />
* [http://www.vanquish.com Vanquish]<br />
<br />
* [http://www.informationweek.com/news/security/vulnerabilities/showArticle.jhtml?articleID=163104354 <i>Information Week</i> article] citing Raymond and Van Alstyne<br />
* [http://weblog.johnlevine.com/Email/attentionbond.html 2004 blog entry] on Vanquish and attention bonds</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Collaborative_filters&diff=1670Collaborative filters2010-10-02T18:51:03Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=2001?<br />
|difficult=Easy<br />
|popular=High<br />
|effective=Varies<br />
|harm=Varies<br />
|where=MTA/MUA<br />
}}<br />
<br />
Collaborative filters rely on users marking traffic as either spam or not-spam. Aggregate scores summarizing the judgments of lots of users are then used as input to simple content filters. Some versions such as [http://razor.sourceforge.net/ Vipul's Razor] also include reports of reporters, so that reliable reporters have more influence, and the effect of sloppy or dishonest reporters is limited. <br />
<br />
== Advantages ==<br />
<br />
Actual human intelligence is harnessed in deciding whether something is spam or not, so many simple spammer tricks are ineffective. (Examples: Spam messages as graphical images, variant spellings.)<br />
<br />
== Disadvantages ==<br />
<br />
If anyone can become a user of the system, spammers will sign up in order to attempt to game it. By using web scripting tools and anonymous proxies, they will attempt to create large numbers of dummy accounts with which to upvote their spam--[http://www.mediapost.com/publications/?fa=Articles.showArticle&art_aid=130320 millions, if necessary]. There are techniques to attempt to adjust for this by rating the raters (example: Slashdot meta-moderation), but they are only partially successful, as spammers can easily vote legitimately part of the time, or rate their other accounts as accurate.<br />
<br />
Collaborative filters can also easily become systems for collaborative censorship. For example, [http://blogs.alternet.org/oleoleolson/2010/08/05/massive-censorship-of-digg-uncovered/ organized political censorship by teams of collaborators] has been observed on Digg.<br />
<br />
Another disadvantage is that large numbers of participating users are required for the system to work. Perhaps the most successful example is Google Gmail, which achieves accuracy because of the extremely large user population.<br />
<br />
In addition, collaborative filtering suffers from a bootstrapping problem. Collaborative filters alone are generally insufficient; the initial volume of spam is so high that it's not worth wading through it all to upvote or downvote anything, so users give up, so the collaborative filtering doesn't work, so the spam volume remains high.<br />
<br />
== Examples ==<br />
* [http://razor.sourceforge.net/ Vipul's Razor]<br />
* [http://www.cloudmark.com/en/serviceproviders/authority-spamassassin.html/ Cloudmark Authority]<br />
* Report spam buttons at AOL, Hotmail, Yahoo! Mail, Google gmail, and other mail systems</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Main_Page&diff=1661Main Page2010-09-30T02:49:29Z<p>Johnl: </p>
<hr />
<div>= The Anti Spam Research Group wiki =<br />
<br />
== Projects ==<br />
<br />
* [[Taxonomy of anti-spam techniques]]<br />
* [[Taxonomy of spamming techniques]]<br />
* [[Description of DNSBLs]]<br />
* [[BCP for DNSBLs]]<br />
* [[Assessing anti-spam techniques]]<br />
* [[Adding a junk button to MUAs]]<br />
<br />
== Events and Notes ==<br />
<br />
* [[ASRG at IETF 71]]<br />
<br />
* [[ASRG at IETF 72]]<br />
<br />
* [[Interesting Papers]] mentioned in ASRG discussions<br />
<br />
== Mailing List ==<br />
<br />
Like all IRTF groups, the ASRG operates primarily by e-mail.<br />
[[https://www.ietf.org/mailman/listinfo/asrg Subscriber page]]<br />
<br />
== Access ==<br />
<br />
ASRG members can edit this wiki. Send a request by e-mail to [[mailto:chair@asrg.sp.am chair@asrg.sp.am]] to get an account. Or if you have an OpenID account, you can use that, and log in via the openid link at the upper right.<br />
<br />
Either way, if you want to edit this wiki, please also join the [[https://www.ietf.org/mailman/listinfo/asrg ASRG mailing list]].</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Bayesian_filters&diff=1660Bayesian filters2010-09-28T17:32:17Z<p>Johnl: Created page with '{{ast |date=2002 |difficult=Medium |popular=High |effective=Varies |harm=Varies |where=MTA/MUA }} Bayesian filters were first proposed in Paul Graham's [http://www.paulgraham.co…'</p>
<hr />
<div>{{ast<br />
|date=2002<br />
|difficult=Medium<br />
|popular=High<br />
|effective=Varies<br />
|harm=Varies<br />
|where=MTA/MUA<br />
}}<br />
<br />
Bayesian filters were first proposed in Paul Graham's [http://www.paulgraham.com/spam.html a plan for spam] in 2002. Starting with two corpora of messages, one of spam, and one of legitimate mail (''ham''), software extracts words or other features from the messages and determines how common each feature is in ham and spam. When subsequent messages arrive, a message with many features common to spam is probably spam, and a message with many features common to legitimate messages is probably legitimate.<br />
<br />
== Advantages ==<br />
<br />
When the corpora accurately reflect a user's message stream, Bayesian filters can be very accurate. As more messages arrive, the filters can adapt to the changing characteristics of messages. (See Sources of Corpora, below.)<br />
<br />
== Disadvantages ==<br />
<br />
It can be difficult to collect enough messages, particularly legitimate messages, to train a Bayesian filter accurately. Both filter training and filtering are slow, since they require significant processing of messages to extract features, to calculate probabilities, and to match message features against the filters.<br />
<br />
When a filter is self-training, that is, it adds messages that it has classified to its corpora, errors tend to cascade.<br />
<br />
== Sources of Corpora ==<br />
<br />
In many environments, it is difficult to find and classify enough messages to create an adequate corpus to train filters. While there are many available archives of spam, there are few archives of legitimate mail, for privacy reasons. Some filters are initially trained from a static corpus, then updated as messages are delivered, automatically as messages are classified by the filter, or as mail is manually classified by users by explicit tagging or implicitly as the move messages between MUA inbox and spam folders. Some systems use a shared filter for groups of users, while others keep separate statistics for groups of users.<br />
<br />
A significant source of corpora is [[Collaborative Filtering]], q.v.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Taxonomy_of_anti-spam_techniques&diff=1659Taxonomy of anti-spam techniques2010-09-28T17:13:17Z<p>Johnl: /* Message content techniques */</p>
<hr />
<div>Many people have invented many anti-spam techniques over the past decade. And a lot of the techniques keep being reinvented. Our goal here is to list all the anti-spam techniques we know, both the good ones and the bad ones.<br />
----<br />
==Message content techniques==<br />
<br />
* [[Static filters]]<br />
<br />
* [[Adaptive filters]]<br />
** [[Bayesian filters]]<br />
** [[Collaborative filters]]<br />
<br />
* [[URL filtering]]<br />
<br />
==SMTP techniques==<br />
<br />
* [[Timing and protocol defects]]<br />
<br />
* [[Greylisting]]<br />
<br />
* [[Callbacks]]<br />
<br />
* [[Rate limits]]<br />
<br />
* [[SMTP pull]]<br />
<br />
==Address management==<br />
<br />
* [[Tagged addresses]]<br />
<br />
* [[Code words]]<br />
<br />
* [[Disposable addresses]]<br />
<br />
==Network techniques==<br />
<br />
* [[DNS blacklists]]<br />
<br />
* [[DNS validation]]<br />
<br />
* [[HELO/EHLO Pattern Matching]]<br />
<br />
* [[Tarpitting and other timing techniques]]<br />
<br />
==Whitelist techniques==<br />
<br />
* [[Recognizing known correspondents]]<br />
<br />
* [[Path validation]]<br />
** [[Path validation manual techniques]]<br />
** [[SPF]] (Sender permitted from)<br />
** [[Sender-ID]]<br />
** [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Marking valid mail client hosts]]<br />
<br />
* [[Signatures]]<br />
<br />
* [[Subject keywords]]<br />
<br />
==[[Message validation techniques]]==<br />
<br />
* [[Cryptographic signatures]]<br />
<br />
* [[Signature callback]]<br />
<br />
==[[Introduction techniques]]==<br />
<br />
* [[Challenges]]<br />
<br />
* [[Proof of work]]<br />
<br />
* [[Postage]]<br />
<br />
* [[Attention bonds]]<br />
<br />
* [[Captchas]]<br />
<br />
==Administrative techniques==<br />
<br />
* [[No spam notices]]<br />
<br />
* [[Do not spam lists]]<br />
<br />
* [[Abuse Reporting]]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adaptive_filters&diff=1658Adaptive filters2010-09-28T17:12:55Z<p>Johnl: </p>
<hr />
<div>Techniques that adjust the features they look for based on experience.<br />
<br />
* [[Bayesian filters]]<br />
* [[Collaborative filters]]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Taxonomy_of_anti-spam_techniques&diff=1651Taxonomy of anti-spam techniques2010-04-17T14:56:34Z<p>Johnl: </p>
<hr />
<div>Many people have invented many anti-spam techniques over the past decade. And a lot of the techniques keep being reinvented. Our goal here is to list all the anti-spam techniques we know, both the good ones and the bad ones.<br />
----<br />
==Message content techniques==<br />
<br />
* [[Static filters]]<br />
<br />
* [[Adaptive filters]]<br />
<br />
* [[URL filtering]]<br />
<br />
==SMTP techniques==<br />
<br />
* [[Timing and protocol defects]]<br />
<br />
* [[Greylisting]]<br />
<br />
* [[Callbacks]]<br />
<br />
* [[Rate limits]]<br />
<br />
* [[SMTP pull]]<br />
<br />
==Address management==<br />
<br />
* [[Tagged addresses]]<br />
<br />
* [[Code words]]<br />
<br />
* [[Disposable addresses]]<br />
<br />
==Network techniques==<br />
<br />
* [[DNS blacklists]]<br />
<br />
* [[DNS validation]]<br />
<br />
* [[HELO/EHLO Pattern Matching]]<br />
<br />
* [[Tarpitting and other timing techniques]]<br />
<br />
==Whitelist techniques==<br />
<br />
* [[Recognizing known correspondents]]<br />
<br />
* [[Path validation]]<br />
** [[Path validation manual techniques]]<br />
** [[SPF]] (Sender permitted from)<br />
** [[Sender-ID]]<br />
** [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Marking valid mail client hosts]]<br />
<br />
* [[Signatures]]<br />
<br />
* [[Subject keywords]]<br />
<br />
==[[Message validation techniques]]==<br />
<br />
* [[Cryptographic signatures]]<br />
<br />
* [[Signature callback]]<br />
<br />
==[[Introduction techniques]]==<br />
<br />
* [[Challenges]]<br />
<br />
* [[Proof of work]]<br />
<br />
* [[Postage]]<br />
<br />
* [[Attention bonds]]<br />
<br />
* [[Captchas]]<br />
<br />
==Administrative techniques==<br />
<br />
* [[No spam notices]]<br />
<br />
* [[Do not spam lists]]<br />
<br />
* [[Abuse Reporting]]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Path_validation_manual_techniques&diff=1650Path validation manual techniques2010-04-17T14:54:54Z<p>Johnl: Created page with '{{ast |date=late 1990s |difficult=Low to do poorly, High to do well |popular=Medium |effective=Low |harm=Low |where=MTA/MUA }} It is possible albeit labor-intensive to make speci…'</p>
<hr />
<div>{{ast<br />
|date=late 1990s<br />
|difficult=Low to do poorly, High to do well<br />
|popular=Medium<br />
|effective=Low<br />
|harm=Low<br />
|where=MTA/MUA<br />
}}<br />
It is possible albeit labor-intensive to make specific checks for known senders. For example, domain paypal.com sends all of its mail from servers with names of the form <i>host</i>.<i>tla</i>.paypal.com where <i>tla</i> is a three letter city code. It is not hard to write a filter that checks the Received: line in mail with paypal.com return addresses to ensure that it came from one of those servers. This is quite effective, at least until Paypal changes server names or outsources some of their mail.<br />
<br />
This can be useful for a few heavily phished domains, but the work involved makes it impractical in general. To the extent it validates the sending host, it is in effect a manual version of [[SPF]].</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1643Adding a junk button to MUAs2010-02-21T16:47:31Z<p>Johnl: /* Via IMAP */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
[[Image:Reportasjunk.png|thumb|right]]<br />
=How many buttons?=<br />
<br />
Existing webmail all has a single junk button, or (gmail) one button that pops up 'do you want to unsub?' if the message looks like it's from a list.<br />
<br />
Some people suggest multiple options, e.g., junk, unsub, don't want any more, etc.<br />
<br />
ARF has many of these options, but as far as we can tell the only ones ever used are junk (which it calls abuse) and virus, which scripts can identify automatically from EXE and ZIP signatures.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT. A ''Consensus Call''<ref>{{list archive|msg=16048|subject=Consensus Call - submission via posting (was Re: Iteration #3)|from=Dave CROCKER|date=Sat, 06 Feb 2010 10:37:05 -0800}}</ref> has had 10 posts for ''normal posting''<br />
<ref>{{list archive|msg=16049|from=Lyndon Nerenberg|date=Sat, 6 Feb 2010 11:50:58 -0700}}</ref><br />
<ref>{{list archive|msg=16051|from=Chris Lewis|date=Sat, 6 Feb 2010 14:36:49 -0500}}</ref><br />
<ref>{{list archive|msg=16057|from=Steve Atkins|date=Sat, 6 Feb 2010 12:42:06 -0800}}</ref><br />
<ref>{{list archive|msg=16065|from=Dotzero|date=Sat, 6 Feb 2010 18:13:58 -0500}}</ref><br />
<ref>{{list archive|msg=16070|from=Derek Diget|date=Sat, 06 Feb 2010 21:18:45 -0500 (EST)}}</ref><br />
<ref>{{list archive|msg=16071|from=Bart Schaefer|date=Sat, 06 Feb 2010 18:46:53 -0800}}</ref><br />
<ref>{{list archive|msg=16073|from=Alessandro Vesely|date=Sun, 07 Feb 2010 07:33:29 +0100}}</ref><br />
<ref>{{list archive|msg=16099|from=John Levine|date=8 Feb 2010 14:14:03 -0000}}</ref><br />
<ref>{{list archive|msg=16114|from=BOBOTEK, ALEX|date=Mon, 8 Feb 2010 08:50:12 -0800}}</ref><br />
<ref>{{list archive|msg=16149|from=Bill Cole|date=Mon, 08 Feb 2010 23:11:57 -0500}}</ref><br />
, 2 for ''specific to the mechanism for retrieving the message''<br />
<ref>{{list archive|msg=16091|from=Ian Eiloart|date=Mon, 08 Feb 2010 12:43:13 +0000}}</ref><br />
<ref>{{list archive|msg=16115|from=Andrew Richards|date=Mon, 08 Feb 2010 17:35:39 +0000}}</ref><br />
, and 1 ambivalent<br />
<ref>{{list archive|msg=16189|from=Jose-Marcio Martins da Cruz|date=Tue, 09 Feb 2010 20:09:24 +0100}}</ref>.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL and in dovecot-antispam, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the reported message and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server. Reports are larger compared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Besides the URL, configuration may also need user/password for authentication.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple. Config can be per-account, which might be useful for servers that handle multiple resellers.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
* Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't provide the server name. All users of the same server get the same metadata, unless they refer to it by different names in which case maybe they don't.<br />
<br />
* If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME poppy.example.net<br />
_report.poppy.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.<br />
<br />
=See also=<br />
* [[Abuse Reporting]]<br />
<br />
=References=<br />
<References /></div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Taxonomy_of_anti-spam_techniques&diff=1636Taxonomy of anti-spam techniques2010-02-15T02:11:14Z<p>Johnl: /* Whitelist techniques */</p>
<hr />
<div>Many people have invented many anti-spam techniques over the past decade. And a lot of the techniques keep being reinvented. Our goal here is to list all the anti-spam techniques we know, both the good ones and the bad ones.<br />
----<br />
==Message content techniques==<br />
<br />
* [[Static filters]]<br />
<br />
* [[Adaptive filters]]<br />
<br />
* [[URL filtering]]<br />
<br />
==SMTP techniques==<br />
<br />
* [[Timing and protocol defects]]<br />
<br />
* [[Greylisting]]<br />
<br />
* [[Callbacks]]<br />
<br />
* [[Rate limits]]<br />
<br />
* [[SMTP pull]]<br />
<br />
==Address management==<br />
<br />
* [[Tagged addresses]]<br />
<br />
* [[Code words]]<br />
<br />
* [[Disposable addresses]]<br />
<br />
==Network techniques==<br />
<br />
* [[DNS blacklists]]<br />
<br />
* [[DNS validation]]<br />
<br />
* [[HELO/EHLO Pattern Matching]]<br />
<br />
* [[Tarpitting and other timing techniques]]<br />
<br />
==Whitelist techniques==<br />
<br />
* [[Recognizing known correspondents]]<br />
<br />
* [[Path validation]]<br />
** [[Path validation manual techniques]]<br />
** [[SPF]] (Sender permitted from)<br />
** [[Sender-ID]]<br />
** [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Marking valid mail client hosts]]<br />
<br />
* [[Signatures]]<br />
<br />
* [[Subject keywords]]<br />
<br />
==[[Message validation techniques]]==<br />
<br />
* [[Cryptographic signatures]]<br />
<br />
* [[Signature callback]]<br />
<br />
==[[Introduction techniques]]==<br />
<br />
* [[Challenges]]<br />
<br />
* [[Proof of work]]<br />
<br />
* [[Postage]]<br />
<br />
* [[Attention bonds]]* [[Path validation manual techniques]]<br />
<br />
* [[SPF]] (Sender permitted from)<br />
<br />
* [[Sender-ID]]<br />
<br />
* [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Captchas]]<br />
<br />
==Administrative techniques==<br />
<br />
* [[No spam notices]]<br />
<br />
* [[Do not spam lists]]<br />
<br />
* [[Abuse Reporting]]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1632Adding a junk button to MUAs2010-02-14T03:32:33Z<p>Johnl: /* Configuring the MUA junk button */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=How many buttons?=<br />
<br />
Existing webmail all has a single junk button, or (gmail) one button that pops up 'do you want to unsub?' if the message looks like it's from a list.<br />
<br />
Some people suggest multiple options, e.g., junk, unsub, don't want any more, etc.<br />
<br />
ARF has many of these options, but as far as we can tell the only ones ever used are junk (which it calls abuse) and virus, which scripts can identify automatically from EXE and ZIP signatures.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the reported message and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple. Config can be per-account, which might be useful for servers that handle multiple resellers.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
* Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't provide the server name. All users of the same server get the same metadata, unless they refer to it by different names in which case maybe they don't.<br />
<br />
* If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME poppy.example.net<br />
_report.poppy.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1631Adding a junk button to MUAs2010-02-14T03:28:20Z<p>Johnl: /* Via a separate path */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=How many buttons?=<br />
<br />
Existing webmail all has a single junk button, or (gmail) one button that pops up 'do you want to unsub?' if the message looks like it's from a list.<br />
<br />
Some people suggest multiple options, e.g., junk, unsub, don't want any more, etc.<br />
<br />
ARF has many of these options, but as far as we can tell the only ones ever used are junk (which it calls abuse) and virus, which scripts can identify automatically from EXE and ZIP signatures.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the reported message and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
* Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
* If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME poppy.example.net<br />
_report.poppy.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1630Adding a junk button to MUAs2010-02-14T00:59:07Z<p>Johnl: </p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=How many buttons?=<br />
<br />
Existing webmail all has a single junk button, or (gmail) one button that pops up 'do you want to unsub?' if the message looks like it's from a list.<br />
<br />
Some people suggest multiple options, e.g., junk, unsub, don't want any more, etc.<br />
<br />
ARF has many of these options, but as far as we can tell the only ones ever used are junk (which it calls abuse) and virus, which scripts can identify automatically from EXE and ZIP signatures.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
* Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
* If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME poppy.example.net<br />
_report.poppy.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1629Adding a junk button to MUAs2010-02-14T00:51:47Z<p>Johnl: /* Metadata in the DNS */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
* Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
* If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME poppy.example.net<br />
_report.poppy.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1628Adding a junk button to MUAs2010-02-14T00:50:12Z<p>Johnl: /* Metadata in the DNS */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems evident to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1627Adding a junk button to MUAs2010-02-14T00:49:48Z<p>Johnl: /* Metadata in the DNS */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
If a server has many CNAMEs pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1626Adding a junk button to MUAs2010-02-14T00:49:27Z<p>Johnl: /* Metadata in the DNS */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that lose the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1625Adding a junk button to MUAs2010-02-14T00:48:04Z<p>Johnl: /* Per message approaches */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
''Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
''Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't pass the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1624Adding a junk button to MUAs2010-02-14T00:47:45Z<p>Johnl: /* Via a separate path */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
* The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
* The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
'Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
'Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't pass the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1623Adding a junk button to MUAs2010-02-14T00:46:55Z<p>Johnl: /* Via IMAP */</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
* Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
* Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
'Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
'Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't pass the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1622Adding a junk button to MUAs2010-02-14T00:46:01Z<p>Johnl: </p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery and retrieval software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
'Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
'Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't pass the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1621Adding a junk button to MUAs2010-02-14T00:45:26Z<p>Johnl: </p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar junk button (JB) to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Any scheme requires an upgrade or at least a plugin to the MUA. Different schemes require different amounts of changes to mail delivery software.<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it by which it was retrieved by the MUA, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the delivery path==<br />
<br />
Most but not all MUAs retrieve mail with IMAP or POP.<br />
<br />
''Overall pro:'' Reports are secure, since they're returned directly to the mail system. Configuration is likely to be automatic.<br />
<br />
''Overall con:'' Requires changes to multiple protocols and to server software. Changes to POP are severe.<br />
<br />
===Via IMAP===<br />
Add an IMAP Junk flag that the MUA can set. <br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and to server software.<br />
<br />
Move messages to a specific subfolder such as Spam.<br />
<br />
''Pro:'' Already implemented at AOL, requires no protocol changes. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires IMAP server (or back end at least) changes, adds a reserved folder name.<br />
<br />
===Via POP===<br />
<br />
Possibilities include:<br />
<br />
* Add a command like XTND JUNK that the MUA can use to send a message back to the server.<br />
* Modify the protocol so that UIDLs or message numbers persist beyond message deletion, and add a command to report the UIDL or session ID number.<br />
<br />
''Pro:'' Self-configuring, MUA can tell automatically if JB is available. Relatively secure since client has already logged in to server.<br />
<br />
''Con:'' Requires changes to protocol and server. XTND JUNK requires servers to accept arbitrarily large uploads, which they don't now. UIDL or ID report requires server to store potentially large amounts of additional mail.<br />
<br />
===Via other paths===<br />
<br />
Some MUAs read mailstore directly, or receive via UUCP or SMTP. Users would be out of luck.<br />
<br />
==Via a separate path==<br />
<br />
The MUA could build an ARF report from the and send that report back to the server.<br />
<br />
''Overall pro:'' Server need not retain messages for potential report. Once configured, reporting system is independent of inbound path or server, not dependent on POP or IMAP.<br />
<br />
''Overall con:'' Less secure than returning directly to incoming server, reports are larger comapared to an IMAP flag change or POP UIDL report.<br />
<br />
===Via SMTP===<br />
<br />
The MUA sends the ARF report to a reporting address at the mail system using standard SUBMIT or SMTP. Configuration only needs the target address.<br />
<br />
''Pro:'' MUAs already know how to create and send MIME messages, leverage existing software, simple configuration.<br />
<br />
''Con:'' SUBMIT or SMTP server might not be under the same management as inbound server, possibly leading to third party snooping or message loss. If systems accept reports other than from their own SUBMIT server, forged reports and DoS are possible.<br />
<br />
===Via HTTP PUT or POST===<br />
<br />
The MUA uploads the ARF report using HTTP PUT or POST. Configuration only needs the URL.<br />
<br />
''Pro:'' Faster and and more secure than SMTP, since communication is directly from MUA to server.<br />
<br />
''Con:'' Requires an HTTP server to accept reports, making an MUA do a PUT or POST may be a challenge.<br />
<br />
=Configuring the MUA junk button=<br />
<br />
The MUA needs to know both whether a message can be reported as junk, and how to report it. We seem to agree that automatic configuration is better than manual.<br />
[<br />
==Per message approaches==<br />
<br />
At delivery time, the MDA adds a header that indicates that a JB is available, with configuration parameters such as the address to which to mail an ARF report. A likely candidate for the header would be an extended version of the [http://tools.ietf.org/html/rfc5451 Authentication-Results] header.<br />
<br />
'Pro:'' No changes to retrieval protocol, independent of retrieval protocol. Can put different flags on different messages. Can potentially remain useful across forwarding or mail systems that consolidate mail by POP retrieval from others.<br />
<br />
'Con:'' Requires changes to MDA. Bad guys could add headers to hijack reports, or do low-level DoS against third parties. Doesn't work well with IMAP junk flag, POP UIDL report, etc.<br />
<br />
==Per server approaches==<br />
<br />
Since the MUA already knows how to find a POP or IMAP server, it can find the JB configuration directly or indirectly from that server.<br />
<br />
===Server queries===<br />
<br />
The POP CAPA command and IMAP CAPABILITY command allow MUAs to query what a server can do. For schemes that report directly to the retrieval server, the MUA can see if the reporting command(s) are available. For schemes that return the report via a different path, a simple command could return configuration info.<br />
<br />
''Pro:'' Secure since MUA already logs into server, protocol and server changes are relatively simple.<br />
<br />
''Con;'' Still requires protocol and server changes.<br />
<br />
===Metadata in the DNS===<br />
<br />
Publish configuration info in the DNS using names based on server names, e.g., <br />
<br />
_report.pop.example.com TXT "arfreport@example.com"<br />
<br />
''Pro:'' No changes to MDA, retrieval protocol or software. Easy to configure in MUAs that can make DNS queries (most of them, presumably).<br />
<br />
''Con:'' New requirement to manage host names that refer to POP or IMAP server. Servers may have multiple names (thousands in some cases) via CNAME or multiple A records requiring a lot of DNS maintenance for any change. Servers may be accessed via tunnels or proxies that don't pass the name used to find the server.<br />
<br />
If a server has many CNAMES pointing at one A record, a variant would be for the MUA to resolve all the CNAMEs and look for the metadata near the name of the A record, e.g.<br />
<br />
pop.example.com CNAME pop.example.net<br />
_report.pop.example.net TXT "arfreport@example.net"<br />
<br />
''Pro:'' Addresses multiple CNAME problem.<br />
<br />
''Con:'' Only partial solution, doesn't address multiple A, tunnel, etc. Unusual DNS usage may be hard to implement or have subtle problems visible only to DNS weeenies.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1620Adding a junk button to MUAs2010-02-13T22:57:26Z<p>Johnl: </p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar button to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Two somewhat separate issues are:<br />
<br />
* How a message that is reported in the MUA is sent back to the mail system<br />
* How a MUA knows when reporting is available<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it arrived, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the arrival path==<br />
<br />
For IMAP clients, it would be straightforward to add a Junk flag IMAP that the MUA can set.<br />
<br />
''Pro:'' Easy to implement with IMAP. Self-configuring with IMAP, MUA can tell if the <br />
<br />
<br />
<br />
==Via a separate path==<br />
<br />
<br />
<br />
=Configuring the MUA junk button=<br />
<br />
==Per message approaches==<br />
<br />
==Per server approaches==</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Adding_a_junk_button_to_MUAs&diff=1619Adding a junk button to MUAs2010-02-13T21:02:50Z<p>Johnl: Created page with 'Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar button to a regular MUA? For th…'</p>
<hr />
<div>Most web mail systems have a button a user can push to mark a message as junk, and report it back to their mail host. How would one add a similar button to a regular MUA? For the purposes of this discussion, the goal is to report the message back to the user's own mail system. They might further report it to the place the message came from, but that's a separate discussion.<br />
<br />
Two somewhat separate issues are:<br />
<br />
* How a message that is reported in the MUA is sent back to the mail system<br />
* How a MUA knows when reporting is available<br />
<br />
=Sending back reported messages=<br />
<br />
The two general approaches seem to be to send it back via the path it arrived, typically POP or IMAP, or to use a separate path, such as SMTP or SUBMIT.<br />
<br />
==Via the arrival path==<br />
<br />
==Via a separate path==<br />
<br />
<br />
<br />
=Configuring the MUA junk button=<br />
<br />
==Per message approaches==<br />
<br />
==Per server approaches==</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Main_Page&diff=1618Main Page2010-02-13T20:46:55Z<p>Johnl: /* Projects */</p>
<hr />
<div>= The Anti Spam Research Group wiki =<br />
<br />
== Projects ==<br />
<br />
* [[Taxonomy of anti-spam techniques]]<br />
* [[Taxonomy of spamming techniques]]<br />
* [[Description of DNSBLs]]<br />
* [[BCP for DNSBLs]]<br />
* [[Assessing anti-spam techniques]]<br />
* [[Adding a junk button to MUAs]]<br />
<br />
== Events and Notes ==<br />
<br />
* [[ASRG at IETF 71]]<br />
<br />
* [[ASRG at IETF 72]]<br />
<br />
* [[Interesting Papers]] mentioned in ASRG discussions<br />
<br />
== Mailing List ==<br />
<br />
Like all IRTF groups, the ASRG operates primarily by e-mail.<br />
[[https://www.ietf.org/mailman/listinfo/asrg Subscriber page]]<br />
<br />
== Access ==<br />
<br />
ASRG members can edit this wiki. Send a request by e-mail to [[mailto:chair@asrg.sp.am chair@asrg.sp.am]] to get an account.<br />
<br />
If you want to edit this wiki, please also join the [[https://www.ietf.org/mailman/listinfo/asrg ASRG mailing list]].</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Taxonomy_of_anti-spam_techniques&diff=1617Taxonomy of anti-spam techniques2010-01-31T17:12:37Z<p>Johnl: </p>
<hr />
<div>Many people have invented many anti-spam techniques over the past decade. And a lot of the techniques keep being reinvented. Our goal here is to list all the anti-spam techniques we know, both the good ones and the bad ones.<br />
----<br />
==Message content techniques==<br />
<br />
* [[Static filters]]<br />
<br />
* [[Adaptive filters]]<br />
<br />
* [[URL filtering]]<br />
<br />
==SMTP techniques==<br />
<br />
* [[Timing and protocol defects]]<br />
<br />
* [[Greylisting]]<br />
<br />
* [[Callbacks]]<br />
<br />
* [[Rate limits]]<br />
<br />
* [[SMTP pull]]<br />
<br />
==Address management==<br />
<br />
* [[Tagged addresses]]<br />
<br />
* [[Code words]]<br />
<br />
* [[Disposable addresses]]<br />
<br />
==Network techniques==<br />
<br />
* [[DNS blacklists]]<br />
<br />
* [[DNS validation]]<br />
<br />
* [[HELO/EHLO Pattern Matching]]<br />
<br />
* [[Tarpitting and other timing techniques]]<br />
<br />
==Whitelist techniques==<br />
<br />
* [[Recognizing known correspondents]]<br />
<br />
* [[Path validation]]<br />
** [[Path validation manual techniques]]<br />
** [[SPF]] (Sender permitted from)<br />
** [[Sender-ID]]<br />
** [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Signatures]]<br />
<br />
* [[Subject keywords]]<br />
<br />
==[[Message validation techniques]]==<br />
<br />
* [[Cryptographic signatures]]<br />
<br />
* [[Signature callback]]<br />
<br />
==[[Introduction techniques]]==<br />
<br />
* [[Challenges]]<br />
<br />
* [[Proof of work]]<br />
<br />
* [[Postage]]<br />
<br />
* [[Attention bonds]]* [[Path validation manual techniques]]<br />
<br />
* [[SPF]] (Sender permitted from)<br />
<br />
* [[Sender-ID]]<br />
<br />
* [[SRS]] (Sender Rewriting Scheme)<br />
<br />
* [[Captchas]]<br />
<br />
==Administrative techniques==<br />
<br />
* [[No spam notices]]<br />
<br />
* [[Do not spam lists]]<br />
<br />
* [[Abuse Reporting]]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Path_validation&diff=1616Path validation2010-01-31T17:09:49Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=late 1990s<br />
|difficult=Low to do poorly, High to do well<br />
|popular=Medium<br />
|effective=Low<br />
|harm=Medium<br />
|where=MTA<br />
}}<br />
Techniques that match up the apparent sender of a message with the path via which it arrived.<br />
<br />
* [[Path validation manual techniques]]<br />
<br />
* [[SPF]] (Sender permitted from)<br />
<br />
* [[Sender-ID]]<br />
<br />
* [[SRS]] (Sender Rewriting Scheme)<br />
<br />
= General issues =<br />
<br />
Internet mail is designed to make message delivery independent of the path a message takes. Although much, perhaps most, mail is sent from hosts managed by the same people that manage domains in a message's header or envelope, there are a variety of well known situations where mail can legitimately arrive via other paths. Some of these situations are:<br />
<br />
* Courtesy forwards, e.g. professional organizations and college alumni assocations. They remail messages to a recipient's actual address, so the recipient system sees the message originating at the forwarding site.<br />
<br />
* Users of courtesy forwards. Since courtesy forwards tend to be inbound only, users send mail from their own ISP with the courtesy forward address.<br />
<br />
* Mailing lists. Messages are resent from the list host.<br />
<br />
* Mail-an-article and kiosks. Web sites allow users to send limited amounts of mail manually entering To: and From: addresses.<br />
<br />
* All-in-one web mail: Web mail systems including Yahoo and Google Gmail allow users to collect mail from other mailboxes via POP or IMAP and send mail with those mailboxes as return addresses.<br />
<br />
= Accountability =<br />
The rationale of path validation techniques is to hold a sending client responsible for what it transmits. Providers may terminate spammers' accounts upon receiving complaints, and damages resulting from spam might hypothetically be claimed before courts. Even if resorting to such extreme measures is sporadic, the theoretical possibility to take that course helps to keep control. To this end, various tokens may be considered in order to learn who is the accountable party.<br />
<br />
== IP addresses ==<br />
The immediate client IP address is the one fact about ''every'' MX-driven attempt at message transport that the receiving MTA can know with very high certainty, even when the message is originated by someone who maliciously tries to hide his identity. IP addresses are trackable and ultimately tied to an ISP, even if that carrier is often an upstream one. However, there is no working global mechanism for identifying an accountable party (i.e. one who explicitly ''accepts'' accountability) from an IP address, due largely to the political and historical variations in how IP addresses have been allocated.<br />
<br />
== HELO name ==<br />
The helo name can be verified to match the IP address in various ways. The CSV paradigm is that the operator of a MTA should exercise some responsibility for what is sends. Its HELO string validation states whether that domain exercises responsibility. However, even if ''mta.example.com'' is the true FQDN of the responsible transmitting client, DNS hierarchy neither follows nor implies administrative hierarchy, as stated in RFC 5507 section 4 "Zone Boundaries are Invisible to Applications".<br />
<br />
== Right Hand Side of MAIL FROM ==<br />
This is the mail domain used for SPF checking. For registered domains, whois data exists, even if it's possibly privacy concealed. SPF allows to verify that an IP address is authorized to relay mail on behalf of that mail domain, but says nothing about possible accountability implications.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=File:Yum.jpeg&diff=1611File:Yum.jpeg2009-12-15T15:05:32Z<p>Johnl: An iconic product</p>
<hr />
<div>An iconic product</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Abuse_Reporting&diff=1604Abuse Reporting2009-12-14T19:45:42Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=Before 2003<br />
|difficult=?<br />
|popular=Medium<br />
|effective=?<br />
|harm=?<br />
|where=External<br />
}}<br />
<br />
'''Abuse Reporting''' is a cooperative technique whereby spam is suppressed at the origin by the administrators of an MSA. Mail domains' reputation can be preserved by limiting outgoing spam, thus operators are motivated to suppress spamming. Probably the earliest anti-spam defense, this technique relies on end users to submit abuse reports timely and correctly. It is supported by huge mailbox providers, who feature a ''This is Spam'' (TiS) button on their web-based mail clients.<br />
<br />
==Abuse Reporting Format (ARF)==<br />
The format for transmitting abuse reports is going to be standardized. See ''abuse-feedback-report'' list.<br />
<br />
<!-- Handy? easier integration with wikipedia is possible by adding a MySQL record, as exemplified in<br />
http://www.mediawiki.org/wiki/Manual:Interwiki#Single_line --><br />
See also the Wikipedia [http://en.wikipedia.org/wiki/Feedback_Loop_%28email%29 page].<br />
<br />
==The Feedback Loop (FBL) and other usage==<br />
Huge mailbox providers may provide a means for ESPs to get abuse reports issued for messages that had been sent by themselves. This tool can be used to automate opt-out. The sender should tag each message, setting an opaque token and possibly signing it, so as to be able to retrieve necessary submission details.<br />
<br />
Another use of abuse reports is to tune an MX server's spam filter. For Bayesian filters this requires a spam/ham indicator. Some hosts automatically convert reports into unsubscribe requests when the reported message contained a ''List-Unsubscribe'' header field.<br />
<br />
While it is felt that proper use of abuse reports may result in reliable reputation recording, how abuse reports should be routed is still a debatable question. Rough consensus indicates that users should send abuse reports to their mailbox providers. FBLs as operated, e.g., by [http://returnpath.net/ ReturnPath], are not scalable and may be questionable from a privacy point of view. The [http://abusix.org/ Abusix] approach provides for routing abuse reports hierarchically, through top level IP assignments.<br />
<br />
==Adding a spam button to MUAs==<br />
Discussed in <ref>List thread on MUA abuse button[http://www.ietf.org/mail-archive/web/asrg/current/msg15780.html]</ref>. Difficulties stem from the fact that a user may have multiple mailbox providers, which rules out single-address configurations.<br />
<br />
The possibility that reports are cascaded back to the spammer may be considered a potential hazard that exposes the reporters and allows list washing. However, it has been pointed out that spammers are not interested in feedback and don't bother. There is also concern that a button might cause a flood of unwanted messages, specially when multiple targets can be determined.<br />
<br />
===Message's header based approach===<br />
An MX server may embed an header field indicating the address where abuse reports may be sent. If the MX doesn't support such field, MUAs may use the field provided by the sender, if any. Otherwise, MX operators may override that by removing the field, or by replacing it with their own FBL reporting address instead.<br />
<br />
Reusing existing field has also been proposed: the topmost ''Received'', any ''Delivered-To'', ''VBR-Info'', or any trusted ''Authentication-Results''. The latter one already requires users to confirm that they trust the relevant ''authserv-id'', thereby clearing that hazard concern. Fields with multiple values allow for multiple targets, which may be needed for forwarded messages in case abuse reports are used to tune spam filters. In addition, using a DNS RR to store one or more addresses allows even more flexibility: verifying that the service is actually supported, using addresses other than ''abuse@'', and directly sending copies to reputation providers seem valid demands that a DNS call can fulfill.<br />
<br />
===Client's protocol based approach===<br />
Setting up an IMAP folder for this purpose has the advantage of requiring no special software on clients. This is in facts the most common solution for servers that maintain per-user Bayesian data, although it may be difficult to interpret users' intent in a non-interactive environment, and notwithstanding possible confusion or conflict with junk folders maintained by the client. When the SUBMIT and IMAP servers coincide, sending the report can be optimized by using BURL or similar techniques.<br />
<br />
POP clients have no folders, hence they'd need special machinery, such as a new command to upload an abuse report or to retrieve the address where it should be sent.<br />
==References==<br />
<References /></div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Description_of_DNSBLs&diff=1598Description of DNSBLs2009-11-10T01:42:09Z<p>Johnl: </p>
<hr />
<div>This draft is now standards track, in IETF Last Call.<br />
<br />
Primary author is John Levine, document shepherd is Alexey Melnikov.<br />
<br />
The latest version is [http://tools.ietf.org/html/draft-irtf-asrg-dnsbl here]. <br />
<br />
Drafts are done, waiting in RFC Editor queue with other IRTF RFCs.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=SMTP_pull&diff=1596SMTP pull2009-08-29T06:07:33Z<p>Johnl: /* Disadvantages */</p>
<hr />
<div>{{ast<br />
|date=?<br />
|difficult=High<br />
|popular=Low<br />
|effective=Low<br />
|harm=?<br />
|where=MTA/MUA/MSA<br />
}}<br />
'''SMTP pull''' is the opposite of ''SMTP push'', the usual store and forward model. SMTP pull provides for the sender to submit only part of a message, an ''offer''. Either the MUA or the MTA will then pull, i.e. connect to the relevant server and fetch the rest of the message's content.<br />
<br />
Pull proposals for anti-spam purposes provide for ''asynchronous pull'', i.e. storing offers in a queue and fetching the relevant content at some later stage. ''Synchronous pull'' allows to fail an SMTP transaction in case fetching the content fails, which is not feasible for the MUA case. A variant of this kind of proposals ([http://www.ietf.org/mail-archive/web/asrg/current/msg15715.html this]) provides for replacing SUBMIT so as to avoid SMTP entirely.<br />
<br />
== Advantages ==<br />
The ''pull'' model might lend itself to traffic optimizations that are not possible otherwise. In case the recipient can determine from the offer that the content is not interesting, its transfer can be avoided altogether. As automated verifications can be done both synchronously and asynchronously, the only residual advantage for the case of the asynchronous MTA-pull is the added shield against DNS poisoning that asynchronous DNS queries imply.<br />
<br />
Psychologically, an apparent advantage may loom from the confusion generated by speaking of clients (or servers) as if they were hosts, rather than processes: Most spam comes from clients, I want to eliminate it, hence I will only pull mail from servers.<br />
<br />
== Disadvantages ==<br />
*Pull requires that senders and receivers all be connected to the same network all the time, which is a lot closer to true now than it was in the 1980s, but still a long way from universally true.<br />
<br />
*MTA-pull does add a burden to both sides of legitimate transactions: keeping track of message offers that are pending a decision to pull and an actual pull attempt. In order to justify that added burden (in addition to the huge development and deployment costs).<br />
<br />
*Asynchronous pull with positive DSNs is likely to infringe the [http://www.google.com/patents?vid=USPAT6192407 Tumbleweed patent]. It covers the idea of putting a message on a web server and sending the recipient a URL to retrieve the message. Tumbleweed enforces it very aggressively (see [http://en.wikipedia.org/wiki/Tumbleweed_Communications#Patents lawsuits list in Wikipedia].)<br />
<br />
== Proposed standards ==<br />
RFC 2017, ''Definition of the URL MIME External-Body Access-Type'', provides for MUA-pull. It is substantially similar to including an http URL in a message, except that a MUA may download the content automatically. It is to be noted that various MUAs' default settings provide for not downloading external images for privacy concerns.<br />
<br />
RFC 2111, ''Content-ID and Message-ID Uniform Resource Locators'', provides for embedding images in html mail messages, and also describes ''mid'' URLs to refer to external messages.<br />
<br />
RFC 4468, ''Message Submission BURL Extension'', provides for MSA-pull, in the SUBMIT variant of SMTP (''forward without downloading.) It addresses traffic optimization by allowing the MSA to synchronously pull the message's content from IMAP URLs.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Path_validation&diff=1592Path validation2009-07-04T11:30:11Z<p>Johnl: /* Accountability */</p>
<hr />
<div>Techniques that match up the apparent sender of a message with the path via which it arrived.<br />
<br />
* [[Path validation manual techniques]]<br />
<br />
* [[SPF]] (Sender permitted from)<br />
<br />
* [[Sender-ID]]<br />
<br />
* [[SRS]] (Sender Rewriting Scheme)<br />
<br />
= General issues =<br />
<br />
Internet mail is designed to make message delivery independent of the path a message takes. Although much, perhaps most, mail is sent from hosts managed by the same people that manage domains in a message's header or envelope, there are a variety of well known situations where mail can legitimately arrive via other paths. Some of these situations are:<br />
<br />
* Courtesy forwards, e.g. professional organizations and college alumni assocations. They remail messages to a recipient's actual address, so the recipient system sees the message originating at the forwarding site.<br />
<br />
* Users of courtesy forwards. Since courtesy forwards tend to be inbound only, users send mail from their own ISP with the courtesy forward address.<br />
<br />
* Mailing lists. Messages are resent from the list host.<br />
<br />
* Mail-an-article and kiosks. Web sites allow users to send limited amounts of mail manually entering To: and From: addresses.<br />
<br />
* All-in-one web mail: Web mail systems including Yahoo and Google Gmail allow users to collect mail from other mailboxes via POP or IMAP and send mail with those mailboxes as return addresses.<br />
<br />
= Accountability =<br />
The rationale of path validation techniques is to hold a sending client responsible for what it transmits. Providers may terminate spammers' accounts upon receiving complaints, and damages resulting from spam might hypothetically be claimed before courts. Even if resorting to such extreme measures is sporadic, the theoretical possibility to take that course helps to keep control. To this end, various tokens may be considered in order to learn who is the accountable party.<br />
<br />
== IP addresses ==<br />
The immediate client IP address is the one fact about ''every'' MX-driven attempt at message transport that the receiving MTA can know with very high certainty, even when the message is originated by someone who maliciously tries to hide his identity. IP addresses are trackable and ultimately tied to an ISP, even if that carrier is often an upstream one. However, there is no working global mechanism for identifying an accountable party (i.e. one who explicitly ''accepts'' accountability) from an IP address, due largely to the political and historical variations in how IP addresses have been allocated.<br />
<br />
== HELO name ==<br />
The helo name can be verified to match the IP address in various ways. The CSV paradigm is that the operator of a MTA should exercise some responsibility for what is sends. Its HELO string validation states whether that domain exercises responsibility. However, even if ''mta.example.com'' is the true FQDN of the responsible transmitting client, the hostmasters at ''example.com'' can unilaterally delegate ''mta.example.com'' the way they wish, with no obligation whatsoever to run a whois service or otherwise publish data about their delegate's admins, if any, and in general do not take accountability for what a transmitting client in the subdomain sends.<br />
<br />
== Right Hand Side of MAIL FROM ==<br />
This is the mail domain used for SPF checking. For registered domains, whois data exists, even if it's possibly privacy concealed. SPF allows to verify that an IP address is authorized to relay mail on behalf of that mail domain, but says nothing about possible accountability implications.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=User:Ianeiloart&diff=1587User:Ianeiloart2009-07-03T11:59:49Z<p>Johnl: New page: Manager of the SMTP and MSA systems at the University of Sussex.</p>
<hr />
<div>Manager of the SMTP and MSA systems at the University of Sussex.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=User:Jdfalk&diff=1586User:Jdfalk2009-07-02T19:59:14Z<p>Johnl: New page: JD Falk works at Return Path, and has experience at just about every mail provider you ever heard of including Yahoo, Hotmail, Critical Path, and more</p>
<hr />
<div>JD Falk works at Return Path, and has experience at just about every mail provider you ever heard of including Yahoo, Hotmail, Critical Path, and more</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Captchas&diff=1585Captchas2009-06-15T13:12:24Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=ca. 2001<br />
|difficult=Medium<br />
|popular=Medium<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Captchas (from "Completely Automated Public Turing test to tell Computers and Humans Apart") is version of mail challenges, adapted from a method originally designed to protect web forms from being completed by robots.<br />
<br />
When using to protect a mailbox, a confirmation message is sent to the sender when he writes to the protected mailbox for the first time, indicating a web page where he will be asked to enter a textual code hidden in an image. If the correct answer is given, the sender is added to a whitelist so subsequent messages will be accepted without any further action.<br />
<br />
Although some defenders of this method claim it provides "100 % spam stopped without loss of messages", in practice this method has many drawbacks including:<br />
<br />
* All of the general problems with [[Challenges]]<br />
* Not user friendly with people with physical deficiencies - blind people may not be able to correctly decode captchas (see [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
<br />
== References ==<br />
<br />
* [http://en.wikipedia.org/wiki/Captcha Wikipedia]<br />
* [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
* [http://www.w3.org/TR/turingtest/ W3C Inaccessibility of CAPTCHA]<br />
* [http://www2.parc.com/istl/projects/captcha/history.htm PARC history of Captchas]</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Challenges&diff=1584Challenges2009-06-15T13:11:32Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=1990s<br />
|difficult=Medium<br />
|popular=Low<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Challenges or Challenge/Response (C/R) attempts to validate unknown senders by replying to their mail with a challenge message. If the sender responds to the challenge in a way the C/R system approves of, the mail is delivered and the sender is whitelisted. The expected responses may be any return e-mail (typically to a coded address in the Reply-To: header), a return message with a particular key phrase, a clickthrough to a web page, or a clickthrough followed by solving a [[Captchas|Captcha]] problem.<br />
<br />
C/R systems are widely considered to be abusive and ineffective. Problems include:<br />
<br />
* They generate backscatter harming innocents - most spam has forged sender addresses, so most challenges are sent to people who had nothing to do with the message being challenged.<br />
* They don't work with messages sent by anything other than an individual human sender: newsletters, receipts and other transactional messages, and so forth.<br />
* Recipients of challenges to forged mail often confirm them, since that's the easiest way to make the challenge spam go away.<br />
* It can generate loops - if two poorly configured systems challenge each other's mail, an indefinite mail loop can result.<br />
* Delays are introduced, which may be larger if the sender doesn't immediately receive the confirmation request or simply doesn't have access to network for some reason.<br />
* Access control is often managed by senders, not by the mailbox owner. One of consequences is that if a spammer wants to send messages to someone it's enough to send a first message from some email address under his control, in order to confirm the captcha.<br />
* Message loss - many senders do not respond to challenges.<br />
<br />
C/R systems invariably include the ability to whitelist correspondents manually, as a partial solution to message loss problems.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Captchas&diff=1583Captchas2009-06-15T13:11:13Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=ca. 2001<br />
|difficult=Medium<br />
|popular=Medium<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Captchas (from "Completely Automated Public Turing test to tell Computers and Humans Apart") is version of mail challenges, adapted from a method originally designed to protect web forms from being completed by robots.<br />
<br />
When using to protect a mailbox, a confirmation message is sent to the sender when he writes to the protected mailbox for the first time, indicating a web page where he will be asked to enter a textual code hidden in an image. If the correct answer is given, the sender is added to a whitelist so subsequent messages will be accepted without any further action.<br />
<br />
Although some defenders of this method claim it provides "100 % spam stopped without loss of messages", in practice this method has many drawbacks including:<br />
<br />
* All of the general problems with [[Challenges]]<br />
* Not user friendly with people with physical deficiencies - blind people may not be able to correctly decode captchas (see [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
<br />
== References ==<br />
<br />
* [http://en.wikipedia.org/wiki/Captcha Wikipedia]<br />
* [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
* [http://www.w3.org/TR/turingtest/ www.w3.org] Inaccessibility of CAPTCHA</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Captchas&diff=1582Captchas2009-06-15T13:10:11Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=ca. 2001<br />
|difficult=Medium<br />
|popular=Medium<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Captchas (from "Completely Automated Public Turing test to tell Computers and Humans Apart") is version of mail challenges, adapted from a method originally designed to protect web forms from being completed by robots.<br />
<br />
When using to protect a mailbox, a confirmation message is sent to the sender when he writes to the protected mailbox for the first time, indicating a web page where he will be asked to enter a textual code hidden in an image. If the correct answer is given, the sender is added to a whitelist so subsequent messages will be accepted without any further action.<br />
<br />
Although some defenders of this method claim it provides "100 % spam stopped without loss of messages", in practice this method has many drawbacks including:<br />
<br />
* All of the general problems with [[Challenges]]<br />
* Not user friendly with people with physical deficiencies - blind people may not be able to correctly decode captchas (see [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
<br />
<br />
The solution proposed by providers of this anti-spam method to some of these drawbacks is an additional white/black list manually managed by the mailbox owner.<br />
<br />
<br />
== References ==<br />
<br />
* [http://en.wikipedia.org/wiki/Captcha Wikipedia]<br />
* [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
* [http://www.w3.org/TR/turingtest/ www.w3.org] Inaccessibility of CAPTCHA</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Challenges&diff=1581Challenges2009-06-15T13:09:16Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=1990s<br />
|difficult=Medium<br />
|popular=Low<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Challenges or Challenge/Response (C/R) attempts to validate unknown senders by replying to their mail with a challenge message. If the sender responds to the challenge in a way the C/R system approves of, the mail is delivered and the sender is whitelisted. The expected responses may be any return e-mail (typically to a coded address in the Reply-To: header), a return message with a particular key phrase, a clickthrough to a web page, or a clickthrough followed by solving a [[Captchas|Captcha]] problem.<br />
<br />
C/R systems are widely considered to be abusive and ineffective. Problems include:<br />
<br />
* They generate backscatter harming innocents - most spam has forged sender addresses, so most challenges are sent to people who had nothing to do with the message being challenged.<br />
* They don't work with messages sent by anything other than an individual human sender: newsletters, receipts and other transactional messages, and so forth.<br />
* Recipients of challenges to forged mail often confirm them, since that's the easiest way to make the challenge spam go away.<br />
* It can generate loops - if two poorly configured systems challenge each other's mail, an indefinite mail loop can result.<br />
* Delays are introduced, which may be larger if the sender doesn't immediately receive the confirmation request or simply doesn't have access to network for some reason.<br />
* Access control is often managed by senders, not by the mailbox owner. One of consequences is that if a spammer wants to send messages to someone it's enough to send a first message from some email address under his control, in order to confirm the captcha.<br />
* Message loss - many senders do not respond to challenges.</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Captchas&diff=1580Captchas2009-06-15T13:02:30Z<p>Johnl: </p>
<hr />
<div>{{ast<br />
|date=2<br />
|difficult=Medium<br />
|popular=Medium<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Captchas (from "Completely Automated Public Turing test to tell Computers and Humans Apart") is version of mail challenges, adapted from a method originally designed to protect web forms from being completed by robots.<br />
<br />
When using to protect a mailbox, a confirmation message is sent to the sender when he writes to the protected mailbox for the first time, indicating a web page where he will be asked to enter a textual code hidden in an image. If the correct answer is given, the sender is added to a whitelist so subsequent messages will be accepted without any further action.<br />
<br />
Although some defenders of this method claim it provides "100 % spam stopped without loss of messages", in practice this method has many drawbacks including:<br />
<br />
* All of the general problems with [[Challenges]]<br />
* It generates backscatter harming innocents - most of spams have a forged sender address, the supposed sender will receive a confirmation request about a spam sent by someone else.<br />
* It doesn't work with messages sent by anything other than an individual human sender: newsletters, travel tickets, bounce management (VERP, BATV, ...)<br />
* Not user friendly with people with physical deficiencies - blind people may not be able to correctly decode captchas (see [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
* It can generate loops - if two poorly configured systems challenge each other's mail, an indefinite mail loop can result.<br />
* Delays are introduced, which may be larger if the sender doesn't immediately receive the confirmation request or simply doesn't have access to network for some reason (message delivery depends on some external uncontrolled resource).<br />
* Access control managed by senders not by the mailbox owner. One of consequences is that if a spammer wants to send messages to someone it's enough to send a first message from some email address under his control, in order to confirm the captcha.<br />
<br />
* Message loss - some senders don't care to confirm his message when it interests only the recipient.<br />
<br />
The solution proposed by providers of this anti-spam method to some of these drawbacks is an additional white/black list manually managed by the mailbox owner.<br />
<br />
<br />
== References ==<br />
<br />
* [http://en.wikipedia.org/wiki/Captcha Wikipedia]<br />
* [http://www.acb.org/board-minutes/bm070802.html American Council of the Blind]<br />
* [http://www.w3.org/TR/turingtest/ www.w3.org] Inaccessibility of CAPTCHA</div>Johnlhttps://wiki.asrg.sp.am/index.php?title=Challenges&diff=1579Challenges2009-06-15T13:02:22Z<p>Johnl: New page: {{ast |date=1990s |difficult=Medium |popular=Medium |effective=Medium |where=MTA or MUA |harm=High }} Challenges or Challenge/Response (C/R) attempts to validate unknown senders by replyin...</p>
<hr />
<div>{{ast<br />
|date=1990s<br />
|difficult=Medium<br />
|popular=Medium<br />
|effective=Medium<br />
|where=MTA or MUA<br />
|harm=High<br />
}}<br />
Challenges or Challenge/Response (C/R) attempts to validate unknown senders by replying to their mail with a challenge message. If the sender responds to the challenge in a way the C/R system approves of, the mail is delivered and the sender is whitelisted.<br />
<br />
C/R systems are widely considered to be abusive and ineffective. Problems include:</div>Johnl