As 2011 comes to an end, I have reflected on many of the blogs, rants, opinions, and post-comments I have thrown out to the Internet this past year. There is, as expected, a common theme and it really does come down to "Save the [Jimmy] Wales!"
For those who don't recognize the name, Jimmy Wales is one of the co-founders of Wikipedia (arguably "The" founder of Wikipedia). It is not that Jimmy is in any kind of trouble, not at all. In fact Wikipedia has grown to become one of the go-to sources of information on the planet. It is wildly popular, usually accurate, and most importantly ... free.
Yes there have been issues with conflicts, inaccuracies, security.. big deal, I can say the same about NASA. Wikipedia is a volunteer driven project funded by a Not For Profit organization that has a goal of bringing free information to the masses. - Awesome.
THAT is what my message has been about and will continue to be about. Be Awesome. Feed the masses. Share the wealth. Wikipedia has done just that and they saved me from dropping $900 on paper encyclopedias that were out-dated before they were printed. If you feel like donating to the cause (as I have done) then you can do so HERE.
However, this is not just an appeal to help fund Wikipedia, it is about ALL open source, freeware, feed-the-masses projects. Jimmy was just an example, and his last name made for a cool title :) There are tons of open source projects in Software, hardware, social engineering, political restructuring, etc, etc you just need to find them and fund them. The Internet is a pure democracy that allows us to vote with clicks and opens and likes and hits... and cash. If you use an open source service on the Internet or have a free program you downloaded from the Internet and you really like it, then please consider sending a donation to support the author. That makes good programs better and feeds the awesomeness engine.
I wrote my first piece of free-ware in 1981 and have been a strong proponent of the concept ever since. I also write commercial software and have no conflict whatsoever selling a thousand-dollar piece of code in the morning, then releasing a different piece of share-ware to the Internet in the evening. This is the new economy, the new political engine, the new audience that will drive the socio-economic reality of the coming year. Embrace it.
Be Awesome - Change the world.
Saturday, December 31, 2011
Saturday, December 24, 2011
Christmas Eve
A time for most Christians around the world to celebrate the life that was given to Jesus Christ 20 centuries ago. A time for people to gather with friends and family, to remember what is really important in life. Jesus Christ walked this earth for 33 years and spent much of that time teaching. His legacy has been translated, into every language on the planet, interpreted and misinterpreted, used and abused, praised and vilified, but has survived none the less.
We light trees and houses to symbolize the bringing of light into the world at the darkest point of the year. We give presents to show our unconditional love and gratitude to the ones we care about. We gather with friends and loved ones to share a meal and song and good times, all to celebrate the life we have knitted together.
To all my friends and family, colleagues and clients, I wish you a very
Merry Christmas and all the best success in the new year.
Thursday, December 22, 2011
email 200 - How NOT to be a Spammer
Today's blog is a follow up to email 110 - DNS - the Domain Name System
DNS (again)
Here I go harping on that DNS thing again. I can't express enough how important DNS to the high volume sender. While just understanding how it works is enough for anyone who sends and receives a few emails a day, the high volume sender needs to understand it completely. There are many things you can do with DNS that will help you be more efficient sending mail. Some of those follow here.
DKIM
Domain Keys Identified Mail is a system of "signing" a message with a fingerprint that can only be connected to you. If a receiver get's your email and it has a DKIM signature, they can look that up in DNS and use it to verify that it was really sent by you. If it was not, then they have the option of rejecting the mail or marking it as possible spam. The way this works is with a pair of cryptographic keys - one is private and the other public. The private key is used to encrypt special information from your server right into the email headers (from part 1). The public key is published in DNS so when a receiver gets your mail, they can use your public key from DNS, to decrypt the signature and verify that the mail really did come from your server.
In DNS, the DKIM entry will look something like this:
d1024._domainkey.example.com. 86400 IN TXT "v=dkim1; k=rsa; h=sha1; t=y; p=MHxwu28eddw…QaB"
When someone at example.com sends a DKIM signed email, a receiver like Yahoo will look to DNS for a "_domainkey" entry in example.com's DNS file. The public key in the "p=" part will be used to decrypt the signature. If it is successful, the mail is marked as "passed".
FBL
Feed Back Loops are best represented by the "This Is Spam" button Yahoo, Hotmail and AOL mail browsers have. A responsible high volume sender can apply for FBL status with many of the larger ISPs but they need to use a different domain name and reporting address. When an end user clicks the "This Is Spam" button, the ISP will generate a new email and send it to the domain listed in DNS. This is actually a special type of mail know as "ARF" or Abuse Reporting Format and is often processed automatically by an abuse reporting system in the sender's network. An FBL mail is NOT a bounce, it is a new email that is generated in response to an end user manually pressing a "This is Spam" button in their email reader. For this reason, you cannot normally rely on getting all of the original message back. Many ISPs will redact any personally identifying information including the email address form the returned email. A bulk sender participating an FBL program will have to use some intelligent processing to manage FBL responses.
SPF
Sender Policy Framework is similar to DKIM in the sense that an ISP can use it to validate the source of the mail. It does not use any kind of cryptographic signing, but rather lists all of the IP addresses that are associated with a mail host (MX) name. This way, a receiver can check in DNS when a mail is received and look up to see if the IP address that sent the email matches the IP addresses listed in DNS for the mail that was sent. If not, it was most likely spam and could potentially be discarded or marked as spam.
SenderID
SenderID is almost synonymous with SPF, but is a slightly different version - you can never have too much security.
WhiteListing
This is a process of simply being a good neighbour in the high volume sending community. When you request whitelisting from an ISP like Yahoo for example, you have to provide proof that you have done all the right things in DNS like SPF and DKIM. You also have to prove that you are a legitimate sender and not a spammer. When they are satisfied that you are one of the good guys, they will list your mail server IPs as "friendly" and will let mail pass though with less security checking - you become a "trusted traveler". This is not necessarily permanent and one offensive mailing can remove you from the list.
Content prescreening with Spamassassin
When an end user clicks on "This is Spam" it is an instant mark against you in the reputation battle. Some ISPs use content scanning and logic to make decisions about what may be obvious spam. These are both very good reasons for content scanning your own messages before sending. This can be done relatively easily with Spamassassin and you will get a score with detail to give you an idea of ways to improve. While no ISP actually uses Spamassassin, they all used some form of ranking that is similar. Odds are that if Spamassassin gives you a score of 10 or more, the ISP will mark it as spam and you mail may never see the inbox.
Double Opt-In vs Purchased Lists
One big factor that can really affect your reputation as a sender is the quality of the email address list you send to. While it is tempting to purchase email address lists for sending to, this can be a dangerous practice. There are some legitimate list vendors out there, but there are also those who will sell you bad lists filled with invalid addresses, spam straps (addresses that are known to be bad and watched by ISPs, and forbidden addresses. Sending mail to any of those can hurt or kill your reputation. One way to avoid that is to double opt-in your own list. First allow people to submit their email address to a web form if they want your information. Then send them a short email asking them to click a verification link if they really did ask for the information - ala double opt-in.
ESPs
One great option for people who have a lot of mail to send, but don't want to manage their own mail servers, is an ESP (Email Service Provider). ESPs can often provide safe distribution lists, FBL management, reputation management, and cost-per-message sending fees that are competitive with running your own systems at lower volumes. There are many reputable ESPs who will provide full service including whitelisting, content checking and reputation consulting. These ESPs often have good relationships with the major ISPs and can help build senders (who are not spammers) a good relationship with them.
Next time we get to the good stuff - advanced magic
DNS (again)
Here I go harping on that DNS thing again. I can't express enough how important DNS to the high volume sender. While just understanding how it works is enough for anyone who sends and receives a few emails a day, the high volume sender needs to understand it completely. There are many things you can do with DNS that will help you be more efficient sending mail. Some of those follow here.
DKIM
Domain Keys Identified Mail is a system of "signing" a message with a fingerprint that can only be connected to you. If a receiver get's your email and it has a DKIM signature, they can look that up in DNS and use it to verify that it was really sent by you. If it was not, then they have the option of rejecting the mail or marking it as possible spam. The way this works is with a pair of cryptographic keys - one is private and the other public. The private key is used to encrypt special information from your server right into the email headers (from part 1). The public key is published in DNS so when a receiver gets your mail, they can use your public key from DNS, to decrypt the signature and verify that the mail really did come from your server.
In DNS, the DKIM entry will look something like this:
d1024._domainkey.example.com. 86400 IN TXT "v=dkim1; k=rsa; h=sha1; t=y; p=MHxwu28eddw…QaB"
When someone at example.com sends a DKIM signed email, a receiver like Yahoo will look to DNS for a "_domainkey" entry in example.com's DNS file. The public key in the "p=" part will be used to decrypt the signature. If it is successful, the mail is marked as "passed".
FBL
Feed Back Loops are best represented by the "This Is Spam" button Yahoo, Hotmail and AOL mail browsers have. A responsible high volume sender can apply for FBL status with many of the larger ISPs but they need to use a different domain name and reporting address. When an end user clicks the "This Is Spam" button, the ISP will generate a new email and send it to the domain listed in DNS. This is actually a special type of mail know as "ARF" or Abuse Reporting Format and is often processed automatically by an abuse reporting system in the sender's network. An FBL mail is NOT a bounce, it is a new email that is generated in response to an end user manually pressing a "This is Spam" button in their email reader. For this reason, you cannot normally rely on getting all of the original message back. Many ISPs will redact any personally identifying information including the email address form the returned email. A bulk sender participating an FBL program will have to use some intelligent processing to manage FBL responses.
SPF
Sender Policy Framework is similar to DKIM in the sense that an ISP can use it to validate the source of the mail. It does not use any kind of cryptographic signing, but rather lists all of the IP addresses that are associated with a mail host (MX) name. This way, a receiver can check in DNS when a mail is received and look up to see if the IP address that sent the email matches the IP addresses listed in DNS for the mail that was sent. If not, it was most likely spam and could potentially be discarded or marked as spam.
SenderID
SenderID is almost synonymous with SPF, but is a slightly different version - you can never have too much security.
WhiteListing
This is a process of simply being a good neighbour in the high volume sending community. When you request whitelisting from an ISP like Yahoo for example, you have to provide proof that you have done all the right things in DNS like SPF and DKIM. You also have to prove that you are a legitimate sender and not a spammer. When they are satisfied that you are one of the good guys, they will list your mail server IPs as "friendly" and will let mail pass though with less security checking - you become a "trusted traveler". This is not necessarily permanent and one offensive mailing can remove you from the list.
Content prescreening with Spamassassin
When an end user clicks on "This is Spam" it is an instant mark against you in the reputation battle. Some ISPs use content scanning and logic to make decisions about what may be obvious spam. These are both very good reasons for content scanning your own messages before sending. This can be done relatively easily with Spamassassin and you will get a score with detail to give you an idea of ways to improve. While no ISP actually uses Spamassassin, they all used some form of ranking that is similar. Odds are that if Spamassassin gives you a score of 10 or more, the ISP will mark it as spam and you mail may never see the inbox.
Double Opt-In vs Purchased Lists
One big factor that can really affect your reputation as a sender is the quality of the email address list you send to. While it is tempting to purchase email address lists for sending to, this can be a dangerous practice. There are some legitimate list vendors out there, but there are also those who will sell you bad lists filled with invalid addresses, spam straps (addresses that are known to be bad and watched by ISPs, and forbidden addresses. Sending mail to any of those can hurt or kill your reputation. One way to avoid that is to double opt-in your own list. First allow people to submit their email address to a web form if they want your information. Then send them a short email asking them to click a verification link if they really did ask for the information - ala double opt-in.
ESPs
One great option for people who have a lot of mail to send, but don't want to manage their own mail servers, is an ESP (Email Service Provider). ESPs can often provide safe distribution lists, FBL management, reputation management, and cost-per-message sending fees that are competitive with running your own systems at lower volumes. There are many reputable ESPs who will provide full service including whitelisting, content checking and reputation consulting. These ESPs often have good relationships with the major ISPs and can help build senders (who are not spammers) a good relationship with them.
Next time we get to the good stuff - advanced magic
Sunday, December 18, 2011
email 110 - DNS - the Domain Name System
Today's blog is a follow up to email 101 - The Basics
This is a subject often overlooked by those who don't understand the mechanics of the email system but it is critical for proper email transport. The reason it is overlooked is that it is really considered part of the TCP/IP network that forms the underlying infrastructure of the Internet.
DNS (the Domain Name System) was developed to help humans locate servers more easily. Computing devices - Servers, PC's, and other devices identify themselves by IP address. For instance, the server this document will be stored on will be a server inside my network with IP 192.168.5.67 and to the public it is known as as 184.71.33.126. remembering numbers like that is difficult for most humans, so DNS keeps a matching "name" for these that we can use. DNS is also smart enough to know that either of those numbers can be called "mairs.ca" so i can just connect to "mairs.ca" and I will find the right server no matter what number is most available.
The email system relies heavily on DNS so that humans don't have to remember all those numbers. Would you rather send an email to john@gmail.com, or john@74.125.115.27 ? More importantly, if that particular server is busy, can you remember the other 5 or 6 IP addresses that are also known as gmail.com? Probably not. This is where DNS really helps out email. When you send that email to john@gmail.com, the mail is attempted for delivery at the first server available. if it is too busy, the next server is tried, and so on, until one is available to take the message, or a special type of bounce message is sent back to say "we are too busy, try again later". That special bounce is called a transient failure and you will usually never see one of those in your POP mail box, but your SMTP MTA sees them all the time. In DNS, we can make several different servers in a network "look" like one mail server to the outside world and we can "weight" them to make sure one is always preferred. When your MTA tries to deliver mail, it looks to DNS first to know where to send the mail to.
The important part of DNS (to email) may look similar to this:
In the example above, sending mail to example.com will actually try to make your MTA communicate with int.mail.server.example.com because it has the lower number (10). That Mail eXchange (MX) may be too busy, so the next one in the list will also accept mail for that same name. In the case of an ISP like Yahoo or Hotmail or Gmail, there may be a dozen of those backup servers listed.
DNS is really a collection of files and look-up tables that are presented by a DNS server. There are many public DNS servers that all work together. Changed information in one DNS server will be passed on to the next DNS server in it's chain and so on until all the DNS servers in the world have been updated. This is how someone in Singapore can send you an email in New York without knowing anything about your servers except the mail domain name. Most people have two or more DNS servers defined in their computer to make sure you get the fastest service. You can tell what your DNS servers are by looking at the properties of your network connection. In there you will see "Primary DNS" and "Secondary DNS" - those are the servers you use to look up this data as well as how to find web pages and other services on the Internet.
DNS can also provide valuable other information to help fight SPAM and unwanted connections. When you sign a message with DKIM (Domain Keys Identified Mail) - add a cryptographic signature to your mail - the receiver will look to DNS to get the public key that matches your domain and will use that to decrypt the signature. If your domain entry does not exist or the public and private keys are not a matching set, then the DKIM check will fail and your mail will likely end up in the trash. A similar thing happens with SPF (Sender Policy Framework) and SenderID where a receiver can look to DNS to see if there is a list of authorized IP addresses that can send mail using a particular domain name.
Feed Back Loops (FBL) rely on DNS as well to have a valid spam reporting domain name available for each registered domain. FBLs are not available for all domains and a bulk sender has to go through an application process to register to use one, but a key ingredient is a valid DNS configuration. In a future session we will discuss DKIM (Domain Keys Identified Mail), FBL (Feed Back Loops), SPF (Sender Policy Framework) and other security components. These all use DNS as a look-up for a receiving MTA to verify that the mail is legitimate.
For high volume senders, setting up the proper DNS entries is critical. ISPs rely heavily on DNS to help determine if mail is real and should be delivered so a misconfiguration in DNS can mean all their email gets dropped or placed in the SPAM folder and is never seen. For the average person it is enough to understand that DNS is used to translate host names into IP addresses.
Next time we cover How NOT to be a Spammer
This is a subject often overlooked by those who don't understand the mechanics of the email system but it is critical for proper email transport. The reason it is overlooked is that it is really considered part of the TCP/IP network that forms the underlying infrastructure of the Internet.
DNS (the Domain Name System) was developed to help humans locate servers more easily. Computing devices - Servers, PC's, and other devices identify themselves by IP address. For instance, the server this document will be stored on will be a server inside my network with IP 192.168.5.67 and to the public it is known as as 184.71.33.126. remembering numbers like that is difficult for most humans, so DNS keeps a matching "name" for these that we can use. DNS is also smart enough to know that either of those numbers can be called "mairs.ca" so i can just connect to "mairs.ca" and I will find the right server no matter what number is most available.
The email system relies heavily on DNS so that humans don't have to remember all those numbers. Would you rather send an email to john@gmail.com, or john@74.125.115.27 ? More importantly, if that particular server is busy, can you remember the other 5 or 6 IP addresses that are also known as gmail.com? Probably not. This is where DNS really helps out email. When you send that email to john@gmail.com, the mail is attempted for delivery at the first server available. if it is too busy, the next server is tried, and so on, until one is available to take the message, or a special type of bounce message is sent back to say "we are too busy, try again later". That special bounce is called a transient failure and you will usually never see one of those in your POP mail box, but your SMTP MTA sees them all the time. In DNS, we can make several different servers in a network "look" like one mail server to the outside world and we can "weight" them to make sure one is always preferred. When your MTA tries to deliver mail, it looks to DNS first to know where to send the mail to.
The important part of DNS (to email) may look similar to this:
| example.com | MX 10 int.mail.server.example.com |
| MX 20 smtp.anotherexample.com |
In the example above, sending mail to example.com will actually try to make your MTA communicate with int.mail.server.example.com because it has the lower number (10). That Mail eXchange (MX) may be too busy, so the next one in the list will also accept mail for that same name. In the case of an ISP like Yahoo or Hotmail or Gmail, there may be a dozen of those backup servers listed.
DNS is really a collection of files and look-up tables that are presented by a DNS server. There are many public DNS servers that all work together. Changed information in one DNS server will be passed on to the next DNS server in it's chain and so on until all the DNS servers in the world have been updated. This is how someone in Singapore can send you an email in New York without knowing anything about your servers except the mail domain name. Most people have two or more DNS servers defined in their computer to make sure you get the fastest service. You can tell what your DNS servers are by looking at the properties of your network connection. In there you will see "Primary DNS" and "Secondary DNS" - those are the servers you use to look up this data as well as how to find web pages and other services on the Internet.
DNS can also provide valuable other information to help fight SPAM and unwanted connections. When you sign a message with DKIM (Domain Keys Identified Mail) - add a cryptographic signature to your mail - the receiver will look to DNS to get the public key that matches your domain and will use that to decrypt the signature. If your domain entry does not exist or the public and private keys are not a matching set, then the DKIM check will fail and your mail will likely end up in the trash. A similar thing happens with SPF (Sender Policy Framework) and SenderID where a receiver can look to DNS to see if there is a list of authorized IP addresses that can send mail using a particular domain name.
Feed Back Loops (FBL) rely on DNS as well to have a valid spam reporting domain name available for each registered domain. FBLs are not available for all domains and a bulk sender has to go through an application process to register to use one, but a key ingredient is a valid DNS configuration. In a future session we will discuss DKIM (Domain Keys Identified Mail), FBL (Feed Back Loops), SPF (Sender Policy Framework) and other security components. These all use DNS as a look-up for a receiving MTA to verify that the mail is legitimate.
For high volume senders, setting up the proper DNS entries is critical. ISPs rely heavily on DNS to help determine if mail is real and should be delivered so a misconfiguration in DNS can mean all their email gets dropped or placed in the SPAM folder and is never seen. For the average person it is enough to understand that DNS is used to translate host names into IP addresses.
Next time we cover How NOT to be a Spammer
Thursday, December 15, 2011
email 101 - The Basics
I work in the field of high volume mail delivery and I get a lot of questions about how email actually works. Actually, more often I am presented with a statement that is inaccurate and have to correct someone who thinks they know how it works. So to save myself from more ad-hoc explanation and to help enlighten anyone who cares, here is a whole series of "papers" on how email works and things you can do with it if you know how.
POP and IMAP for end users
When you sit down to check your mail, you most likely either log into Hotmail or Yahoo or you may open your Zimbra or Outlook mail reader. These are usually (but not always) samples of accessing email with IMAP or POP respectively. Both are systems where a server you or someone you trust manages a set of mailboxes that are accessed for reading by a specific user account. For example, bob@yahoo.com will be a specific directory on a server located at yahoo.com that only "bob" has access to read. Corporate mail systems work the same way, but the mail boxes (folders, directories) will be located on the company's own servers.
When you use a web browser to access your mail, you are most likely using IMAP (Internet Message Access Protocol) and when you use a reader like Zimbra or Outlook, you are likely using POP or Post Office Protocol. Either way, you will have a set of "folders" which represent directories on the server in your user space that you have access to. When you select "leave messages on server" in your settings, all the messages that you download to your computer to read are still kept at the server as well. This means that you can retrieve them with a different computer later if you need to, but also means that you take up more space on the server and most mail services have space restrictions you need to pay attention to. When you run out of space, senders will get a bounce message that your mailbox is full and you will not receive any more mail.
IMAP is the normal mail protocol for ISPs (Internet Service Providers) like Yahoo, Hotmail, Cox, Comcast, Earthlink, Shaw, Telus, Bell, etc, who provide access through a web interface. These also typically allow you to use a mail reader like Outlook to access them and you can often use POP to gather your mail as well. Corporate email is almost always POP but can also be IMAP. The important thing to remember is that POP and IMAP are protocols that are intended to be accessed by a human being who is reading and sending email. There are usually different folder options and view preferences that can be associated with a user account to help you better manage your email.
SMTP - the cargo train
The end user protocols POP and IMAP would be very barren places if there were no way to get email between them. If mary@hotmail.com wants to send an email to bob@yahoo.com, the email is first created in her mail reader or IMAP account, then she presses the SEND button. What happens next is all about SMTP (Simple Mail Transfer Protocol), DNS ( Domain Name System), and MTAs (Mail Transport Agents). WOW - that is a lot of acronyms - get used to it, this is an industry of acronyms.
SMTP is a protocol for delivering messages from one MTA server to another. Taking the example above, when Mary hits the SEND button, the server she is connected to will use a Mail Transfer Agent (MTA) to turn her message into something the SMTP protocol will understand. That way the MTA at Hotmail can pass the message on to the MTA at Yahoo in a form that all MTA's understand. Yahoo can then change the message back into whatever format Bob uses for his mail reader.
SMTP really only has 2 parts - the envelope and the data. The data is then broken into 2 separate parts called the headers and the body. "Headers" and "Body" are familiar to most email users who will recognize the headers as the part where the TO and FROM are and the body is where the actual message is. It is actually a bit more complex that that, but you are on the right track.
The SMTP envelope part has one and only one MAIL FROM part. This is where the message actually originated from and is usually the MTA's address on the internet. It may be incomprehensible and that is ok because a human rarely sees it. Mary's envelope MAIL FROM address may be something like 12992j3hdnji8ej88277@bounce.e140.outbound.hotmail..com which would not be a useful address for a human to send mail to, but makes perfect sense to the MTA at Yahoo receiving the message. Yahoo will use that address to determine if the mail really did come from Hotmail and validate a "return path" in case a bounce message has to be sent back.
Also in the envelope is one or more RCPT TO addresses. These are the addresses of the people who will receive the mail. Again, this is not normally revealed to the receiver but is used by the MTA to route messages. All destination addresses go in this area so if Mary also BCC'd Bob's boss, you would also find a "bobsboss@yahoo.com" in the RCPT TO area even though Bob will never see it in his mail reader. That is the entire envelope space. The next line in what we call the SMTP conversation is DATA and is where the headers and body live.
The DATA section contains the TO, FROM, and SUBJECT lines which everyone is familiar with. These are usually readable in any mail reader but there are other lines that are possible here you may not be aware of and are typically not viewable. This first section - everything before and including the SUBJECT line is called the "Headers" and can be huge. A sender can add items here in a specific format to direct an MTA to do certain things , or allow your mail reader more options. For instance, you can add a REPLY TO header that will allow a different address to be used if anyone replies to your mail. Marketing companies who have passed all the "i'm not a spammer" tests can add specific headers to tell an MTA they are a friendly sender. Google (gmail) for instance allows legitimate bulk senders to use a "Precedence: Bulk" header to indicate non-spam bulk email. Security signatures and unsubscribe processing instructions can also be stored here so your receiving MTA know how to handle the mail under certain conditions.
After the SUBJECT line, comes the body, or what most people think of as the actual 'message'. This can be plain, readable text, or it can follow the MIME standards and be available in sections. MIME (Multipurpose Internet Mail Extensions) is a way for a single message to contain different formats of the same message as well as to allow for attachments and embedded images. When a message is sent in a proper MIME format, the end user can read a plain text version or an HTML version with images attached or in-line all depending on the settings of the receivers local mail reader.
All SMTP conversations (we will get to the 'conversation' bit in a minute) end with a single dot (period) on a single line. When an MTA sees that, it closes that message and calls it complete telling the sending MTA that the message was received in good order. That does not close the connection between the servers however, so that same connection can be used to pass hundred of other messages as well before the connection is closed.
The SMTP conversation is called such because that is really what happens. Two MTA's will 'talk' to each other and perform 'handshaking'. It goes something like this:
Hi, I am hotmail.com can I send you email?
Hi hotmail.com, I am yahoo.com, sure you can send email.
This is from user "mary" on server1.hotmail.com
OK, that sender is allowed
This is going to user "bob" on server.yahoo.com
OK, that recipient exists
Here comes the DATA….
I am waiting...
TO: BOB
FROM: Mary
REPLY TO: mary21@hotmail.com
SUBJECT: This is my subject line
This is a test message
.
OK, I am done
Thanks, I got the message OK
In reality it looks like this:
EHLO yahoo.com
250-yahoo.com says EHLO to hotmail.com
MAIL FROM: mary@hotmail.com
250 MAIL FROM accepted
RCPT TO: bob@yahoo.com
250 RCPT TO accepted
RCPT TO: bobsboss@hotmail.com
250 RCPT TO accepted
DATA
354 continue. finished with "\r\n.\r\n"
FROM: mary@hotmail.com
TO: bob@yahoo.com
X-jobid: This is a special header
Subject: test
This is a test message
.
250 OK 50/00-11937-185D7EE4
quit
221 c1n1 closing connection
BOUNCES - what happens when the conversation fails
The above SMTP conversation is a perfect scenario and is what happens most of the time, but sometimes you run into errors like a bad address or a slow responding MTA. When that happens, the receiving MTA will send back a bounce. Bounces can either be In-Band (synchronous) where they happen during the SMTP conversation, or Out-of-Band (asynchronous) where they happen after the SMTP conversation has completed. If the MAIL FROM is rejected for instance, the receiving MTA will immediately return a 500 error code - often a "relaying denied" error. You may also get 500 code errors when defining a recipient that does not exist in the receiving system. In some cases the SMTP conversation is completed and some time later, the receiving MTA will discover an error (like the recipient not existing) and will return a separate message to the sender, also known as an Out Of Band bounce. In technical terms it is called a DSN (Delivery Status Notification). These are captured by the sending MTA and usually relayed back to the originate of the message.
Next time we tackle DNS - the Domain Name System
POP and IMAP for end users
When you sit down to check your mail, you most likely either log into Hotmail or Yahoo or you may open your Zimbra or Outlook mail reader. These are usually (but not always) samples of accessing email with IMAP or POP respectively. Both are systems where a server you or someone you trust manages a set of mailboxes that are accessed for reading by a specific user account. For example, bob@yahoo.com will be a specific directory on a server located at yahoo.com that only "bob" has access to read. Corporate mail systems work the same way, but the mail boxes (folders, directories) will be located on the company's own servers.
When you use a web browser to access your mail, you are most likely using IMAP (Internet Message Access Protocol) and when you use a reader like Zimbra or Outlook, you are likely using POP or Post Office Protocol. Either way, you will have a set of "folders" which represent directories on the server in your user space that you have access to. When you select "leave messages on server" in your settings, all the messages that you download to your computer to read are still kept at the server as well. This means that you can retrieve them with a different computer later if you need to, but also means that you take up more space on the server and most mail services have space restrictions you need to pay attention to. When you run out of space, senders will get a bounce message that your mailbox is full and you will not receive any more mail.
IMAP is the normal mail protocol for ISPs (Internet Service Providers) like Yahoo, Hotmail, Cox, Comcast, Earthlink, Shaw, Telus, Bell, etc, who provide access through a web interface. These also typically allow you to use a mail reader like Outlook to access them and you can often use POP to gather your mail as well. Corporate email is almost always POP but can also be IMAP. The important thing to remember is that POP and IMAP are protocols that are intended to be accessed by a human being who is reading and sending email. There are usually different folder options and view preferences that can be associated with a user account to help you better manage your email.
SMTP - the cargo train
The end user protocols POP and IMAP would be very barren places if there were no way to get email between them. If mary@hotmail.com wants to send an email to bob@yahoo.com, the email is first created in her mail reader or IMAP account, then she presses the SEND button. What happens next is all about SMTP (Simple Mail Transfer Protocol), DNS ( Domain Name System), and MTAs (Mail Transport Agents). WOW - that is a lot of acronyms - get used to it, this is an industry of acronyms.
SMTP is a protocol for delivering messages from one MTA server to another. Taking the example above, when Mary hits the SEND button, the server she is connected to will use a Mail Transfer Agent (MTA) to turn her message into something the SMTP protocol will understand. That way the MTA at Hotmail can pass the message on to the MTA at Yahoo in a form that all MTA's understand. Yahoo can then change the message back into whatever format Bob uses for his mail reader.
SMTP really only has 2 parts - the envelope and the data. The data is then broken into 2 separate parts called the headers and the body. "Headers" and "Body" are familiar to most email users who will recognize the headers as the part where the TO and FROM are and the body is where the actual message is. It is actually a bit more complex that that, but you are on the right track.
The SMTP envelope part has one and only one MAIL FROM part. This is where the message actually originated from and is usually the MTA's address on the internet. It may be incomprehensible and that is ok because a human rarely sees it. Mary's envelope MAIL FROM address may be something like 12992j3hdnji8ej88277@bounce.e140.outbound.hotmail..com which would not be a useful address for a human to send mail to, but makes perfect sense to the MTA at Yahoo receiving the message. Yahoo will use that address to determine if the mail really did come from Hotmail and validate a "return path" in case a bounce message has to be sent back.
Also in the envelope is one or more RCPT TO addresses. These are the addresses of the people who will receive the mail. Again, this is not normally revealed to the receiver but is used by the MTA to route messages. All destination addresses go in this area so if Mary also BCC'd Bob's boss, you would also find a "bobsboss@yahoo.com" in the RCPT TO area even though Bob will never see it in his mail reader. That is the entire envelope space. The next line in what we call the SMTP conversation is DATA and is where the headers and body live.
The DATA section contains the TO, FROM, and SUBJECT lines which everyone is familiar with. These are usually readable in any mail reader but there are other lines that are possible here you may not be aware of and are typically not viewable. This first section - everything before and including the SUBJECT line is called the "Headers" and can be huge. A sender can add items here in a specific format to direct an MTA to do certain things , or allow your mail reader more options. For instance, you can add a REPLY TO header that will allow a different address to be used if anyone replies to your mail. Marketing companies who have passed all the "i'm not a spammer" tests can add specific headers to tell an MTA they are a friendly sender. Google (gmail) for instance allows legitimate bulk senders to use a "Precedence: Bulk" header to indicate non-spam bulk email. Security signatures and unsubscribe processing instructions can also be stored here so your receiving MTA know how to handle the mail under certain conditions.
After the SUBJECT line, comes the body, or what most people think of as the actual 'message'. This can be plain, readable text, or it can follow the MIME standards and be available in sections. MIME (Multipurpose Internet Mail Extensions) is a way for a single message to contain different formats of the same message as well as to allow for attachments and embedded images. When a message is sent in a proper MIME format, the end user can read a plain text version or an HTML version with images attached or in-line all depending on the settings of the receivers local mail reader.
All SMTP conversations (we will get to the 'conversation' bit in a minute) end with a single dot (period) on a single line. When an MTA sees that, it closes that message and calls it complete telling the sending MTA that the message was received in good order. That does not close the connection between the servers however, so that same connection can be used to pass hundred of other messages as well before the connection is closed.
The SMTP conversation is called such because that is really what happens. Two MTA's will 'talk' to each other and perform 'handshaking'. It goes something like this:
Hi, I am hotmail.com can I send you email?
Hi hotmail.com, I am yahoo.com, sure you can send email.
This is from user "mary" on server1.hotmail.com
OK, that sender is allowed
This is going to user "bob" on server.yahoo.com
OK, that recipient exists
Here comes the DATA….
I am waiting...
TO: BOB
FROM: Mary
REPLY TO: mary21@hotmail.com
SUBJECT: This is my subject line
This is a test message
.
OK, I am done
Thanks, I got the message OK
In reality it looks like this:
EHLO yahoo.com
250-yahoo.com says EHLO to hotmail.com
MAIL FROM: mary@hotmail.com
250 MAIL FROM accepted
RCPT TO: bob@yahoo.com
250 RCPT TO accepted
RCPT TO: bobsboss@hotmail.com
250 RCPT TO accepted
DATA
354 continue. finished with "\r\n.\r\n"
FROM: mary@hotmail.com
TO: bob@yahoo.com
X-jobid: This is a special header
Subject: test
This is a test message
.
250 OK 50/00-11937-185D7EE4
quit
221 c1n1 closing connection
BOUNCES - what happens when the conversation fails
The above SMTP conversation is a perfect scenario and is what happens most of the time, but sometimes you run into errors like a bad address or a slow responding MTA. When that happens, the receiving MTA will send back a bounce. Bounces can either be In-Band (synchronous) where they happen during the SMTP conversation, or Out-of-Band (asynchronous) where they happen after the SMTP conversation has completed. If the MAIL FROM is rejected for instance, the receiving MTA will immediately return a 500 error code - often a "relaying denied" error. You may also get 500 code errors when defining a recipient that does not exist in the receiving system. In some cases the SMTP conversation is completed and some time later, the receiving MTA will discover an error (like the recipient not existing) and will return a separate message to the sender, also known as an Out Of Band bounce. In technical terms it is called a DSN (Delivery Status Notification). These are captured by the sending MTA and usually relayed back to the originate of the message.
Next time we tackle DNS - the Domain Name System
Saturday, December 3, 2011
Occupy This
There is a revolution happening planet-wide - unfortunately the message is getting mangled. I was in Manhattan during the Occupy Wall Street rally on October 5th (have the flyer to prove it) and the message at that time seemed clear - enforce fair taxation on the uber-rich so "they pay their fair share" into to social good. I agree with the core of this argument, but there is a great deal that is being misrepresented by the protesters and misinterpreted by the media. Don't get me wrong, I am not trashing the movement as a whole, there is a real problem here, what I take exception to is misinformation and obfuscation of the truth.
Really, that is what it is all about - truth, fairness, social justice, and global responsibility. I get that. I wish the majority of "occupy" protesters did too. By the time Occupy Calgary happened a few of weeks ago, the message had been perverted into "I want my fair share" - That is different. That is Marxism - and it doesn't work. Here's a newsflash - the fact that you exist does not entitle you to jack shit. Get a job, prove your worth, help the planet by reducing your consumption, be generous with what you have even if all you have is time. Begging for attention and handouts while trashing shared public space is not doing the movement any favours. While there may be a very small percentage of these protesters who actually understand the under pinning of the movement, the vast majority show up because it is what everyone else is doing. Lemmings.
Don't even get me started on Occupy Edmonton. Really? Protesting big oil companies in a city that was founded and is still primarily funded by big oil companies? This is the epitome of the bastardization of the movement. The majority of the protesters are students that are there because their parent could afford to send them to university on money made in the oil industry. Astounding hypocrisy.
I find it particularly amusing that the "Occupy Wall Street" concept - specifically condemning big business for not sharing the wealth - was originally started by the CANADIAN activist movement "Adbusters" in an attempt to effect global social change - awesome idea. The thing is that it was supposed to be about effecting global social change and it was quickly twisted into an "us verses them" conflict. The 1% verses the 99%. The uber-rich verses the impoverished.
What seems to be getting lost is that it is about GLOBAL social change, not fixing the US economy. The 1% you hear referred to is specifically a number representing the US economy which is a capitalist economic experiment gone sadly out of control. On a global scale *I* am in the 1% and there is a very strong probability that 99% of the people reading this are too.
I agree with Occupy Wall Street protesters in the argument that the US government should enact changes to capture some of the top end wealth and have it distributed to productive social causes like funding universities or eliminating extreme poverty. In the same vein I agree with the Occupy Berkley protesters in the concept of reducing student debt to encourage economic growth. All the other "occupy" protests have gone sadly awry, and even those two are missing the important part of the equation that is personal social responsibility.
While the US government may not be collecting tax and forcing the uber-rich to pay into social programs, many of those in the upper echelon do so voluntarily - something that has been glaringly absent from this discussion. While Bill Gates may not be paying millions of dollars to the government tax coffers, the Bill and Melinda Gates foundation has generated over $26 billion (yes that was BILLION) in grants to effect social and economic change globally since its inception in 1994, seeded by their own personal income. While Mark Zuckerberg may not have been forced to pay millions in federal tax last year, he voluntarily donated $100 million to the most needy school district in the US - something the US government itself failed to do despite the fact that it is their responsibility to look after education. The reality is that the majority of these large corporations and individually wealthy people do in fact pay a great deal toward making direct and effective change where it is needed. Can the occupy protesters say the same?
Really now - answer the question. When was the last time you (occupy protester) spent a day on the volunteer side of a food bank operation or soup kitchen? Did you donate 10% of your earnings to help someone in a third world country last year? Did you help out in a local shelter to make sure the homeless in your neighbourhood had a warm bed to sleep in? Don't tell me you are too poor because it takes less then $20 per month to feed, clothe and provide an education for a child in impoverished Burkina Faso, Africa. It costs nothing to donate your time to a homeless shelter. While you are sipping your Starbucks triple shot, skinny, decaf latte and Tweeting your daily protest activities on your iPhone through your ATT or Rogers account, maybe you should think about who is *really* affecting social change.
If you want to occupy something, try occupying a new state of mind. Try thinking about the social responsibility revolution that was behind the original "occupy" protest. In the US it is mostly exemplified by a small percentage of individuals and corporations who have exempted themselves out of paying a fair portion of their income to taxation. On a global scale it is about governments that exploit the local population for cheap labor in return for poor living standards. Its about big global corporations moving jobs to locations that allow them to exploit cheap labor and dangerous work standards. It is also about populations who are unlucky enough to live in areas of the world that have been hit with decades of war, famine, drought and ongoing political stress. If you want to be part of the solution then do something to effect real social change globally.
I work 80 to 90 hours every week to maintain my place in the 1% (globally) and am not afraid or ashamed to say so. On the contrary, I am proud of the fact that my hard work had allowed me to build a nice home and help my son through University. I also use some of that to help a young girl and her family in Africa make ends meet so she can complete her education in a safe environment. The odd goat and cash for school books does not mean a lot in New York or Toronto, but in Burkina Faso, Africa, it is the difference between despair and a life of hope and success. On the other side of the planet, I use some of that hard earned money to help a dozen homeless people in Calgary with a hot meal and a place to sleep 3 or 4 times a year. So when I see dozens (or hundreds) of young people using the "Occupy" movement as an excuse to party, rebel, and generally be useless, I get a little miffed.
The Mayans had it absolutely right. This is a time of change, resurrection, new beginnings. This is not a time of doom and despair - unless you feed the fear monsters. There is definitely upheaval in the forecast - geological, astrophysical, and interpersonal. We are evolving as a species and we need to pay attention to the fact that we exist as one consolidated people of Earth. The time for political and economic separation is past, the future is all about moving forward as a planet of people not a collection of separated castes.
Change is good - embrace it, but understand what you are embracing. Making a political stand about economic fairness in the US… in Edmonton… is just missing the whole point. There *is* a need for social change and we *do* need to challenge our governments, but we need to challenge ourselves first. After all there really is no "them" there is only "us" because "we" are the government and "we" do have the power to change it, but only if we understand the change we are asking for and act on it by not supporting offensive companies and actively supporting those in need directly.
There will be massive political change in the next year in a global sense - of that I have no doubt. We have been stagnant as a race for too long. Europe is already feeling this and North America (the "new world") is not far behind. The "Occupy" movement and the social unrest in the Middle East is merely a symptom, not a cause. Leaders will fail, people will rise and geopolitical fallout will be extreme, but the resulting global society will be stronger for it for an eon. So are you part of the problem or are you part of the solution?
Be awesome. Change the world.
Really, that is what it is all about - truth, fairness, social justice, and global responsibility. I get that. I wish the majority of "occupy" protesters did too. By the time Occupy Calgary happened a few of weeks ago, the message had been perverted into "I want my fair share" - That is different. That is Marxism - and it doesn't work. Here's a newsflash - the fact that you exist does not entitle you to jack shit. Get a job, prove your worth, help the planet by reducing your consumption, be generous with what you have even if all you have is time. Begging for attention and handouts while trashing shared public space is not doing the movement any favours. While there may be a very small percentage of these protesters who actually understand the under pinning of the movement, the vast majority show up because it is what everyone else is doing. Lemmings.
Don't even get me started on Occupy Edmonton. Really? Protesting big oil companies in a city that was founded and is still primarily funded by big oil companies? This is the epitome of the bastardization of the movement. The majority of the protesters are students that are there because their parent could afford to send them to university on money made in the oil industry. Astounding hypocrisy.
I find it particularly amusing that the "Occupy Wall Street" concept - specifically condemning big business for not sharing the wealth - was originally started by the CANADIAN activist movement "Adbusters" in an attempt to effect global social change - awesome idea. The thing is that it was supposed to be about effecting global social change and it was quickly twisted into an "us verses them" conflict. The 1% verses the 99%. The uber-rich verses the impoverished.
What seems to be getting lost is that it is about GLOBAL social change, not fixing the US economy. The 1% you hear referred to is specifically a number representing the US economy which is a capitalist economic experiment gone sadly out of control. On a global scale *I* am in the 1% and there is a very strong probability that 99% of the people reading this are too.
I agree with Occupy Wall Street protesters in the argument that the US government should enact changes to capture some of the top end wealth and have it distributed to productive social causes like funding universities or eliminating extreme poverty. In the same vein I agree with the Occupy Berkley protesters in the concept of reducing student debt to encourage economic growth. All the other "occupy" protests have gone sadly awry, and even those two are missing the important part of the equation that is personal social responsibility.
While the US government may not be collecting tax and forcing the uber-rich to pay into social programs, many of those in the upper echelon do so voluntarily - something that has been glaringly absent from this discussion. While Bill Gates may not be paying millions of dollars to the government tax coffers, the Bill and Melinda Gates foundation has generated over $26 billion (yes that was BILLION) in grants to effect social and economic change globally since its inception in 1994, seeded by their own personal income. While Mark Zuckerberg may not have been forced to pay millions in federal tax last year, he voluntarily donated $100 million to the most needy school district in the US - something the US government itself failed to do despite the fact that it is their responsibility to look after education. The reality is that the majority of these large corporations and individually wealthy people do in fact pay a great deal toward making direct and effective change where it is needed. Can the occupy protesters say the same?
Really now - answer the question. When was the last time you (occupy protester) spent a day on the volunteer side of a food bank operation or soup kitchen? Did you donate 10% of your earnings to help someone in a third world country last year? Did you help out in a local shelter to make sure the homeless in your neighbourhood had a warm bed to sleep in? Don't tell me you are too poor because it takes less then $20 per month to feed, clothe and provide an education for a child in impoverished Burkina Faso, Africa. It costs nothing to donate your time to a homeless shelter. While you are sipping your Starbucks triple shot, skinny, decaf latte and Tweeting your daily protest activities on your iPhone through your ATT or Rogers account, maybe you should think about who is *really* affecting social change.
If you want to occupy something, try occupying a new state of mind. Try thinking about the social responsibility revolution that was behind the original "occupy" protest. In the US it is mostly exemplified by a small percentage of individuals and corporations who have exempted themselves out of paying a fair portion of their income to taxation. On a global scale it is about governments that exploit the local population for cheap labor in return for poor living standards. Its about big global corporations moving jobs to locations that allow them to exploit cheap labor and dangerous work standards. It is also about populations who are unlucky enough to live in areas of the world that have been hit with decades of war, famine, drought and ongoing political stress. If you want to be part of the solution then do something to effect real social change globally.
I work 80 to 90 hours every week to maintain my place in the 1% (globally) and am not afraid or ashamed to say so. On the contrary, I am proud of the fact that my hard work had allowed me to build a nice home and help my son through University. I also use some of that to help a young girl and her family in Africa make ends meet so she can complete her education in a safe environment. The odd goat and cash for school books does not mean a lot in New York or Toronto, but in Burkina Faso, Africa, it is the difference between despair and a life of hope and success. On the other side of the planet, I use some of that hard earned money to help a dozen homeless people in Calgary with a hot meal and a place to sleep 3 or 4 times a year. So when I see dozens (or hundreds) of young people using the "Occupy" movement as an excuse to party, rebel, and generally be useless, I get a little miffed.
The Mayans had it absolutely right. This is a time of change, resurrection, new beginnings. This is not a time of doom and despair - unless you feed the fear monsters. There is definitely upheaval in the forecast - geological, astrophysical, and interpersonal. We are evolving as a species and we need to pay attention to the fact that we exist as one consolidated people of Earth. The time for political and economic separation is past, the future is all about moving forward as a planet of people not a collection of separated castes.
Change is good - embrace it, but understand what you are embracing. Making a political stand about economic fairness in the US… in Edmonton… is just missing the whole point. There *is* a need for social change and we *do* need to challenge our governments, but we need to challenge ourselves first. After all there really is no "them" there is only "us" because "we" are the government and "we" do have the power to change it, but only if we understand the change we are asking for and act on it by not supporting offensive companies and actively supporting those in need directly.
There will be massive political change in the next year in a global sense - of that I have no doubt. We have been stagnant as a race for too long. Europe is already feeling this and North America (the "new world") is not far behind. The "Occupy" movement and the social unrest in the Middle East is merely a symptom, not a cause. Leaders will fail, people will rise and geopolitical fallout will be extreme, but the resulting global society will be stronger for it for an eon. So are you part of the problem or are you part of the solution?
Be awesome. Change the world.
Subscribe to:
Posts (Atom)