Sunday, November 10, 2024
Saturday, August 3, 2013
Gpop to it
This post will attempt to describe the unique aspects of the POP3 implementation available in Gmail. Gmail was introduced with a very large quota for message storage and, from the beginning, could hold many, many messages. When POP3 access was provided, they needed to deal with the problem of scale. The solutions chosen are straightforward, but unexpected and can be confusing.
RFC1939, which describes the POP3 protocol, recognised the problems that the protocol had with scaling to large mailboxes, and made some specific suggestions for controlling the size. Gmail simulates two of these suggestions, depending how you access the account.
One of the suggestions is to limit the amount of time that messages can be retained on the server. Gmail simulates this with "recent mode". In this mode, Gmail's POP server provides a list of all messages added to the account in the last thirty days. Older messages do not appear in the POP server message list, so they appear, to the POP client, to have been deleted from the server. With recent mode, messages deleted by the client are moved to the Gmail Trash, and messages downloaded but not deleted are unchanged on the server. If the messages are not deleted within thirty days, they remain in the Gmail account, and can no longer be downloaded or deleted by the POP client.
Another suggestion is to delete the messages after they are downloaded, whether the client requests it or not. This is simulated by the default Gmail POP server mode, and is a little more complicated to describe. In this mode, a few hundred of the oldest "new" messages are added to the POP3 mailbox. I like to think of this as a download queue, and it has a limited number of slots. When the POP3 client downloads or deletes messages, they are removed from the queue, and the slots are filled by newer messages for the next session. When they are removed from the queue, the action chosen in Gmail's POP3 settings for "When messages are accessed via POP" is applied. Note that the email client can only delete messages from the queue, not from the Gmail mailbox, unless delete was chosen in the account settings in Gmail.
The next time the client connects, the previously downloaded messages are not listed, so they appear to have been deleted. If the session is lost before the client signs out, or the client does not download the entire message, the messages will remain in the queue until the client deletes them. If there are enough messages in the queue, it can slow down or stop the incoming messages, because there are not enough slots in the queue to add them. For this reason, it is very important that, in this mode, the email client deletes the messages after they are downloaded.
Because messages can only be downloaded once in normal mode, Gmail provides an option to reset POP to "Enable POP for all mail (even mail that's already been downloaded)". Choosing this option allows you to download everything to a new email client. There is no way to enable downloads for selected messages only.
RFC1939, which describes the POP3 protocol, recognised the problems that the protocol had with scaling to large mailboxes, and made some specific suggestions for controlling the size. Gmail simulates two of these suggestions, depending how you access the account.
One of the suggestions is to limit the amount of time that messages can be retained on the server. Gmail simulates this with "recent mode". In this mode, Gmail's POP server provides a list of all messages added to the account in the last thirty days. Older messages do not appear in the POP server message list, so they appear, to the POP client, to have been deleted from the server. With recent mode, messages deleted by the client are moved to the Gmail Trash, and messages downloaded but not deleted are unchanged on the server. If the messages are not deleted within thirty days, they remain in the Gmail account, and can no longer be downloaded or deleted by the POP client.
Another suggestion is to delete the messages after they are downloaded, whether the client requests it or not. This is simulated by the default Gmail POP server mode, and is a little more complicated to describe. In this mode, a few hundred of the oldest "new" messages are added to the POP3 mailbox. I like to think of this as a download queue, and it has a limited number of slots. When the POP3 client downloads or deletes messages, they are removed from the queue, and the slots are filled by newer messages for the next session. When they are removed from the queue, the action chosen in Gmail's POP3 settings for "When messages are accessed via POP" is applied. Note that the email client can only delete messages from the queue, not from the Gmail mailbox, unless delete was chosen in the account settings in Gmail.
The next time the client connects, the previously downloaded messages are not listed, so they appear to have been deleted. If the session is lost before the client signs out, or the client does not download the entire message, the messages will remain in the queue until the client deletes them. If there are enough messages in the queue, it can slow down or stop the incoming messages, because there are not enough slots in the queue to add them. For this reason, it is very important that, in this mode, the email client deletes the messages after they are downloaded.
Because messages can only be downloaded once in normal mode, Gmail provides an option to reset POP to "Enable POP for all mail (even mail that's already been downloaded)". Choosing this option allows you to download everything to a new email client. There is no way to enable downloads for selected messages only.
Hop on POP3
I am going to start this blog with some technical descriptions, because I was asked, and because I think they can be interesting. I will try to keep the descriptions as simple as I can without sacrificing clarity. The first entry will be cover the popular POP3 protocol for receiving mail in an email program.
POP is the Post Office Protocol, and it was designed to work like a real post office box. Incoming mail is placed in the box, and you periodically open the box and remove the mail. I am going describe the simple process, and a typical "leave on server" version, because it helps in understanding what can go wrong, and in understanding the unique features of some implementations.
A basic POP3 session starts with the email program (client) connecting and signing in. The sign-in identifies the user and selects the mailbox. The client will ask for the number and size of the messages, and may ask for each message's unique identifier, if it is available. Each message is downloaded and deleted from the server, and the client signs out. To protect the user from losing messages when the client crashes, if the connection is lost before the client signs out, none of the messages will be deleted, and they will be available in the next session. If the client recorded the unique ids of the messages, they can be deleted without downloading them again, otherwise they will simply be duplicated. (It is better to have two copies than none.)
There are several reasons for leaving messages on the server. The user might want to access the same messages from multiple email programs, or to review the addresses and subject of the messages so that unwanted messages can be deleted without downloading. Some email program also support limiting the size of offline messages, and can download just part of larger messages until they can be reviewed. In any case, this is accomplished by asking the server for the unique ID (UID) of each message and maintaining a list of messages known to be on the server. After deleting or downloading each message, the UID is recorded in the list. (Remember that deleted messages may need to be retained.) In future sessions, the list from the server is compared against the local list. Any messages no longer on the server are removed from the local list, and any messages that are still there, but should have been deleted are deleted again. New messages are identified and processed by the client.
POP3 does not recognise any concept of folders or other organisational features, and it does not contain any commands for controlling the read/unread status of the messages, so email servers that provide both POP3 access and other types of access, like webmail or IMAP, must choose how to handle those details. Most webmail providers only make Inbox messages available through POP3, but some, like Gmail, access everything that is not deleted or Spam.
POP3 is s simple protocol, and is widely supported, but it has some disadvantages. Each email program that accesses the account is an independent mailbox and there is no simple, reliable way to keep them synchronised. There is no required sequence to the UIDs assigned to the messages, so comparing two lists normally starts with sorting, so the process does not scale well to very large mailboxes. It is possible to download part of a message by the number of lines, but it is not possible to download identified parts of a message, like the body or attachments. Also, because it is not synchronised with the server, it is easy for the user to lose track of the messages that were left behind.
POP is the Post Office Protocol, and it was designed to work like a real post office box. Incoming mail is placed in the box, and you periodically open the box and remove the mail. I am going describe the simple process, and a typical "leave on server" version, because it helps in understanding what can go wrong, and in understanding the unique features of some implementations.
A basic POP3 session starts with the email program (client) connecting and signing in. The sign-in identifies the user and selects the mailbox. The client will ask for the number and size of the messages, and may ask for each message's unique identifier, if it is available. Each message is downloaded and deleted from the server, and the client signs out. To protect the user from losing messages when the client crashes, if the connection is lost before the client signs out, none of the messages will be deleted, and they will be available in the next session. If the client recorded the unique ids of the messages, they can be deleted without downloading them again, otherwise they will simply be duplicated. (It is better to have two copies than none.)
There are several reasons for leaving messages on the server. The user might want to access the same messages from multiple email programs, or to review the addresses and subject of the messages so that unwanted messages can be deleted without downloading. Some email program also support limiting the size of offline messages, and can download just part of larger messages until they can be reviewed. In any case, this is accomplished by asking the server for the unique ID (UID) of each message and maintaining a list of messages known to be on the server. After deleting or downloading each message, the UID is recorded in the list. (Remember that deleted messages may need to be retained.) In future sessions, the list from the server is compared against the local list. Any messages no longer on the server are removed from the local list, and any messages that are still there, but should have been deleted are deleted again. New messages are identified and processed by the client.
POP3 does not recognise any concept of folders or other organisational features, and it does not contain any commands for controlling the read/unread status of the messages, so email servers that provide both POP3 access and other types of access, like webmail or IMAP, must choose how to handle those details. Most webmail providers only make Inbox messages available through POP3, but some, like Gmail, access everything that is not deleted or Spam.
POP3 is s simple protocol, and is widely supported, but it has some disadvantages. Each email program that accesses the account is an independent mailbox and there is no simple, reliable way to keep them synchronised. There is no required sequence to the UIDs assigned to the messages, so comparing two lists normally starts with sorting, so the process does not scale well to very large mailboxes. It is possible to download part of a message by the number of lines, but it is not possible to download identified parts of a message, like the body or attachments. Also, because it is not synchronised with the server, it is easy for the user to lose track of the messages that were left behind.
Sunday, July 7, 2013
I can't shut up, so I might was well post
Several people have been trying to get me to write about the things I like to talk about, and I thought it was about time. This blog is intended to hold comments about communicating, connecting and otherwise reaching other people through technology.
"Whispering in the Storm" is meant to evoke the image of trying to be heard in the midst of a lot of noise, and, I think, describes what we encounter on the internet. Wish me luck.
"Whispering in the Storm" is meant to evoke the image of trying to be heard in the midst of a lot of noise, and, I think, describes what we encounter on the internet. Wish me luck.
Subscribe to:
Posts (Atom)