Geoff Chappell - Software Analyst
Outlook from Microsoft Office 2016 may compose emails to spammers as notice that their emails to you were deleted without being read. Outlook creates these emails despite being configured never to send read receipts. Outlook does not show these not-read receipts in the Outbox for you to review or delete.
The same behaviour is observed from at least as far back as Office 2007. For how long has Microsoft been helping the spammers by confirming for them that they have the address of an active account?
The problem is observed when Send/Receive is used to download just the email headers so that you can identify spam at the server and delete it there. Of three known workarounds, the best may be to insert a clumsy extra step: before marking the header for deletion at the server, mark it as read.
Once upon a time good practice for computer security meant mostly to stay away from danger. For instance, since floppy disks can have boot-sector viruses, make it your habit not to start a computer without checking that the floppy-disk drive is empty. In modern times such defence seems not just hopelessly naive but beyond impractical. The internet is dark and full of terrors but only a few diehards would think to deal with this by keeping their computers disconnected from the internet. Instead, a whole industry has risen around the idea that the best defence is not to reduce our exposure by carefully selecting what use we make of great new functionality but is instead that we should commit fully to the software and trust that we’re in safe hands.
This article looks at a particular method for having Outlook download email in a way that you might think protects you by exposing you less. It turns out, however, that Outlook misbehaves. If you use this method, then email can be contrived by its sender to induce Outlook to respond to the sender against your expressed wishes. Leave aside whether a bad actor’s ability to get a response from your computer against your wishes by exploiting a program’s misbehaviour is a security flaw. It certainly is a privacy violation.
In other circumstances, Microsoft does have Outlook care about your privacy. For instance, Outlook will warn you if replying to a message would download content from another server which might in turn “inform the sender, who could be a spammer, that your e-mail address is valid and that you opened the message.” For the circumstance of this article, however, Microsoft has been completely and knowingly unwilling to attend to the implication for privacy. Instead, and I kid you not, Microsoft’s view is that you do it yourself by operating Outlook in a way that doesn’t trust Microsoft enough.
Another old-fashioned perspective is that security and privacy go hand in hand. Two fundamental principles of secure coding—ask for no more than you need and expose nothing that doesn’t have to be—are just basic habits of private people (and, some would say, of good citizens). Now an industry that hungers for our personal data encourages us to prepare even our private thoughts in the cloud and to trust that these things we would once have kept to ourselves will be kept securely by others. The inevitability that things go wrong with this is no longer seen as reason not to do it or even just to be careful about it, but is instead turned into a marketing opportunity: trust us rather some other provider.
Being more than a little guarded with my trust and anyway leaning very much towards the diehards, I have a computer that would, if I had my way, connect to the internet only for the very limited purpose of email (and an occasional upload to this website). Microsoft, through Windows 10, has very different ideas. As part of their mission to run my computer as if it’s theirs, they are forever grumbling about how their attempts at providing me with good service are frustrated by having no internet connection—not that their service looks like it would be anything but astonishingly poor even if I did keep the computer connected for them. But back to email…
From an old-fashioned perspective, a first defence for the particular threat that emails pose for security is to start by downloading just the headers. If an email looks suspicious just from its header, delete it at the server. Then, since nothing in its body—no HTML, no rich text, and especially no attachment—ever makes it to your computer, none of it can be the slightest bit capable of causing any harm.
This defence has been explicitly supported by Outlook, from the Microsoft Office suite, for decades. In all my long memory of using different versions of this software, the feature is named Send/Receive. Outlook from Office 2016 has not just a Send/Receive tab on the ribbon but a Send/Receive group on the Home tab, which presumably there would not be if Microsoft regarded the feature as obscure or unimportant. Older versions have a large Send/Receive button on the Standard toolbar and a Send/Receive menu that expands from Tools (and they also have a Help menu that provides ample description even while you’re not connected to the internet, but that’s another story).
In all versions, the main way to configure the feature is through a dialog box named Send/Receive Groups, which can be reached through the menu entry for Define Send/Receive Groups and more quickly through the Ctrl+Alt+S shortcut. From this dialog, you can organise your email accounts into groups or just stick with the All Accounts group that’s created for you. Editing a group gives you a Send/Receive Settings dialog. For each suitable account you can then specify that when Outlook gets new email for this account through this group it should “Download headers only”.
How this works in practice once you’ve set it up is that getting email from a POP3 account at an email server now takes two exchanges. You can set these to be done automatically and less visibly, but let’s go slower and direct the process through the Send/Receive button or its F9 shortcut. A first click on Send/Receive sends emails from your Outbox but receives just the headers of whatever emails are waiting for you at your server. You then mark each header with your choice of what to do with the whole message:
You can take your time over this and even do it while disconnected. When you’re done with reviewing the headers (and have reconnected), a second click on Send/Receive processes your choices. You have then downloaded only the messages that you explicitly wanted. Those that you explicitly did not want are gone. Your only exposure to each such unwanted email was to its relatively tiny header. Though you might have trusted Outlook with the message body, you’ve avoided having to.
To the Outlook user who’s conscious of either privacy or security, downloading just the headers and deleting unwanted emails at the server is the obviously superior way to isolate yourself from spam. Or would be, if it worked the way that Microsoft says it works.
A spammer sends you an email. By using Send/Receive as above, you delete this email at your server instead of downloading it to your computer. So far, so good, but when you next do a Send/Receive, whether to check for new email or to send some, you see that Outlook sends more emails than are in your Outbox.
This problem is most noticeable, of course, when you know your Outbox is empty. As if it’s not disturbing enough to see Outlook sending emails from an empty Outbox, another way you may notice the problem is that you start receiving notifications of delivery failures for emails that you supposedly sent. You’re sure, of course, that you never did send them. Perhaps you worry for a time that your computer has been infected by malware. What actually happens, which you might reasonably think is not much different, is that Outlook sent emails on your behalf.
The problem starts with a spammer’s email that asks for a read receipt. The message that Outlook sends without having shown it in your Outbox is Outlook telling the spammer that their message to you “was deleted without being read”. Literally.
You can see this for yourself by playing the role of spammer and sending yourself a message that requests a read receipt. A first Send/Receive sends the message. A second gets you the header as if from a spammer. Mark it for deletion, just as you would for real spam. Click on Send/Receive and your simulated spam is gone, again just as you would want for real spam, but click on Send/Receive some more times and your Inbox gets the header for Outlook’s not-read receipt. Mark this for downloading, then click on Send/Receive once more and you get to see what Outlook would otherwise have sent to a spammer:
From: Geoff Chappell [mailto:recipient] Sent: Sunday, January 27, 2019 11:10 To: 'Geoff Chappell' <sender> Subject: Not read: subject Your message To: recipient Subject: subject Sent: 01/27/19 11:08 was deleted without being read on 01/27/19 11:10.
Any sender of email to you can request a receipt. You, however, do not have to cooperate. Outlook helps with settings under Tracking on the Mail tab of the Outlook Options dialog (or, in older versions, in a Tracking Options dialog box reached from a button under “Message handling” in an E-mail Options dialog box, in turn reached from a button under E-mail in the Options dialog box). On both new versions and old, you can choose between:
Asking is the default and is useful for alerting you to which of your correspondents, even the welcome ones, ask you for read receipts. You can also, of course, tell Outlook to “Always send a read receipt”, but the point to this article is that you tell Outlook not to do this.
As an aside, if you ever do want to send a read receipt, then you may want to know that clicking Yes when asked is not quite the same as having told Outlook to send always. A sufficiently competent recipient of your receipt can tell which. This is possible because the receipt, as Outlook sends it, has a MIME part with a field named Disposition that contains “manual-action/MDN-sent-automatically” ordinarily but “manual-action/MDN-sent-manually” if you click Yes, thus distinguishing that the receipt was sent with your explicit approval. For more on this—in general, not specifically for Outlook—see RFC8098.
A sender who gets a receipt has confirmation that their message got through, and even that the message was read or not, but the sender who gets no receipt learns nothing. As a privacy-minded user of Outlook, you will naturally prefer not to send read receipts, certainly not to strangers and possibly not ever. Yet here, despite your telling Outlook never to send read receipts, your price for deleting the spammer’s email at your earliest chance is that Outlook sends the spammer a not-read receipt. How very helpful. To the spammer. Not to you.
Three ways are known to defeat Outlook’s surprise generation of a not-read receipt for a message that you delete just as a header:
To download the whole message is less a work-around than a give-in. Its one merit is that you now use Outlook in what Microsoft perhaps expects as the ordinary way or even as the preferred way. All other things being equal, if you get emails that you’ll most likely just delete, then downloading just the headers gives the software less opportunity to go wrong. Not only do you feed the software less data from an untrusted external source but you avoid giving it that part of the data that has by far the greater known potential for danger. But all things are not equal. If most users download whole messages, not just headers, then the handling of whole messages will inevitably get more attention in the software’s continuing development and testing. Commit to the software more fully, and in exchange for trusting it more than you should need to, you do get better behaviour from it because you execute it in ways that have been better thought through and better tested.
Depending on how you delete the downloaded message, what seems to be deletion may just move the message to a folder for Deleted Items, pending its true deletion. You can go straight to true deletion by pressing Shift-Delete instead of just Delete or indeed by holding down the Shift key while clicking on any Delete button or menu item. You also get true deletion if the message you delete is already in the Deleted Items folder. Deletion from the Deleted Items folder can also be done for you automatically as Outlook exits, if you configure it to. No matter how the whole message sooner or later gets deleted for real, if it has not yet been read (or at least marked as read) and you have the read-receipt option set to “Ask each time”, you will be told that the sender “requested a notification be sent when this message is deleted” and you will finally have your say on whether Outlook plays along. And if you have the read-receipt option set to “Never send”, then Outlook will indeed not send.
See that this workaround has a not insubstantial price. By downloading the whole message, just to keep your privacy in case it asks for a read receipt, you potentially add a lot to your bandwidth. Not only that, but unless you make a point of truly deleting the message immediately, you also keep it on your computer for possibly a very long time. For as long as the message remains on your computer, you trust both yourself and Outlook never to mishandle it and especially never to read the message body in any way that exposes you to harm. Even if Outlook is safe to trust for this, do not miss the irony of trusting Outlook for security when it demonstrably is not trustable for privacy.
The other known workarounds allow you to keep to the principle of reducing risk by trying not to invite it. You still delete the spammer’s email at the server for the security of knowing that the message body never gets to your computer, but to keep your privacy you insert one or another inconvenient and arguably counter-intuitive step when you mark the header for deletion.
To see what might work as a different way to mark a header for deletion in the hope that Outlook will respond correctly regarding read receipts, look back at how Outlook handles the generation of read receipts when deleting a whole message. What happens then is just Outlook behaving as it should. Read receipts are a well-defined feature and have explicit support subject to the recipient’s control. At the first sign that a message is read, including just if it is marked as read, Outlook has every proper reason to send a read receipt if one was requested and the recipient approves. If a message hasn’t yet been read and all chance of its ever being read is about to vanish, then Outlook has every proper reason to send a not-read receipt if one was requested and the recipient approves. The workarounds play to two conditions that invoke this proper handling:
If you mark the header as read, Outlook acts on your preference there and then, just as if you marked a whole message as read. If you have configured Outlook to “Ask each time” about read receipts, you will be told that the sender “requested a notification be sent when this message is read” and you can say yes or no to cooperating. And if you configured Outlook to “Never send”, then it will respect your direction and not prepare a read receipt. Whether you then mark the header for deletion or you download the whole message, the read-receipt question is done and dusted.
Holding down the Shift key while marking a header for deletion induces Outlook to treat the deletion as final, just as if you do this while deleting a whole message. You will typically be warned that “This will be permanently deleted.” Let’s assume that you have turned that off or that you click Yes to continue. If you have configured Outlook to “Ask each time”, you will be told that the sender “requested a notification be sent when this message is deleted” and you can say yes or no to cooperating. And again, if you have configured Outlook to “Never send”, then Outlook will indeed not send. Whether you then Process Marked Headers to delete the message at the server or you change your mind and unmark the header—and even if you then download the message—the read-receipt question is done and dusted.
But make no mistake: though these workarounds are simple, they are clumsy things of a sort that nobody would ever think to do except to compensate for a program’s misbehaviour.
Marking a header as read is at least counter-intuitive: it’s not as if a header is something that Outlook lets you read, so in what sense should Outlook permit you to mark it as read? If you try to read a header, Outlook complains that the “item has not been retrieved from the server” and asks what you want to do with it. In the resulting dialog, marking the header as read is not an option. Where the Send/Receive ribbon’s Server group provides controls for what to do with a message at the server, there is no control for marking a header as read. Though it does show on the context menu for a header (in Outlook 2016 but not in Outlook 2007), this looks to be incidental: see that it duplicates the keyboard shortcut for the earlier and obviously more intended “Mark to Download”. Still, marking the header as read before marking it for deletion is an easy workaround. You can set it up to be as easy as one extra click. In no sense, however, can it plausibly have been anyone’s design that anyone should insert this extra step.
Holding down the Shift key while marking a header for deletion, as if to indicate that you want permanent deletion instead of deletion that can be undone, is worse than counter-intuitive: it’s counter-factual. The deletion that results is not at all permanent, no matter that Outlook may warn you differently. The header remains but with a red X over its Header Status to show that it is merely marked for deletion, and this marking is as easily undone whether you held the Shift key down or not. Not only does this make a liar of Outlook’s warning that the deletion will be permanent but it also makes nonsense of the read receipt’s text: a sender who receives one of Outlook’s notifications that a message “was deleted without being read” can know their message got through to a live account but cannot safely infer that the message actually was deleted. Such inference is anyway not supported by RFC8089, but see that Microsoft has gone to some trouble to aim for the receipt’s text to be true when deleting a whole message. When all Outlook has to work with is a header, we trick it into composing the read receipt earlier than it otherwise would. It then respects our wishes without realising that the message is not truly being deleted. That we can do this is plainly not by design.
That Microsoft does not—or will not on reflection—regard Outlook’s surprise generation of a not-read receipt as a serious breach of privacy, if not also of security, is just not credible. Presumably, this highly unwelcome behaviour from Outlook happens only by oversight and will get fixed. Indeed, since I think I have better things to do than keep downloading and installing Office updates for a computer that connects to the internet for only a few minutes each day, there’s even a possibility that Microsoft has fixed this problem already and I just don’t know yet.
Against this possibility that the problem is already known to Microsoft and is already fixed must be placed that a fix might understandably sit for a while in a too-hard basket. Users are certainly due a fix, but some forbearance is in order. If the general design of read receipts is to generate one at the first indication that the message is read but the last of its being deleted, then the trigger moment for a read receipt in response to deleting a header does not occur when the header is marked for deletion but when a subsequent Send/Receive processes this header so that the message is deleted at the server and the header is deleted locally. This is trickier for Outlook than is the final deletion of a whole message.
When a whole message is deleted, OUTLOOK.EXE itself responds to the user-interface input and sees that deletion is wanted. If this is a final deletion of a message that has not yet been marked as read and is still marked as wanting a read receipt, then it is OUTLOOK.EXE that considers whether the recipient approves, including to ask through the user interface. If the user does not approve, then the flags for requesting a read receipt are cleared from the message (by two calls to the IMessage::SetReadFlag method). As the deletion works its way through OLMAPI32.DLL and MSPST32.DLL (typically), the latter’s handling of an IMAPIFolder::DeleteMessages method has no cause to call a IMAPISupport::ReadReceipt method and so no read receipt is composed.
The essence of Outlook’s mishandling of read receipts for a downloaded header is that final deletion occurs at a different time and is overlooked. The proper time for considering whether a read receipt should be sent is not when the header is marked for deletion but when the mark is processed. Because this is missed, if the flags for requesting a read receipt haven’t been cleared through some intervention such as the workarounds presented above, the DLLs do what they’re told, DeleteMessages calls ReadReceipt, and a read receipt is composed for sending whether the user would have wanted this or not.
The difficulty for Outlook is in the contrast. For final deletion of a whole message, certainly when invoked by holding down the Shift key in combination with the Delete key or with some user-interface control such as a Delete button, a user is present to answer a prompt. It also looks like final deletion of a whole message can always be done off-line. In contrast, processing a header for deletion requires a connection with the server. This connection may be metered and the Send/Receive may anyway have been automated. There might easily be no user to ask. I, for one, would not want to be a Microsoft programmer who has to sort out what to do about this!
With that summary of the fix’s difficulty, sympathy with Microsoft and allowances for the human frailty of its programmers must end. Though it’s only recently that I happen to have noticed Outlook sending hidden emails, Outlook all too plausibly has been doing this to me—and to more than a few others—for very many years. Though my first conscious experience of the problem was for Outlook from Office 2016, once I knew enough about the problem to know what to search for, I soon learnt from the internet that problems with read receipts have a very long history. Many of the reported problems are not exactly the same as what I present above, but Outlook: How do I disable “NOT READ” receipt sent back to spammers? seems to be and is from as long ago as March 2007. Sure enough, my steps above for reproducing the problem work also on an old installation from Office 2007.
Microsoft cannot, of course, be expected to have known of a report on an obscure forum (where the report anyway attracted no reply). But now consider the long discussion of an Outlook 2007 IMAP Bug at one of Microsoft’s own websites. It started in December 2007 and continued for several years. As with any long thread contributed to by users with widely ranging competence and intention, this is not one that offers much to rely on. But two things are clear and relevant. First, there are several, even many, different experiences, evidently of different problems each with a different cause, some of which may have got addressed in the years since, but all involve the surprise of finding that Outlook sometimes sends not-read receipts. One observes explicitly that downloading just headers seems to be a factor. Others suggest that problems reach back at least to Outlook 2003. Second—and sadly much too typical of Microsoft’s so-called community support—a response by someone who supposedly represents Microsoft has been marked as the answer even though it obviously goes nowhere near to providing a solution. That the “issue has been reproduced and has been filed as a bug” is progress of a sort but in no honest classification is it any sort of solution. An update “when this issue is fixed” might have been.
So, how about it, Microsoft? Is a decade long enough to fix a problem that you reproduce and file as a bug? Or is your attitude that if you can wait out a decade, then the problem doesn’t need to be fixed?
I can’t help wonder if problems with read receipts could have persisted for so long if the receipts themselves were more openly displayed. That they do not show in the Outbox looks to me like begging for trouble to go unnoticed.
Yet hiding them is plainly by design. See above that I provide links to Microsoft’s documentation of a few relevant MAPI methods. It’s not my usual practice to do so. I usually take Microsoft’s documentation as well-known to this site’s expected readership and as being largely unnecessary to refer to explicitly unless it’s erroneous. Here, I cite the documentation to highlight something it tells of how Microsoft disregards its users. At the end of the page for the ReadReceipt method is the following advice for what callers of the method might do with any receipt that the message produces:
You can either hide or display read and nonread reports generated by stores in your folders. Storing read and nonread reports in hidden folders enables you to implement tighter security.
My take on this is that Microsoft’s interpretation of security for read receipts is about ensuring delivery to the sender. Perhaps that’s fair enough in the context. Given that the recipient has approved the sending of a receipt, Outlook’s responsibility is no longer to the recipient but to the integrity of the receipt. The recipient had their chance in general through the Tracking Options and specifically through a prompt. Well, except when they didn’t. Then, Outlook doesn’t look to be tightening security but abusing it.
For the record, my first observation of this problem is with Outlook from Microsoft Office 2016. In my installation, OUTLOOK.EXE is version 16.0.6741.2048 and was signed by Microsoft on 13th June 2016. The problem may be fixed in some later build. How long the problem has been present in Outlook is not yet known. It is seen in Outlook from Office 2007, which I happen to have retained on an old computer. Its version of OUTLOOK.EXE is 12.0.4518.1014, signed by Microsoft on 27th October 2006.