email library could "recover" from bad mime boundary like (some?) email clients do

[FedeleMantuano]

BPO	27010
Nosy	@warsaw, @bitdancer, @soltysh, @adepasquale
Files	mail mail.json issue27010-notuniqueboundary.patch: NotUniqueBoundaryInMultipartDefect

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-05-12.15:41:58.399>
labels = ['type-feature', '3.7', 'expert-email']
title = 'email library could "recover" from bad mime boundary like (some?) email clients do'
updated_at = <Date 2016-09-21.13:21:01.103>
user = 'https://proxy-ga.blitzz.co/proxy/123456/bugs.python.org/FedeleMantuano'

bugs.python.org fields:

activity = <Date 2016-09-21.13:21:01.103>
actor = 'Fedele Mantuano'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2016-05-12.15:41:58.399>
creator = 'Fedele Mantuano'
dependencies = []
files = ['42830', '42831', '43016']
hgrepos = []
issue_num = 27010
keywords = ['patch']
message_count = 17.0
messages = ['265413', '265416', '265417', '265419', '265420', '266372', '266382', '266438', '266440', '268558', '268909', '271882', '274878', '275012', '277114', '277136', '277140']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'maciej.szulik', 'Fedele Mantuano', 'adepasquale']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://proxy-ga.blitzz.co/proxy/123456/bugs.python.org/issue27010'
versions = ['Python 3.7']

[FedeleMantuano]

We are receiving a lot of mail with attachments not detected from email library.
I also tested Tika parser and it have the same issue:

mail: https://proxy-ga.blitzz.co/proxy/123456/pastebin.com/kSEJnzSa
mail parsed: https://proxy-ga.blitzz.co/proxy/123456/pastebin.com/7HaVPcTq

I can read only these content types:
multipart/mixed
multipart/alternative
text/plain
text/html

there isn't Content-Type: application/zip.

With a normal mail client I can read the attachment.

Where is the issue?

[bitdancer]

When you say the attachment is not detected, what do you mean? What call are you making to the email library that you are expecting to see the attachment in that it is not in? Your 'parsed' pastebin isn't something the library produces, so I assume that's the Tika output.

(By the way, pastbin links are problematic in tracker issues, since they may expire. Please paste directly in to the issue, or attach files to the issue.)

Oh, wait. Looking at the email I think I see the problem:

----------------------------------------
</BODY>
</HTML>

--51a14337d8625bb8ce4a5b1667f--

--51a14337d8625bb8ce4a5b1667f
<attachment content>
----------------------------------------

That line that ends with '--' signals the end of the last MIME part in the message. So by RFC standards the remainder of the message is part of the 'epilogue'. If you check msg.epilogue I think you'll find that it contains the raw text of the remainder of the message.

It is interesting that your email client treats it as an actual attachment. It would be possible to have the email library recognize such out of place mime dividers and register it as an error. I would review a patch for that if someone wants to propose one.

--David

[FedeleMantuano]

Hi David,

I use email library to detect malicious attachments, so:

message = email.message_from_file(open('mail'))
for i in message.walk():
   do somethings

Not detected means that in for loop I can't see these attachments.

The same problem there is with tika parser (now I attached file).

I think that all automatics tools that using email library can't extract and post analyze these mails.

[FedeleMantuano]

I test your hypothesis:

for i in message.walk():
    print i.get_content_type()
    print("#################################################################")
    print i.epilogue

multipart/mixed
#################################################################

--31a14337d8625bb8ce4a5b1667f
Content-Type: application/zip; name="n.41056 0002 02 107413 del 11.05.2016.zip"
Content-Transfer-Encoding: base64
Content-ID: <008601d1ac89$01f7f760$0d00a8c0@D25LND1N>

UEsDBBQAAAAIALNQrEi/ST/WbSsBAABAAgAtAAAAbi40MTA0NiAwMDA0IDAyIDEwNzIwMyBk
ZWwgMTEuMDUuMjAxNi5wZGYuZXhl7FNnjExRGL1vDAZjZ1Zd0YbookeUIIgRYocdjBq9r766
GG2ZeJ7RrZroJXrvYtUhIUqIXhLEYMJisJJhnPPePjt6+CeZLzvn3nfv+c733XPvOjrNFVmE

And for me it's right.

[bitdancer]

I'm going to change the title of this and see if anyone wants to propose a patch. It'll probably end up getting closed as not a bug if no one does for a while, though.

[adepasquale]

Isn't this covered by the following test case?

Lib/test/test_email/test_defect_handling.py:18

[bitdancer]

Yes. The current behavior is not a bug, the question is, do we want to deal with that XXX comment in the test by detecting the duplicate and reconizing the "extra" mime part? The defect detection would remain.

[adepasquale]

How about the following patch? If it's different from what you had in mind, please let me know.

[bitdancer]

Thanks for the patch. I'll take a look at this during the PyCon sprints.

[adepasquale]

Hello,
did you have a chance to look at my patch?

[bitdancer]

Unfortunatley no, things were too busy. I'm hoping to have time to review email patches in the not too distant future, though.

[adepasquale]

Ok thanks, please kindly let me know.