Problem

I am working on a project where we have to classify e-mails. For this project I need to extract ALL text from the e-mails and their attachments.

My problem is that some attachments are of the type "application/pkcs7-mime", and I am not sure how to handle those.

What I have tried

import email, base64# Opening messageeml_file = '/path/to/file.eml'message = email.message_from_file(open(eml_file))# Printing content typesfor part in message.walk():    print(part.get_content_type())>multipart/mixed text/plain message/rfc822 application/pkcs7-mime

The part that is giving the problems is "application/pkcs7-mime". Next I try to extract data from the payload.

# Ensuring we got the right payloadmessage.get_payload(1).get_payload(0).get_content_type()>application/pkcs7-mime# Getting payloadmessage.get_payload(1).get_payload(0).get_payload()MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwGggxDhqwSDEOGmQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PV8wMDRfRkVGNjMyNEIyNDFFNDA5MEFBMUQ3RUQ5QUZBQkZFMDdic2Vka187DQogbWljYWxnPXNoYTI1Ng0KDQotLV8wMDRfRkVGNjMyNEIyNDFFNDA5MEFBMUQ3RUQ5QUZBQkZFMDdic2Vka18NCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUlEOiA8Mzg3NTg2Nzk0M0UzNkY0OUEzQjcy.......

It looks like the payload is encoded in base64, so I try to decode it:

# Decoding messagemessage.get_payload(1).get_payload(0).get_payload(decode=True).decode("iso-8859-1")# Top of output á«á¦Content-Type: multipart/mixed; boundary=_004_FEF6324B241E4090AA1D7ED9AFABFE07bsedk_; micalg=sha256--_004_FEF6324B241E4090AA1D7ED9AFABFE07bsedk_Content-Type: text/html; charset=utf-8Content-ID: <3875867943E36F49A3B72D0FAEB1DC35@eurprd01.prod.exchangelabs.com>Content-Transfer-Encoding: base64PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJodHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo# Bottom of output0U 0U00, * (&http://crl.oces.trust2408.com/oces.crl0\ Z X¤V0T10 UDK10U×ÿÑE|g¢¿ºÊ]6¾ãñJfYéÿBû´s²J7ª¡-*¨    Ø p@6¨Ë9ǦíæýÕUB9Íõ £àg&7î!Ö®?sÒ8Wè>º|mIh +|2¨Ü%«O´|%o¬¦¥ð¶oòÝ¿t³ é¾ÏPÒëqûW®ö8|¼èÇçqQL ½þ¹Ù`¶¿*7ÒýsÎKb!ÿ¶?:{¸½Ú\õ{ÕþyÅI.­éÃÂKèj36q°lD}|RÎàIzÙ/^[j;3¶Ðà+®r$¥!1ß0Û0I0A10 UDK10UªÏÒ¶,`§rÌbèñy5&FJÃ(êí°&¬,ÚêX$á.=ïïæN$®]Þ½yU0+*§ÔÚ´í6azL(!DÏÝ6ÂNê,Ä5FsíXEó§î_»SG]Úüåt Ô¼'âröÓg!ðSÐ,O¶x><s5ÖRv«N¸¡¿<ý>¼VBñ¤f[ÔÏàø7¿ÂûÊuidÊUS!ÂÕÜÚæòÜíþµüâeüLü'^[¦/d{«oäp¹ÁNî÷Ž½Oq$Øà-WDsüèXÀÎ}á¾9À̹ÙÎhAÎ ¯P¾ñäK!VIrÏ̯Ín,º~¿IÐ{[D¢=ý%Úîckr¿·_³EÙ]¨g0âk:`vÄÖ</È»HBà4%ª]|;~wÐ=·&;êºÕAr«Á¡GÅòØ)wÂd{Ù    BvÞ·3ºàCÔ

Conclusion

Some of the message is decoded correctly, while the rest is a mess. I can, however, not figure out where things go wrong.

I have written a UDF to decode the endcode base64 text in sap pi mapping. But after decoding the xml tags, open and close tags all are getting replaced by > <

Please find below screenshot of the code.

public String Base64Decoder(String EncodedString, Container container) throws StreamTransformationException{//Variable to hold decoded stringString decodedString = null; byte[] decodedBytes = DatatypeConverter.parseBase64Binary(EncodedString);try{//Convert base64 schema to byte arrayreturn new String(decodedBytes,"UTF-8");//Get the string of decoded byte array}catch(Exception e) {}//Export decoded stringreturn decodedString;}

Output after decoding:

<?xml version="1.0" encoding="UTF-8"?><ns0:Log xmlns:ns0="http://world.com/pi/M/Log"><Data>&lt;OUTPUT&gt;&lt;item&gt;&lt;VBELN&gt;0180001027&lt;/VBELN&gt;&lt;WADAT&gt;00000000&lt;/WADAT&gt;&lt;VSTEL&gt;1000&lt;/VSTEL&gt;&lt;KUNNR/&gt;&lt;ITEM&gt;&lt;YITEM&gt;&lt;MATNR&gt;000000000011000082&lt;/MATNR&gt;&lt;LFIMG&gt;1.0&lt;/LFIMG&gt;&lt;MEINS&gt;L&lt;/MEINS&gt;&lt;/YITEM&gt;&lt;/ITEM&gt;&lt;/item&gt;&lt;item&gt;&lt;VBELN&gt;0180001233&lt;/VBELN&gt;&lt;WADAT&gt;00000000&lt;/WADAT&gt;&lt;VSTEL&gt;1000&lt;/VSTEL&gt;&lt;KUNNR/&gt;&lt;ITEM&gt;&lt;YITEM&gt;&lt;MATNR&gt;000000000041000000&lt;/MATNR&gt;&lt;LFIMG&gt;10.0&lt;/LFIMG&gt;&lt;MEINS&gt;LM&lt;/MEINS&gt;&lt;/YITEM&gt;&lt;/ITEM&gt;&lt;/item&gt;&lt;/OUTPUT&gt;</Data></ns0:Log>

The data inside <Data> tag is decoded text. But as you can see all its tags are replaced. For example, instead of <OUTPUT> it is printing <OUTPUT>

Can you please let me know what changes should I do in the UDF to avoid this.

Encoded String in Base64:

PE9VVFBVVD48aXRlbT48VkJFTE4+MDE4MDAwMTAyNzwvVkJFTE4+PFdBREFUPjAwMDAwMDAwPC9XQURBVD48VlNURUw+MTAwMDwvVlNURUw+PEtVTk5SLz48SVRFTT48WUlURU0+PE1BVE5SPjAwMDAwMDAwMDAxMTAwMDA4MjwvTUFUTlI+PExGSU1HPjEuMDwvTEZJTUc+PE1FSU5TPkw8L01FSU5TPjwvWUlURU0+PC9JVEVNPjwvaXRlbT48aXRlbT48VkJFTE4+MDE4MDAwMTIzMzwvVkJFTE4+PFdBREFUPjAwMDAwMDAwPC9XQURBVD48VlNURUw+MTAwMDwvVlNURUw+PEtVTk5SLz48SVRFTT48WUlURU0+PE1BVE5SPjAwMDAwMDAwMDA0MTAwMDAwMDwvTUFUTlI+PExGSU1HPjEwLjA8L0xGSU1HPjxNRUlOUz5MTTwvTUVJTlM+PC9ZSVRFTT48L0lURU0+PC9pdGVtPjwvT1VUUFVUPg==

I am trying to decode data in an xml format into bytes base64 and I am having an issues. My method is in java which takes a String data and converts it into bytes like as bellow.

String data = "......"; //string of data in xml formatbyte[] dataBytes = Base64.getDecoder().decode(data);

Which failed and gave the exception like bellow.

java.lang.IllegalArgumentException: Illegal base64 character 3c    at java.util.Base64$Decoder.decode0(Base64.java:714)    at java.util.Base64$Decoder.decode(Base64.java:526)    at java.util.Base64$Decoder.decode(Base64.java:549)    at XmlReader.main(XmlReader.java:61)

Is the xml format not compatible with base64?

I have a simple script where the user clicks a button and the code will execute a shell command that cuts base64 encrypted data from a csv and places it inside of a textarea. Then, I simply assign that value to a variable and echo the decoded result. However, my result is nothing. If I echo $encoded_data I do see the encoded strings.

My data is laid out like this within the textarea:

ZGF2aWRAZW1haWwuY29tZGF2aWRAZW1haWwuY29tZGF2aWRAZW1haWwuY29tZGF2aWRAZW1haWwuY29t...

There are hundreds of lines of encoded strings.

If I pass a string, the decoder works. But, with the text inside of the textarea it returns nothing at all. I'm assuming the amount of strings passed causes a memory issue. I also noticed that if i try to pass multiple strings I get weird characters likely because of the line breaks. How can I avoid this as well?

<form name="decode" action="" method="post" enctype="multipart/form-data"><textarea id="decode_field" name="decode_field" rows="10" cols="50" autofocus><?if (isset($_POST['decode_btn'])) {       // Cut base64 data from csv file. Selected data is in col 2 of    each row.       $encoded_data = shell_exec("cut -d, -f2 test.csv");       // Display data to user.       echo base64_decode($encoded_data);}?></textarea><input type="submit" name="decode_btn" value="Get Decoded Data" /></form>

output should be the base64 strings decoded to regular strings line by line.

Below is my requirement:

1. Program will have an xml file as input with 3 tags: , and . All these data are Base64 encoded. Note: Program is using BC jars
2. Program needs to decode them and verify the data for its authenticity using the signature and certificate
3. Verified data should be Base64 decoded and written into another file

Below is my code which tries to decode the certificate:

public void executeTask(InputStream arg0, OutputStream arg1) throws SomeException{try{    BufferedReader br = null;    br = new BufferedReader(new InputStreamReader(arg0));    String orgContent  = "", splitData = "", signContent = "", certContent = "";    DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();    DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();    Document doc = docBuilder.parse(arg0);    doc.getDocumentElement().normalize();    NodeList originalContent = doc.getElementsByTagName("OrgContent");    Element originalElement = (Element)originalContent.item(0);    NodeList textOrgContent = originalElement.getChildNodes();    orgContent = ((Node)textOrgContent.item(0)).getNodeValue().trim();    NodeList signature = doc.getElementsByTagName("Signature");    Element signatureElement = (Element)signature.item(0);    NodeList signatureContent = signatureElement.getChildNodes();    signContent = ((Node)signatureContent.item(0)).getNodeValue().trim();    NodeList certificate = doc.getElementsByTagName("Certificate");    Element certificateElement = (Element)certificate.item(0);    NodeList certificateContent = certificateElement.getChildNodes();    certContent = ((Node)certificateContent.item(0)).getNodeValue().trim();    String decodedCertContent = new String(Base64.decode(certContent),StandardCharsets.UTF_8);    byte[] certByteValue = Base64.decode(certContent);    CertificateFactory certFactory = CertificateFactory.getInstance("X.509");    System.out.println("certContent:\n" + new String(certByteValue,StandardCharsets.UTF_8));    InputStream inputStream = new ByteArrayInputStream(Base64.decode(certContent));    X509Certificate cert = (X509Certificate)certFactory.generateCertificate(inputStream);    arg1.write(decodedOrgData.getBytes());    arg1.flush();   }catch (ParserConfigurationException e){    e.printStackTrace();}catch (IOException e){    e.printStackTrace();}catch (org.xml.sax.SAXException e){    e.printStackTrace();}catch (CertificateException e){    e.printStackTrace();}

}

When I print the value of new String(certByteValue,StandardCharsets.UTF_8) the program is printing some unrecognizable text. When executing the last line of the code X509Certificate cert = (X509Certificate)certFactory.generateCertificate(inputStream); system is throwing java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Invalid BER/DER data (too huge?).

Since I am a newbie to these Certificates thing, I have hit a deadlock. I am unable to proceed with the requirement. I would like to know how to achieve my above said requirements.

EditThe input stream to the above code will be an xml file. Another program creates that xml file with base64 encoded data with signature and certificate. In that program, for encoding the certificate the below code is used:

        KeyStore keyStore = KeyStore.getInstance("JKS");    keyStore.load(new FileInputStream("Filepath/certificate.p12"), "password".toCharArray());    PrivateKey privateKey = (PrivateKey)keyStore.getKey(alias, "password".toCharArray());        CertificateFactory factory = CertificateFactory.getInstance("X.509");        X509Certificate certificate = (X509Certificate) factory.generateCertificate(new FileInputStream("D:/Sujai/Implementation Team/PI/Axis Treds/Certificates/PI_7.5_Cert/Arteria_Certificate-cert.cert"));    byte[] encodedCert = certificate.getEncoded();    String encodedStringCert = new String(Base64.encode(new String(encodedCert).getBytes(StandardCharsets.UTF_8)));

The variable encodedStringCert is passed as the certificate value inside a tag. In the program shared at the top of this question, I need to decode this certificate value.

Sample certificate content: -----BEGIN CERTIFICATE-----MIIDBjCCAe6....IM1g==-----END CERTIFICATE-----