Online Base64 Encoder

I am trying to compose an XML document that will get sent to some target system. (There isn't a particular system, I'm just trying to implement a messaging standard.) The document schema gives the type of a particular field as xsd:base64Binary. No problem, I just base64-encode the content of the field when serializing. But what if the original value of the field contains Unicode characters? In my case I'm dealing with Japanese characters.

The programming language I'm using represents these characters internally with their Unicode values. Ex:

"た" (HIRAGANA LETTER TA) = 0x305F.

So before I can base64-encode this value I need a way to transform it from a double-wide byte to a series of single-width bytes. UTF-8 encoding comes to mind, so the value would become:

0xE3 0x81 0x9F

This base64-encodes to:

44Gf

So presumably that's what would go in my document. There's just one wrinkle to this. The system that is receiving the XML document will have to "know" somehow that the base64-decoded value of the field is UTF-8 encoded. I can't find anything in the standard for the messaging format that says that once the value of the field is base64-decoded, the resulting value is UTF-8 (or any other format) encoded. I was hoping the correct thing to do in this scenario is described in the XML spec, but I haven't been able to find anything definitive.

One thought that comes to mind is that maybe the character encoding of the base64-decoded value is supposed to be the same as the character encoding named in the XML prolog, for example:

<?xml version="1.0" encoding="UTF-8" ?>

So if my XML document has this, then is the receiving system supposed to know that once the field in question is base64-decoded, that it is still UTF-8 encoded?

I have this code:

static char Base64Digits[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";std::string ToBase64Simple( const unsigned char* pSrc, int nLenSrc){   std::string pDst;   int nLenOut= 0;   int count = 0;   while ( nLenSrc > 0 ) {      unsigned char s1= pSrc[0];       unsigned char s2= 0; if (nLenSrc>1) s2=pSrc[1];       unsigned char s3= 0; if (nLenSrc>2) s3=pSrc[2];      unsigned int n;      n =  s1;          n <<= 8;         n |= s2;         n <<= 8;         n |= s3;        unsigned char m4= n & 0x3f;  n >>= 6;      unsigned char m3= n & 0x3f;  n >>= 6;      unsigned char m2= n & 0x3f;  n >>= 6;      unsigned char m1= n & 0x3f;        unsigned char b1 = Base64Digits[m1];      unsigned char b2 = Base64Digits[m2];      unsigned char b3 = Base64Digits[m3];      unsigned char b4 = Base64Digits[m4];      pDst.push_back(b1);      pDst.push_back(b2);      if ( nLenSrc >= 3 ) {          pDst.push_back(b3);         pDst.push_back(b4);      }      if ( nLenSrc == 2 ) {         pDst.push_back(b3);         pDst.push_back('=');      }      if ( nLenSrc == 1 ) {          pDst.push_back('=');         pDst.push_back('=');      }      pSrc    += 3;      nLenSrc -= 3;      nLenOut += 4;   }   return pDst;}static unsigned char LookupDigits[] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, //gap: ctrl chars0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, //gap: ctrl chars0,0,0,0,0,0,0,0,0,0,0,           //gap: spc,!"#$%'()*62,                   // + 0, 0, 0,             // gap ,-.63,                   // /52, 53, 54, 55, 56, 57, 58, 59, 60, 61, // 0-9 0, 0, 0,             // gap: :;<99,                   //  = (end padding) 0, 0, 0,             // gap: >?@ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25, // A-Z 0, 0, 0, 0, 0, 0,    // gap: [\]^_`26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51, // a-z     0, 0, 0, 0,          // gap: {|}~ (and the rest...)0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};int FromBase64Simple( std::string pSrc, std::vector<unsigned char> vec){   for( int j=0; j<pSrc.size(); j+=4 ) {      unsigned char s1= LookupDigits[pSrc.at(j)];      unsigned char s2= LookupDigits[pSrc.at(j+1)];      unsigned char s3= LookupDigits[pSrc.at(j+2)];      unsigned char s4= LookupDigits[pSrc.at(j+3)];      unsigned char d1= ((s1 & 0x3f) << 2) | ((s2 & 0x30) >> 4);      unsigned char d2= ((s2 & 0x0f) << 4) | ((s3 & 0x3c) >> 2);      unsigned char d3= ((s3 & 0x03) << 6) | ((s4 & 0x3f) >> 0);      vec.push_back(d1);        if (s3==99) break;           vec.push_back(d2);        if (s4==99) break;        vec.push_back(d3);     }   return 0;}

I found it online and modified it to use std::strings and unsigned char vectors instead of pointers.

Encoding works perfectly: I've been checking it with an online encoder and decoder. Decoding returns an empty vector each time. Why is this?

I call it using this code:

std::vector<unsigned char> byteVec;byteVec.push_back('a');std::string str = ToBase64Simple(&byteVec[0], byteVec.size());std::cout<<str<<std::endl;std::vector<unsigned char> vec;FromBase64Simple(str, vec);std::cout<<vec.size()<<std::endl;

a correctly translates to YQ==, but not back.

Imagine there is one numeric str:

Followings are the list of id, I need to merge them together to str and store it into DB, and when I need to use it, I need to translate it into the raw ids.So I need to shorten the ids size....

7376,89962,18299,74081,4094197,74445,807343,309251102,594649‌​555

I use base64 encode the str to：NzM3Njg5OTE2MjE4Mjk5MTc0MDgxNDAxOTQxOTc3MTQ0NDU4MDcxMzQzMzA5MjE1MTEwMjU5MTQ2NDk1NTU=

But it is too long actually.Is there any ways I can shorten(encode) the str's length and decode it to the before?

I was exploiting a web application for my lab and I found these hashes

B8qE6WbhBO1F53Cjj+O42TbczyFFqrWpG+BcfS70piQQiI2yASVRfhBiLRtIEU7PxdprFtkzDVLeVxfL7z5vTQ==

8NxO3RXifKUPNw9f7jjp9T6TA8UePPK5jn6QdoPTgEkOElVdNSAG8xp2Y/xJJ/dlCcOLxaYIEbE0n/1onvyAlA==

I have read the answers that we can't identify a hash type because of randomness but I wanted to know if it is possible to guess based on some properties, on the first look this looks like base 64 encoding(because of padding and + and / signs) but its some binary data if I decode it. So if it is a hash why is it base 64 encoded and if decoded why is it binary data and not some hash

I am very confused

I'm using Grails 1.3.7. I have some code that uses the built-in base64Encode function and base64Decode function. It all works fine in simple test cases where I encode some binary data and then decode the resulting string and write it to a new file. In this case the files are identical.

But then I wrote a web service that took the base64 encoded data as a parameter in a POST call. Although the length of the base64 data is identical to the string I passed into the function, the contents of the base64 data are being modified. I spend DAYS debugging this and finally wrote a test controller that passed the data in base64 to post and also took the name of a local file with the correct base64 encoded data, as in:

data=AAA-base-64-data...&testFilename=/name/of/file/with/base64data

Within the test function I compared every byte in the incoming data parameter with the appropriate byte in the test file. I found that somehow every "+" character in the input data parameter had been replaced with a " " (space, ordinal ascii 32). Huh? What could have done that?

To be sure I was correct, I added a line that said:

data = data.replaceAll(' ', '+')

and sure enough the data decoded exactly right. I tried it with arbitrarily long binary files and it now works every time. But I can't figure out for the life of me what would be modifying the data parameter in the post to convert the ord(43) character to ord(32)? I know that the plus sign is one of the 2 somewhat platform dependent characters in the base64 spec, but given that I am doing the encoding and decoding on the same machine for now I am super puzzled what caused this. Sure I have a "fix" since I can make it work, but I am nervous about "fixes" that I don't understand.

The code is too big to post here, but I get the base64 encoding like so:

def inputFile = new File(inputFilename)def rawData =  inputFile.getBytes()def encoded = rawData.encodeBase64().toString()

I then write that encoded string out to new a file so I can use it for testing later. If I load that file back in as so I get the same rawData:

def encodedFile = new File(encodedFilename)String encoded = encodedFile.getText()byte[] rawData = encoded.decodeBase64()

So all that is good. Now assume I take the "encoded" variable and add it to a param to a POST function like so:

String queryString = "data=$encoded"String url = "http://localhost:8080/some_web_service"def results = urlPost(url, queryString)def urlPost(String urlString, String queryString) {    def url = new URL(urlString)    def connection = url.openConnection()    connection.setRequestMethod("POST")    connection.doOutput = true    def writer = new OutputStreamWriter(connection.outputStream)    writer.write(queryString)    writer.flush()    writer.close()    connection.connect()    return (connection.responseCode == 200) ? connection.content.text : "error                         $connection.responseCode, $connection.responseMessage"}

on the web service side, in the controller I get the parameter like so:

String data = params?.dataprintln "incoming data parameter has length of ${data.size()}" //confirm right size//unless I run the following line, the data does not decode to the same sourcedata = data.replaceAll(' ', '+')//as long as I replace spaces with plus, this decodes correctly, why?byte[] bytedata = data.decodeBase64()

Sorry for the long rant, but I'd really love to understand why I had to do the "replace space with plus sign" to get this to decode correctly. Is there some problem with the plus sign being used in a request parameter?