php - cURL gets response with utf-8 BOM

Question

In my script I send data with cURL, and enabled CURLOPT_RETURNTRANSFER. The response is json encoded data. When I'm trying to json_decode, it returns null. Then I found that response contains utf-8 BOM symbols at the beginning of string (???).

There is some experiments:

---

$data = $data = curl_exec($ch);
echo $data;

the result is {"field_1":"text_1","field_2":"text_2","field_3":"text_3"}

$data = $data = curl_exec($ch);
echo mb_detect_encoding($data);

result - UTF-8

$data = $data = curl_exec($ch);
echo mb_convert_encoding($data, 'UTF-8', mb_detect_encoding($data));
// identical to echo mb_convert_encoding($data, 'UTF-8', 'UTF-8');

result - ???{"field_1":"text_1","field_2":"text_2","field_3":"text_3"}

---

The one thing that helps is removing first 3 symbols:

if (substr($data, 0, 3) == pack('CCC', 239, 187, 191)) {
    $data = substr($data, 3);
}

But what if there will be another BOM? So the question is: How to detect right encoding of cURL response? OR how to detect what BOM has arrrived? Or maybe how to convert the response with BOM?

Answer 1

I'm afraid you already found the answer by yourself - it's bad news in that there is no better answer that I know of.

The BOM should not be there, and it's the sender's responsibility to not send it along.

But I can reassure you, the BOM is either there or there is not, and if it is, it's those three bytes you know.

You can have a slightly faster and handle another N BOMs with a small alteration:

$__BOM = pack('CCC', 239, 187, 191);
// Careful about the three ='s -- they're all needed.
while(0 === strpos($data, $__BOM))
    $data = substr($data, 3);

A third-party BOM detector wouldn't do any different. This way you're covered even if at a later time cURL began stripping unneeded BOMs.

Possible causes

Some JSON optimizers and filters may decide the output requires a BOM. Also, perhaps more simply, whoever wrote the script generating the JSON inadvertently included a BOM before the opening PHP tag. Apache, not caring what the BOM is, sees there is data before the opening tag, so sends it along and hides it from the PHP stream itself. This can occasionally also cause the "Cannot add headers: output already started" error.

Content detection

You can verify the JSON is valid UTF-8, BOM or not BOM, but need mb_string support and you must use strict mode to get some edge cases:

if (false === mb_detect_encoding($data, 'UTF-8', true)) {
    // JSON contains invalid sequences (erroneously NOT JSON encoded)
}

I would advise against trying to correct a possible encoding error; you risk breaking your own code, and also having to maintain someone else's work.