http - Detecting character encoding in HTML

Question

I download an HTML page. The HTTP content-type header specifies one character encoding, and the page has a meta tag that specifies another. What's the correct way to handle that?

I guess 'correct' isn't the right word, since nobody follows the damn standards anyway... so what's the way that will cause me the least problems?

Answer 1

Do the same as webbrowsers do: use the response header. When HTML is served over HTTP, the meta tag is ignored when the response header is present. Only when the HTML is read from local disk file system, the meta tag is been used. This is also explicitly specified by w3 HTML spec.

To sum up, conforming user agents must observe the following priorities when determining a document's character encoding (from highest priority to lowest):

1. An HTTP "charset" parameter in a "Content-Type" field.
2. A META declaration with "http-equiv" set to "Content-Type" and a value set for "charset".
3. The charset attribute set on an element that designates an external resource.

Any existing decent HTML parser in whatever language you use should already take this into account. According your question history you're familiar with Java, I'd then suggest to grab Jsoup for this.