A UTF-8 specific character encoder that handles cleaning and transforming.
More...
|
static | muteErrorHandler () |
| Error-handler that mutes errors, alternative to shut-up operator. More...
|
|
static | unsafeIconv ($in, $out, $text) |
| iconv wrapper which mutes errors, but doesn't work around bugs. More...
|
|
static | iconv ($in, $out, $text, $max_chunk_size=8000) |
| iconv wrapper which mutes errors and works around bugs. More...
|
|
static | cleanUTF8 ($str, $force_php=false) |
| Cleans a UTF-8 string for well-formedness and SGML validity. More...
|
|
static | unichr ($code) |
| Translates a Unicode codepoint into its corresponding UTF-8 character. More...
|
|
static | iconvAvailable () |
|
static | convertToUTF8 ($str, $config, $context) |
| Convert a string to UTF-8 based on configuration. More...
|
|
static | convertFromUTF8 ($str, $config, $context) |
| Converts a string from UTF-8 based on configuration. More...
|
|
static | convertToASCIIDumbLossless ($str) |
| Lossless (character-wise) conversion of HTML to ASCII. More...
|
|
static | testIconvTruncateBug () |
| glibc iconv has a known bug where it doesn't handle the magic //IGNORE stanza correctly. More...
|
|
static | testEncodingSupportsASCII ($encoding, $bypass=false) |
| This expensive function tests whether or not a given character encoding supports ASCII. More...
|
|
|
const | ICONV_OK = 0 |
| No bugs detected in iconv. More...
|
|
const | ICONV_TRUNCATES = 1 |
| Iconv truncates output if converting from UTF-8 to another character set with //IGNORE, and a non-encodable character is found. More...
|
|
const | ICONV_UNUSABLE = 2 |
| Iconv does not support //IGNORE, making it unusable for transcoding purposes. More...
|
|
|
| __construct () |
| Constructor throws fatal error if you attempt to instantiate class. More...
|
|
A UTF-8 specific character encoder that handles cleaning and transforming.
- Note
- All functions in this class should be static.
Definition at line 7 of file Encoder.php.
◆ __construct()
HTMLPurifier_Encoder::__construct |
( |
| ) |
|
|
private |
Constructor throws fatal error if you attempt to instantiate class.
Definition at line 13 of file Encoder.php.
15 trigger_error(
'Cannot instantiate encoder, call methods statically', E_USER_ERROR);
◆ cleanUTF8()
static HTMLPurifier_Encoder::cleanUTF8 |
( |
|
$str, |
|
|
|
$force_php = false |
|
) |
| |
|
static |
Cleans a UTF-8 string for well-formedness and SGML validity.
It will parse according to UTF-8 and return a valid UTF8 string, with non-SGML codepoints excluded.
Specifically, it will permit: {9}{A}{D}{20}-{7E}{A0}-{D7FF}{E000}-{FFFD}{10000}-{10FFFF} Source: https://www.w3.org/TR/REC-xml/#NT-Char Arguably this function should be modernized to the HTML5 set of allowed characters: https://www.w3.org/TR/html5/syntax.html#preprocessing-the-input-stream which simultaneously expand and restrict the set of allowed characters.
- Parameters
-
string | $str | The string to clean |
bool | $force_php | |
- Returns
- string
- Note
- Just for reference, the non-SGML code points are 0 to 31 and 127 to 159, inclusive. However, we allow code points 9, 10 and 13, which are the tab, line feed and carriage return respectively. 128 and above the code points map to multibyte UTF-8 representations.
-
Fallback code adapted from utf8ToUnicode by Henri Sivonen and hsivo.nosp@m.nen@.nosp@m.iki.f.nosp@m.i at http://iki.fi/hsivonen/php-utf8/ under the LGPL license. Notes on what changed are inside, but in general, the original code transformed UTF-8 text into an array of integer Unicode codepoints. Understandably, transforming that back to a string would be somewhat expensive, so the function was modded to directly operate on the string. However, this discourages code reuse, and the logic enumerated here would be useful for any function that needs to be able to understand UTF-8 characters. As of right now, only smart lossless character encoding converters would need that, and I'm probably not going to implement them.
Definition at line 134 of file Encoder.php.
References $i, $in, and $out.
Referenced by HTMLPurifier_Printer\escape(), HTMLPurifier_AttrDef\expandCSSEscape(), and HTMLPurifier_Lexer\normalize().
140 '/^[\x{9}\x{A}\x{D}\x{20}-\x{7E}\x{A0}-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]*$/Du',
161 for (
$i = 0;
$i < $len;
$i++) {
167 if (0 == (0x80 & (
$in))) {
169 if ((
$in <= 31 ||
$in == 127) &&
179 } elseif (0xC0 == (0xE0 & (
$in))) {
182 $mUcs4 = ($mUcs4 & 0x1F) << 6;
185 } elseif (0xE0 == (0xF0 & (
$in))) {
188 $mUcs4 = ($mUcs4 & 0x0F) << 12;
191 } elseif (0xF0 == (0xF8 & (
$in))) {
194 $mUcs4 = ($mUcs4 & 0x07) << 18;
197 } elseif (0xF8 == (0xFC & (
$in))) {
208 $mUcs4 = ($mUcs4 & 0x03) << 24;
211 } elseif (0xFC == (0xFE & (
$in))) {
215 $mUcs4 = ($mUcs4 & 1) << 30;
229 if (0x80 == (0xC0 & (
$in))) {
231 $shift = ($mState - 1) * 6;
233 $tmp = ($tmp & 0x0000003F) << $shift;
236 if (0 == --$mState) {
243 if (((2 == $mBytes) && ($mUcs4 < 0x0080)) ||
244 ((3 == $mBytes) && ($mUcs4 < 0x0800)) ||
245 ((4 == $mBytes) && ($mUcs4 < 0x10000)) ||
248 (($mUcs4 & 0xFFFFF800) == 0xD800) ||
253 } elseif (0xFEFF != $mUcs4 &&
259 (0x20 <= $mUcs4 && 0x7E >= $mUcs4) ||
262 (0xA0 <= $mUcs4 && 0xD7FF >= $mUcs4) ||
263 (0xE000 <= $mUcs4 && 0xFFFD >= $mUcs4) ||
264 (0x10000 <= $mUcs4 && 0x10FFFF >= $mUcs4)
if(php_sapi_name() !='cli') $in
◆ convertFromUTF8()
static HTMLPurifier_Encoder::convertFromUTF8 |
( |
|
$str, |
|
|
|
$config, |
|
|
|
$context |
|
) |
| |
|
static |
Converts a string from UTF-8 based on configuration.
- Parameters
-
- Returns
- string
- Note
- Currently, this is a lossy conversion, with unexpressable characters being omitted.
Definition at line 426 of file Encoder.php.
References $config.
Referenced by HTMLPurifier\purify().
428 $encoding =
$config->get(
'Core.Encoding');
429 if ($escape =
$config->get(
'Core.EscapeNonASCIICharacters')) {
430 $str = self::convertToASCIIDumbLossless($str);
432 if ($encoding ===
'utf-8') {
435 static $iconv = null;
436 if ($iconv === null) {
437 $iconv = self::iconvAvailable();
439 if ($iconv && !
$config->get(
'Test.ForceNoIconv')) {
441 $ascii_fix = self::testEncodingSupportsASCII($encoding);
442 if (!$escape && !empty($ascii_fix)) {
443 $clear_fix = array();
444 foreach ($ascii_fix as $utf8 => $native) {
445 $clear_fix[$utf8] =
'';
447 $str = strtr($str, $clear_fix);
449 $str = strtr($str, array_flip($ascii_fix));
451 $str = self::iconv(
'utf-8', $encoding .
'//IGNORE', $str);
453 } elseif ($encoding ===
'iso-8859-1') {
454 $str = utf8_decode($str);
457 trigger_error(
'Encoding not supported', E_USER_ERROR);
◆ convertToASCIIDumbLossless()
static HTMLPurifier_Encoder::convertToASCIIDumbLossless |
( |
|
$str | ) |
|
|
static |
Lossless (character-wise) conversion of HTML to ASCII.
- Parameters
-
string | $str | UTF-8 string to be converted to ASCII |
- Returns
- string ASCII encoded string with non-ASCII character entity-ized
- Warning
- Adapted from MediaWiki, claiming fair use: this is a common algorithm. If you disagree with this license fudgery, implement it yourself.
- Note
- Uses decimal numeric entities since they are best supported.
-
This is a DUMB function: it has no concept of keeping character entities that the projected character encoding can allow. We could possibly implement a smart version but that would require it to also know which Unicode codepoints the charset supported (not an easy task).
-
Sort of with cleanUTF8() but it assumes that $str is well-formed UTF-8
Definition at line 480 of file Encoder.php.
References $i, and $result.
486 for (
$i = 0;
$i < $len;
$i++) {
487 $bytevalue = ord($str[
$i]);
488 if ($bytevalue <= 0x7F) {
491 } elseif ($bytevalue <= 0xBF) {
492 $working = $working << 6;
493 $working += ($bytevalue & 0x3F);
495 if ($bytesleft <= 0) {
496 $result .=
"&#" . $working .
";";
498 } elseif ($bytevalue <= 0xDF) {
499 $working = $bytevalue & 0x1F;
501 } elseif ($bytevalue <= 0xEF) {
502 $working = $bytevalue & 0x0F;
505 $working = $bytevalue & 0x07;
◆ convertToUTF8()
static HTMLPurifier_Encoder::convertToUTF8 |
( |
|
$str, |
|
|
|
$config, |
|
|
|
$context |
|
) |
| |
|
static |
Convert a string to UTF-8 based on configuration.
- Parameters
-
- Returns
- string
Definition at line 378 of file Encoder.php.
References $config, and testIconvTruncateBug().
Referenced by HTMLPurifier\purify().
380 $encoding =
$config->get(
'Core.Encoding');
381 if ($encoding ===
'utf-8') {
384 static $iconv = null;
385 if ($iconv === null) {
386 $iconv = self::iconvAvailable();
388 if ($iconv && !
$config->get(
'Test.ForceNoIconv')) {
390 $str = self::unsafeIconv($encoding,
'utf-8//IGNORE', $str);
391 if ($str ===
false) {
393 trigger_error(
'Invalid encoding ' . $encoding, E_USER_ERROR);
399 $str = strtr($str, self::testEncodingSupportsASCII($encoding));
401 } elseif ($encoding ===
'iso-8859-1') {
402 $str = utf8_encode($str);
406 if ($bug == self::ICONV_OK) {
407 trigger_error(
'Encoding not supported, please install iconv', E_USER_ERROR);
410 'You have a buggy version of iconv, see https://bugs.php.net/bug.php?id=48147 ' .
411 'and http://sourceware.org/bugzilla/show_bug.cgi?id=13541',
static testIconvTruncateBug()
glibc iconv has a known bug where it doesn't handle the magic //IGNORE stanza correctly.
◆ iconv()
static HTMLPurifier_Encoder::iconv |
( |
|
$in, |
|
|
|
$out, |
|
|
|
$text, |
|
|
|
$max_chunk_size = 8000 |
|
) |
| |
|
static |
iconv wrapper which mutes errors and works around bugs.
- Parameters
-
string | $in | Input encoding |
string | $out | Output encoding |
string | $text | The text to convert |
int | $max_chunk_size | |
- Returns
- string
Definition at line 48 of file Encoder.php.
References $c, $code, $i, $in, $out, $r, and $text.
Referenced by unsafeIconv().
50 $code = self::testIconvTruncateBug();
51 if (
$code == self::ICONV_OK) {
53 } elseif (
$code == self::ICONV_TRUNCATES) {
57 if ($max_chunk_size < 4) {
58 trigger_error(
'max_chunk_size is too small', E_USER_WARNING);
63 if ((
$c = strlen(
$text)) <= $max_chunk_size) {
69 if (
$i + $max_chunk_size >=
$c) {
74 if (0x80 != (0xC0 & ord(
$text[
$i + $max_chunk_size]))) {
75 $chunk_size = $max_chunk_size;
76 } elseif (0x80 != (0xC0 & ord(
$text[
$i + $max_chunk_size - 1]))) {
77 $chunk_size = $max_chunk_size - 1;
78 } elseif (0x80 != (0xC0 & ord(
$text[
$i + $max_chunk_size - 2]))) {
79 $chunk_size = $max_chunk_size - 2;
80 } elseif (0x80 != (0xC0 & ord(
$text[
$i + $max_chunk_size - 3]))) {
81 $chunk_size = $max_chunk_size - 3;
85 $chunk = substr(
$text,
$i, $chunk_size);
86 $r .= self::unsafeIconv(
$in,
$out, $chunk);
if(php_sapi_name() !='cli') $in
◆ iconvAvailable()
static HTMLPurifier_Encoder::iconvAvailable |
( |
| ) |
|
|
static |
- Returns
- bool
Definition at line 362 of file Encoder.php.
364 static $iconv = null;
365 if ($iconv === null) {
366 $iconv = function_exists(
'iconv') && self::testIconvTruncateBug() != self::ICONV_UNUSABLE;
◆ muteErrorHandler()
static HTMLPurifier_Encoder::muteErrorHandler |
( |
| ) |
|
|
static |
Error-handler that mutes errors, alternative to shut-up operator.
Definition at line 21 of file Encoder.php.
◆ testEncodingSupportsASCII()
static HTMLPurifier_Encoder::testEncodingSupportsASCII |
( |
|
$encoding, |
|
|
|
$bypass = false |
|
) |
| |
|
static |
This expensive function tests whether or not a given character encoding supports ASCII.
7/8-bit encodings like Shift_JIS will fail this test, and require special processing. Variable width encodings shouldn't ever fail.
- Parameters
-
string | $encoding | Encoding name to test, as per iconv format |
bool | $bypass | Whether or not to bypass the precompiled arrays. |
- Returns
- Array of UTF-8 characters to their corresponding ASCII, which can be used to "undo" any overzealous iconv action.
Definition at line 571 of file Encoder.php.
References $c, $i, $r, and $ret.
578 static $encodings = array();
580 if (isset($encodings[$encoding])) {
581 return $encodings[$encoding];
583 $lenc = strtolower($encoding);
586 return array(
"\xC2\xA5" =>
'\\',
"\xE2\x80\xBE" =>
'~');
588 return array(
"\xE2\x82\xA9" =>
'\\');
590 if (strpos($lenc,
'iso-8859-') === 0) {
595 if (self::unsafeIconv(
'UTF-8', $encoding,
'a') ===
false) {
598 for (
$i = 0x20;
$i <= 0x7E;
$i++) {
600 $r = self::unsafeIconv(
'UTF-8',
"$encoding//IGNORE",
$c);
604 (
$r ===
$c && self::unsafeIconv($encoding,
'UTF-8//IGNORE',
$r) !==
$c)
609 $ret[self::unsafeIconv($encoding,
'UTF-8//IGNORE',
$c)] =
$c;
612 $encodings[$encoding] =
$ret;
◆ testIconvTruncateBug()
static HTMLPurifier_Encoder::testIconvTruncateBug |
( |
| ) |
|
|
static |
glibc iconv has a known bug where it doesn't handle the magic //IGNORE stanza correctly.
In particular, rather than ignore characters, it will return an EILSEQ after consuming some number of characters, and expect you to restart iconv as if it were an E2BIG. Old versions of PHP did not respect the errno, and returned the fragment, so as a result you would see iconv mysteriously truncating output. We can work around this by manually chopping our input into segments of about 8000 characters, as long as PHP ignores the error code. If PHP starts paying attention to the error code, iconv becomes unusable.
- Returns
- int Error code indicating severity of bug.
Definition at line 537 of file Encoder.php.
References $c, $code, and $r.
Referenced by convertToUTF8().
540 if (
$code === null) {
542 $r = self::unsafeIconv(
'utf-8',
'ascii//IGNORE',
"\xCE\xB1" . str_repeat(
'a', 9000));
544 $code = self::ICONV_UNUSABLE;
545 } elseif ((
$c = strlen(
$r)) < 9000) {
546 $code = self::ICONV_TRUNCATES;
547 } elseif (
$c > 9000) {
549 'Your copy of iconv is extremely buggy. Please notify HTML Purifier maintainers: ' .
550 'include your iconv version as per phpversion()',
554 $code = self::ICONV_OK;
◆ unichr()
static HTMLPurifier_Encoder::unichr |
( |
|
$code | ) |
|
|
static |
Translates a Unicode codepoint into its corresponding UTF-8 character.
- Note
- Based on Feyd's function at http://forums.devnetwork.net/viewtopic.php?p=191404#191404, which is in public domain.
-
While we're going to do code point parsing anyway, a good optimization would be to refuse to translate code points that are non-SGML characters. However, this could lead to duplication.
-
This is very similar to the unichr function in maintenance/generate-entity-file.php (although this is superior, due to its sanity checks).
Definition at line 315 of file Encoder.php.
References $code, $ret, $w, $x, and $y.
Referenced by HTMLPurifier_EntityParser\entityCallback(), HTMLPurifier_AttrDef\expandCSSEscape(), and HTMLPurifier_EntityParser\nonSpecialEntityCallback().
332 $y = ((
$code & 2047) >> 6) | 192;
334 $y = ((
$code & 4032) >> 6) | 128;
336 $z = ((
$code >> 12) & 15) | 224;
338 $z = ((
$code >> 12) & 63) | 128;
339 $w = ((
$code >> 18) & 7) | 240;
◆ unsafeIconv()
static HTMLPurifier_Encoder::unsafeIconv |
( |
|
$in, |
|
|
|
$out, |
|
|
|
$text |
|
) |
| |
|
static |
iconv wrapper which mutes errors, but doesn't work around bugs.
- Parameters
-
string | $in | Input encoding |
string | $out | Output encoding |
string | $text | The text to convert |
- Returns
- string
Definition at line 32 of file Encoder.php.
References $in, $out, $r, $text, and iconv().
34 set_error_handler(array(
'HTMLPurifier_Encoder',
'muteErrorHandler'));
36 restore_error_handler();
static iconv($in, $out, $text, $max_chunk_size=8000)
iconv wrapper which mutes errors and works around bugs.
if(php_sapi_name() !='cli') $in
◆ ICONV_OK
const HTMLPurifier_Encoder::ICONV_OK = 0 |
No bugs detected in iconv.
Definition at line 513 of file Encoder.php.
◆ ICONV_TRUNCATES
const HTMLPurifier_Encoder::ICONV_TRUNCATES = 1 |
Iconv truncates output if converting from UTF-8 to another character set with //IGNORE, and a non-encodable character is found.
Definition at line 517 of file Encoder.php.
◆ ICONV_UNUSABLE
const HTMLPurifier_Encoder::ICONV_UNUSABLE = 2 |
Iconv does not support //IGNORE, making it unusable for transcoding purposes.
Definition at line 521 of file Encoder.php.
The documentation for this class was generated from the following file:
- libs/composer/vendor/ezyang/htmlpurifier/library/HTMLPurifier/Encoder.php