UtfNormal Class Reference
Collaboration diagram for UtfNormal:
Public Member Functions | |
| cleanUp ( $string) | |
| The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition. More... | |
| toNFC ( $string) | |
| Convert a UTF-8 string to normal form C, canonical composition. More... | |
| toNFD ( $string) | |
| Convert a UTF-8 string to normal form D, canonical decomposition. More... | |
| toNFKC ( $string) | |
| Convert a UTF-8 string to normal form KC, compatibility composition. More... | |
| toNFKD ( $string) | |
| Convert a UTF-8 string to normal form KD, compatibility decomposition. More... | |
| loadData () | |
| Load the basic composition data if necessary @access private. More... | |
| quickIsNFC ( $string) | |
| Returns true if the string is definitely in NFC. More... | |
| quickIsNFCVerify (&$string) | |
| Returns true if the string is definitely in NFC. More... | |
| NFC ( $string) | |
| NFD ( $string) | |
| NFKC ( $string) | |
| NFKD ( $string) | |
| fastDecompose ( $string, &$map) | |
| Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us). More... | |
| fastCombiningSort ( $string) | |
| Sorts combining characters into canonical order. More... | |
| fastCompose ( $string) | |
| Produces canonically composed sequences, i.e. More... | |
| placebo ( $string) | |
| This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance. More... | |
Static Public Member Functions | |
| static | cleanUp ( $string) |
| The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition. More... | |
| static | toNFC ( $string) |
| Convert a UTF-8 string to normal form C, canonical composition. More... | |
| static | toNFD ( $string) |
| Convert a UTF-8 string to normal form D, canonical decomposition. More... | |
| static | toNFKC ( $string) |
| Convert a UTF-8 string to normal form KC, compatibility composition. More... | |
| static | toNFKD ( $string) |
| Convert a UTF-8 string to normal form KD, compatibility decomposition. More... | |
| static | quickIsNFC ( $string) |
| Returns true if the string is definitely in NFC. More... | |
| static | quickIsNFCVerify (&$string) |
| Returns true if the string is definitely in NFC. More... | |
| static | placebo ( $string) |
| This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance. More... | |
Static Private Member Functions | |
| static | loadData () |
| Load the basic composition data if necessary. More... | |
| static | NFC ( $string) |
| static | NFD ( $string) |
| static | NFKC ( $string) |
| static | NFKD ( $string) |
| static | fastDecompose ( $string, $map) |
| Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us). More... | |
| static | fastCombiningSort ( $string) |
| Sorts combining characters into canonical order. More... | |
| static | fastCompose ( $string) |
| Produces canonically composed sequences, i.e. More... | |
Detailed Description
Definition at line 117 of file UtfNormal.php.
Member Function Documentation
◆ cleanUp() [1/2]
| UtfNormal::cleanUp | ( | $string | ) |
The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition.
Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters. Not as fast as toNFC().
- Parameters
-
string $string a UTF-8 string
- Returns
- string a clean, shiny, normalized UTF-8 string
Definition at line 128 of file UtfNormal.php.
128 {
130 # We exclude a few chars that ICU would not.
131 $string = preg_replace(
132 '/[\x00-\x08\x0b\x0c\x0e-\x1f]/',
133 UTF8_REPLACEMENT,
134 $string );
137
138 # UnicodeString constructor fails if the string ends with a
139 # head byte. Add a junk char at the end, we'll strip it off.
141 } elseif( UtfNormal::quickIsNFCVerify( $string ) ) {
142 # Side effect -- $string has had UTF-8 errors cleaned up.
143 return $string;
144 } else {
146 }
147 }
quickIsNFCVerify(&$string)
Returns true if the string is definitely in NFC.
Definition: UtfNormal.php:273
References NFC(), NORMALIZE_ICU, quickIsNFCVerify(), UNORM_NFC, UTF8_FFFE, UTF8_FFFF, and UTF8_REPLACEMENT.
Referenced by CleanUpTest\doTestBytes(), CleanUpTest\doTestDoubleBytes(), CleanUpTest\doTestTripleBytes(), CleanUpTest\testAscii(), CleanUpTest\testBomRegression(), CleanUpTest\testChunkRegression(), CleanUpTest\testForbiddenRegression(), CleanUpTest\testHangulRegression(), CleanUpTest\testInterposeRegression(), CleanUpTest\testLatin(), CleanUpTest\testLatinNormal(), CleanUpTest\testNull(), CleanUpTest\testOverlongRegression(), CleanUpTest\testSurrogateRegression(), and CleanUpTest\XtestAllChars().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ cleanUp() [2/2]
|
static |
The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition.
Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters. Not as fast as toNFC().
- Parameters
-
string $string a UTF-8 string
- Returns
- string a clean, shiny, normalized UTF-8 string
Definition at line 124 of file UtfNormal.php.
124 {
126 # We exclude a few chars that ICU would not.
127 $string = preg_replace(
128 '/[\x00-\x08\x0b\x0c\x0e-\x1f]/',
129 UTF8_REPLACEMENT,
130 $string );
133
134 # UnicodeString constructor fails if the string ends with a
135 # head byte. Add a junk char at the end, we'll strip it off.
137 } elseif( UtfNormal::quickIsNFCVerify( $string ) ) {
138 # Side effect -- $string has had UTF-8 errors cleaned up.
139 return $string;
140 } else {
142 }
143 }
References NFC(), NORMALIZE_ICU, quickIsNFCVerify(), UNORM_NFC, UTF8_FFFE, UTF8_FFFF, and UTF8_REPLACEMENT.
Here is the call graph for this function:
◆ fastCombiningSort() [1/2]
| UtfNormal::fastCombiningSort | ( | $string | ) |
Sorts combining characters into canonical order.
This is the final step in creating decomposed normal forms D and KD. @access private
- Parameters
-
string $string a valid, decomposed UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string with combining characters sorted in canonical order
Definition at line 600 of file UtfNormal.php.
600 {
601 UtfNormal::loadData();
602 global $utfCombiningClass;
603 $len = strlen( $string );
605 $combiners = array();
606 $lastClass = -1;
607 for( $i = 0; $i < $len; $i++ ) {
608 $c = $string{$i};
609 $n = ord( $c );
612 $c = substr( $string, $i, 4 );
613 $i += 3;
614 } elseif( $n >= 0xe0 ) {
615 $c = substr( $string, $i, 3 );
616 $i += 2;
617 } elseif( $n >= 0xc0 ) {
618 $c = substr( $string, $i, 2 );
619 $i++;
620 }
622 $lastClass = $utfCombiningClass[$c];
623 @$combiners[$lastClass] .= $c;
624 continue;
625 }
626 }
627 if( $lastClass ) {
628 ksort( $combiners );
630 $combiners = array();
631 }
632 $out .= $c;
633 $lastClass = 0;
634 }
635 if( $lastClass ) {
636 ksort( $combiners );
638 }
640 }
loadData()
Load the basic composition data if necessary @access private.
Definition: UtfNormal.php:220
References $n, $out, $utfCombiningClass, and loadData().
Referenced by NFD(), and NFKD().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ fastCombiningSort() [2/2]
|
staticprivate |
Sorts combining characters into canonical order.
This is the final step in creating decomposed normal forms D and KD.
- Parameters
-
string $string a valid, decomposed UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string with combining characters sorted in canonical order
Definition at line 610 of file UtfNormal.php.
610 {
611 UtfNormal::loadData();
612 global $utfCombiningClass;
613 $len = strlen( $string );
615 $combiners = array();
616 $lastClass = -1;
617 for( $i = 0; $i < $len; $i++ ) {
618 $c = $string{$i};
619 $n = ord( $c );
622 $c = substr( $string, $i, 4 );
623 $i += 3;
624 } elseif( $n >= 0xe0 ) {
625 $c = substr( $string, $i, 3 );
626 $i += 2;
627 } elseif( $n >= 0xc0 ) {
628 $c = substr( $string, $i, 2 );
629 $i++;
630 }
632 $lastClass = $utfCombiningClass[$c];
633 if( isset( $combiners[$lastClass] ) ) {
634 $combiners[$lastClass] .= $c;
635 } else {
636 $combiners[$lastClass] = $c;
637 }
638 continue;
639 }
640 }
641 if( $lastClass ) {
642 ksort( $combiners );
644 $combiners = array();
645 }
646 $out .= $c;
647 $lastClass = 0;
648 }
649 if( $lastClass ) {
650 ksort( $combiners );
652 }
654 }
References $n, $out, $utfCombiningClass, and loadData().
Here is the call graph for this function:
◆ fastCompose() [1/2]
| UtfNormal::fastCompose | ( | $string | ) |
Produces canonically composed sequences, i.e.
normal form C or KC.
@access private
- Parameters
-
string $string a valid UTF-8 string in sorted normal form D or KD. Input is not validated.
- Returns
- string a UTF-8 string with canonical precomposed characters used where possible
Definition at line 649 of file UtfNormal.php.
649 {
650 UtfNormal::loadData();
652 $len = strlen( $string );
654 $lastClass = -1;
655 $lastHangul = 0;
656 $startChar = '';
657 $combining = '';
658 $x1 = ord(substr(UTF8_HANGUL_VBASE,0,1));
659 $x2 = ord(substr(UTF8_HANGUL_TEND,0,1));
660 for( $i = 0; $i < $len; $i++ ) {
661 $c = $string{$i};
662 $n = ord( $c );
664 # No combining characters here...
665 $out .= $startChar;
666 $out .= $combining;
667 $startChar = $c;
668 $combining = '';
669 $lastClass = 0;
670 continue;
671 } elseif( $n >= 0xf0 ) {
672 $c = substr( $string, $i, 4 );
673 $i += 3;
674 } elseif( $n >= 0xe0 ) {
675 $c = substr( $string, $i, 3 );
676 $i += 2;
677 } elseif( $n >= 0xc0 ) {
678 $c = substr( $string, $i, 2 );
679 $i++;
680 }
681 $pair = $startChar . $c;
684 # A combining char; see what we can do with it
685 $class = $utfCombiningClass[$c];
686 if( !empty( $startChar ) &&
687 $lastClass < $class &&
688 $class > 0 &&
689 isset( $utfCanonicalComp[$pair] ) ) {
690 $startChar = $utfCanonicalComp[$pair];
691 $class = 0;
692 } else {
693 $combining .= $c;
694 }
695 $lastClass = $class;
696 $lastHangul = 0;
697 continue;
698 }
699 }
700 # New start char
701 if( $lastClass == 0 ) {
703 $startChar = $utfCanonicalComp[$pair];
704 $lastHangul = 0;
705 continue;
706 }
708 # WARNING: Hangul code is painfully slow.
709 # I apologize for this ugly, ugly code; however
710 # performance is even more teh suck if we call
711 # out to nice clean functions. Lookup tables are
712 # marginally faster, but require a lot of space.
713 #
715 $c <= UTF8_HANGUL_VEND &&
716 $startChar >= UTF8_HANGUL_LBASE &&
717 $startChar <= UTF8_HANGUL_LEND ) {
718 #
719 #$lIndex = utf8ToCodepoint( $startChar ) - UNICODE_HANGUL_LBASE;
720 #$vIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_VBASE;
721 $lIndex = ord( $startChar{2} ) - 0x80;
722 $vIndex = ord( $c{2} ) - 0xa1;
723
724 $hangulPoint = UNICODE_HANGUL_FIRST +
725 UNICODE_HANGUL_TCOUNT *
726 (UNICODE_HANGUL_VCOUNT * $lIndex + $vIndex);
727
728 # Hardcode the limited-range UTF-8 conversion:
729 $startChar = chr( $hangulPoint >> 12 & 0x0f | 0xe0 ) .
730 chr( $hangulPoint >> 6 & 0x3f | 0x80 ) .
731 chr( $hangulPoint & 0x3f | 0x80 );
732 $lastHangul = 0;
733 continue;
734 } elseif( $c >= UTF8_HANGUL_TBASE &&
735 $c <= UTF8_HANGUL_TEND &&
736 $startChar >= UTF8_HANGUL_FIRST &&
737 $startChar <= UTF8_HANGUL_LAST &&
738 !$lastHangul ) {
739 # $tIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_TBASE;
740 $tIndex = ord( $c{2} ) - 0xa7;
741 if( $tIndex < 0 ) $tIndex = ord( $c{2} ) - 0x80 + (0x11c0 - 0x11a7);
742
743 # Increment the code point by $tIndex, without
744 # the function overhead of decoding and recoding UTF-8
745 #
746 $tail = ord( $startChar{2} ) + $tIndex;
747 if( $tail > 0xbf ) {
748 $tail -= 0x40;
749 $mid = ord( $startChar{1} ) + 1;
750 if( $mid > 0xbf ) {
751 $startChar{0} = chr( ord( $startChar{0} ) + 1 );
752 $mid -= 0x40;
753 }
754 $startChar{1} = chr( $mid );
755 }
756 $startChar{2} = chr( $tail );
757
758 # If there's another jamo char after this, *don't* try to merge it.
759 $lastHangul = 1;
760 continue;
761 }
762 }
763 }
764 $out .= $startChar;
765 $out .= $combining;
766 $startChar = $c;
767 $combining = '';
768 $lastClass = 0;
769 $lastHangul = 0;
770 }
771 $out .= $startChar . $combining;
773 }
References $n, $out, $utfCanonicalComp, $utfCombiningClass, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_TCOUNT, UNICODE_HANGUL_VCOUNT, UTF8_HANGUL_FIRST, UTF8_HANGUL_LAST, UTF8_HANGUL_LBASE, UTF8_HANGUL_LEND, UTF8_HANGUL_TBASE, UTF8_HANGUL_TEND, UTF8_HANGUL_VBASE, and UTF8_HANGUL_VEND.
Referenced by NFC(), and NFKC().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ fastCompose() [2/2]
|
staticprivate |
Produces canonically composed sequences, i.e.
normal form C or KC.
- Parameters
-
string $string a valid UTF-8 string in sorted normal form D or KD. Input is not validated.
- Returns
- string a UTF-8 string with canonical precomposed characters used where possible
Definition at line 664 of file UtfNormal.php.
664 {
665 UtfNormal::loadData();
667 $len = strlen( $string );
669 $lastClass = -1;
670 $lastHangul = 0;
671 $startChar = '';
672 $combining = '';
673 $x1 = ord(substr(UTF8_HANGUL_VBASE,0,1));
674 $x2 = ord(substr(UTF8_HANGUL_TEND,0,1));
675 for( $i = 0; $i < $len; $i++ ) {
676 $c = $string{$i};
677 $n = ord( $c );
679 # No combining characters here...
680 $out .= $startChar;
681 $out .= $combining;
682 $startChar = $c;
683 $combining = '';
684 $lastClass = 0;
685 continue;
686 } elseif( $n >= 0xf0 ) {
687 $c = substr( $string, $i, 4 );
688 $i += 3;
689 } elseif( $n >= 0xe0 ) {
690 $c = substr( $string, $i, 3 );
691 $i += 2;
692 } elseif( $n >= 0xc0 ) {
693 $c = substr( $string, $i, 2 );
694 $i++;
695 }
696 $pair = $startChar . $c;
699 # A combining char; see what we can do with it
700 $class = $utfCombiningClass[$c];
701 if( !empty( $startChar ) &&
702 $lastClass < $class &&
703 $class > 0 &&
704 isset( $utfCanonicalComp[$pair] ) ) {
705 $startChar = $utfCanonicalComp[$pair];
706 $class = 0;
707 } else {
708 $combining .= $c;
709 }
710 $lastClass = $class;
711 $lastHangul = 0;
712 continue;
713 }
714 }
715 # New start char
716 if( $lastClass == 0 ) {
718 $startChar = $utfCanonicalComp[$pair];
719 $lastHangul = 0;
720 continue;
721 }
723 # WARNING: Hangul code is painfully slow.
724 # I apologize for this ugly, ugly code; however
725 # performance is even more teh suck if we call
726 # out to nice clean functions. Lookup tables are
727 # marginally faster, but require a lot of space.
728 #
730 $c <= UTF8_HANGUL_VEND &&
731 $startChar >= UTF8_HANGUL_LBASE &&
732 $startChar <= UTF8_HANGUL_LEND ) {
733 #
734 #$lIndex = utf8ToCodepoint( $startChar ) - UNICODE_HANGUL_LBASE;
735 #$vIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_VBASE;
736 $lIndex = ord( $startChar{2} ) - 0x80;
737 $vIndex = ord( $c{2} ) - 0xa1;
738
739 $hangulPoint = UNICODE_HANGUL_FIRST +
740 UNICODE_HANGUL_TCOUNT *
741 (UNICODE_HANGUL_VCOUNT * $lIndex + $vIndex);
742
743 # Hardcode the limited-range UTF-8 conversion:
744 $startChar = chr( $hangulPoint >> 12 & 0x0f | 0xe0 ) .
745 chr( $hangulPoint >> 6 & 0x3f | 0x80 ) .
746 chr( $hangulPoint & 0x3f | 0x80 );
747 $lastHangul = 0;
748 continue;
749 } elseif( $c >= UTF8_HANGUL_TBASE &&
750 $c <= UTF8_HANGUL_TEND &&
751 $startChar >= UTF8_HANGUL_FIRST &&
752 $startChar <= UTF8_HANGUL_LAST &&
753 !$lastHangul ) {
754 # $tIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_TBASE;
755 $tIndex = ord( $c{2} ) - 0xa7;
756 if( $tIndex < 0 ) $tIndex = ord( $c{2} ) - 0x80 + (0x11c0 - 0x11a7);
757
758 # Increment the code point by $tIndex, without
759 # the function overhead of decoding and recoding UTF-8
760 #
761 $tail = ord( $startChar{2} ) + $tIndex;
762 if( $tail > 0xbf ) {
763 $tail -= 0x40;
764 $mid = ord( $startChar{1} ) + 1;
765 if( $mid > 0xbf ) {
766 $startChar{0} = chr( ord( $startChar{0} ) + 1 );
767 $mid -= 0x40;
768 }
769 $startChar{1} = chr( $mid );
770 }
771 $startChar{2} = chr( $tail );
772
773 # If there's another jamo char after this, *don't* try to merge it.
774 $lastHangul = 1;
775 continue;
776 }
777 }
778 }
779 $out .= $startChar;
780 $out .= $combining;
781 $startChar = $c;
782 $combining = '';
783 $lastClass = 0;
784 $lastHangul = 0;
785 }
786 $out .= $startChar . $combining;
788 }
References $n, $out, $utfCanonicalComp, $utfCombiningClass, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_TCOUNT, UNICODE_HANGUL_VCOUNT, UTF8_HANGUL_FIRST, UTF8_HANGUL_LAST, UTF8_HANGUL_LBASE, UTF8_HANGUL_LEND, UTF8_HANGUL_TBASE, UTF8_HANGUL_TEND, UTF8_HANGUL_VBASE, and UTF8_HANGUL_VEND.
Here is the call graph for this function:
◆ fastDecompose() [1/2]
|
staticprivate |
Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us).
Input is assumed to be valid UTF-8. Invalid code will break.
- Parameters
-
string $string Valid UTF-8 string array $map hash of expanded decomposition map
- Returns
- string a UTF-8 string decomposed, not yet normalized (needs sorting)
Definition at line 549 of file UtfNormal.php.
549 {
550 UtfNormal::loadData();
551 $len = strlen( $string );
553 for( $i = 0; $i < $len; $i++ ) {
554 $c = $string{$i};
555 $n = ord( $c );
557 # ASCII chars never decompose
558 # THEY ARE IMMORTAL
559 $out .= $c;
560 continue;
561 } elseif( $n >= 0xf0 ) {
562 $c = substr( $string, $i, 4 );
563 $i += 3;
564 } elseif( $n >= 0xe0 ) {
565 $c = substr( $string, $i, 3 );
566 $i += 2;
567 } elseif( $n >= 0xc0 ) {
568 $c = substr( $string, $i, 2 );
569 $i++;
570 }
571 if( isset( $map[$c] ) ) {
572 $out .= $map[$c];
573 continue;
574 } else {
576 # Decompose a hangul syllable into jamo;
577 # hardcoded for three-byte UTF-8 sequence.
578 # A lookup table would be slightly faster,
579 # but adds a lot of memory & disk needs.
580 #
581 $index = ( (ord( $c{0} ) & 0x0f) << 12
582 | (ord( $c{1} ) & 0x3f) << 6
583 | (ord( $c{2} ) & 0x3f) )
584 - UNICODE_HANGUL_FIRST;
591 } elseif( $t ) {
593 }
594 continue;
595 }
596 }
597 $out .= $c;
598 }
600 }
References $l, $n, $out, $t, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_NCOUNT, UNICODE_HANGUL_TCOUNT, UTF8_HANGUL_FIRST, and UTF8_HANGUL_LAST.
Here is the call graph for this function:
◆ fastDecompose() [2/2]
| UtfNormal::fastDecompose | ( | $string, | |
| & | $map | ||
| ) |
Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us).
Input is assumed to be valid UTF-8. Invalid code will break. @access private
- Parameters
-
string $string Valid UTF-8 string array $map hash of expanded decomposition map
- Returns
- string a UTF-8 string decomposed, not yet normalized (needs sorting)
Definition at line 540 of file UtfNormal.php.
540 {
541 UtfNormal::loadData();
542 $len = strlen( $string );
544 for( $i = 0; $i < $len; $i++ ) {
545 $c = $string{$i};
546 $n = ord( $c );
548 # ASCII chars never decompose
549 # THEY ARE IMMORTAL
550 $out .= $c;
551 continue;
552 } elseif( $n >= 0xf0 ) {
553 $c = substr( $string, $i, 4 );
554 $i += 3;
555 } elseif( $n >= 0xe0 ) {
556 $c = substr( $string, $i, 3 );
557 $i += 2;
558 } elseif( $n >= 0xc0 ) {
559 $c = substr( $string, $i, 2 );
560 $i++;
561 }
562 if( isset( $map[$c] ) ) {
563 $out .= $map[$c];
564 continue;
565 } else {
567 # Decompose a hangul syllable into jamo;
568 # hardcoded for three-byte UTF-8 sequence.
569 # A lookup table would be slightly faster,
570 # but adds a lot of memory & disk needs.
571 #
572 $index = ( (ord( $c{0} ) & 0x0f) << 12
573 | (ord( $c{1} ) & 0x3f) << 6
574 | (ord( $c{2} ) & 0x3f) )
575 - UNICODE_HANGUL_FIRST;
582 } elseif( $t ) {
584 }
585 continue;
586 }
587 }
588 $out .= $c;
589 }
591 }
References $l, $n, $out, $t, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_NCOUNT, UNICODE_HANGUL_TCOUNT, UTF8_HANGUL_FIRST, and UTF8_HANGUL_LAST.
Referenced by NFD(), and NFKD().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ loadData() [1/2]
| UtfNormal::loadData | ( | ) |
Load the basic composition data if necessary @access private.
Definition at line 220 of file UtfNormal.php.
220 {
223 require_once( 'UtfNormalData.inc' );
224 }
225 }
References $utfCanonicalComp, $utfCanonicalDecomp, and $utfCombiningClass.
Referenced by fastCombiningSort(), fastCompose(), fastDecompose(), NFD(), quickIsNFC(), and quickIsNFCVerify().
Here is the caller graph for this function:
◆ loadData() [2/2]
|
staticprivate |
Load the basic composition data if necessary.
Definition at line 221 of file UtfNormal.php.
221 {
222 global $utfCombiningClass;
224 require_once( dirname(__FILE__) . '/UtfNormalData.inc' );
225 }
226 }
References $utfCombiningClass.
◆ NFC() [1/2]
| UtfNormal::NFC | ( | $string | ) |
- Parameters
-
string $string
- Returns
- string @access private
Definition at line 491 of file UtfNormal.php.
491 {
493 }
References fastCompose(), and NFD().
Referenced by cleanUp(), CleanUpTest\doTestDoubleBytes(), CleanUpTest\doTestTripleBytes(), toNFC(), and CleanUpTest\XtestAllChars().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ NFC() [2/2]
|
staticprivate |
- Parameters
-
string $string
- Returns
- string
Definition at line 496 of file UtfNormal.php.
References fastCompose(), and NFD().
Here is the call graph for this function:
◆ NFD() [1/2]
| UtfNormal::NFD | ( | $string | ) |
- Parameters
-
string $string
- Returns
- string @access private
Definition at line 500 of file UtfNormal.php.
500 {
501 UtfNormal::loadData();
502 global $utfCanonicalDecomp;
505 }
fastCombiningSort( $string)
Sorts combining characters into canonical order.
Definition: UtfNormal.php:600
fastDecompose( $string, &$map)
Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition ma...
Definition: UtfNormal.php:540
References $utfCanonicalDecomp, fastCombiningSort(), fastDecompose(), and loadData().
Referenced by NFC(), and toNFD().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ NFD() [2/2]
|
staticprivate |
- Parameters
-
string $string
- Returns
- string
Definition at line 506 of file UtfNormal.php.
References $utfCanonicalDecomp, fastCombiningSort(), fastDecompose(), and loadData().
Here is the call graph for this function:
◆ NFKC() [1/2]
| UtfNormal::NFKC | ( | $string | ) |
- Parameters
-
string $string
- Returns
- string @access private
Definition at line 512 of file UtfNormal.php.
512 {
514 }
References fastCompose(), and NFKD().
Referenced by toNFKC().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ NFKC() [2/2]
|
staticprivate |
- Parameters
-
string $string
- Returns
- string
Definition at line 519 of file UtfNormal.php.
References fastCompose(), and NFKD().
Here is the call graph for this function:
◆ NFKD() [1/2]
| UtfNormal::NFKD | ( | $string | ) |
- Parameters
-
string $string
- Returns
- string @access private
Definition at line 521 of file UtfNormal.php.
References $utfCompatibilityDecomp, fastCombiningSort(), and fastDecompose().
Referenced by NFKC(), and toNFKD().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ NFKD() [2/2]
|
staticprivate |
- Parameters
-
string $string
- Returns
- string
Definition at line 529 of file UtfNormal.php.
References $utfCompatibilityDecomp, fastCombiningSort(), and fastDecompose().
Here is the call graph for this function:
◆ placebo() [1/2]
| UtfNormal::placebo | ( | $string | ) |
This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance.
- Parameters
-
string $string
- Returns
- string
Definition at line 781 of file UtfNormal.php.
781 {
782 $len = strlen( $string );
784 for( $i = 0; $i < $len; $i++ ) {
785 $out .= $string{$i};
786 }
788 }
References $out.
◆ placebo() [2/2]
|
static |
This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance.
- Parameters
-
string $string
- Returns
- string
Definition at line 797 of file UtfNormal.php.
797 {
798 $len = strlen( $string );
800 for( $i = 0; $i < $len; $i++ ) {
801 $out .= $string{$i};
802 }
804 }
References $out.
◆ quickIsNFC() [1/2]
| UtfNormal::quickIsNFC | ( | $string | ) |
Returns true if the string is definitely in NFC.
Returns false if not or uncertain.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- bool
Definition at line 233 of file UtfNormal.php.
233 {
234 # ASCII is always valid NFC!
235 # If it's pure ASCII, let it through.
236 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
237
238 UtfNormal::loadData();
239 global $utfCheckNFC, $utfCombiningClass;
240 $len = strlen( $string );
241 for( $i = 0; $i < $len; $i++ ) {
242 $c = $string{$i};
243 $n = ord( $c );
245 continue;
246 } elseif( $n >= 0xf0 ) {
247 $c = substr( $string, $i, 4 );
248 $i += 3;
249 } elseif( $n >= 0xe0 ) {
250 $c = substr( $string, $i, 3 );
251 $i += 2;
252 } elseif( $n >= 0xc0 ) {
253 $c = substr( $string, $i, 2 );
254 $i++;
255 }
256 if( isset( $utfCheckNFC[$c] ) ) {
257 # If it's NO or MAYBE, bail and do the slow check.
258 return false;
259 }
261 # Combining character? We might have to do sorting, at least.
262 return false;
263 }
264 }
265 return true;
266 }
References $n, $utfCombiningClass, and loadData().
Referenced by toNFC().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ quickIsNFC() [2/2]
|
static |
Returns true if the string is definitely in NFC.
Returns false if not or uncertain.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- bool
Definition at line 235 of file UtfNormal.php.
235 {
236 # ASCII is always valid NFC!
237 # If it's pure ASCII, let it through.
238 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
239
240 UtfNormal::loadData();
241 global $utfCheckNFC, $utfCombiningClass;
242 $len = strlen( $string );
243 for( $i = 0; $i < $len; $i++ ) {
244 $c = $string{$i};
245 $n = ord( $c );
247 continue;
248 } elseif( $n >= 0xf0 ) {
249 $c = substr( $string, $i, 4 );
250 $i += 3;
251 } elseif( $n >= 0xe0 ) {
252 $c = substr( $string, $i, 3 );
253 $i += 2;
254 } elseif( $n >= 0xc0 ) {
255 $c = substr( $string, $i, 2 );
256 $i++;
257 }
258 if( isset( $utfCheckNFC[$c] ) ) {
259 # If it's NO or MAYBE, bail and do the slow check.
260 return false;
261 }
263 # Combining character? We might have to do sorting, at least.
264 return false;
265 }
266 }
267 return true;
268 }
References $n, $utfCombiningClass, and loadData().
Here is the call graph for this function:
◆ quickIsNFCVerify() [1/2]
| UtfNormal::quickIsNFCVerify | ( | & | $string | ) |
Returns true if the string is definitely in NFC.
Returns false if not or uncertain.
- Parameters
-
string $string a UTF-8 string, altered on output to be valid UTF-8 safe for XML.
Definition at line 273 of file UtfNormal.php.
273 {
274 # Screen out some characters that eg won't be allowed in XML
276
277 # ASCII is always valid NFC!
278 # If we're only ever given plain ASCII, we can avoid the overhead
279 # of initializing the decomposition tables by skipping out early.
280 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
281
282 static $checkit = null, $tailBytes = null, $utfCheckOrCombining = null;
283 if( !isset( $checkit ) ) {
284 # Load/build some scary lookup tables...
285 UtfNormal::loadData();
286 global $utfCheckNFC, $utfCombiningClass;
287
288 $utfCheckOrCombining = array_merge( $utfCheckNFC, $utfCombiningClass );
289
290 # Head bytes for sequences which we should do further validity checks
291 $checkit = array_flip( array_map( 'chr',
292 array( 0xc0, 0xc1, 0xe0, 0xed, 0xef,
293 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
294 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff ) ) );
295
296 # Each UTF-8 head byte is followed by a certain
297 # number of tail bytes.
298 $tailBytes = array();
301 $remaining = 0;
302 } elseif( $n < 0xe0 ) {
303 $remaining = 1;
304 } elseif( $n < 0xf0 ) {
305 $remaining = 2;
306 } elseif( $n < 0xf8 ) {
307 $remaining = 3;
308 } elseif( $n < 0xfc ) {
309 $remaining = 4;
310 } elseif( $n < 0xfe ) {
311 $remaining = 5;
312 } else {
313 $remaining = 0;
314 }
315 $tailBytes[chr($n)] = $remaining;
316 }
317 }
318
319 # Chop the text into pure-ASCII and non-ASCII areas;
320 # large ASCII parts can be handled much more quickly.
321 # Don't chop up Unicode areas for punctuation, though,
322 # that wastes energy.
323 preg_match_all(
324 '/([\x00-\x7f]+|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*)/',
325 $string, $matches );
326
327 $looksNormal = true;
328 $base = 0;
329 $replace = array();
330 foreach( $matches[1] as $str ) {
331 $chunk = strlen( $str );
332
333 if( $str{0} < "\x80" ) {
334 # ASCII chunk: guaranteed to be valid UTF-8
335 # and in normal form C, so skip over it.
336 $base += $chunk;
337 continue;
338 }
339
340 # We'll have to examine the chunk byte by byte to ensure
341 # that it consists of valid UTF-8 sequences, and to see
342 # if any of them might not be normalized.
343 #
344 # Since PHP is not the fastest language on earth, some of
345 # this code is a little ugly with inner loop optimizations.
346
347 $head = '';
348 $len = $chunk + 1; # Counting down is faster. I'm *so* sorry.
349
350 for( $i = -1; --$len; ) {
351 if( $remaining = $tailBytes[$c = $str{++$i}] ) {
352 # UTF-8 head byte!
353 $sequence = $head = $c;
354 do {
355 # Look for the defined number of tail bytes...
356 if( --$len && ( $c = $str{++$i} ) >= "\x80" && $c < "\xc0" ) {
357 # Legal tail bytes are nice.
358 $sequence .= $c;
359 } else {
360 if( 0 == $len ) {
361 # Premature end of string!
362 # Drop a replacement character into output to
363 # represent the invalid UTF-8 sequence.
364 $replace[] = array( UTF8_REPLACEMENT,
365 $base + $i + 1 - strlen( $sequence ),
366 strlen( $sequence ) );
367 break 2;
368 } else {
369 # Illegal tail byte; abandon the sequence.
370 $replace[] = array( UTF8_REPLACEMENT,
371 $base + $i - strlen( $sequence ),
372 strlen( $sequence ) );
373 # Back up and reprocess this byte; it may itself
374 # be a legal ASCII or UTF-8 sequence head.
375 --$i;
376 ++$len;
377 continue 2;
378 }
379 }
380 } while( --$remaining );
381
382 if( isset( $checkit[$head] ) ) {
383 # Do some more detailed validity checks, for
384 # invalid characters and illegal sequences.
385 if( $head == "\xed" ) {
386 # 0xed is relatively frequent in Korean, which
387 # abuts the surrogate area, so we're doing
388 # this check separately to speed things up.
389
391 # Surrogates are legal only in UTF-16 code.
392 # They are totally forbidden here in UTF-8
393 # utopia.
394 $replace[] = array( UTF8_REPLACEMENT,
395 $base + $i + 1 - strlen( $sequence ),
396 strlen( $sequence ) );
397 $head = '';
398 continue;
399 }
400 } else {
401 # Slower, but rarer checks...
402 $n = ord( $head );
403 if(
404 # "Overlong sequences" are those that are syntactically
405 # correct but use more UTF-8 bytes than are necessary to
406 # encode a character. Naïve string comparisons can be
407 # tricked into failing to see a match for an ASCII
408 # character, for instance, which can be a security hole
409 # if blacklist checks are being used.
413
414 # U+FFFE and U+FFFF are explicitly forbidden in Unicode.
415 || ($n == 0xef &&
416 ($sequence == UTF8_FFFE)
417 || ($sequence == UTF8_FFFF) )
418
419 # Unicode has been limited to 21 bits; longer
420 # sequences are not allowed.
422
423 $replace[] = array( UTF8_REPLACEMENT,
424 $base + $i + 1 - strlen( $sequence ),
425 strlen( $sequence ) );
426 $head = '';
427 continue;
428 }
429 }
430 }
431
432 if( isset( $utfCheckOrCombining[$sequence] ) ) {
433 # If it's NO or MAYBE, we'll have to rip
434 # the string apart and put it back together.
435 # That's going to be mighty slow.
436 $looksNormal = false;
437 }
438
439 # The sequence is legal!
440 $head = '';
441 } elseif( $c < "\x80" ) {
442 # ASCII byte.
443 $head = '';
444 } elseif( $c < "\xc0" ) {
445 # Illegal tail bytes
446 if( $head == '' ) {
447 # Out of the blue!
448 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
449 } else {
450 # Don't add if we're continuing a broken sequence;
451 # we already put a replacement character when we looked
452 # at the broken sequence.
453 $replace[] = array( '', $base + $i, 1 );
454 }
455 } else {
456 # Miscellaneous freaks.
457 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
458 $head = '';
459 }
460 }
461 $base += $chunk;
462 }
463 if( count( $replace ) ) {
464 # There were illegal UTF-8 sequences we need to fix up.
466 $last = 0;
467 foreach( $replace as $rep ) {
468 list( $replacement, $start, $length ) = $rep;
469 if( $last < $start ) {
470 $out .= substr( $string, $last, $start - $last );
471 }
472 $out .= $replacement;
473 $last = $start + $length;
474 }
475 if( $last < strlen( $string ) ) {
476 $out .= substr( $string, $last );
477 }
478 $string = $out;
479 }
480 return $looksNormal;
481 }
References $n, $utfCombiningClass, loadData(), UTF8_FFFE, UTF8_FFFF, UTF8_MAX, UTF8_OVERLONG_A, UTF8_OVERLONG_B, UTF8_OVERLONG_C, UTF8_REPLACEMENT, and UTF8_SURROGATE_FIRST.
Referenced by cleanUp().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ quickIsNFCVerify() [2/2]
|
static |
Returns true if the string is definitely in NFC.
Returns false if not or uncertain.
- Parameters
-
string $string a UTF-8 string, altered on output to be valid UTF-8 safe for XML.
Definition at line 276 of file UtfNormal.php.
276 {
277 # Screen out some characters that eg won't be allowed in XML
279
280 # ASCII is always valid NFC!
281 # If we're only ever given plain ASCII, we can avoid the overhead
282 # of initializing the decomposition tables by skipping out early.
283 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
284
285 static $checkit = null, $tailBytes = null, $utfCheckOrCombining = null;
286 if( !isset( $checkit ) ) {
287 # Load/build some scary lookup tables...
288 UtfNormal::loadData();
289 global $utfCheckNFC, $utfCombiningClass;
290
291 $utfCheckOrCombining = array_merge( $utfCheckNFC, $utfCombiningClass );
292
293 # Head bytes for sequences which we should do further validity checks
294 $checkit = array_flip( array_map( 'chr',
295 array( 0xc0, 0xc1, 0xe0, 0xed, 0xef,
296 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
297 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff ) ) );
298
299 # Each UTF-8 head byte is followed by a certain
300 # number of tail bytes.
301 $tailBytes = array();
304 $remaining = 0;
305 } elseif( $n < 0xe0 ) {
306 $remaining = 1;
307 } elseif( $n < 0xf0 ) {
308 $remaining = 2;
309 } elseif( $n < 0xf8 ) {
310 $remaining = 3;
311 } elseif( $n < 0xfc ) {
312 $remaining = 4;
313 } elseif( $n < 0xfe ) {
314 $remaining = 5;
315 } else {
316 $remaining = 0;
317 }
318 $tailBytes[chr($n)] = $remaining;
319 }
320 }
321
322 # Chop the text into pure-ASCII and non-ASCII areas;
323 # large ASCII parts can be handled much more quickly.
324 # Don't chop up Unicode areas for punctuation, though,
325 # that wastes energy.
326 $matches = array();
327 preg_match_all(
328 '/([\x00-\x7f]+|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*)/',
329 $string, $matches );
330
331 $looksNormal = true;
332 $base = 0;
333 $replace = array();
334 foreach( $matches[1] as $str ) {
335 $chunk = strlen( $str );
336
337 if( $str{0} < "\x80" ) {
338 # ASCII chunk: guaranteed to be valid UTF-8
339 # and in normal form C, so skip over it.
340 $base += $chunk;
341 continue;
342 }
343
344 # We'll have to examine the chunk byte by byte to ensure
345 # that it consists of valid UTF-8 sequences, and to see
346 # if any of them might not be normalized.
347 #
348 # Since PHP is not the fastest language on earth, some of
349 # this code is a little ugly with inner loop optimizations.
350
351 $head = '';
352 $len = $chunk + 1; # Counting down is faster. I'm *so* sorry.
353
354 for( $i = -1; --$len; ) {
355 if( $remaining = $tailBytes[$c = $str{++$i}] ) {
356 # UTF-8 head byte!
357 $sequence = $head = $c;
358 do {
359 # Look for the defined number of tail bytes...
360 if( --$len && ( $c = $str{++$i} ) >= "\x80" && $c < "\xc0" ) {
361 # Legal tail bytes are nice.
362 $sequence .= $c;
363 } else {
364 if( 0 == $len ) {
365 # Premature end of string!
366 # Drop a replacement character into output to
367 # represent the invalid UTF-8 sequence.
368 $replace[] = array( UTF8_REPLACEMENT,
369 $base + $i + 1 - strlen( $sequence ),
370 strlen( $sequence ) );
371 break 2;
372 } else {
373 # Illegal tail byte; abandon the sequence.
374 $replace[] = array( UTF8_REPLACEMENT,
375 $base + $i - strlen( $sequence ),
376 strlen( $sequence ) );
377 # Back up and reprocess this byte; it may itself
378 # be a legal ASCII or UTF-8 sequence head.
379 --$i;
380 ++$len;
381 continue 2;
382 }
383 }
384 } while( --$remaining );
385
386 if( isset( $checkit[$head] ) ) {
387 # Do some more detailed validity checks, for
388 # invalid characters and illegal sequences.
389 if( $head == "\xed" ) {
390 # 0xed is relatively frequent in Korean, which
391 # abuts the surrogate area, so we're doing
392 # this check separately to speed things up.
393
395 # Surrogates are legal only in UTF-16 code.
396 # They are totally forbidden here in UTF-8
397 # utopia.
398 $replace[] = array( UTF8_REPLACEMENT,
399 $base + $i + 1 - strlen( $sequence ),
400 strlen( $sequence ) );
401 $head = '';
402 continue;
403 }
404 } else {
405 # Slower, but rarer checks...
406 $n = ord( $head );
407 if(
408 # "Overlong sequences" are those that are syntactically
409 # correct but use more UTF-8 bytes than are necessary to
410 # encode a character. Naïve string comparisons can be
411 # tricked into failing to see a match for an ASCII
412 # character, for instance, which can be a security hole
413 # if blacklist checks are being used.
417
418 # U+FFFE and U+FFFF are explicitly forbidden in Unicode.
419 || ($n == 0xef &&
420 ($sequence == UTF8_FFFE)
421 || ($sequence == UTF8_FFFF) )
422
423 # Unicode has been limited to 21 bits; longer
424 # sequences are not allowed.
426
427 $replace[] = array( UTF8_REPLACEMENT,
428 $base + $i + 1 - strlen( $sequence ),
429 strlen( $sequence ) );
430 $head = '';
431 continue;
432 }
433 }
434 }
435
436 if( isset( $utfCheckOrCombining[$sequence] ) ) {
437 # If it's NO or MAYBE, we'll have to rip
438 # the string apart and put it back together.
439 # That's going to be mighty slow.
440 $looksNormal = false;
441 }
442
443 # The sequence is legal!
444 $head = '';
445 } elseif( $c < "\x80" ) {
446 # ASCII byte.
447 $head = '';
448 } elseif( $c < "\xc0" ) {
449 # Illegal tail bytes
450 if( $head == '' ) {
451 # Out of the blue!
452 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
453 } else {
454 # Don't add if we're continuing a broken sequence;
455 # we already put a replacement character when we looked
456 # at the broken sequence.
457 $replace[] = array( '', $base + $i, 1 );
458 }
459 } else {
460 # Miscellaneous freaks.
461 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
462 $head = '';
463 }
464 }
465 $base += $chunk;
466 }
467 if( count( $replace ) ) {
468 # There were illegal UTF-8 sequences we need to fix up.
470 $last = 0;
471 foreach( $replace as $rep ) {
472 list( $replacement, $start, $length ) = $rep;
473 if( $last < $start ) {
474 $out .= substr( $string, $last, $start - $last );
475 }
476 $out .= $replacement;
477 $last = $start + $length;
478 }
479 if( $last < strlen( $string ) ) {
480 $out .= substr( $string, $last );
481 }
482 $string = $out;
483 }
484 return $looksNormal;
485 }
References $n, $utfCombiningClass, loadData(), UTF8_FFFE, UTF8_FFFF, UTF8_MAX, UTF8_OVERLONG_A, UTF8_OVERLONG_B, UTF8_OVERLONG_C, UTF8_REPLACEMENT, and UTF8_SURROGATE_FIRST.
Here is the call graph for this function:
◆ toNFC() [1/2]
| UtfNormal::toNFC | ( | $string | ) |
Convert a UTF-8 string to normal form C, canonical composition.
Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form C
Definition at line 157 of file UtfNormal.php.
157 {
160 elseif( UtfNormal::quickIsNFC( $string ) )
161 return $string;
162 else
164 }
References NFC(), NORMALIZE_ICU, quickIsNFC(), and UNORM_NFC.
Referenced by ilDAVServer\davDeslashify(), ilDAVServer\davUrlEncode(), ilTree\getNodePathForTitlePath(), and ilStr\normalizeUtf8String().
Here is the call graph for this function:
Here is the caller graph for this function:
◆ toNFC() [2/2]
|
static |
Convert a UTF-8 string to normal form C, canonical composition.
Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form C
Definition at line 154 of file UtfNormal.php.
References NFC(), NORMALIZE_ICU, quickIsNFC(), and UNORM_NFC.
Here is the call graph for this function:
◆ toNFD() [1/2]
| UtfNormal::toNFD | ( | $string | ) |
Convert a UTF-8 string to normal form D, canonical decomposition.
Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form D
Definition at line 173 of file UtfNormal.php.
173 {
176 elseif( preg_match( '/[\x80-\xff]/', $string ) )
178 else
179 return $string;
180 }
References NFD(), NORMALIZE_ICU, and UNORM_NFD.
Here is the call graph for this function:
◆ toNFD() [2/2]
|
static |
Convert a UTF-8 string to normal form D, canonical decomposition.
Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form D
Definition at line 171 of file UtfNormal.php.
171 {
174 elseif( preg_match( '/[\x80-\xff]/', $string ) )
176 else
177 return $string;
178 }
References NFD(), NORMALIZE_ICU, and UNORM_NFD.
Here is the call graph for this function:
◆ toNFKC() [1/2]
| UtfNormal::toNFKC | ( | $string | ) |
Convert a UTF-8 string to normal form KC, compatibility composition.
This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form KC
Definition at line 190 of file UtfNormal.php.
190 {
193 elseif( preg_match( '/[\x80-\xff]/', $string ) )
195 else
196 return $string;
197 }
References NFKC(), NORMALIZE_ICU, and UNORM_NFKC.
Here is the call graph for this function:
◆ toNFKC() [2/2]
|
static |
Convert a UTF-8 string to normal form KC, compatibility composition.
This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form KC
Definition at line 189 of file UtfNormal.php.
189 {
192 elseif( preg_match( '/[\x80-\xff]/', $string ) )
194 else
195 return $string;
196 }
References NFKC(), NORMALIZE_ICU, and UNORM_NFKC.
Here is the call graph for this function:
◆ toNFKD() [1/2]
| UtfNormal::toNFKD | ( | $string | ) |
Convert a UTF-8 string to normal form KD, compatibility decomposition.
This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form KD
Definition at line 207 of file UtfNormal.php.
207 {
210 elseif( preg_match( '/[\x80-\xff]/', $string ) )
212 else
213 return $string;
214 }
References NFKD(), NORMALIZE_ICU, and UNORM_NFKD.
Here is the call graph for this function:
◆ toNFKD() [2/2]
|
static |
Convert a UTF-8 string to normal form KD, compatibility decomposition.
This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.
- Parameters
-
string $string a valid UTF-8 string. Input is not validated.
- Returns
- string a UTF-8 string in normal form KD
Definition at line 207 of file UtfNormal.php.
207 {
210 elseif( preg_match( '/[\x80-\xff]/', $string ) )
212 else
213 return $string;
214 }
References NFKD(), NORMALIZE_ICU, and UNORM_NFKD.
Here is the call graph for this function:
The documentation for this class was generated from the following files:
- include/Unicode/UtfNormal.php
- Modules/Wiki/mediawiki/normal/UtfNormal.php
