ILIAS  release_5-2 Revision v5.2.25-18-g3f80b828510
Static Public Member Functions | Static Private Member Functions
UtfNormal Class Reference
+ Collaboration diagram for UtfNormal:

Static Public Member Functions

static cleanUp ( $string)
 The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition. More...
 
static toNFC ( $string)
 Convert a UTF-8 string to normal form C, canonical composition. More...
 
static toNFD ( $string)
 Convert a UTF-8 string to normal form D, canonical decomposition. More...
 
static toNFKC ( $string)
 Convert a UTF-8 string to normal form KC, compatibility composition. More...
 
static toNFKD ( $string)
 Convert a UTF-8 string to normal form KD, compatibility decomposition. More...
 
static quickIsNFC ( $string)
 Returns true if the string is definitely in NFC. More...
 
static quickIsNFCVerify (&$string)
 Returns true if the string is definitely in NFC. More...
 
static placebo ( $string)
 This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance. More...
 

Static Private Member Functions

static loadData ()
 Load the basic composition data if necessary. More...
 
static NFC ( $string)
 
static NFD ( $string)
 
static NFKC ( $string)
 
static NFKD ( $string)
 
static fastDecompose ( $string, $map)
 Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us). More...
 
static fastCombiningSort ( $string)
 Sorts combining characters into canonical order. More...
 
static fastCompose ( $string)
 Produces canonically composed sequences, i.e. More...
 

Detailed Description

Definition at line 112 of file UtfNormal.php.

Member Function Documentation

◆ cleanUp()

static UtfNormal::cleanUp (   $string)
static

The ultimate convenience function! Clean up invalid UTF-8 sequences, and convert to normal form C, canonical composition.

Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters. Not as fast as toNFC().

Parameters
string$stringa UTF-8 string
Returns
string a clean, shiny, normalized UTF-8 string

Definition at line 124 of file UtfNormal.php.

124 {
125 if( NORMALIZE_ICU ) {
126 # We exclude a few chars that ICU would not.
127 $string = preg_replace(
128 '/[\x00-\x08\x0b\x0c\x0e-\x1f]/',
129 UTF8_REPLACEMENT,
130 $string );
131 $string = str_replace( UTF8_FFFE, UTF8_REPLACEMENT, $string );
132 $string = str_replace( UTF8_FFFF, UTF8_REPLACEMENT, $string );
133
134 # UnicodeString constructor fails if the string ends with a
135 # head byte. Add a junk char at the end, we'll strip it off.
136 return rtrim( utf8_normalize( $string . "\x01", UNORM_NFC ), "\x01" );
137 } elseif( UtfNormal::quickIsNFCVerify( $string ) ) {
138 # Side effect -- $string has had UTF-8 errors cleaned up.
139 return $string;
140 } else {
141 return UtfNormal::NFC( $string );
142 }
143 }
UNORM_NFC
const UNORM_NFC
Definition: UtfNormal.php:91
UTF8_REPLACEMENT
const UTF8_REPLACEMENT
Definition: UtfNormal.php:68
UTF8_FFFF
const UTF8_FFFF
Definition: UtfNormal.php:79
UTF8_FFFE
const UTF8_FFFE
Definition: UtfNormal.php:78
NORMALIZE_ICU
const NORMALIZE_ICU
Definition: UtfNormal.php:96
UtfNormal\NFC
static NFC( $string)
Definition: UtfNormal.php:496
UtfNormal\quickIsNFCVerify
static quickIsNFCVerify(&$string)
Returns true if the string is definitely in NFC.
Definition: UtfNormal.php:276

References NFC(), NORMALIZE_ICU, quickIsNFCVerify(), UNORM_NFC, UTF8_FFFE, UTF8_FFFF, and UTF8_REPLACEMENT.

Referenced by CleanUpTest\doTestBytes(), CleanUpTest\doTestDoubleBytes(), CleanUpTest\doTestTripleBytes(), CleanUpTest\testAscii(), CleanUpTest\testBomRegression(), CleanUpTest\testChunkRegression(), CleanUpTest\testForbiddenRegression(), CleanUpTest\testHangulRegression(), CleanUpTest\testInterposeRegression(), CleanUpTest\testLatin(), CleanUpTest\testLatinNormal(), CleanUpTest\testNull(), CleanUpTest\testOverlongRegression(), CleanUpTest\testSurrogateRegression(), and CleanUpTest\XtestAllChars().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ fastCombiningSort()

static UtfNormal::fastCombiningSort (   $string)
staticprivate

Sorts combining characters into canonical order.

This is the final step in creating decomposed normal forms D and KD.

Parameters
string$stringa valid, decomposed UTF-8 string. Input is not validated.
Returns
string a UTF-8 string with combining characters sorted in canonical order

Definition at line 610 of file UtfNormal.php.

610 {
611 UtfNormal::loadData();
612 global $utfCombiningClass;
613 $len = strlen( $string );
614 $out = '';
615 $combiners = array();
616 $lastClass = -1;
617 for( $i = 0; $i < $len; $i++ ) {
618 $c = $string{$i};
619 $n = ord( $c );
620 if( $n >= 0x80 ) {
621 if( $n >= 0xf0 ) {
622 $c = substr( $string, $i, 4 );
623 $i += 3;
624 } elseif( $n >= 0xe0 ) {
625 $c = substr( $string, $i, 3 );
626 $i += 2;
627 } elseif( $n >= 0xc0 ) {
628 $c = substr( $string, $i, 2 );
629 $i++;
630 }
631 if( isset( $utfCombiningClass[$c] ) ) {
632 $lastClass = $utfCombiningClass[$c];
633 if( isset( $combiners[$lastClass] ) ) {
634 $combiners[$lastClass] .= $c;
635 } else {
636 $combiners[$lastClass] = $c;
637 }
638 continue;
639 }
640 }
641 if( $lastClass ) {
642 ksort( $combiners );
643 $out .= implode( '', $combiners );
644 $combiners = array();
645 }
646 $out .= $c;
647 $lastClass = 0;
648 }
649 if( $lastClass ) {
650 ksort( $combiners );
651 $out .= implode( '', $combiners );
652 }
653 return $out;
654 }
$n
$n
Definition: RandomTest.php:80
$utfCombiningClass
global $utfCombiningClass
Definition: UtfNormal.php:23
UtfNormal\loadData
static loadData()
Load the basic composition data if necessary.
Definition: UtfNormal.php:221

References $n, $out, $utfCombiningClass, and loadData().

Referenced by NFD(), and NFKD().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ fastCompose()

static UtfNormal::fastCompose (   $string)
staticprivate

Produces canonically composed sequences, i.e.

normal form C or KC.

Parameters
string$stringa valid UTF-8 string in sorted normal form D or KD. Input is not validated.
Returns
string a UTF-8 string with canonical precomposed characters used where possible

Definition at line 664 of file UtfNormal.php.

664 {
665 UtfNormal::loadData();
666 global $utfCanonicalComp, $utfCombiningClass;
667 $len = strlen( $string );
668 $out = '';
669 $lastClass = -1;
670 $lastHangul = 0;
671 $startChar = '';
672 $combining = '';
673 $x1 = ord(substr(UTF8_HANGUL_VBASE,0,1));
674 $x2 = ord(substr(UTF8_HANGUL_TEND,0,1));
675 for( $i = 0; $i < $len; $i++ ) {
676 $c = $string{$i};
677 $n = ord( $c );
678 if( $n < 0x80 ) {
679 # No combining characters here...
680 $out .= $startChar;
681 $out .= $combining;
682 $startChar = $c;
683 $combining = '';
684 $lastClass = 0;
685 continue;
686 } elseif( $n >= 0xf0 ) {
687 $c = substr( $string, $i, 4 );
688 $i += 3;
689 } elseif( $n >= 0xe0 ) {
690 $c = substr( $string, $i, 3 );
691 $i += 2;
692 } elseif( $n >= 0xc0 ) {
693 $c = substr( $string, $i, 2 );
694 $i++;
695 }
696 $pair = $startChar . $c;
697 if( $n > 0x80 ) {
698 if( isset( $utfCombiningClass[$c] ) ) {
699 # A combining char; see what we can do with it
700 $class = $utfCombiningClass[$c];
701 if( !empty( $startChar ) &&
702 $lastClass < $class &&
703 $class > 0 &&
704 isset( $utfCanonicalComp[$pair] ) ) {
705 $startChar = $utfCanonicalComp[$pair];
706 $class = 0;
707 } else {
708 $combining .= $c;
709 }
710 $lastClass = $class;
711 $lastHangul = 0;
712 continue;
713 }
714 }
715 # New start char
716 if( $lastClass == 0 ) {
717 if( isset( $utfCanonicalComp[$pair] ) ) {
718 $startChar = $utfCanonicalComp[$pair];
719 $lastHangul = 0;
720 continue;
721 }
722 if( $n >= $x1 && $n <= $x2 ) {
723 # WARNING: Hangul code is painfully slow.
724 # I apologize for this ugly, ugly code; however
725 # performance is even more teh suck if we call
726 # out to nice clean functions. Lookup tables are
727 # marginally faster, but require a lot of space.
728 #
729 if( $c >= UTF8_HANGUL_VBASE &&
730 $c <= UTF8_HANGUL_VEND &&
731 $startChar >= UTF8_HANGUL_LBASE &&
732 $startChar <= UTF8_HANGUL_LEND ) {
733 #
734 #$lIndex = utf8ToCodepoint( $startChar ) - UNICODE_HANGUL_LBASE;
735 #$vIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_VBASE;
736 $lIndex = ord( $startChar{2} ) - 0x80;
737 $vIndex = ord( $c{2} ) - 0xa1;
738
739 $hangulPoint = UNICODE_HANGUL_FIRST +
740 UNICODE_HANGUL_TCOUNT *
741 (UNICODE_HANGUL_VCOUNT * $lIndex + $vIndex);
742
743 # Hardcode the limited-range UTF-8 conversion:
744 $startChar = chr( $hangulPoint >> 12 & 0x0f | 0xe0 ) .
745 chr( $hangulPoint >> 6 & 0x3f | 0x80 ) .
746 chr( $hangulPoint & 0x3f | 0x80 );
747 $lastHangul = 0;
748 continue;
749 } elseif( $c >= UTF8_HANGUL_TBASE &&
750 $c <= UTF8_HANGUL_TEND &&
751 $startChar >= UTF8_HANGUL_FIRST &&
752 $startChar <= UTF8_HANGUL_LAST &&
753 !$lastHangul ) {
754 # $tIndex = utf8ToCodepoint( $c ) - UNICODE_HANGUL_TBASE;
755 $tIndex = ord( $c{2} ) - 0xa7;
756 if( $tIndex < 0 ) $tIndex = ord( $c{2} ) - 0x80 + (0x11c0 - 0x11a7);
757
758 # Increment the code point by $tIndex, without
759 # the function overhead of decoding and recoding UTF-8
760 #
761 $tail = ord( $startChar{2} ) + $tIndex;
762 if( $tail > 0xbf ) {
763 $tail -= 0x40;
764 $mid = ord( $startChar{1} ) + 1;
765 if( $mid > 0xbf ) {
766 $startChar{0} = chr( ord( $startChar{0} ) + 1 );
767 $mid -= 0x40;
768 }
769 $startChar{1} = chr( $mid );
770 }
771 $startChar{2} = chr( $tail );
772
773 # If there's another jamo char after this, *don't* try to merge it.
774 $lastHangul = 1;
775 continue;
776 }
777 }
778 }
779 $out .= $startChar;
780 $out .= $combining;
781 $startChar = $c;
782 $combining = '';
783 $lastClass = 0;
784 $lastHangul = 0;
785 }
786 $out .= $startChar . $combining;
787 return $out;
788 }
UTF8_HANGUL_TEND
const UTF8_HANGUL_TEND
Definition: UtfNormal.php:63
UNICODE_HANGUL_VCOUNT
const UNICODE_HANGUL_VCOUNT
Definition: UtfNormal.php:40
UTF8_HANGUL_LEND
const UTF8_HANGUL_LEND
Definition: UtfNormal.php:61
UTF8_HANGUL_TBASE
const UTF8_HANGUL_TBASE
Definition: UtfNormal.php:59
UTF8_HANGUL_VEND
const UTF8_HANGUL_VEND
Definition: UtfNormal.php:62
UNICODE_HANGUL_TCOUNT
const UNICODE_HANGUL_TCOUNT
Definition: UtfNormal.php:41
UNICODE_HANGUL_FIRST
const UNICODE_HANGUL_FIRST
Definition: UtfNormal.php:32
UTF8_HANGUL_VBASE
const UTF8_HANGUL_VBASE
Definition: UtfNormal.php:58
UTF8_HANGUL_LAST
const UTF8_HANGUL_LAST
Definition: UtfNormal.php:55
UTF8_HANGUL_LBASE
const UTF8_HANGUL_LBASE
Definition: UtfNormal.php:57
$utfCanonicalComp
global $utfCanonicalComp
Definition: UtfNormal.php:23
UTF8_HANGUL_FIRST
const UTF8_HANGUL_FIRST
Definition: UtfNormal.php:54

References $n, $out, $utfCanonicalComp, $utfCombiningClass, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_TCOUNT, UNICODE_HANGUL_VCOUNT, UTF8_HANGUL_FIRST, UTF8_HANGUL_LAST, UTF8_HANGUL_LBASE, UTF8_HANGUL_LEND, UTF8_HANGUL_TBASE, UTF8_HANGUL_TEND, UTF8_HANGUL_VBASE, and UTF8_HANGUL_VEND.

Referenced by NFC(), and NFKC().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ fastDecompose()

static UtfNormal::fastDecompose (   $string,
  $map 
)
staticprivate

Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition map is passed to us).

Input is assumed to be valid UTF-8. Invalid code will break.

Parameters
string$stringValid UTF-8 string
array$maphash of expanded decomposition map
Returns
string a UTF-8 string decomposed, not yet normalized (needs sorting)

Definition at line 549 of file UtfNormal.php.

549 {
550 UtfNormal::loadData();
551 $len = strlen( $string );
552 $out = '';
553 for( $i = 0; $i < $len; $i++ ) {
554 $c = $string{$i};
555 $n = ord( $c );
556 if( $n < 0x80 ) {
557 # ASCII chars never decompose
558 # THEY ARE IMMORTAL
559 $out .= $c;
560 continue;
561 } elseif( $n >= 0xf0 ) {
562 $c = substr( $string, $i, 4 );
563 $i += 3;
564 } elseif( $n >= 0xe0 ) {
565 $c = substr( $string, $i, 3 );
566 $i += 2;
567 } elseif( $n >= 0xc0 ) {
568 $c = substr( $string, $i, 2 );
569 $i++;
570 }
571 if( isset( $map[$c] ) ) {
572 $out .= $map[$c];
573 continue;
574 } else {
575 if( $c >= UTF8_HANGUL_FIRST && $c <= UTF8_HANGUL_LAST ) {
576 # Decompose a hangul syllable into jamo;
577 # hardcoded for three-byte UTF-8 sequence.
578 # A lookup table would be slightly faster,
579 # but adds a lot of memory & disk needs.
580 #
581 $index = ( (ord( $c{0} ) & 0x0f) << 12
582 | (ord( $c{1} ) & 0x3f) << 6
583 | (ord( $c{2} ) & 0x3f) )
584 - UNICODE_HANGUL_FIRST;
585 $l = intval( $index / UNICODE_HANGUL_NCOUNT );
586 $v = intval( ($index % UNICODE_HANGUL_NCOUNT) / UNICODE_HANGUL_TCOUNT);
587 $t = $index % UNICODE_HANGUL_TCOUNT;
588 $out .= "\xe1\x84" . chr( 0x80 + $l ) . "\xe1\x85" . chr( 0xa1 + $v );
589 if( $t >= 25 ) {
590 $out .= "\xe1\x87" . chr( 0x80 + $t - 25 );
591 } elseif( $t ) {
592 $out .= "\xe1\x86" . chr( 0xa7 + $t );
593 }
594 continue;
595 }
596 }
597 $out .= $c;
598 }
599 return $out;
600 }
UNICODE_HANGUL_NCOUNT
const UNICODE_HANGUL_NCOUNT
Definition: UtfNormal.php:42
$l
global $l
Definition: afr.php:30

References $l, $n, $out, $t, loadData(), UNICODE_HANGUL_FIRST, UNICODE_HANGUL_NCOUNT, UNICODE_HANGUL_TCOUNT, UTF8_HANGUL_FIRST, and UTF8_HANGUL_LAST.

Referenced by NFD(), and NFKD().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ loadData()

static UtfNormal::loadData ( )
staticprivate

Load the basic composition data if necessary.

Definition at line 221 of file UtfNormal.php.

221 {
222 global $utfCombiningClass;
223 if( !isset( $utfCombiningClass ) ) {
224 require_once( dirname(__FILE__) . '/UtfNormalData.inc' );
225 }
226 }

References $utfCombiningClass.

Referenced by fastCombiningSort(), fastCompose(), fastDecompose(), NFD(), quickIsNFC(), and quickIsNFCVerify().

+ Here is the caller graph for this function:

◆ NFC()

static UtfNormal::NFC (   $string)
staticprivate
Parameters
string$string
Returns
string

Definition at line 496 of file UtfNormal.php.

496 {
497 return UtfNormal::fastCompose( UtfNormal::NFD( $string ) );
498 }
UtfNormal\NFD
static NFD( $string)
Definition: UtfNormal.php:506
UtfNormal\fastCompose
static fastCompose( $string)
Produces canonically composed sequences, i.e.
Definition: UtfNormal.php:664

References fastCompose(), and NFD().

Referenced by cleanUp(), CleanUpTest\doTestDoubleBytes(), CleanUpTest\doTestTripleBytes(), toNFC(), and CleanUpTest\XtestAllChars().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ NFD()

static UtfNormal::NFD (   $string)
staticprivate
Parameters
string$string
Returns
string

Definition at line 506 of file UtfNormal.php.

506 {
507 UtfNormal::loadData();
508 global $utfCanonicalDecomp;
509 return UtfNormal::fastCombiningSort(
510 UtfNormal::fastDecompose( $string, $utfCanonicalDecomp ) );
511 }
$utfCanonicalDecomp
global $utfCanonicalDecomp
Definition: UtfNormal.php:23
UtfNormal\fastCombiningSort
static fastCombiningSort( $string)
Sorts combining characters into canonical order.
Definition: UtfNormal.php:610
UtfNormal\fastDecompose
static fastDecompose( $string, $map)
Perform decomposition of a UTF-8 string into either D or KD form (depending on which decomposition ma...
Definition: UtfNormal.php:549

References $utfCanonicalDecomp, fastCombiningSort(), fastDecompose(), and loadData().

Referenced by NFC(), and toNFD().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ NFKC()

static UtfNormal::NFKC (   $string)
staticprivate
Parameters
string$string
Returns
string

Definition at line 519 of file UtfNormal.php.

519 {
520 return UtfNormal::fastCompose( UtfNormal::NFKD( $string ) );
521 }
UtfNormal\NFKD
static NFKD( $string)
Definition: UtfNormal.php:529

References fastCompose(), and NFKD().

Referenced by toNFKC().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ NFKD()

static UtfNormal::NFKD (   $string)
staticprivate
Parameters
string$string
Returns
string

Definition at line 529 of file UtfNormal.php.

529 {
530 global $utfCompatibilityDecomp;
531 if( !isset( $utfCompatibilityDecomp ) ) {
532 require_once( 'UtfNormalDataK.inc' );
533 }
534 return UtfNormal::fastCombiningSort(
535 UtfNormal::fastDecompose( $string, $utfCompatibilityDecomp ) );
536 }
$utfCompatibilityDecomp
global $utfCompatibilityDecomp
Definition: UtfNormal.php:29

References $utfCompatibilityDecomp, fastCombiningSort(), and fastDecompose().

Referenced by NFKC(), and toNFKD().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ placebo()

static UtfNormal::placebo (   $string)
static

This is just used for the benchmark, comparing how long it takes to interate through a string without really doing anything of substance.

Parameters
string$string
Returns
string

Definition at line 797 of file UtfNormal.php.

797 {
798 $len = strlen( $string );
799 $out = '';
800 for( $i = 0; $i < $len; $i++ ) {
801 $out .= $string{$i};
802 }
803 return $out;
804 }

References $out.

◆ quickIsNFC()

static UtfNormal::quickIsNFC (   $string)
static

Returns true if the string is definitely in NFC.

Returns false if not or uncertain.

Parameters
string$stringa valid UTF-8 string. Input is not validated.
Returns
bool

Definition at line 235 of file UtfNormal.php.

235 {
236 # ASCII is always valid NFC!
237 # If it's pure ASCII, let it through.
238 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
239
240 UtfNormal::loadData();
241 global $utfCheckNFC, $utfCombiningClass;
242 $len = strlen( $string );
243 for( $i = 0; $i < $len; $i++ ) {
244 $c = $string{$i};
245 $n = ord( $c );
246 if( $n < 0x80 ) {
247 continue;
248 } elseif( $n >= 0xf0 ) {
249 $c = substr( $string, $i, 4 );
250 $i += 3;
251 } elseif( $n >= 0xe0 ) {
252 $c = substr( $string, $i, 3 );
253 $i += 2;
254 } elseif( $n >= 0xc0 ) {
255 $c = substr( $string, $i, 2 );
256 $i++;
257 }
258 if( isset( $utfCheckNFC[$c] ) ) {
259 # If it's NO or MAYBE, bail and do the slow check.
260 return false;
261 }
262 if( isset( $utfCombiningClass[$c] ) ) {
263 # Combining character? We might have to do sorting, at least.
264 return false;
265 }
266 }
267 return true;
268 }

References $n, $utfCombiningClass, and loadData().

Referenced by toNFC().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ quickIsNFCVerify()

static UtfNormal::quickIsNFCVerify ( &  $string)
static

Returns true if the string is definitely in NFC.

Returns false if not or uncertain.

Parameters
string$stringa UTF-8 string, altered on output to be valid UTF-8 safe for XML.

Definition at line 276 of file UtfNormal.php.

276 {
277 # Screen out some characters that eg won't be allowed in XML
278 $string = preg_replace( '/[\x00-\x08\x0b\x0c\x0e-\x1f]/', UTF8_REPLACEMENT, $string );
279
280 # ASCII is always valid NFC!
281 # If we're only ever given plain ASCII, we can avoid the overhead
282 # of initializing the decomposition tables by skipping out early.
283 if( !preg_match( '/[\x80-\xff]/', $string ) ) return true;
284
285 static $checkit = null, $tailBytes = null, $utfCheckOrCombining = null;
286 if( !isset( $checkit ) ) {
287 # Load/build some scary lookup tables...
288 UtfNormal::loadData();
289 global $utfCheckNFC, $utfCombiningClass;
290
291 $utfCheckOrCombining = array_merge( $utfCheckNFC, $utfCombiningClass );
292
293 # Head bytes for sequences which we should do further validity checks
294 $checkit = array_flip( array_map( 'chr',
295 array( 0xc0, 0xc1, 0xe0, 0xed, 0xef,
296 0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
297 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff ) ) );
298
299 # Each UTF-8 head byte is followed by a certain
300 # number of tail bytes.
301 $tailBytes = array();
302 for( $n = 0; $n < 256; $n++ ) {
303 if( $n < 0xc0 ) {
304 $remaining = 0;
305 } elseif( $n < 0xe0 ) {
306 $remaining = 1;
307 } elseif( $n < 0xf0 ) {
308 $remaining = 2;
309 } elseif( $n < 0xf8 ) {
310 $remaining = 3;
311 } elseif( $n < 0xfc ) {
312 $remaining = 4;
313 } elseif( $n < 0xfe ) {
314 $remaining = 5;
315 } else {
316 $remaining = 0;
317 }
318 $tailBytes[chr($n)] = $remaining;
319 }
320 }
321
322 # Chop the text into pure-ASCII and non-ASCII areas;
323 # large ASCII parts can be handled much more quickly.
324 # Don't chop up Unicode areas for punctuation, though,
325 # that wastes energy.
326 $matches = array();
327 preg_match_all(
328 '/([\x00-\x7f]+|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*)/',
329 $string, $matches );
330
331 $looksNormal = true;
332 $base = 0;
333 $replace = array();
334 foreach( $matches[1] as $str ) {
335 $chunk = strlen( $str );
336
337 if( $str{0} < "\x80" ) {
338 # ASCII chunk: guaranteed to be valid UTF-8
339 # and in normal form C, so skip over it.
340 $base += $chunk;
341 continue;
342 }
343
344 # We'll have to examine the chunk byte by byte to ensure
345 # that it consists of valid UTF-8 sequences, and to see
346 # if any of them might not be normalized.
347 #
348 # Since PHP is not the fastest language on earth, some of
349 # this code is a little ugly with inner loop optimizations.
350
351 $head = '';
352 $len = $chunk + 1; # Counting down is faster. I'm *so* sorry.
353
354 for( $i = -1; --$len; ) {
355 if( $remaining = $tailBytes[$c = $str{++$i}] ) {
356 # UTF-8 head byte!
357 $sequence = $head = $c;
358 do {
359 # Look for the defined number of tail bytes...
360 if( --$len && ( $c = $str{++$i} ) >= "\x80" && $c < "\xc0" ) {
361 # Legal tail bytes are nice.
362 $sequence .= $c;
363 } else {
364 if( 0 == $len ) {
365 # Premature end of string!
366 # Drop a replacement character into output to
367 # represent the invalid UTF-8 sequence.
368 $replace[] = array( UTF8_REPLACEMENT,
369 $base + $i + 1 - strlen( $sequence ),
370 strlen( $sequence ) );
371 break 2;
372 } else {
373 # Illegal tail byte; abandon the sequence.
374 $replace[] = array( UTF8_REPLACEMENT,
375 $base + $i - strlen( $sequence ),
376 strlen( $sequence ) );
377 # Back up and reprocess this byte; it may itself
378 # be a legal ASCII or UTF-8 sequence head.
379 --$i;
380 ++$len;
381 continue 2;
382 }
383 }
384 } while( --$remaining );
385
386 if( isset( $checkit[$head] ) ) {
387 # Do some more detailed validity checks, for
388 # invalid characters and illegal sequences.
389 if( $head == "\xed" ) {
390 # 0xed is relatively frequent in Korean, which
391 # abuts the surrogate area, so we're doing
392 # this check separately to speed things up.
393
394 if( $sequence >= UTF8_SURROGATE_FIRST ) {
395 # Surrogates are legal only in UTF-16 code.
396 # They are totally forbidden here in UTF-8
397 # utopia.
398 $replace[] = array( UTF8_REPLACEMENT,
399 $base + $i + 1 - strlen( $sequence ),
400 strlen( $sequence ) );
401 $head = '';
402 continue;
403 }
404 } else {
405 # Slower, but rarer checks...
406 $n = ord( $head );
407 if(
408 # "Overlong sequences" are those that are syntactically
409 # correct but use more UTF-8 bytes than are necessary to
410 # encode a character. Naïve string comparisons can be
411 # tricked into failing to see a match for an ASCII
412 # character, for instance, which can be a security hole
413 # if blacklist checks are being used.
414 ($n < 0xc2 && $sequence <= UTF8_OVERLONG_A)
415 || ($n == 0xe0 && $sequence <= UTF8_OVERLONG_B)
416 || ($n == 0xf0 && $sequence <= UTF8_OVERLONG_C)
417
418 # U+FFFE and U+FFFF are explicitly forbidden in Unicode.
419 || ($n == 0xef &&
420 ($sequence == UTF8_FFFE)
421 || ($sequence == UTF8_FFFF) )
422
423 # Unicode has been limited to 21 bits; longer
424 # sequences are not allowed.
425 || ($n >= 0xf0 && $sequence > UTF8_MAX) ) {
426
427 $replace[] = array( UTF8_REPLACEMENT,
428 $base + $i + 1 - strlen( $sequence ),
429 strlen( $sequence ) );
430 $head = '';
431 continue;
432 }
433 }
434 }
435
436 if( isset( $utfCheckOrCombining[$sequence] ) ) {
437 # If it's NO or MAYBE, we'll have to rip
438 # the string apart and put it back together.
439 # That's going to be mighty slow.
440 $looksNormal = false;
441 }
442
443 # The sequence is legal!
444 $head = '';
445 } elseif( $c < "\x80" ) {
446 # ASCII byte.
447 $head = '';
448 } elseif( $c < "\xc0" ) {
449 # Illegal tail bytes
450 if( $head == '' ) {
451 # Out of the blue!
452 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
453 } else {
454 # Don't add if we're continuing a broken sequence;
455 # we already put a replacement character when we looked
456 # at the broken sequence.
457 $replace[] = array( '', $base + $i, 1 );
458 }
459 } else {
460 # Miscellaneous freaks.
461 $replace[] = array( UTF8_REPLACEMENT, $base + $i, 1 );
462 $head = '';
463 }
464 }
465 $base += $chunk;
466 }
467 if( count( $replace ) ) {
468 # There were illegal UTF-8 sequences we need to fix up.
469 $out = '';
470 $last = 0;
471 foreach( $replace as $rep ) {
472 list( $replacement, $start, $length ) = $rep;
473 if( $last < $start ) {
474 $out .= substr( $string, $last, $start - $last );
475 }
476 $out .= $replacement;
477 $last = $start + $length;
478 }
479 if( $last < strlen( $string ) ) {
480 $out .= substr( $string, $last );
481 }
482 $string = $out;
483 }
484 return $looksNormal;
485 }
UTF8_OVERLONG_C
const UTF8_OVERLONG_C
Definition: UtfNormal.php:73
UTF8_MAX
const UTF8_MAX
Definition: UtfNormal.php:67
UTF8_SURROGATE_FIRST
const UTF8_SURROGATE_FIRST
Definition: UtfNormal.php:65
UTF8_OVERLONG_A
const UTF8_OVERLONG_A
Definition: UtfNormal.php:71
UTF8_OVERLONG_B
const UTF8_OVERLONG_B
Definition: UtfNormal.php:72
down
down()
Definition: down.php:2

References $n, $utfCombiningClass, down(), loadData(), UTF8_FFFE, UTF8_FFFF, UTF8_MAX, UTF8_OVERLONG_A, UTF8_OVERLONG_B, UTF8_OVERLONG_C, UTF8_REPLACEMENT, and UTF8_SURROGATE_FIRST.

Referenced by cleanUp().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ toNFC()

static UtfNormal::toNFC (   $string)
static

Convert a UTF-8 string to normal form C, canonical composition.

Fast return for pure ASCII strings; some lesser optimizations for strings containing only known-good characters.

Parameters
string$stringa valid UTF-8 string. Input is not validated.
Returns
string a UTF-8 string in normal form C

Definition at line 154 of file UtfNormal.php.

154 {
155 if( NORMALIZE_ICU )
156 return utf8_normalize( $string, UNORM_NFC );
157 elseif( UtfNormal::quickIsNFC( $string ) )
158 return $string;
159 else
160 return UtfNormal::NFC( $string );
161 }
UtfNormal\quickIsNFC
static quickIsNFC( $string)
Returns true if the string is definitely in NFC.
Definition: UtfNormal.php:235

References NFC(), NORMALIZE_ICU, quickIsNFC(), and UNORM_NFC.

Referenced by ilDAVServer\davDeslashify(), ilDAVServer\davUrlEncode(), ilTree\getNodePathForTitlePath(), and ilStr\normalizeUtf8String().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ toNFD()

static UtfNormal::toNFD (   $string)
static

Convert a UTF-8 string to normal form D, canonical decomposition.

Fast return for pure ASCII strings.

Parameters
string$stringa valid UTF-8 string. Input is not validated.
Returns
string a UTF-8 string in normal form D

Definition at line 171 of file UtfNormal.php.

171 {
172 if( NORMALIZE_ICU )
173 return utf8_normalize( $string, UNORM_NFD );
174 elseif( preg_match( '/[\x80-\xff]/', $string ) )
175 return UtfNormal::NFD( $string );
176 else
177 return $string;
178 }
UNORM_NFD
const UNORM_NFD
Definition: UtfNormal.php:89

References NFD(), NORMALIZE_ICU, and UNORM_NFD.

+ Here is the call graph for this function:

◆ toNFKC()

static UtfNormal::toNFKC (   $string)
static

Convert a UTF-8 string to normal form KC, compatibility composition.

This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.

Parameters
string$stringa valid UTF-8 string. Input is not validated.
Returns
string a UTF-8 string in normal form KC

Definition at line 189 of file UtfNormal.php.

189 {
190 if( NORMALIZE_ICU )
191 return utf8_normalize( $string, UNORM_NFKC );
192 elseif( preg_match( '/[\x80-\xff]/', $string ) )
193 return UtfNormal::NFKC( $string );
194 else
195 return $string;
196 }
UNORM_NFKC
const UNORM_NFKC
Definition: UtfNormal.php:93
UtfNormal\NFKC
static NFKC( $string)
Definition: UtfNormal.php:519

References NFKC(), NORMALIZE_ICU, and UNORM_NFKC.

+ Here is the call graph for this function:

◆ toNFKD()

static UtfNormal::toNFKD (   $string)
static

Convert a UTF-8 string to normal form KD, compatibility decomposition.

This may cause irreversible information loss, use judiciously. Fast return for pure ASCII strings.

Parameters
string$stringa valid UTF-8 string. Input is not validated.
Returns
string a UTF-8 string in normal form KD

Definition at line 207 of file UtfNormal.php.

207 {
208 if( NORMALIZE_ICU )
209 return utf8_normalize( $string, UNORM_NFKD );
210 elseif( preg_match( '/[\x80-\xff]/', $string ) )
211 return UtfNormal::NFKD( $string );
212 else
213 return $string;
214 }
UNORM_NFKD
const UNORM_NFKD
Definition: UtfNormal.php:90

References NFKD(), NORMALIZE_ICU, and UNORM_NFKD.

+ Here is the call graph for this function:
The documentation for this class was generated from the following file: