ILIAS  release_5-1 Revision 5.0.0-5477-g43f3e3fab5f
HTMLPurifier_Lexer Class Reference

Forgivingly lexes HTML (SGML-style) markup into tokens. More...

+ Inheritance diagram for HTMLPurifier_Lexer:
+ Collaboration diagram for HTMLPurifier_Lexer:

Public Member Functions

 __construct ()
 
 parseData ($string)
 Parses special entities into the proper characters. More...
 
 tokenizeHTML ($string, $config, $context)
 Lexes an HTML string into tokens. More...
 
 normalize ($html, $config, $context)
 Takes a piece of HTML and normalizes it by converting entities, fixing encoding, extracting bits, and other good stuff. More...
 
 extractBody ($html)
 Takes a string of HTML (fragment or document) and returns the content. More...
 

Static Public Member Functions

static create ($config)
 Retrieves or sets the default Lexer as a Prototype Factory. More...
 

Data Fields

 $tracksLineNumbers = false
 Whether or not this lexer implements line-number/column-number tracking. More...
 

Static Protected Member Functions

static escapeCDATA ($string)
 Translates CDATA sections into regular sections (through escaping). More...
 
static escapeCommentedCDATA ($string)
 Special CDATA case that is especially convoluted for <script> More...
 
static removeIEConditional ($string)
 Special Internet Explorer conditional comments should be removed. More...
 
static CDATACallback ($matches)
 Callback function for escapeCDATA() that does the work. More...
 

Protected Attributes

 $_special_entity2str
 Most common entity to raw value conversion table for special entities. More...
 

Detailed Description

Forgivingly lexes HTML (SGML-style) markup into tokens.

A lexer parses a string of SGML-style markup and converts them into corresponding tokens. It doesn't check for well-formedness, although its internal mechanism may make this automatic (such as the case of HTMLPurifier_Lexer_DOMLex). There are several implementations to choose from.

A lexer is HTML-oriented: it might work with XML, but it's not recommended, as we adhere to a subset of the specification for optimization reasons. This might change in the future. Also, most tokenizers are not expected to handle DTDs or PIs.

This class should not be directly instantiated, but you may use create() to retrieve a default copy of the lexer. Being a supertype, this class does not actually define any implementation, but offers commonly used convenience functions for subclasses.

Note
The unit tests will instantiate this class for testing purposes, as many of the utility functions require a class to be instantiated. This means that, even though this class is not runnable, it will not be declared abstract.
Note
We use tokens rather than create a DOM representation because DOM would:
  1. Require more processing and memory to create,
  2. Is not streamable, and
  3. Has the entire document structure (html and body not needed).
However, DOM is helpful in that it makes it easy to move around nodes without a lot of lookaheads to see when a tag is closed. This is a limitation of the token system and some workarounds would be nice.

Definition at line 42 of file Lexer.php.

Constructor & Destructor Documentation

◆ __construct()

HTMLPurifier_Lexer::__construct ( )

Reimplemented in HTMLPurifier_Lexer_DOMLex.

Definition at line 152 of file Lexer.php.

153 {
154 $this->_entity_parser = new HTMLPurifier_EntityParser();
155 }
Handles referencing and derefencing character entities.

Member Function Documentation

◆ CDATACallback()

static HTMLPurifier_Lexer::CDATACallback (   $matches)
staticprotected

Callback function for escapeCDATA() that does the work.

Warning
Though this is public in order to let the callback happen, calling it directly is not recommended.
Parameters
array$matchesPCRE matches array, with index 0 the entire match and 1 the inside of the CDATA section.
Returns
string Escaped internals of the CDATA section.

Definition at line 279 of file Lexer.php.

280 {
281 // not exactly sure why the character set is needed, but whatever
282 return htmlspecialchars($matches[1], ENT_COMPAT, 'UTF-8');
283 }

◆ create()

static HTMLPurifier_Lexer::create (   $config)
static

Retrieves or sets the default Lexer as a Prototype Factory.

By default HTMLPurifier_Lexer_DOMLex will be returned. There are a few exceptions involving special features that only DirectLex implements.

Note
The behavior of this class has changed, rather than accepting a prototype object, it now accepts a configuration object. To specify your own prototype, set Core.LexerImpl to it. This change in behavior de-singletonizes the lexer object.
Parameters
HTMLPurifier_Config$config
Returns
HTMLPurifier_Lexer
Exceptions
HTMLPurifier_Exception

Definition at line 69 of file Lexer.php.

70 {
71 if (!($config instanceof HTMLPurifier_Config)) {
72 $lexer = $config;
73 trigger_error(
74 "Passing a prototype to
75 HTMLPurifier_Lexer::create() is deprecated, please instead
76 use %Core.LexerImpl",
77 E_USER_WARNING
78 );
79 } else {
80 $lexer = $config->get('Core.LexerImpl');
81 }
82
83 $needs_tracking =
84 $config->get('Core.MaintainLineNumbers') ||
85 $config->get('Core.CollectErrors');
86
87 $inst = null;
88 if (is_object($lexer)) {
89 $inst = $lexer;
90 } else {
91 if (is_null($lexer)) {
92 do {
93 // auto-detection algorithm
94 if ($needs_tracking) {
95 $lexer = 'DirectLex';
96 break;
97 }
98
99 if (class_exists('DOMDocument') &&
100 method_exists('DOMDocument', 'loadHTML') &&
101 !extension_loaded('domxml')
102 ) {
103 // check for DOM support, because while it's part of the
104 // core, it can be disabled compile time. Also, the PECL
105 // domxml extension overrides the default DOM, and is evil
106 // and nasty and we shan't bother to support it
107 $lexer = 'DOMLex';
108 } else {
109 $lexer = 'DirectLex';
110 }
111 } while (0);
112 } // do..while so we can break
113
114 // instantiate recognized string names
115 switch ($lexer) {
116 case 'DOMLex':
117 $inst = new HTMLPurifier_Lexer_DOMLex();
118 break;
119 case 'DirectLex':
120 $inst = new HTMLPurifier_Lexer_DirectLex();
121 break;
122 case 'PH5P':
123 $inst = new HTMLPurifier_Lexer_PH5P();
124 break;
125 default:
126 throw new HTMLPurifier_Exception(
127 "Cannot instantiate unrecognized Lexer type " .
128 htmlspecialchars($lexer)
129 );
130 }
131 }
132
133 if (!$inst) {
134 throw new HTMLPurifier_Exception('No lexer was instantiated');
135 }
136
137 // once PHP DOM implements native line numbers, or we
138 // hack out something using XSLT, remove this stipulation
139 if ($needs_tracking && !$inst->tracksLineNumbers) {
140 throw new HTMLPurifier_Exception(
141 'Cannot use lexer that does not support line numbers with ' .
142 'Core.MaintainLineNumbers or Core.CollectErrors (use DirectLex instead)'
143 );
144 }
145
146 return $inst;
147
148 }
Configuration object that triggers customizable behavior.
Definition: Config.php:18
Global exception class for HTML Purifier; any exceptions we throw are from here.
Definition: Exception.php:8
Parser that uses PHP 5's DOM extension (part of the core).
Definition: DOMLex.php:28
Our in-house implementation of a parser.
Definition: DirectLex.php:14
Experimental HTML5-based parser using Jeroen van der Meer's PH5P library.
Definition: PH5P.php:14

Referenced by HTMLPurifier\purify().

+ Here is the caller graph for this function:

◆ escapeCDATA()

static HTMLPurifier_Lexer::escapeCDATA (   $string)
staticprotected

Translates CDATA sections into regular sections (through escaping).

Parameters
string$stringHTML string to process.
Returns
string HTML with CDATA sections escaped.

Definition at line 233 of file Lexer.php.

234 {
235 return preg_replace_callback(
236 '/<!\[CDATA\[(.+?)\]\]>/s',
237 array('HTMLPurifier_Lexer', 'CDATACallback'),
238 $string
239 );
240 }

Referenced by normalize().

+ Here is the caller graph for this function:

◆ escapeCommentedCDATA()

static HTMLPurifier_Lexer::escapeCommentedCDATA (   $string)
staticprotected

Special CDATA case that is especially convoluted for <script>

Parameters
string$stringHTML string to process.
Returns
string HTML with CDATA sections escaped.

Definition at line 247 of file Lexer.php.

248 {
249 return preg_replace_callback(
250 '#<!--//--><!\[CDATA\[//><!--(.+?)//--><!\]\]>#s',
251 array('HTMLPurifier_Lexer', 'CDATACallback'),
252 $string
253 );
254 }

Referenced by normalize().

+ Here is the caller graph for this function:

◆ extractBody()

HTMLPurifier_Lexer::extractBody (   $html)

Takes a string of HTML (fragment or document) and returns the content.

Todo:
Consider making protected

Definition at line 345 of file Lexer.php.

346 {
347 $matches = array();
348 $result = preg_match('!<body[^>]*>(.*)</body>!is', $html, $matches);
349 if ($result) {
350 return $matches[1];
351 } else {
352 return $html;
353 }
354 }
$result
$html
Definition: example_001.php:87

References $html, and $result.

Referenced by normalize().

+ Here is the caller graph for this function:

◆ normalize()

HTMLPurifier_Lexer::normalize (   $html,
  $config,
  $context 
)

Takes a piece of HTML and normalizes it by converting entities, fixing encoding, extracting bits, and other good stuff.

Parameters
string$htmlHTML.
HTMLPurifier_Config$config
HTMLPurifier_Context$context
Returns
string
Todo:
Consider making protected

Definition at line 294 of file Lexer.php.

295 {
296 // normalize newlines to \n
297 if ($config->get('Core.NormalizeNewlines')) {
298 $html = str_replace("\r\n", "\n", $html);
299 $html = str_replace("\r", "\n", $html);
300 }
301
302 if ($config->get('HTML.Trusted')) {
303 // escape convoluted CDATA
305 }
306
307 // escape CDATA
308 $html = $this->escapeCDATA($html);
309
311
312 // extract body from document if applicable
313 if ($config->get('Core.ConvertDocumentToFragment')) {
314 $e = false;
315 if ($config->get('Core.CollectErrors')) {
316 $e =& $context->get('ErrorCollector');
317 }
318 $new_html = $this->extractBody($html);
319 if ($e && $new_html != $html) {
320 $e->send(E_WARNING, 'Lexer: Extracted body');
321 }
322 $html = $new_html;
323 }
324
325 // expand entities that aren't the big five
326 $html = $this->_entity_parser->substituteNonSpecialEntities($html);
327
328 // clean into wellformed UTF-8 string for an SGML context: this has
329 // to be done after entity expansion because the entities sometimes
330 // represent non-SGML characters (horror, horror!)
332
333 // if processing instructions are to removed, remove them now
334 if ($config->get('Core.RemoveProcessingInstructions')) {
335 $html = preg_replace('#<\?.+?\?>#s', '', $html);
336 }
337
338 return $html;
339 }
static cleanUTF8($str, $force_php=false)
Cleans a UTF-8 string for well-formedness and SGML validity.
Definition: Encoder.php:127
static escapeCDATA($string)
Translates CDATA sections into regular sections (through escaping).
Definition: Lexer.php:233
static removeIEConditional($string)
Special Internet Explorer conditional comments should be removed.
Definition: Lexer.php:261
extractBody($html)
Takes a string of HTML (fragment or document) and returns the content.
Definition: Lexer.php:345
static escapeCommentedCDATA($string)
Special CDATA case that is especially convoluted for <script>
Definition: Lexer.php:247

References $html, HTMLPurifier_Encoder\cleanUTF8(), escapeCDATA(), escapeCommentedCDATA(), extractBody(), and removeIEConditional().

Referenced by HTMLPurifier_Lexer_DirectLex\tokenizeHTML(), HTMLPurifier_Lexer_DOMLex\tokenizeHTML(), and HTMLPurifier_Lexer_PH5P\tokenizeHTML().

+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ parseData()

HTMLPurifier_Lexer::parseData (   $string)

Parses special entities into the proper characters.

This string will translate escaped versions of the special characters into the correct ones.

Warning
You should be able to treat the output of this function as completely parsed, but that's only because all other entities should have been handled previously in substituteNonSpecialEntities()
Parameters
string$stringString character data to be parsed.
Returns
string Parsed character data.

Definition at line 186 of file Lexer.php.

187 {
188 // following functions require at least one character
189 if ($string === '') {
190 return '';
191 }
192
193 // subtracts amps that cannot possibly be escaped
194 $num_amp = substr_count($string, '&') - substr_count($string, '& ') -
195 ($string[strlen($string) - 1] === '&' ? 1 : 0);
196
197 if (!$num_amp) {
198 return $string;
199 } // abort if no entities
200 $num_esc_amp = substr_count($string, '&amp;');
201 $string = strtr($string, $this->_special_entity2str);
202
203 // code duplication for sake of optimization, see above
204 $num_amp_2 = substr_count($string, '&') - substr_count($string, '& ') -
205 ($string[strlen($string) - 1] === '&' ? 1 : 0);
206
207 if ($num_amp_2 <= $num_esc_amp) {
208 return $string;
209 }
210
211 // hmm... now we have some uncommon entities. Use the callback.
212 $string = $this->_entity_parser->substituteSpecialEntities($string);
213 return $string;
214 }

Referenced by HTMLPurifier_Lexer_DOMLex\createStartNode(), HTMLPurifier_Lexer_DirectLex\parseAttributeString(), and HTMLPurifier_Lexer_DirectLex\tokenizeHTML().

+ Here is the caller graph for this function:

◆ removeIEConditional()

static HTMLPurifier_Lexer::removeIEConditional (   $string)
staticprotected

Special Internet Explorer conditional comments should be removed.

Parameters
string$stringHTML string to process.
Returns
string HTML with conditional comments removed.

Definition at line 261 of file Lexer.php.

262 {
263 return preg_replace(
264 '#<!--\[if [^>]+\]>.*?<!\[endif\]-->#si', // probably should generalize for all strings
265 '',
266 $string
267 );
268 }

Referenced by normalize().

+ Here is the caller graph for this function:

◆ tokenizeHTML()

HTMLPurifier_Lexer::tokenizeHTML (   $string,
  $config,
  $context 
)

Lexes an HTML string into tokens.

Parameters
$stringString HTML.
HTMLPurifier_Config$config
HTMLPurifier_Context$context
Returns
HTMLPurifier_Token[] array representation of HTML.

Reimplemented in HTMLPurifier_Lexer_DirectLex, HTMLPurifier_Lexer_DOMLex, and HTMLPurifier_Lexer_PH5P.

Definition at line 223 of file Lexer.php.

224 {
225 trigger_error('Call to abstract class', E_USER_ERROR);
226 }

Field Documentation

◆ $_special_entity2str

HTMLPurifier_Lexer::$_special_entity2str
protected
Initial value:
=
array(
'&quot;' => '"',
'&amp;' => '&',
'&lt;' => '<',
'&gt;' => '>',
'&#39;' => "'",
'&#039;' => "'",
'&#x27;' => "'"
)

Most common entity to raw value conversion table for special entities.

@type array

Definition at line 161 of file Lexer.php.

◆ $tracksLineNumbers

HTMLPurifier_Lexer::$tracksLineNumbers = false

Whether or not this lexer implements line-number/column-number tracking.

If it does, set to true.

Definition at line 49 of file Lexer.php.


The documentation for this class was generated from the following file: