Forgivingly lexes HTML (SGML-style) markup into tokens.
A lexer parses a string of SGML-style markup and converts them into corresponding tokens. It doesn't check for well-formedness, although its internal mechanism may make this automatic (such as the case of HTMLPurifier_Lexer_DOMLex). There are several implementations to choose from.
A lexer is HTML-oriented: it might work with XML, but it's not recommended, as we adhere to a subset of the specification for optimization reasons. This might change in the future. Also, most tokenizers are not expected to handle DTDs or PIs.
This class should not be directly instantiated, but you may use create() to retrieve a default copy of the lexer. Being a supertype, this class does not actually define any implementation, but offers commonly used convenience functions for subclasses.
- Note
- The unit tests will instantiate this class for testing purposes, as many of the utility functions require a class to be instantiated. This means that, even though this class is not runnable, it will not be declared abstract.
- Note
- We use tokens rather than create a DOM representation because DOM would:
- Require more processing and memory to create,
- Is not streamable, and
- Has the entire document structure (html and body not needed).
- However, DOM is helpful in that it makes it easy to move around nodes without a lot of lookaheads to see when a tag is closed. This is a limitation of the token system and some workarounds would be nice.
Definition at line 42 of file Lexer.php.
HTMLPurifier_Lexer::normalize |
( |
|
$html, |
|
|
|
$config, |
|
|
|
$context |
|
) |
| |
Takes a piece of HTML and normalizes it by converting entities, fixing encoding, extracting bits, and other good stuff.
- Parameters
-
- Returns
- string
- Todo:
- Consider making protected
Definition at line 294 of file Lexer.php.
References $html, HTMLPurifier_Encoder\cleanUTF8(), escapeCDATA(), escapeCommentedCDATA(), extractBody(), and removeIEConditional().
Referenced by HTMLPurifier_Lexer_PH5P\tokenizeHTML(), HTMLPurifier_Lexer_DirectLex\tokenizeHTML(), and HTMLPurifier_Lexer_DOMLex\tokenizeHTML().
297 if ($config->get(
'Core.NormalizeNewlines')) {
302 if ($config->get(
'HTML.Trusted')) {
313 if ($config->get(
'Core.ConvertDocumentToFragment')) {
315 if ($config->get(
'Core.CollectErrors')) {
316 $e =& $context->get(
'ErrorCollector');
319 if ($e && $new_html !=
$html) {
320 $e->send(E_WARNING,
'Lexer: Extracted body');
326 $html = $this->_entity_parser->substituteNonSpecialEntities(
$html);
334 if ($config->get(
'Core.RemoveProcessingInstructions')) {
335 $html = preg_replace(
'#<\?.+?\?>#s',
'',
$html);
static removeIEConditional($string)
Special Internet Explorer conditional comments should be removed.
static cleanUTF8($str, $force_php=false)
Cleans a UTF-8 string for well-formedness and SGML validity.
static escapeCDATA($string)
Translates CDATA sections into regular sections (through escaping).
extractBody($html)
Takes a string of HTML (fragment or document) and returns the content.
static escapeCommentedCDATA($string)
Special CDATA case that is especially convoluted for <script>