latexnodes.parsers — Latex Construct Parsers

Collection of Parser objects that can parse specific types of LaTeX constructs.

Parser base class

class pylatexenc.latexnodes.parsers.LatexParserBase

The base class for pylatexenc.latexnodes.parsers parsers.

Parsers are objects that are designed to parse a specific type of latex construct, such as content enclosed in curly braces, into a node tree.

When invoked, parse objects return a tuple (nodes, parsing_state_delta). The first element, nodes, is the result nodes. It is usually a LatexNodeList instance, but it can also be a specific node instance, or another related object like a ParsedArguments instance. The second element, parsing_state_delta, encode any changes in the parsing state that should be caused by parsing the given construct. The parsing_state_delta should be either None (no parsing state changes) or a ParsingStateDelta instance. For instance, if the parser encountered a \newcommand it can relay the corresponding state change through the parsing_state_delta object.

The main functionality of the parser is implemented in the parse() method.

Parser objects should be invoked via the latex walker instance, using LatexWalker.parse_content() (see LatexWalkerBase and pylatexenc.latexwalker.LatexWalker):

my_latex_walker = LatexWalker(....)
my_parser = .... # some LatexParserBase subclass

token_reader = my_latex_walker.make_token_reader()
parsing_state = my_latex_walker.make_parsing_state()

# parse that specific construct:
nodes, parsing_state_delta = my_latex_walker.parse_content(
    my_parser,
    token_reader,
    parsing_state
)

parse(latex_walker, token_reader, parsing_state, **kwargs)

The main functionality of the parser is implemented in this method.

Parser objects should not be called directly, but rather be invoked via the latex walker instance, using LatexWalker.parse_content(). (See class doc above.)

Subclasses should implement this method to construct the relevant node tree by reading tokens from the token_reader (use token_reader.next_token() and friends, see LatexTokenReaderBase)

Subclasses should return a tuple pair (nodes, parsing_state_delta).

The nodes is the node list, node, or object that resulted from the parsing.

The parsing_state_delta encodes any parsing state changes that resulted during the parsing of this construct. If there are no parsing state changes, parsing_state_delta can be set to None.

contents_can_be_empty()

If absorbing no tokens is a valid option for the thing this object is meant to parse, then we should return True here. This would be the case, for instance, for group contents, for optional arguments, etc. But a parser for a mandatory argument would return False here.

This is used in certain special situations, for instance if a closing brace is immediately encountered after a macro that expected an argument (say \mymacro} — it’s an error if \mymacro requires a mandatory argument but it’s ok if it accepts an optional argument). In this case, we need to check all the macro arguments’ parser to see if it is okay that they have no contents.

General nodes

class pylatexenc.latexnodes.parsers.LatexGeneralNodesParser(stop_token_condition=None, stop_nodelist_condition=None, require_stop_condition_met=True, stop_condition_message=None, make_child_parsing_state=None, handle_stop_condition_token=None, include_stop_token_pre_space_chars=True, handle_stop_data=None, **kwargs)

Bases: LatexParserBase

Parse general nodes, either until a stopping condition is met, or until the end of stream is reached.

This is the general-purpose parser that parses the bulk of LaTeX content. It will parse content, using a nodes collector instance, where the latter will instantiate specialized parsers whenever specific constructs (such as macros, environments, arguments, etc.) are identified.

Nodes are parsed with a LatexNodesCollector instance provided by the latex walker instance (see LatexWalker.make_nodes_collector()). This class can be seen as a thin wrapper around a LatexNodesCollector instance to provide a LatexParserBase interface.

Arguments:

• stop_token_condition, stop_nodelist_condition, make_child_parsing_state, include_stop_token_pre_space_chars are passed on directly to create a nodes collector instance (see also parse())

• If require_stop_condition_met is True (the default), then any stopping condition must be eventually met; a parse error is raised if the end of stream is reached. This option has no effect if no stopping conditions are specified.

• The stop_condition_message argument can be used to specify a more human-friendly error message to report in case a specified stopping condition is not met. E.g., “expected ‘}’ after …” is probably more illuminating than the default message “stopping condition not met…”.

• The handle_stop_condition_token argument accepts a callable. When a token stopping condition is met, the given callable is invoked as handle_stop_condition_token(token, latex_walker=latex_walker, token_reader=token_reader, parsing_state=parsing_state), where token is the token that caused the token stop condition to fire. This callback is typically used to set the tokenreader’s position appropriately (e.g., past the token that ends a group).

• The handle_stop_data argument accepts a callable. After a stopping condition is met (whether token or nodelist), this callable is invoked with the return value of the stop callback function (stop_data) as argument. This enables the stop condition callback to specify more detailed information about what caused the processing to stop. The callback is invoked with the syntax handle_stop_data(stop_data, latex_walker=latex_walker, token_reader=token_reader, parsing_state=parsing_state).

make_nodes_collector(latex_walker, token_reader, parsing_state)

Create the nodes collector instance that will do the main parsing.

parse(latex_walker, token_reader, parsing_state, **kwargs)

The main parsing routine. The nodes collector instance is created using self.make_nodes_collector().

We check that stop conditions are met, if applicable, and call the relevant handlers.

class pylatexenc.latexnodes.parsers.LatexSingleNodeParser(stop_on_comment=True, **kwargs)

Bases: LatexGeneralNodesParser

A parser that collects a single logical node.

Inherits LatexGeneralNodesParser. Additional keyword arguments are provided to the LatexGeneralNodesParser constructor.

This class is a simple LatexGeneralNodesParser where the stopping condition is set to whenever the node list reaches one node. (If stop_on_comment is False, then we don’t count comment nodes).

The parser always returns a node list, and never a single node instance.

If the end of stream is reached, an empty node list is returned.

Arguments:

• stop_on_comment: If True, then a single comment node will count as a single node read. If False, then processing will continue until a non-comment node is reached.

contents_can_be_empty()

Return False, because no content would not satisfy the requirements of this parser.

Delimited expressions

class pylatexenc.latexnodes.parsers.LatexDelimitedExpressionParserInfo(delimited_expression_parser, opening_delimiter_tokens, group_parsing_state, parsing_state, delimiters, latex_walker)

Class that specifies how to parse a LaTeX chunk of code that is delimited by specific delimiters (a “delimited group”). Helper class for LatexDelimitedExpressionParser.

This class determines specific aspects of what type of delimited LaTeX code we should be reading, such as whether the delimiters are actual LaTeX group delimiters, or if they are math mode delimiters, or if they are verbatim delimiters, etc.

Some static/class methods are crucial for determining how the beginning of the delimited expression is read. E.g., determine the parsing state to use for the group itself, as well as if a token is an acceptable opening delimiter. After reading the token that we determined corresponds to the opening delimiter, an instance of the class is created to handle the rest of the group processing.

Important class methods, called when beginning to parse a delimited expression and before an instance of this class is created, are:

• get_group_parsing_state() — Create the parsing state used for the group node itself. One reason it might differ from the surrounding parsing state is for instance if the group is a LaTeX group with delimiters that are not ‘{’, ‘}’ and which need to be added to the list of LaTeX group delimiters.

• is_opening_delimiter() — Determine whether a token that was read is indeed an opening delimiter that the current delimited expression parser is meant to handle.

• get_acceptable_open_delimiter_list() — This class method is called only to generate friendlier error messages.

The important class instance properties, that should be set by initialize(), are:

• group_parsing_state — The group’s parsing state (e.g., that associated with the LatexGroupNode, this might be the same parsing state as surrounding code, but it might also be modified if it is to include special delimiters, etc.).

  The group_parsing_state (not the contents_parsing_state) is used to parse the opening delimiter.

  NOTE: the group_parsing_state is determined by the LatexDelimitedGroupParser object before this LatexDelimitedExpressionParserInfo is constructed. See the get_group_parsing_state() static method.

• contents_parsing_state — The group content’s parsing state, which might differ from surrounding parsing state. E.g., the group’s contents might be in math mode.

• child_parsing_state_delta - Any state changes to set when recursing down to children of the contents of this delimited content. By default, the delta is applied with respect to the group_parsing_state, not the contents_parsing_state. [Rationale: this attribute is usually used to “undo” some effects in the contents parsing state, so it’s more useful to have the reference parsing state be the group_parsing_state.]

• parsed_delimiters — This object is also responsible for actually determining which delimiters were used (if they weren’t predetermined exactly). Subclasses can use get_parsed_delimiters(), along with a custom implementation of get_matching_delimiter() if necessary.

  The parsed_delimiters is assumed to always be a 2-item tuple. The values can be placeholders, if necessary. They are only used in user messages until the final call to the make_group_node_and_parsing_state_delta() method. In that method, the value of the parsed_delimiters is used to set the final group node’s delimiters attribute.

Further responsibilities of this object include:

• Implement the stopping condition to detect the end delimiter (you might need to reimplement stop_token_condition()). Also handle the final token that caused the stop (the default implementation of handle_stop_condition_token(), which is to place the token reader past that token, should be sufficient in most cases).

• Create a relevant parser to parse the contents of the delimited expression. See make_content_parser(). The default implementation should suffice in most cases.

• make_group_node_and_parsing_state_delta() — prepare the parsing state changes information that will be left over after the group is fully parsed.

Note

When subclasses reimplement the methods in this class, they must ensure that they accept **kwargs to accommodate any additional arguments that I might think of adding in the future.

Here are some attributes that are set on the object instance:

delimited_expression_parser

A pointer to the main parser instance class, provided for convenience. E.g., methods can use this object pointer to look up any custom attributes that were set on the main parser instance.

opening_delimiter_tokens

The tokens associated with the opening delimiter. This is the list returned by parse_initial().

first_token

This attribute holds the first token in opening_delimiter_tokens. The first token is used at a couple places, such as for determining the initial position of the final group node, or for reporting the token associated with the open parser context representing this delimited expression. (Subclasses may choose to set this attribute to something else in their initialize() method, but I wouldn’t see why that is necessary.)

parsed_delimiters

A 2-item tuple of the actual delimiters that are applicable to this particular delimited expression that we are parsing. Normally this attribute is set in initialize(), possibly using the help of get_parsed_delimiters(), because once we know the opening delimiter we also know what closing delimiter to expect. See doc for get_parsed_delimiters(), initialize(), and make_group_node_and_parsing_state_delta() for more info on if the closing delimiter can’t be deduced from the opening delimiter.

This attribute is set in the constructor to the placeholder tuple (None, None), and should be set to an appropriate value in initialize().

group_parsing_state

The parsing state returned by get_group_parsing_state()

contents_parsing_state

The parsing state to use for parsing the contents of the delimited expression. Defaults to group_parsing_state; can be set by subclasses in their initialize() reimplementation.

child_parsing_state_delta

Any state changes to set when calling parsers for children of the contents of this delimited expression. This can be left to None to keep the same parsing state as the contents. It should be set to a ParsingStateDelta instance to specify parsing state changes if applicable.

parsing_state

The original parsing_state in which the delimited expression main parser instance was asked to operate in. This is the parsing state that was provided as an argument to get_group_parsing_state().

classmethod get_group_parsing_state(parsing_state, delimiters, delimited_expression_parser, latex_walker)

Return the parsing state object to use for the overall group. This is the parsing state that will be attached to the resulting LatexGroupNode. (The group contents might have a different parsing state.)

This default implementation simply returns parsing_state as is.

classmethod get_acceptable_open_delimiter_list(delimiters, group_parsing_state, delimited_expression_parser, latex_walker)

Return a list of strings representing acceptable opening delimiters.

The result is only used for error messages; this method’s return value does not influence parsing behavior or the final node structure.

classmethod parse_initial(delimiters, allow_pre_space, latex_walker, token_reader, group_parsing_state, delimited_expression_parser)

Attempt to parse the beginning of the delimited group.

This method is responsible for inspecting new tokens using token_reader to to ensure that the first token (or first few tokens) indeed correspond to an opening delimiter of the type of expression this parser info is meant to handle.

If an opening delimiter is successfully encountered, then this method should return a list of tokens that are associated with the opening delimiter that was read. (Generally this is a single token, like a ‘{’ char.) The returned list will serve as argument to the constructor’s opening_delimiter_tokens argument. The token reader’s position should be left after the initial delimiter tokens, so that the contents can immediately be read without having to move the token reader’s position.

If an opening delimiter is not successfully encountered, or the opening delimiter has space and allow_pre_space is not set, then the LatexDelimitedExpressionParserOpeningDelimiterNotFound should be raised, with relevant information for error messages and recovery. (The exception should be raised regardless of whether or not the delimited expression is optional.) It is not necessary to reset the token reader’s position, but it is important to set the first_tokens argument of that exception class to those tokens that were read so far, because it is used to reset the token reader correctly if the delimited expression was in fact optional or if we are trying to recover from a parse error.

The default implementation of parse_initial() provided here inspects a single token and uses the is_opening_delimiter() class function to determine whether or not the token that was read is a valid opening delimiter. You may reimplement this function to customize this behavior, e.g., if you need to inspect multiple tokens. The default implementation uses get_acceptable_open_delimiter_list() to help generate a human-friendly error message in case an opening delimiter is not successfully parsed.

classmethod is_opening_delimiter(delimiters, first_token, group_parsing_state, delimited_expression_parser, latex_walker)

Return True if the token first_token that was just read does indeed correspond to an opening delimiter that this parser is intended to read, including any constraints provided by the delimiters argument to the parser instance (provided here as the delimiters argument).

classmethod check_opening_delimiter(delimiters, parsed_opening_delimiter, latex_walker)

A helper convenience function for subclasses’ optional use in their is_opening_delimiter() method reimplementations. Returns True or False according to whether or not the parsed opening delimiter (parsed_opening_delimiter, a string) matches with the given delimiters argument, which was the delimiters constraint specified to the main parser instance.

I.e., if the delimiters is None, True is returned because there was no delimiters constraint. If the delimiters is a single string, it is compared to the parsed delimiter and returns True only if the strings are equal. Similarly, if delimiters is a 2-item tuple, the first item is compared to the parsed delimiter.

initialize()

This method is called after the instance is created, so that subclasses can initialize the important relevant attributes for this class.

By default, this method initializes the parsed_delimiters attribute using the get_parsed_delimiters() method.

stop_token_condition(token)

This method must be reimplemented so that the parser knows when a closing delimiter was encountered.

handle_stop_condition_token(token, latex_walker, token_reader, parsing_state)

Called to take action after the token was read and determined to satisfy the stopping condition. By default, the token_reader is positioned immediately after the stopping token.

make_child_parsing_state(parsing_state, node_class, token)

When parsing the delimited expression’s contents, we might need to recurse into child groups etc.; in such cases, this method is used to create a relevant parsing state for child nodes. It is directly used as the expression’s internal node collector’s make_child_parsing_state() handler.

You may reimplement this method if you need a more detailed mechanism for determining what parsing state the child parser should be initiated with, e.g., if the parsing state should vary according to what type of child is encountered.

get_matching_delimiter(opening_delimiter)

Subclasses can reimplement this method instead of reimplementing get_parsed_delimiters(). This method will determine what delimiter should be used as closing delimiter for the given read opening_delimiter. This can be used to match parentheses or braces, or LaTeX group delimiters, or math delimiters, etc.

The default implementation returns opening_delimiter as is.

get_parsed_delimiters()

Determine what the actual delimiters of this expression will be, based off the opening delimiter. This method is usually called at the beginning of the delimited expression to help in information messages, etc.

This function is called by the default initialize() method to set the parsed_delimiters attribute.

The default implementation inspects the delimiters attribute. Recall that delimiters is the argument that was provided to the LatexDelimitedExpressionParser() instance, specifying a requirement for what delimiters we want to accept. If delimiters is already a pair, this method simply returns that pair. If delimiters is a string, it attempts to find the matching closing delimiter by calling get_matching_delimiter(). If delimiters is None, then the opening delimiter is set to the first token’s arg attribute, and a matching delimiter is sought via the get_matching_delimiter() method.

Usually there shouldn’t be any need to reimplement this method. Instead, you should reimplement get_matching_delimiter() to instruct this method how to deducing the closing delimiter that is associated with a specific opening delimiter.

In the event where there might be multiple acceptable closing delimiters, then the closing delimiter cannot be known in advance. You need to handle this case manually. I.e., return an empty closing delimiter here, or reimplement initialize() to assign a temporary placeholder value to the parsed_delimiters attribute. This attribute is only used for information messages anyway before the final make_group_node_and_parsing_state_delta() anyways. However, you need to either reimplement make_group_node_and_parsing_state_delta(), or to set the property parsed_delimiters correctly before that method gets called.

make_content_parser(latex_walker, token_reader)

This method is responsible for creating a relevant parser to parse the contents of the delimited expression.

The default implementation creates a LatexGeneralNodesParser instance that is properly initialized call the stop_token_condition(), handle_stop_condition_token(), and make_child_parsing_state() callbacks on this instance object.

You should only need to reimplement this method if you need more detailed control over the contents parser, e.g., if you need a stopping condition on the node list instead of the token only (stop_nodelist_condition).

get_open_context_description()

Return a 2-item tuple (msg, token) or None. The value must be suitable for passing as an argument to the open_context parameter of LatexWalkerBase.parse_content().

make_group_node_and_parsing_state_delta(latex_walker, token_reader, nodelist, parsing_state_delta)

Actually create the final node object and the associated parsing_state_delta that will be returned by the delimited expression parser.

The default implementation creates a LatexGroupNode instance. The delimiters field is set to the parsed_delimiters field that must have been set to an appropriate value by this point.

class pylatexenc.latexnodes.parsers.LatexDelimitedExpressionParser(delimiters, delimited_expression_parser_info_class, optional=False, allow_pre_space=False, discard_parsing_state_delta=True, **kwargs)

Bases: LatexParserBase

A general-purpose parser / parser base class to handle LaTeX content that is enclosed by some form of delimiters. This can include a standard TeX group ({ ... }), or math mode blocks ($...$), or environments (\begin{xyz}...\end{xyz}), etc.

This class is a general base class that groups common functionality of various parsers. Usually you shouldn’t use LatexDelimitedExpressionParser parser instances directly. Rather, you should use one of the more specialized parsers such as LatexDelimitedGroupParser or LatexMathParser, etc.

Many specifics by which the delimited group is parsed is determined by a separate helper class, which must be a LatexDelimitedExpressionParserInfo subclass. That info class is given through the delimited_expression_parser_info_class constructor argument. Specialized delimiter-based parsers, such as LatexDelimitedGroupParser, provide their own info classes delimited_expression_parser_info_class; when using those specialized parsers, you don’t have to worry about that argument.

The main parse() method normally returns a LatexGroupNode instance with the delimited LaTeX contents that was parsed according to the provided info class. The exact object that is returned can be customized by the info class, see LatexDelimitedExpressionParserInfo.make_group_node_and_parsing_state_delta().

Constructor arguments:

• delimiters can be either:

  • None to auto-detect delimiters. If the first char token is one of the auto_delimiters, then the corresponding closing delimiter is determined from the parsing state.

  • A single <char1> indicating an opening delimiter. The corresponding closing delimiter will be obtained by inspecting the parsing state.

  • A pair (<char1>, <char2>) of opening and closing delimiters.

• If optional is True, then no error is raised if the expected opening delimiter is not present and a None node is returned by the parser. By default, not encountering an expected opening delimiter causes a parse error.

• If allow_pre_space is True, any space preceding the group is ignored (it’s False by default). This can be useful for instance in certain edge cases where you’d want an optional argument to immediately follow a command, so that an opening bracket after whitespace doesn’t get parsed as an optional argument. For instance, IIRC AMS redefines the command \\ to require its optional argument (vertical spacing) to immediately follow the \\ without any whitespace so that the code A+B=C \\ [A,B] = 0 doesn’t parse [A,B] as an argument to \\.

• If discard_parsing_state_delta is True, any parsing state changes information from parsing the content of the group is discarded. This is the default behavior and mirrors the behavior of (La)TeX that most definitions are local to a group

class pylatexenc.latexnodes.parsers.LatexDelimitedGroupParserInfo(delimited_expression_parser, opening_delimiter_tokens, group_parsing_state, parsing_state, delimiters, latex_walker)

Bases: LatexDelimitedExpressionParserInfo

Helper class for LatexDelimitedGroupParser. See also LatexDelimitedExpressionParserInfo.

classmethod get_group_parsing_state(parsing_state, delimiters, delimited_expression_parser, latex_walker, **kwargs)

Return the parsing state object to use for the overall group. This is the parsing state that will be attached to the resulting LatexGroupNode. (The group contents might have a different parsing state.)

The default implementation inspects the parsing_state to ensure that the current delimiters are in the list of latex group delimiters. It will add them if necessary and if it is able to (i.e., if the delimiters property specifies both opening and closing delimiters). If delimiters only contains the opening delimiter, and if it is not listed as a latex group delimiter, a parse error is raised.

You should reimplement this method if the delimiters you want to capture are not latex group delimiters. See LatexMathParser for an example.

Note

This method assumes that the delimiters are latex group type (e.g., curly brace chars), as specified in the parsing state’s latex group delimiter list. If not, then you need to reimplement this function in a subclass.

classmethod get_acceptable_open_delimiter_list(delimiters, group_parsing_state, delimited_expression_parser, latex_walker, **kwargs)

Only to be used for error messages.

get_matching_delimiter(opening_delimiter)

Return the closing delimiter that matches the given opening_delimiter, by looking it up in the group parsing state’s list of latex group delimiters (see latex_group_delimiters).

Reimplemented from LatexDelimitedExpressionParserInfo.get_matching_delimiter().

This method assumes that the delimiters are latex group type (e.g., curly brace chars). If not, then you need to reimplement this function in a subclass.

class pylatexenc.latexnodes.parsers.LatexDelimitedGroupParser(delimiters, delimited_expression_parser_info_class=<class 'pylatexenc.latexnodes.parsers._delimited.LatexDelimitedGroupParserInfo'>, **kwargs)

Bases: LatexDelimitedExpressionParser

Parse a LaTeX group, i.e., contents enclosed in a pair of LaTeX group delimiters such as { ... }.

This is the parser that a latex walker uses whenever the tokenizer reports a group opening token; see make_latex_group_parser(). The result of parse() is a LatexGroupNode instance.

The delimiters constructor argument, inherited from LatexDelimitedExpressionParser, restricts which delimiters are accepted: specify None to accept any of the group delimiters that are declared in the parsing state, a single character to require that specific opening delimiter, or a pair of characters to require that specific pair of opening and closing delimiters. All other constructor arguments of LatexDelimitedExpressionParser are also accepted.

In all cases, the first token read (after possible whitespace) must be a ‘brace_open’ type. If delimiters is a pair of characters, the parsing state is inspected to ensure that these delimiters are recorded as latex group delimiters, and will add them if necessary for parsing the group contents (but not its parsed children).

If the delimiters are not defined as LaTeX group delimiters in the parsing state, then a new parsing state is created that includes these delimiters. The new parsing state is used to parse the group contents but not any child nodes. This behavior ensures that expressions of the type \macro[A[B]] and \macro[{[}] are parsed correctly respectively as an optional argument A[B] to \macro and as a single opening bracket as an optional argument to \macro.

class pylatexenc.latexnodes.parsers.LatexDelimitedMultiDelimGroupParserInfo(delimited_expression_parser, opening_delimiter_tokens, group_parsing_state, parsing_state, delimiters, latex_walker)

Bases: LatexDelimitedGroupParserInfo

Reimplementation of the LatexDelimitedGroupParserInfo class for groups that can be delimited by any pair out of a list of candidate delimiter pairs, for LatexDelimitedMultiDelimGroupParser.

classmethod get_group_parsing_state(parsing_state, delimiters, delimited_expression_parser, latex_walker, **kwargs)

Return a parsing state in which the group delimiters are exactly the candidate delimiter pairs listed in the parser’s delimiters_list attribute, so that any of them can open this group.

The delimiters argument must be None here, because which delimiters are acceptable is specified through the parser’s delimiters_list.

Reimplemented from LatexDelimitedGroupParserInfo.get_group_parsing_state().

initialize()

Set up the parsing state for the group contents.

After the base class implementation has determined which delimiter pair actually opened this group, the candidate delimiters are no longer all recognized as group delimiters for the contents. The contents parsing state keeps the group delimiters of the surrounding parsing state, plus the pair that was actually parsed for this group.

This way, the code <a {b} c> parses the inner {b} as a group, while <a [b c> doesn’t see the [ as opening a group, even though [–] is one of the candidate delimiter pairs for the outer group.

Reimplemented from LatexDelimitedExpressionParserInfo.initialize().

class pylatexenc.latexnodes.parsers.LatexDelimitedMultiDelimGroupParser(delimiters_list=(('{', '}'), ('[', ']'), ('(', ')'), ('<', '>')), delimited_expression_parser_info_class=<class 'pylatexenc.latexnodes.parsers._delimited.LatexDelimitedMultiDelimGroupParserInfo'>, **kwargs)

Bases: LatexDelimitedExpressionParser

Parse a group that can be delimited by any pair out of a list of candidate delimiter pairs.

Whereas LatexDelimitedGroupParser accepts the group delimiters that the parsing state declares, this parser accepts the pairs listed in the delimiters_list constructor argument regardless of the parsing state. The default list is [ ('{','}'), ('[',']'), ('(',')'), ('<','>') ]. This parser is what the standard argument types 'AnyDelimited' and 'AnyDelimitedOptional' use, see LatexStandardArgumentParser.

Whichever pair actually opened the group must also close it; the remaining candidate pairs are not treated as group delimiters while parsing the group contents (see LatexDelimitedMultiDelimGroupParserInfo.initialize()).

The delimiters argument of LatexDelimitedExpressionParser is not available here (it is always set to None); use delimiters_list instead. The other constructor arguments of LatexDelimitedExpressionParser are accepted.

class pylatexenc.latexnodes.parsers.LatexDelimitedExpressionParserOpeningDelimiterNotFound(first_tokens, msg, **kwargs)

Bases: Exception

Exception class used by a LatexDelimitedExpressionParserInfo object to communicate to the associated LatexDelimitedExpressionParser instance that the initial delimiter was not found.

You won’t need to worry about handling this exception unless you are writing a very customized subclass of LatexDelimitedExpressionParserInfo, see doc there.

class pylatexenc.latexnodes.parsers.LatexMathParser(math_mode_delimiters, **kwargs)

Bases: LatexDelimitedExpressionParser

Single expression parser

class pylatexenc.latexnodes.parsers.LatexExpressionParser(allow_pre_space=True, allow_pre_comments=True, return_full_node_list=True, single_token_requiring_arg_is_error=True, parse_callable_arguments=False, **kwargs)

Bases: LatexParserBase

The parsed result is a LatexNodeList.

If return_full_node_list is True, then a LatexNodeList is returned as the content returned by parse(). The node list contains all nodes read while parsing the given LaTeX expression (including comment nodes and whitespace reported as whitespace-only chars node).

If return_full_node_list is False, then only the single node that contained the expression we’re interested in is returned in the contents returned by the parse() method. While you directly get the expression you’re interested in, you might lose information about how to recompose the node into its source LaTeX string.

If parse_callable_arguments is True, then a macro or a “latex specials” that forms the expression is parsed along with its own arguments. For instance, the expression in x^\mathrm{max} is then the entire \mathrm{max}. By default (parse_callable_arguments=False) the macro or specials is picked up on its own, without its arguments, and single_token_requiring_arg_is_error determines whether or not a macro that requires arguments is reported as an error.

Optional expression parser

class pylatexenc.latexnodes.parsers.LatexOptionalSquareBracketsParser(delimiters=('[', ']'), optional=True, **kwargs)

Bases: LatexDelimitedGroupParser

A shorthand for reading an optional argument placed in square brackets.

class pylatexenc.latexnodes.parsers.LatexOptionalCharsMarkerParser(chars_list, following_arg_parser=None, include_chars_node_before_following_arg=True, return_none_instead_of_empty=True, allow_pre_space=True, return_full_node_list=True, collect_chars_with_following_arg_as_delimited_group=False, max_num_args=None, **kwargs)

Bases: LatexParserBase

Parse an optional marker consisting of one or more specific characters, such as the star in \section*, optionally along with an argument that follows the marker.

The parser looks for a marker repeatedly, so that several markers can be picked up one after the other. Each marker in chars_list can be matched at most once; after a marker was read, only the remaining markers are looked for. The parser stops as soon as the next tokens don’t match any of the remaining markers, once all the markers have been read, or once max_num_args markers have been read. If no marker at all was found, the parser reports that the optional argument was not provided (see return_none_instead_of_empty).

Constructor arguments:

• chars_list is the list of markers to look for. Each entry is a string of one or more characters. Whitespace within an entry is normalized, and matches any run of whitespace in the source; entries that are empty after this normalization are dropped, as they could never be matched. If a single string is given instead of a list, it is interpreted as the list of its individual characters. A “specials” token counts as its characters here, so that a marker can be made of characters that are declared as specials.

• following_arg_parser, if non-None, is the parser that is used to read an argument that follows the marker. It is obtained via get_following_arg_parser(), so that subclasses can pick a different parser depending on which marker was matched.

• include_chars_node_before_following_arg determines whether or not the marker itself is reported as a chars node before the nodes of the following argument.

• return_none_instead_of_empty determines what is returned if no marker was found: None (the default), or an empty node list.

• allow_pre_space determines whether whitespace is allowed before the marker. If it is False and whitespace is encountered, the marker counts as not provided.

• collect_chars_with_following_arg_as_delimited_group wraps the marker and the following argument into a single LatexGroupNode whose delimiters are the marker and an empty string. This requires return_full_node_list=True.

• max_num_args, if non-None, is the maximum number of markers to read.

contents_can_be_empty()

Return True; the marker that this parser reads is always optional.

Reimplemented from LatexParserBase.contents_can_be_empty().

get_following_arg_parser(chars)

Return the parser to use to read the argument that follows the marker chars that was just matched, or None if this marker does not take an argument.

The default implementation returns the following_arg_parser that was specified to the constructor, regardless of chars. Reimplement this method if different markers should take different arguments.

parse(latex_walker, token_reader, parsing_state, **kwargs)

Read the optional marker(s) and any argument that follows them. See class doc.

Returns a tuple (nodes, None). The nodes is None if no marker was found and return_none_instead_of_empty is set; otherwise it is a LatexNodeList, or a single node if return_full_node_list is False.

Reimplemented from LatexParserBase.parse().

class pylatexenc.latexnodes.parsers.LatexOptionalEmbellishmentArgsParser(embellishment_chars, allow_pre_space=True, **kwargs)

Bases: LatexOptionalCharsMarkerParser

Parse a sequence of optional “embellishments”, i.e., single characters that each introduce an argument, such as the ^ and _ in x^i_j.

This parser is inspired by xparse’s e{tokens} argument type.

The embellishment_chars argument lists the characters that can introduce an embellishment; it is used as the chars_list of the base class LatexOptionalCharsMarkerParser, so a plain string is interpreted as the list of its individual characters. Each embellishment character that is encountered is followed by a single expression, which is read with a LatexExpressionParser.

Each embellishment is reported as a LatexGroupNode whose opening delimiter is the embellishment character and whose closing delimiter is an empty string; all of them are collected into a single node list. If no embellishment at all is present, the parser reports None.

Verbatim/literal expressions

class pylatexenc.latexnodes.parsers.LatexVerbatimBaseParser(**kwargs)

Bases: LatexParserBase

Base class for parsers that read verbatim content, i.e., content that is taken literally rather than parsed as LaTeX code.

Instead of reading tokens, these parsers read the input one character at a time and hand each character to new_char_check_stop_condition(), which decides when the verbatim content ends. Subclasses reimplement that method (and possibly finalize_verbatim_string()) to implement a specific verbatim construct; see LatexDelimitedVerbatimParser and LatexVerbatimEnvironmentContentsParser.

This base class implementation stops after a single character.

Note: this parser requires the token reader to provide character-level access to the input string.

class VerbatimInfo

Bases: object

A simple object on which the verbatim parser stores the state associated with a single parse() call, so that the parser object itself can be reused.

The fields are set by the parser as it goes along and include parsed_delimiters (the opening and closing delimiters of this verbatim construct, if applicable), original_pos (the position at which parsing started), content_pos_start (the position at which the verbatim content itself starts), and pos_start/pos_end (the span of the resulting chars node, set by finalize_verbatim_string()).

new_char_check_stop_condition(char, verbatim_string, verbatim_info, parsing_state)

Called for each character that is read, to decide whether the verbatim content ends here. The char is the character that was just read (or None if the end of the stream was reached), and verbatim_string is the verbatim content that was accumulated so far, not including char.

Return a false value to continue reading; char is then appended to the verbatim content. Return True to stop, in which case char is consumed and dropped, or return a dictionary {'put_back_char': True} to stop and have char put back onto the input so that it is read again by whatever parses the following content.

The default implementation in this base class is to read a single verbatim char. Reimplement this method in a subclass for more advanced behavior.

error_end_of_stream(pos, recovery_nodes, latex_walker, verbatim_info)

Called when the end of the stream was reached before the stop condition fired. Raises a LatexWalkerNodesParseError with recovery_nodes set to the verbatim content that was read so far, so that parsing can continue in tolerant parsing mode.

Subclasses reimplement this method in order to wrap recovery_nodes into the same node structure that a successful parse would have produced.

finalize_verbatim_string(verbatim_string, verbatim_info)

Return the string to include in the verbatim chars node.

Also, this method should assign the fields pos_start and pos_end in verbatim_info to set the start and the end positions of the node.

parse(latex_walker, token_reader, parsing_state, **kwargs)

Read the verbatim content at the current position and return it as a chars node, with no surrounding delimiters.

Reimplemented from LatexParserBase.parse().

read_verbatim_content(latex_walker, token_reader, parsing_state, verbatim_info, **kwargs)

Read the verbatim content itself, character by character, until new_char_check_stop_condition() says to stop.

The accumulated string is then handed to finalize_verbatim_string(), and the result is returned as a single LatexCharsNode. That node’s parsing state has macros, environments, specials, comments, groups, math mode and paragraph breaks all disabled, reflecting the fact that its contents are not LaTeX code.

If the end of the stream is reached before the stop condition fires, error_end_of_stream() is called with the chars node offered as recovery nodes.

Return a tuple (chars_node, None).

The token_reader is left after the character that caused the processing to stop.

class pylatexenc.latexnodes.parsers.LatexDelimitedVerbatimParser(delimiters=None, auto_delimiters=None, **kwargs)

Bases: LatexVerbatimBaseParser

Parse verbatim content specified between token delimiters (e.g., \verb|...|).

Any whitespace before the opening delimiter is skipped. How the delimiters are determined depends on the delimiters constructor argument:

• If delimiters is None (the default), the delimiters are detected automatically, as \verb does: the first character that follows is the opening delimiter, and the closing delimiter is looked up in the auto_delimiters dictionary, falling back to the opening delimiter itself. The default auto_delimiters maps {, [, < and ( to their respective closing counterparts.

• If delimiters is a pair (open_delimiter, close_delimiter), then exactly that pair is expected; a LatexWalkerParseError is raised if the opening delimiter is not there.

If the opening and closing delimiters are different characters, then nested occurrences of the delimiter pair are tracked, so that \verb{a{b}c} yields the verbatim content a{b}c.

The result is a LatexGroupNode with the parsed delimiters, containing a single chars node with the verbatim content.

new_char_check_stop_condition(char, verbatim_string, verbatim_info, parsing_state)

The default implementation in this base class is to read a single verbatim char. Reimplement this method in a subclass for more advanced behavior.

class pylatexenc.latexnodes.parsers.LatexVerbatimEnvironmentContentsParser(environment_name='verbatim', **kwargs)

Bases: LatexVerbatimBaseParser

Parse verbatim content given as an environment body contents.

The content is read literally until the string \end{environment_name} is encountered (using the parsing state’s macro escape character in place of the backslash). That terminator is not part of the verbatim content, but it is consumed: the token reader is left immediately after it, so that the enclosing environment node spans the closing \end{...} as usual.

A single newline character immediately following \begin{environment_name} is dropped, as LaTeX does.

The environment_name constructor argument gives the name of the environment to look for and defaults to 'verbatim'.

The result is a LatexNodeList containing a single chars node with the verbatim content, which is the shape expected for an environment body.

Typical macro arguments

pylatexenc.latexnodes.parsers.get_standard_argument_parser(arg_spec, **kwargs)

Return an argument parser instance associated with the string shorthand arg_spec and given keyword arguments. This creates a LatexStandardArgumentParser instance and caches it (with the given arg_spec and any given keyword arguments) for future use.

class pylatexenc.latexnodes.parsers.LatexStandardArgumentParser(arg_spec='{', return_full_node_list=False, expression_single_token_requiring_arg_is_error=True, allow_pre_space=True, **kwargs)

Bases: LatexParserBase

Node parser that can parse standard macro argument types, such as a mandatory argument in braces, an optional argument in square braces, an optional star, as well as more advanced macro argument constructs.

Which argument type to parse is selected by the arg_spec string, whose syntax is inspired by that of the LaTeX xparse package. The recognized values are:

• 'm' or '{' — a mandatory argument, given either as a single token or as a braced group. See LatexExpressionParser.

• 'o' or '[' — an optional argument delimited by square brackets, as in \mymacro[optional].

• 's' or '*' — an optional star, as in \section*.

• 't<char>' — an optional single-character marker <char>; like 's', but for a character other than the star.

• 'r<char1><char2>' — a mandatory argument delimited by the two given characters.

• 'd<char1><char2>' — an optional argument delimited by the two given characters.

• 'e{<chars>}' — a sequence of optional embellishments, each introduced by one of the given characters, as the ^ and _ in x^i_j. See LatexOptionalEmbellishmentArgsParser.

• 'v' — a verbatim argument whose delimiters are whichever characters follow the macro (the behavior of \verb); or 'v<char1><char2>' — a verbatim argument delimited by the two given characters. See LatexDelimitedVerbatimParser.

• 'AnyDelimited' — a mandatory argument delimited by any of the standard delimiter pairs {}, [], () and <>; or 'AnyDelimitedOptional' for the same thing as an optional argument. See LatexDelimitedMultiDelimGroupParser.

Any other value of arg_spec causes a ValueError to be raised. The relevant parser instance is created lazily, the first time parse() is called, and is then reused for subsequent calls.

Further constructor arguments:

• return_full_node_list is relayed to the parsers used for the 'm' and 's'/'*' argument types. It determines whether all the nodes that were read are reported (including preceding whitespace and comment nodes), or only the single node that carries the argument itself.

• expression_single_token_requiring_arg_is_error is relayed as single_token_requiring_arg_is_error to the LatexExpressionParser used for the 'm' argument type. It determines whether it is an error if the single token picked up as the argument is a macro that itself requires arguments.

• allow_pre_space determines whether whitespace is allowed between the callable and its argument. It is relayed to all the argument parsers above except the verbatim ones.

arg_spec

The argument type specification that was given to the constructor.

get_arg_parser_instance(arg_spec)

Create and return the parser instance that reads an argument of the type given by the arg_spec string. See class doc for the recognized values.

Raises ValueError if arg_spec is not a recognized argument type specification.

Reimplement this method in a subclass if you’d like to support further argument types of your own.

parse(latex_walker, token_reader, parsing_state, **kwargs)

Read the argument by handing over to the parser that corresponds to this object’s arg_spec, and return that parser’s result unchanged.

The relevant parser instance is created on the first call to this method (see get_arg_parser_instance()) and reused afterwards.

Reimplemented from LatexParserBase.parse().

class pylatexenc.latexnodes.parsers.LatexCharsCommaSeparatedListParser(comma_char=',', delimiters=('{', '}'), enable_comments=True, enable_groups=True, keep_empty_parts=False, **kwargs)

Bases: LatexDelimitedGroupParser

Parse a delimited group that contains a list of items separated by a given character, e.g. {one,two,three}.

As for LatexCharsGroupParser, the contents are read as plain characters: macros, environments, specials, and math mode are disabled, while comments and nested groups can be enabled or disabled with the enable_comments and enable_groups constructor arguments (both are enabled by default). The separator is the comma_char constructor argument, which defaults to ',', and the group delimiters are given by delimiters, which defaults to ('{','}').

The result is a LatexGroupNode for the group as a whole, whose node list has one LatexGroupNode per item. Each item node has an empty opening delimiter, and its closing delimiter is the separator character that terminated it (an empty string for the last item).

Empty items are skipped unless keep_empty_parts is set to True. An entirely empty group ({}) never produces any item node, even when keep_empty_parts is set.

NOTE: It might be better to use a standard argument, and parse the argument as a comma-separated list using split_at_chars().

Careful, when accessing the value of this group via a ParsedArgumentsInfo and get_contents_nodelist(), make sure to use unwrap_double_group=False as otherwise the inner group is going to be unwrapped if a single group is present.

class pylatexenc.latexnodes.parsers.LatexCharsGroupParser(delimiters=('{', '}'), enable_comments=True, enable_groups=True, **kwargs)

Bases: LatexDelimitedGroupParser

Parse a delimited group whose contents are read as plain characters, i.e., a group in which macros, environments, specials, and math mode are all disabled.

This is what you want for an argument that is a piece of text rather than LaTeX code, such as a label name or a file name. Comments and nested groups can be enabled or disabled with the enable_comments and enable_groups constructor arguments (both are enabled by default); everything else is reported as ordinary characters.

The delimiters constructor argument is as for LatexDelimitedGroupParser and defaults to ('{','}'). The parsing state changes described above apply to the group contents only, not to any children of nested groups: those are parsed with the parsing state that surrounds this group.

Very similar to a verbatim parser, but works with tokens instead of chars. You can use comments and recursive groups, too.

Will result in a group node that contains one char node (and possibly a combination of group, chars, and comment nodes if those were enabled)

class pylatexenc.latexnodes.parsers.LatexTackOnInformationFieldMacrosParser(macronames, allow_multiple=False, macro_arg_parsers=None, default_macro_arg_parser=True, **kwargs)

Bases: LatexParserBase

• macronames is iterable/list/set of names of macros for which information can be specified. (Name without backslash.)

• allow_multiple can be either True, False, or an iterable/list/set. Set to True to allow any information field macro to be specified multiple times; set to False to disallow all multiple occurrences of an information field macro. Providing a set (or iterable or list) of macro names (w/o backslash) will allow repeated specification of those macro names only and not the other macro names in macronames.

• default_macro_arg_parser is a LatexParserBase instance or True (the default) or None. It is used to parse arguments to each of the tacked-on macros that we encounter, except for ones for which a specific parser is found in macro_arg_parsers. If set to True (the default), a standard single-expression parser is used (single token or braced group). If set to None, no arguments are expected to the given tack-on macros for which the default parser is used.

• macro_arg_parsers can be set to a dictionary containing specific parsers to use for specific macro names of tacked-on macros. Specifying macro_arg_parsers=None is equivalent to specifying an empty dictionary. The keys in the dictionaries are the macro names. Each value is either None (no argument is expected), True (use the default parser), or a parser instance.

get_macro_arg_parser(macroname)

You can reimplement this method if you want to be able to read more complex macro call syntaxes. It is probably simpler, however, to use the macro_arg_parsers constructor argument. If you reimplement this function, then macro_arg_parsers and default_macro_arg_parser have no effect unless you call the base implementation.