tyrian.lexer¶

class tyrian.lexer.Lexer(token_defs: dict)[source]¶

Code to perform lexing according to token definitions

Parameters:	token_defs – dictionary containing token definitions, see load_token_definitions for definitions

match_with(left: str)[source]¶

Convenience function.

returns an object with a match attribute partial’ed operator.eq, configured to match left with the supplied right

Parameters:	left – const for returned function to be configured to compare against
Return type:	object with match attribute

load_token_definitions(defs: dict)[source]¶

Iterates through the supplied token_defs dictionary, creates wrappers for literals and compiles regex’s

Parameters:	token_defs – contains token definitions; see GrammarParser.load_token_definitions for format

lex(content: str, filename: str) → list[source]¶

Takes a string to lex according to token definition loaded via load_token_definitions

Parameters:	content – content of file being lexed filename – name of file being lexed

_lex(line: str, line_no: int, filename: str) → dict[source]¶

used internally by lex, does actual lexing

Parameters:	line – line from source file line_no – line number of provided line filename – name of file from which the line originates

yields tokens of format

{
    "name": str,
    "token": str,
    "line_no": int,
    'filename': str
}