Skip to content

Commit 08cf72a

Browse files
authored
Simplify location tracking in the lexer. (#41)
Simplify location tracking in the lexer. This commit changes the location tracking in lexer.rs to be almost completely implicit, except for line tracking. The offset can be computed from the start of the &str and the current position of the innermost iterator. The final location information is only created in Lexer::next, at the start and the end of the currently build token. To simplify the control flow for each char consumed, inner iterators are changed to not use Peekable but instead do copies of inner iterators as needed.
1 parent 96b93d9 commit 08cf72a

File tree

2 files changed

+241
-188
lines changed

2 files changed

+241
-188
lines changed

0 commit comments

Comments
 (0)