Module lexer

This module provides functionality to tokenize a line of markdown text into a vector of Token enums.

Functions

is_punctuation 🔒

Helper function to determine if a string is a single punctuation character.

tokenize

Tokenizes a line of markdown text into a vector of Token enums.