Function tokenize

pub fn tokenize(markdown_line: &str) -> Vec<Token>

Tokenizes a line of markdown text into a vector of Token enums.

Arguments

• markdown_line - A string slice representing a line of markdown text.

Returns

A vector of Token enums representing the tokenized line.

Example

use lexer::tokenize;
use types::Token;
let tokens = tokenize("This is *italic* and **bold** text.");
assert_eq!(tokens.len(), 9);
assert_eq!(tokens[4], Token::EmphasisRun { delimiter: '*', length: 1 });