Scanner and Lexical analysis

tags: compiler design, systems

Scanning or lexical analysis is responsible for converting from stream of characters to tokens. The rules of the lexer are defined with a regular expression.

Token typed are defined in an enum, for example:

typedef enum {
	IF,
	THEN,
	ELSE,
	PLUS,
	NUM,
	ID
} TokenType;

Tokens are divided into categories like reserved words for if and then, or operators like + or tokens that can represent multiple things like NUM type

The value of the token is stored in the symbol table and a pointer to the value is returned, according to the doctor this happens only when the token can have multiple values

Tokens as logical entities must be distinguished from the value they represent, the value is defined as its string value or lexeme, some tokens have only one lexeme (reserved words)

The scanner computes several attributes for a single token, a single token can have the following struct

typedef struct {
	TokenType type;
	char* stringval;
	int numval;
} TokenRecord

Note: although to job of the scanner is to convert the whole source code is not converted at once, it happens in respect with the parser, if the parser needs the next token it can call a function maybe defined as getToken()