Lexical analyzer
:
Example
Source program:
Position := Initial + Rate * 60
Tokens:
id
E
Position
:=
id
E
I
nitial
+
id
E
Rate
*
int
60
Position
:=
Initial
+
Rate
*
60
Lexemes:
7/17