if (i == j)
z = 0;
else
z = 1;
\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;
Token Class(or Class)
-
In English
- Noun, Verb, Adjective...
-
In a programming language:
- Identifier, keywords,'(', ')', Numbers
Token classes correspond to sets of strings.
-
Identifier:
- strings of lettters or digits, starting with a letter.
-
Integer:
- a non-empty string of digits.
-
Keyword
- "else" or "if" or "begin" or...
-
Whitespace:
- a non-empty sequence of blanks, newlines, and tabs.
Classify program substrings according to role.
Communicate tokens to the parser.
\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;
- Operator
- Whitespace
- Keywords
- Identifiers
- Numbers
- '(', ')', ; , =
WKWIWOWI)WIW=N...
An implementation must do two things:
- Recognize substrings corresponding to tokens
- The lexemes
- Identify the token class of each lexeme
- <token class, lexeme> is called a token.
