Token

Definition

Token classification and representation for expression parsing in fpx.

This module provides the lightweight but robust token infrastructure used by the fpx preprocessor when evaluating constant expressions in #if / #elif directives.

It defines:

• A clean enumeration of token kinds (tokens_enum)
• A simple token derived type that carries both the lexical value and its semantic category

These types are used internally by evaluate_expression() (from fpx_token) to parse and compute #if DEBUG > 1 && defined(USE_MPI)-style conditions.

Key design goals

• Minimal memory footprint
• Clear separation between lexical scanning and semantic interpretation
• Easy extensibility for future operators or functions

Examples

1. Manual token creation (mostly for testing/debugging):

   use fpx_token
    
   type(token) :: t1, t2, t3
    
   t1 = token('42',   number)         ! numeric literal
   t2 = token('DEBUG', identifier)    ! macro name
   t3 = token('>',    operator)       ! comparison operator
    
   print *, 'Token: ', t1%value, ' type=', t1%type   ! -> 42 type=0

2. Typical internal usage during #if evaluation:

   ! (inside evaluate_expression)
   tokens = tokenize('defined(USE_MPI) && MPI_VERSION >= 3')
   ! tokens(1) -> value=vdefined'  type=identifier
   ! tokens(2) -> value='('        type=parenthesis
   ! tokens(3) -> value='USE_MPI'  type=identifier
   ! ...

Token kinds overview

Enumerator	Value	Meaning
unknown	-1	Invalid / unrecognized token
number	0	Integer or floating-point literal

| operator | 1 | ?:, +, -, *, /, ==, !=, &&, ||, !, >, <, etc.| | identifier | 2 | Macro name or function name (e.g. defined) | | parenthesis| 3 | ( or ) | | defined | 4 | Special keyword defined (treated specially)|

Data Types
type	token
	Represents a single token in a parsed expression. Holds the string value of the token and its classified type. More...

Variables

tokens_enum

integer, parameter, public tokens_enum = kind(unknown)

Kind parameter for token type enumeration. Values are (unknown, number, operation, identifier, parenthesis, defined).

Definition at line 87 of file token.f90.

Methods

tokenize()

subroutine, public tokenize	(	character(*), intent(in)	expr,
		type(token), dimension(:), intent(out), allocatable	tokens,
		integer, intent(out)	ntokens )

Tokenizes a preprocessor expression into an array of token structures. Handles whitespace, multi-character operators (&&, ||, ==, etc.), the defined operator (with or without parentheses), numbers in various bases, identifiers, and parentheses.

Parameters

[in]	expr	Expression string to tokenize
[out]	tokens	Allocated array receiving the tokens
[out]	ntokens	Number of tokens produced

Remarks

Definition at line 161 of file token.f90.