Token

Definition

Token classification and representation for expression parsing in fpx.

This module provides the lightweight but robust token infrastructure used by the fpx preprocessor when evaluating constant expressions in #if / #elif directives.

It defines:

• A clean enumeration of token kinds (tokens_enum)
• A simple token derived type that carries both the lexical value and its semantic category

These types are used internally by evaluate_expression() (from fpx_token) to parse and compute #if DEBUG > 1 && defined(USE_MPI)-style conditions.

Key design goals

• Minimal memory footprint
• Clear separation between lexical scanning and semantic interpretation
• Easy extensibility for future operators or functions

Examples

1. Manual token creation (mostly for testing/debugging):

   use fpx_token
    
   type(token) :: t1, t2, t3
    
   t1 = token('42',   number)         ! numeric literal
   t2 = token('DEBUG', identifier)    ! macro name
   t3 = token('>',    operator)       ! comparison operator
    
   print *, 'Token: ', t1%value, ' type=', t1%type   ! → 42 type=0

2. Typical internal usage during #if evaluation:

   ! (inside evaluate_expression)
   tokens = tokenize('defined(USE_MPI) && MPI_VERSION >= 3')
   ! tokens(1) → value=vdefined'  type=identifier
   ! tokens(2) → value='('        type=parenthesis
   ! tokens(3) → value='USE_MPI'  type=identifier
   ! ...

Token kinds overview

Enumerator	Value	Meaning
unknown	-1	Invalid / unrecognized token
number	0	Integer or floating-point literal

| operator | 1 | ?:, +, -, *, /, ==, !=, &&, ||, !, >, <, etc.| | identifier | 2 | Macro name or function name (e.g. defined) | | parenthesis| 3 | ( or ) | | defined | 4 | Special keyword defined (treated specially)|

Data Types
type	token
	Represents a single token in a parsed expression. Holds the string value of the token and its classified type. More...

Enumerations
enum
	Token kinds used in expression parsing. Enumeration defining the possible types of tokens recognized by the tokenizer. More...

Variables

tokens_enum

integer, parameter, public tokens_enum = kind(unknown)

Kind parameter for token type enumeration. Values are (unknown, number, operation, identifier, parenthesis, defined).

Definition at line 87 of file token.f90.

Methods

is_digit()

logical elemental function is_digit ( character(*), intent(in) ch )

private

Tests whether a single character is a decimal digit ('0'-'9').

Parameters

[in]

ch

Character to test

Returns

.true. if ch is a digit

Remarks

Definition at line 309 of file token.f90.

is_typeless()

logical function is_typeless ( character(*), intent(in) str, integer, intent(out) pos )

private

Detects whether a string starts a typeless constant (hex, octal, binary). Used to avoid treating them as identifiers during tokenization.

Parameters

[in]	str	Input string starting at current position
[out]	pos	Length of the typeless constant (0 if not typeless)

Returns

.true. if the prefix is a valid typeless constant in non-base-10

Remarks

Definition at line 323 of file token.f90.

tokenize()

subroutine, public tokenize	(	character(*), intent(in)	expr,
		type(token), dimension(:), intent(out), allocatable	tokens,
		integer, intent(out)	ntokens )

Tokenizes a preprocessor expression into an array of token structures. Handles whitespace, multi-character operators (&&, ||, ==, etc.), the defined operator (with or without parentheses), numbers in various bases, identifiers, and parentheses.

Parameters

[in]	expr	Expression string to tokenize
[out]	tokens	Allocated array receiving the tokens
[out]	ntokens	Number of tokens produced

Remarks

Definition at line 161 of file token.f90.