Token

Definition

Token classification and representation for expression parsing in fpx

This module provides the lightweight but robust token infrastructure used by the fpx preprocessor when evaluating constant expressions in #if / #elif directives.

It defines:

• A clean enumeration of token kinds (tokens_enum)
• A simple token derived type that carries both the lexical value and its semantic category

These types are used internally by evaluate_expression() (from fpx_token) to parse and compute #if DEBUG > 1 && defined(USE_MPI)-style conditions.

Key design goals

• Minimal memory footprint
• Clear separation between lexical scanning and semantic interpretation
• Easy extensibility for future operators or functions

Examples

1. Manual token creation (mostly for testing/debugging):

   use fpx_token
    
   type(token) :: t1, t2, t3
    
   t1 = token('42',   number)         ! numeric literal
   t2 = token('DEBUG', identifier)    ! macro name
   t3 = token('>',    operator)       ! comparison operator
    
   print *, 'Token: ', t1%value, ' type=', t1%type   ! → 42 type=0

2. Typical internal usage during #if evaluation:

   ! (inside evaluate_expression)
   tokens = tokenize('defined(USE_MPI) && MPI_VERSION >= 3')
   ! tokens(1) → value=vdefined'  type=identifier
   ! tokens(2) → value='('        type=parenthesis
   ! tokens(3) → value='USE_MPI'  type=identifier
   ! ...

Token kinds overview

Enumerator	Value	Meaning
unknown	-1	Invalid / unrecognized token
number	0	Integer or floating-point literal

| operator | 1 | ?:, +, -, *, /, ==, !=, &&, ||, !, >, <, etc.| | identifier | 2 | Macro name or function name (e.g. defined) | | parenthesis| 3 | ( or ) | | defined | 4 | Special keyword defined (treated specially)|

Data Types
type	token
	Represents a single token in a parsed expression. Holds the string value of the token and its classified type. More...

Enumerations
enum	{   unknown = -1 , number = 0 , operator = 1 , identifier = 2 ,   parenthesis = 3 , defined = 4 }
	Token kinds used in expression parsing. Enumeration defining the possible types of tokens recognized by the tokenizer. More...

Variables

tokens_enum

integer, parameter, public tokens_enum = kind(unknown)

Kind parameter for token type enumeration. Values are (unknown, number, operator, identifier, parenthesis, defined)

Definition at line 87 of file token.f90.

Methods

is_digit()

logical elemental function is_digit ( character(*), intent(in) ch )

private

Tests whether a single character is a decimal digit ('0'-'9').

Parameters

[in]

ch

Character to test

Returns

.true. if ch is a digit

Remarks

Definition at line 298 of file token.f90.

is_typeless()

logical function is_typeless ( character(*), intent(in) str, integer, intent(out) pos )

private

Detects whether a string starts a typeless constant (hex, octal, binary). Used to avoid treating them as identifiers during tokenization.

Parameters

[in]	str	Input string starting at current position
[out]	pos	Length of the typeless constant (0 if not typeless)

Returns

.true. if the prefix is a valid typeless constant in non-base-10

Remarks

Definition at line 312 of file token.f90.

tokenize()

subroutine, public tokenize	(	character(*), intent(in)	expr,
		type(token), dimension(:), intent(out), allocatable	tokens,
		integer, intent(out)	ntokens )

Tokenizes a preprocessor expression into an array of token structures. Handles whitespace, multi-character operators (&&, ||, ==, etc.), the defined operator (with or without parentheses), numbers in various bases, identifiers, and parentheses.

Parameters

[in]	expr	Expression string to tokenize
[out]	tokens	Allocated array receiving the tokens
[out]	ntokens	Number of tokens produced

Remarks

Definition at line 160 of file token.f90.