Ionflux::Tools::Utf8TokenTypeMap Class Reference
[String tokenizer]

Token type map with UTF-8 support. More...

#include <Utf8TokenTypeMap.hpp>

Inheritance diagram for Ionflux::Tools::Utf8TokenTypeMap:

[legend]Collaboration diagram for Ionflux::Tools::Utf8TokenTypeMap:


Public Member Functions
	Utf8TokenTypeMap ()
	Constructor.
virtual	~Utf8TokenTypeMap ()
	Destructor.
virtual void	clear ()
	Clear token type map.
virtual unsigned int	getNumTokenTypes ()
	Get number of token types.
virtual void	addTokenType (const Utf8TokenType &newTokenType)
	Add token type.
virtual void	removeTokenType (const Utf8TokenType &removeTokenType)
	Remove token type.
virtual void	setDefaultTokenType (const Utf8TokenType &newDefaultTokenType)
	Set default token type.
virtual Utf8TokenType	getTokenType (unsigned int testChar)
	Get token type.
Static Public Member Functions
static bool	isValid (const Utf8TokenType &checkTokenType)
	Validate token type.
Static Public Attributes
static const Utf8TokenTypeMapClassInfo	utf8TokenTypeMapClassInfo
	Class information instance.
static const Ionflux::Tools::ClassInfo *	CLASS_INFO
	Class information.
Protected Attributes
std::vector< Utf8TokenType * >	tokenTypes
	Token types.
ImplTokenTypeMap	typeMap
	The actual token type map.
Utf8TokenType *	defaultTokenType
	The default token type.

Detailed Description

Token type map with UTF-8 support.

A lookup map for mapping Unicode characters to token types.

Constructor & Destructor Documentation

Ionflux::Tools::Utf8TokenTypeMap::Utf8TokenTypeMap ( )

Constructor.
Construct new Utf8TokenTypeMap object.

Ionflux::Tools::Utf8TokenTypeMap::~Utf8TokenTypeMap ( ) [virtual]

Destructor.
Destruct Utf8TokenTypeMap object.

Member Function Documentation

void Ionflux::Tools::Utf8TokenTypeMap::addTokenType ( const Utf8TokenType & newTokenType ) [virtual]

Add token type.
Add a token type to the map

Parameters:

newTokenType Token type to be added.

void Ionflux::Tools::Utf8TokenTypeMap::clear ( ) [virtual]

Clear token type map.
Remove all token types from the map.

unsigned int Ionflux::Tools::Utf8TokenTypeMap::getNumTokenTypes ( ) [virtual]

Get number of token types.
Get the number of token types in the map.

Returns:
Number of token types.

Utf8TokenType Ionflux::Tools::Utf8TokenTypeMap::getTokenType ( unsigned int testChar ) [virtual]

Get token type.
Perform a token type lookup for the specified character.

Parameters:

testChar Unicode character.

Returns:
Token type.

bool Ionflux::Tools::Utf8TokenTypeMap::isValid ( const Utf8TokenType & checkTokenType ) [static]

Validate token type.
Check whether the specified token type represents a valid token (i.e. it is not invalid or empty).

Parameters:

checkTokenType Token type to be checked.

Returns:
true if the specified token type is valid, false otherwise.

void Ionflux::Tools::Utf8TokenTypeMap::removeTokenType ( const Utf8TokenType & removeTokenType ) [virtual]

Remove token type.
Remove a token type from the map

Parameters:

removeTokenType Token type to be removed.

void Ionflux::Tools::Utf8TokenTypeMap::setDefaultTokenType ( const Utf8TokenType & newDefaultTokenType ) [virtual]

Set default token type.
Set the token type to be returned if a character does not match any token type in the map.

Parameters:

newDefaultTokenType Default token type.