AALanguage: LexicalAnalyzer Class Reference

AALanguage

The best language for those who have nothing to do

Loading...

Searching...

No Matches

A class that allows you to break program source code into indivisible grammar tokens. More...

#include <LexicalAnalyzer.h>

Public Types
enum	literal_type { logical , integer , real , symbol , string , none }
	Enumeration for a literal category. More...

enum	token_type { service , identifier , literal , operation , punctuation , comma , brace , comment , unknown }
	Enumeration for possible types of language tokens. Tokens with the comment type are ignored. More...

Public Member Functions
Token	get_token ()
	Getting the next token from the generated array.

Token	prev_token ()
	Getting the previous token from the generated array.

bool	get_ok () const
	Gains information about the success of lexical analysis.

literal_type	is_literal (std::string s)
	Checks if the string is a valid literal from the language grammar.

std::vector< Token >	get_tokens ()
	Getting all tokens from the analyzed file.

	LexicalAnalyzer (std::string path, LibParser &lib_parser)
	A class constructor that performs lexical analysis on the specified file.

Private Member Functions
bool	is_service (Trie *service_trie, std::string s)

bool	is_identifier (std::string s)

bool	is_operation (std::string s)

bool	is_punctuation (std::string s)

bool	is_comma (std::string s)

bool	is_brace (std::string s)

bool	is_comment (std::string s)

token_type	get_token_type (Trie *service_trie, std::string s)

void	print_token (Trie *service_trie, std::string buffer)

std::vector< Token >	merge (std::vector< Token > &f, std::vector< Token > &s, int &ind, int)

Private Attributes
int	line_ = 1

int	curr_ = 0

bool	ok_ = true

std::vector< Token >	tokens_

Detailed Description

A class that allows you to break program source code into indivisible grammar tokens.

Definition at line 11 of file LexicalAnalyzer.h.

Member Enumeration Documentation

◆ literal_type

enum LexicalAnalyzer::literal_type

Enumeration for a literal category.

Enumerator
logical
integer
real
symbol
string
none

Definition at line 14 of file LexicalAnalyzer.h.

◆ token_type

enum LexicalAnalyzer::token_type

Enumeration for possible types of language tokens. Tokens with the comment type are ignored.

Enumerator
service
identifier
literal
operation
punctuation
comma
brace
comment
unknown

Definition at line 18 of file LexicalAnalyzer.h.

Constructor & Destructor Documentation

◆ LexicalAnalyzer()

LexicalAnalyzer::LexicalAnalyzer	(	std::string	path,
		LibParser &	lib_parser )

A class constructor that performs lexical analysis on the specified file.

Uses Trie to create a dictionary with all the reserved words of a language.

Greedily types characters into the current token until an error occurs due to an unexpected character. Amortized asymptotics of O(n) partitioning.

Causes an exception if parsing a token with type unknown.

Replaces all using directives with an array of tokens of libraries used in these directives.

Parameters

path	Path to the file with the source code of the program in AALanguage
lib_parser	Reference to an instance of the LibParser class for parsing code from built-in libraries

Returns: Vector array with all tokens obtained during lexical analysis.

Definition at line 115 of file LexicalAnalyzer.cpp.

References Trie::add(), LibParser::add_lib(), brace, comma, comment, get_token_type(), is_punctuation(), LexicalAnalyzer(), line_, merge(), LibParser::name_to_path(), operation, print_token(), punctuation, tokens_, and unknown.

Referenced by LexicalAnalyzer().

Member Function Documentation

◆ get_ok()

bool LexicalAnalyzer::get_ok ( ) const

Gains information about the success of lexical analysis.

Returns: True if the analysis was error-free - no unexpected character set was encountered that is not a token of the language.

Definition at line 92 of file LexicalAnalyzer.cpp.

References ok_.

◆ get_token()

Token LexicalAnalyzer::get_token ( )

Getting the next token from the generated array.

Returns: Token class instance for the following lexeme.

Definition at line 80 of file LexicalAnalyzer.cpp.

References curr_, tokens_, and unknown.

Referenced by Analyzer::assignment_expression(), Analyzer::bitwise_and_expression(), Analyzer::bitwise_consequence_expression(), Analyzer::bitwise_or_expression(), Analyzer::bitwise_shift_expression(), Analyzer::bitwise_xor_expression(), Analyzer::comparison_expression(), Analyzer::construct_expression(), Analyzer::convert_statement(), Analyzer::default_type(), Analyzer::equality_expression(), Analyzer::execute(), Analyzer::exit_statement(), Analyzer::expression(), Analyzer::field(), Analyzer::for_statement(), Analyzer::function(), Analyzer::function_or_var_definitions(), Analyzer::goto_statement(), Analyzer::if_statement(), Analyzer::logical_and_expression(), Analyzer::logical_or_expression(), Analyzer::may_be_semicolon(), Analyzer::multiply_expression(), Analyzer::namespace_definition(), Analyzer::parameter_list(), Analyzer::plus_expression(), Analyzer::preprocessor(), Analyzer::print_statement(), Analyzer::rand_statement(), Analyzer::read_statement(), Analyzer::readln_statement(), Analyzer::return_statement(), Analyzer::semicolon(), Analyzer::statement(), Analyzer::strlen_statement(), Analyzer::switch_statement(), Analyzer::time_statement(), Analyzer::type(), Analyzer::unary_expression(), Analyzer::var_definition(), Analyzer::var_definitions(), and Analyzer::while_statement().

◆ get_token_type()

LexicalAnalyzer::token_type LexicalAnalyzer::get_token_type	(	Trie *	service_trie,
		std::string	s )

private

Definition at line 56 of file LexicalAnalyzer.cpp.

References brace, comma, comment, identifier, is_brace(), is_comma(), is_comment(), is_identifier(), is_literal(), is_operation(), is_punctuation(), is_service(), literal, none, operation, punctuation, service, and unknown.

Referenced by LexicalAnalyzer(), and print_token().

◆ get_tokens()

std::vector< Token > LexicalAnalyzer::get_tokens ( )

Getting all tokens from the analyzed file.

Returns: Vector array with all tokens obtained during lexical analysis.

Definition at line 96 of file LexicalAnalyzer.cpp.

References tokens_.

◆ is_brace()

bool LexicalAnalyzer::is_brace ( std::string s )

private

Referenced by get_token_type().

◆ is_comma()

bool LexicalAnalyzer::is_comma ( std::string s )

private

Referenced by get_token_type().

◆ is_comment()

bool LexicalAnalyzer::is_comment ( std::string s )

private

Referenced by get_token_type().

◆ is_identifier()

bool LexicalAnalyzer::is_identifier ( std::string s )

private

Definition at line 9 of file LexicalAnalyzer.cpp.

Referenced by get_token_type().

◆ is_literal()

LexicalAnalyzer::literal_type LexicalAnalyzer::is_literal ( std::string s )

Checks if the string is a valid literal from the language grammar.

Performs a check for each literal type using std::regex_match.

Parameters

s	String to validate

Returns: The appropriate literal enumerator, or none if the string is not a literal.

Definition at line 14 of file LexicalAnalyzer.cpp.

References integer, logical, none, real, string, and symbol.

Referenced by Analyzer::function(), get_token_type(), Semantic::literal_to_type(), Analyzer::var_definition(), and Analyzer::var_definitions().

◆ is_operation()

bool LexicalAnalyzer::is_operation ( std::string s )

private

Referenced by get_token_type().

◆ is_punctuation()

bool LexicalAnalyzer::is_punctuation ( std::string s )

private

Referenced by get_token_type(), and LexicalAnalyzer().

◆ is_service()

bool LexicalAnalyzer::is_service	(	Trie *	service_trie,
		std::string	s )

private

Definition at line 5 of file LexicalAnalyzer.cpp.

References Trie::find().

Referenced by get_token_type().

◆ merge()

std::vector< Token > LexicalAnalyzer::merge	(	std::vector< Token > &	f,
		std::vector< Token > &	s,
		int &	ind,
		int	start_ptr )

private

Definition at line 100 of file LexicalAnalyzer.cpp.

Referenced by LexicalAnalyzer().

◆ prev_token()

Token LexicalAnalyzer::prev_token ( )

Getting the previous token from the generated array.

Returns: Token class instance for the previous lexeme.

Definition at line 87 of file LexicalAnalyzer.cpp.

References curr_, and tokens_.

Referenced by Analyzer::for_statement(), Analyzer::function_or_var_definitions(), Analyzer::semicolon(), Analyzer::statement(), Analyzer::var_definition(), and Analyzer::var_definitions().

◆ print_token()

void LexicalAnalyzer::print_token	(	Trie *	service_trie,
		std::string	buffer )

private

Definition at line 68 of file LexicalAnalyzer.cpp.

References comment, get_token_type(), line_, ok_, tokens_, and unknown.

Referenced by LexicalAnalyzer().

Member Data Documentation

◆ curr_

int LexicalAnalyzer::curr_ = 0

private

Definition at line 75 of file LexicalAnalyzer.h.

Referenced by get_token(), and prev_token().

◆ line_

int LexicalAnalyzer::line_ = 1

private

Definition at line 74 of file LexicalAnalyzer.h.

Referenced by LexicalAnalyzer(), and print_token().

◆ ok_

bool LexicalAnalyzer::ok_ = true

private

Definition at line 76 of file LexicalAnalyzer.h.

Referenced by get_ok(), and print_token().

◆ tokens_

std::vector<Token> LexicalAnalyzer::tokens_

private

Definition at line 77 of file LexicalAnalyzer.h.

Referenced by get_token(), get_tokens(), LexicalAnalyzer(), prev_token(), and print_token().

The documentation for this class was generated from the following files:

D:/GitHub/AALanguage2/AALanguage/LexicalAnalyzer.h
D:/GitHub/AALanguage2/AALanguage/LexicalAnalyzer.cpp

LexicalAnalyzer
Generated by 1.10.0