Tokens processing utilities. More...

Functions
void	postprocess_tokens (tokens_t &tokens)
	Perform postprocessing on a vector of tokens.

void	insert_tokens (tokens_t &tokens, tokens_t::iterator &pos, const tokens_t &sub_tokens)
	Insert a sequence of tokens into a vector of tokens on a given position.

tokens_t	split_words (const std::string_view input)
	Perform word splitting on the given input string, breaking it down into a vector of WORD tokens.

void	set_variables (tokens_t &tokens)
	Sets internal variables based on VAR_DECL tokens in a vector of tokens.

void	expand_aliases (tokens_t &tokens)
	Expand command aliases within a vector of tokens.

void	expand_vars (tokens_t &tokens)
	Expand variables within a vector of tokens.

void	substitute_commands (tokens_t &tokens)
	Perform command substitution within a vector of tokens.

void	expand_glob (tokens_t &tokens)
	Expand glob patterns within a vector of tokens.

void	squash_tokens (tokens_t &tokens)
	Squash the adjacent tokens flagged as WORD_LIKE.

void	check_syntax (const tokens_t &tokens)
	Check the syntax of a vector of tokens.

simple_command_ptr	make_simple_command (const tokens_t &tokens)
	Construct a shared pointer to a simple command from a vector of tokens.

command	split_commands (tokens_t &tokens)
	Split a vector of tokens into a tree structure of commands.

void	process_tokens (tokens_t &tokens)
	Process a vector of tokens.

Detailed Description

Tokens processing utilities.

Function Documentation

◆ check_syntax()

void check_syntax ( const tokens_t & tokens )

Check the syntax of a vector of tokens.

If syntax is incorrect, throws an exception.

Parameters

tokens A vector of tokens to check.

Exceptions

msh_exception if syntax is incorrect.

See also: Token; token_flags

◆ expand_aliases()

void expand_aliases ( tokens_t & tokens )

Expand command aliases within a vector of tokens.

Takes a vector of tokens and expands command aliases by replacing alias, defined with malias, with their corresponding token sequences. Uses a stack-based approach to perform alias expansion, ensuring that nested aliases are also expanded. Only COMMAND tokens are eligible for alias expansion.

Parameters

tokens A vector of tokens to expand aliases within.

Note: Alias expansion is performed in-place, i.e. the input vector is modified.

See also: malias; Token; token_flags; aliases; lexer()

◆ expand_glob()

void expand_glob ( tokens_t & tokens )

Expand glob patterns within a vector of tokens.

Iterates through a vector of tokens and expands glob patterns within tokens by matching them against files in the file system. If no files are matched, the glob pattern is left unchanged. Token with matched glob pattern is replaced with a sequence of WORD tokens containing the matched file names.

Tokens flagged as GLOB_NO_EXPAND are not expanded.

Parameters

tokens A vector of tokens to process and expand glob patterns within.

Note: Glob expansion is performed in-place, i.e. the input vector is modified.

See also: Token; token_flags; glob

◆ expand_vars()

void expand_vars ( tokens_t & tokens )

Expand variables within a vector of tokens.

Iterates through a vector of tokens and expands variables within tokens by replacing them with their corresponding values. The expansion priority is given to internal variables. If no variable with the given name is found, expansion result to an empty string.

Tokens flagged as VAR_NO_EXPAND are not expanded. The word splitting is performed on the resulting string, unless the token is flagged as NO_WORD_SPLIT.

Parameters

tokens A vector of tokens to process and expand variables within.

Note: Variable expansion is performed in-place, i.e. the input vector is modified.

See also: Token; token_flags; get_variable

◆ insert_tokens()

void insert_tokens	(	tokens_t &	tokens,
		tokens_t::iterator &	pos,
		const tokens_t &	sub_tokens )

Insert a sequence of tokens into a vector of tokens on a given position.

Parameters

tokens	A vector of tokens to insert into.
pos	An iterator pointing to the position to insert tokens at.
sub_tokens	A vector of tokens to insert.

Note: The iterator is updated to point to the last inserted token. The token on the given position is erased.

See also: Token; std::advance

◆ make_simple_command()

simple_command_ptr make_simple_command ( const tokens_t & tokens )

Construct a shared pointer to a simple command from a vector of tokens.

Parameters

tokens A vector of tokens to construct a simple command from.

Returns: A shared pointer to a simple command.

See also: simple_command

◆ postprocess_tokens()

void postprocess_tokens ( tokens_t & tokens )

Perform postprocessing on a vector of tokens.

Postprocessing can include some specific operations that must be performed after tokenization, but before any other processing steps. For example, variable declarations that appear as malias or mexport commands are not eligible for word splitting and filename expansion.

Parameters

tokens A vector of tokens to postprocess.

See also: Token; token_flags; lexer()

◆ process_tokens()

void process_tokens ( tokens_t & tokens )

Process a vector of tokens.

Performs all necessary token processing steps in the correct order.

Parameters

tokens A vector of tokens to process.

Exceptions

msh_exception if an error occurs during token processing. Must be handled elsewhere.

See also: expand_vars; substitute_commands; set_variables; expand_glob; squash_tokens

◆ set_variables()

void set_variables ( tokens_t & tokens )

Sets internal variables based on VAR_DECL tokens in a vector of tokens.

Iterates through a vector of tokens, identifies VAR_DECL tokens, and sets corresponding variables based on their values. If a VAR_DECL token is followed by a token flagged as WORD_LIKE without separators, the two tokens are combined and the resulting string is assigned to the variable.

Parameters

tokens A vector of tokens to process and update variable values.

Note: VAR_DECL tokens are removed from the vector after processing.

See also: Token; token_flags; set_variable

◆ split_commands()

command split_commands ( tokens_t & tokens )

Split a vector of tokens into a tree structure of commands.

Each node of the tree is a command object, which can be either a simple command or a connection command. The connection command contains a connector token and two command objects, lhs and rhs. Returns the root of the tree.

Note: Alias expansion is performed on whole command sequence before splitting.

Parameters

tokens A vector of tokens to split.

Returns: The root of the tree of commands.

See also: command; simple_command; connection_command

◆ split_words()

tokens_t split_words ( const std::string_view input )

Perform word splitting on the given input string, breaking it down into a vector of WORD tokens.

Parameters

input The input string to be split.

Returns: A vector of WORD tokens.

Note: WORD tokens are separated according to the value of the IFS environment variable, if set, otherwise the delimiters are <space>, <tab> and <newline>. Embedded newlines can be removed during this operation.

◆ squash_tokens()

void squash_tokens ( tokens_t & tokens )

Squash the adjacent tokens flagged as WORD_LIKE.

Parameters

tokens A vector of tokens to process.

◆ substitute_commands()

void substitute_commands ( tokens_t & tokens )

Perform command substitution within a vector of tokens.

Iterates through a vector of tokens and replaces tokens of type COM_SUB with the output of the underlying command by executing it in a subshell.

Any trailing newlines are removed from the output. The word splitting is performed on the resulting string, unless the token is flagged as NO_WORD_SPLIT.

Parameters

tokens A vector of tokens to process and substitute command substitutions within.

Note: Command substitution is performed in-place, i.e. the input vector is modified.

Exceptions

msh_exception if an error occurs during command execution.

See also: Token; token_flags; parse_input

Functions

Detailed Description

Function Documentation

◆ check_syntax()

◆ expand_aliases()

◆ expand_glob()

◆ expand_vars()

◆ insert_tokens()

◆ make_simple_command()

◆ postprocess_tokens()

◆ process_tokens()

◆ set_variables()

◆ split_commands()

◆ split_words()

◆ squash_tokens()

◆ substitute_commands()