Stages of compiler design

Learning objective:

12.5.1.3 describe program compilation stages: lexical and syntactic analysis, code generation, and optimization​

When a programmer uses a computer language (high-level) to write a program, the statements are called source code.

Source code is a human-readable text written in a specific programming language.

The compiler translates source code into a low-level language. The compiled code is saved as an object file.

Stages in the compilation of a program:

Lexical analysis

Lexical analysis is the process of analyzing a stream of individual characters (normally arranged as lines) into a sequence of lexical tokens (tokenization of words and symbols) to feed into the parser that the compiler will understand. 

Source code is written by programmers using ASCII characters. During lexical analysis, the compiler breaks down this stream of ASCII characters into its parts, called "lexemes".

A lexeme is the smallest unit of language. Lexemes cannot be broken down further without losing meaning.

Each lexeme defines a token corresponding to it.
Tokens:

Tokens

Lexical analysis

• Identifies lexemes within the source code;
• White space and comments are removed;
• Generates a stream of tokens;
• Each token is passed to the parser on request;
• Creates a symbol table of 'names' used in the source code;
• Limited error checking at this stage.

Video - Compilation - Part Two: Lexical Analysis

Syntactic analysis

This is alternatively known as parsing. This stage analyses the syntax of the statements to ensure they conform to the rules of grammar for the computer language in question.
It is roughly the equivalent of checking that some ordinary text is written in a natural language (e.g., English) and is grammatically correct (without worrying about meaning).
The purpose of syntax analysis or parsing is to check that we have a valid sequence of tokens. Tokens are a valid sequence of symbols, keywords, identifiers, etc. 

Syntax analyzers receive their inputs, in the form of tokens, from lexical analyzers. Lexical analyzers are responsible for the validity of a token supplied by the syntax analyzer.

Drawbacks of syntax analyzers:

• It cannot determine if a token is valid,
• It cannot determine if a token is declared before it is used,
• It cannot determine if a token is initialized before it is used,
• It cannot determine if an operation performed on a token type is valid or not.

Abstract Syntax Tree (AST) – the Program’s Family Tree

After lexical and syntactic analysis, the compiler does not just keep the code as raw text. Instead, it builds a special structure called an Abstract Syntax Tree (AST). Think of the AST as a “family tree” of your program: each node represents a construct (like an operation or a variable), and the connections between nodes show how these constructs relate to one another.

Unlike the raw source code, the AST ignores details like punctuation or parentheses placement. What matters is the hierarchy of meaning. For example, the expression:

X - 2 * Y

In plain text, this looks linear. But in the AST, the structure reveals the true order of operations:

This “tree view” makes it easier for the compiler to perform optimizations and generate efficient machine code. In other words, the AST is the bridge between human-readable code and the machine’s execution plan.
Video - Compilation - Part Three: Syntax Analysis

Code optimization

Making the compile time as short as possible. Optimization is a program transformation technique that tries to improve the code by making it consume fewer resources (i.e., CPU, Memory) and deliver high speed. The system programmer is responsible for optimization.

Optimizations provided by a compiler include:

• Inlining small functions 
• Code hoisting moves loop-invariant code out of loops.
• Reduction in strength means code optimization obtained by the user of simpler machine instructions.
• Dead store/code elimination removes values that never get used.
• Eliminating common sub-expressions 
• Loop unrolling avoids tests at every iteration of the loop.
• Loop optimizations: Code motion, Induction variable elimination, and Reduction in strength.

Code generation

The code generated by the compiler is an object file.  In compiler design, object code refers to the code that the compiler generates after analyzing the program's structure and meaning and optimizing it. This code is a machine-readable version of the source code, allowing the computer’s CPU to run it directly. When the file runs, the machine code is processed by the CPU. 

Minimum properties of low-level object code:​

• It should carry the exact meaning of the source code.​
• It should be efficient in terms of CPU usage and memory management.​

Using resources: Syntax Analysis 

Stages of compilation (Isaac CS)

Questions:

Exercises:

Ex. 1 Drag and Drop

Ex. 2 Fill the table

Ex. 3 "Order the stages of compilation"

Ex. 4 "Fill the gaps" (Author: Litvinova Olga - CS teacher of NIS Pavlodar)

Ex. 5 

Exam questions: