Compare commits

...

10 Commits

12 changed files with 415 additions and 32 deletions

View File

@ -0,0 +1,132 @@
## Phase 4
I implemented a basic compiler for the phase 4. This compiler translates the given program file into GNU assembly that is then compiled using GCC and linked against system libc. Libc is used for printing values and time manipulation.
### Semantic checking
Before the compilation check, full semantic check is performed on the code. All types are statically checked to be valid.
#### Levels of interpretation
1. All arithmetic expressions and print statements are implemented
- Arithmetic expressions are checked during semantic checking.
- Valid addition expressions
- [int] + [int] -> [int]
- [date] + [int] -> [date]
- Valid subtraction expression
- [int] - [int] -> [int]
- [date] - [int] -> [date]
- [date] - [date] -> [int]
- Valid multiplication expressions
- [int] * [int] -> [int]
- Valid division expressions
- [int] / [int] -> [int]
- Valid comparisons
- [int] = [int] -> [bool]
- [date] = [date] -> [bool]
- [int] < [int] -> [bool]
- [date] < [date] -> [bool]
- Print statement can take a list of [int], [date] or [string] types
2. Global variables can be defined and they are accessible from anywhere in the code
3. For [date] variables, reading a field is possible. Writing is not because of limitations of assembly language.
- Readable fields include
- [date]'day -> [int], day number in month (1-31)
- [date]'month -> [int], month number in year (1-12)
- [date]'year -> [int], year number
- [date]'weekday -> [int], number of the day in week (1-7)
- [date]'weeknum -> [int], number of the week in year (1-53)
- Writable fields include
- [date].day -> [int], day number in month (1-31)
- [date].month -> [int], month number in year (1-12)
- [date].year -> [int], year number
- Writing to date will pass semantic checking but fail during compilation
4. Unless-statement, unless-expression and until-loop all work as expected
5. Function and procedure definitions and calls are properly handled and work as expected
- Function and procedure return types are validated during semantic check
- Auto keyword works. In this case the return type is determined using the type of returned value.
6. There is no runtime checking done as my compiler has static typing. All variables and functions are typed, so there is no need for doing it during runtime.
7. Recursion and proper local variables are implemented. This happens pretty much automatically when local variables and function arguments are kept on stack.
#### Test cases
All tests in _public\_examples/04\_semantics\_and\_running/running\_examples_ pass semantic checking and output the correct values.
All tests in _public\_examples/04\_semantics\_and\_running/semantic\_error\_examples_ fail during semantic checking.
### Compiler ABI and generated code
Some insights on how different data types are handled and how the compiler works in general.
- Integer literals are stored as 64 bit values and arithmetic is done using basic assembly instructions
- Date literals are stored as an Unix timestamp (seconds since 1970). To add or subtract days, the right operand is multiplied by 86400 (seconds in 24 hours) for the timestamp to work expecedly. Subtracting dates is done using basic integer arithmetic and dividing the result by 86400 which gives number of days as the difference.
- String literals are stored in the .data section with automatically generated labels.
- Printing
- Integers are printed using `printf("%lld", value)`, so `printf` takes care of the formatting
- Strings are printed using `printf("%s", label)`, where label is the strings read-only label in .data section
- Dates are printed by first calling `localtime(timestamp)` to get `struct tm*` pointer containing broken down time. Then dates are formatted into a fixed buffer in .bss section using `strftime(date_buffer, date_buffer_size, "%Y-%m-%d", tm)`, to format the string into the wanted form. After the date buffer has formatted string, it can printed using `printf("%s", date_buffer)`.
- `Today()` builtin function calls `__builtin_today` which calls `time(NULL)` from libc. This function returns current unix timestamp, so it is really suitable for this usage.
- Reading attributes from date objects is kind of hacky. It is mostly done as printing date objects. First date is converted into `struct tm*` with `localtime(timestamp)`, and formatted to wanted attribute (day -> "%d", month -> "%m", ...). This is then formatted into the date buffer using `strftime`. After formatting date buffer should contain only the asked part of the date. This can be converted into integer using the `atoi()` function.
- Writing date attributes is not implemented as that would require modifying `struct tm*` which is not feasible in pure assembly as the actual structure of it is unknown. There doesn't seem to be easy way do determine size and offset of a field in structure in GNU assembly.
- Conditional statements are done comparing the low byte of rax. After which a conditional jump happens to a label defined in the code.
- Calling convension of generated assembly code is as follows
- Arguments are pushed to stack before function call
- Local variables are initialized to proper values in the stack after as the first step in function call
- Rest of the code is generated
- Using either arguments or local variables happens through offset to rbp which contains the current stack frame.
- Return value of each operation of pythons `compile_ast()` is stored in rax register
- Stack is always aligned to 16-byte boundary according to _System V ABI_. This is because called libc functions might use sse-instructions that depend on this alignment.
- Global variables are stored in .bss section. The first thing main function does, is initialize global variables to proper values.
### Optimization
There is also an optional `-O` flag for the "compiler". This does a simple optimization pass over the generated assembly and optimizes some obvious instructions. These are the current optimizations implemented
- Moving from a register to the same register is removed
- `movq %rax, %rax` => `nop`
- Moving into rax and then pusing into rax is optimized to a single push instruction
- `movq $1, %rax; pushq %rax` => `pushq $1`
- Moving into rax ant then moving out of rax is optimized to single move instruction without accessing rax
- `movq $1, %rax; movq %rax, %rcx` => `movq $1, %rcx`
- Repeated addition and subtraction instruction with immediate values on the same register are combined into a signle instruction
- `addq $5, %rax; subq $2, %rax` => `addq $3, %rax`
- Moving immediate into register followed by add or subtract on the same register is optimized to single move
- `movq $5, %rax; subq $2, %rax` => `movq $3, %rax`
- Adding or subtracting immediate 1 is optimized to corresponding inc or dec instruction
- `addq $1, %rax` => `incq %rax`
- Moving immediate zero to register is optimized to xor instruction
- `movq $0, %rax` => `xorq %rax, %rax`
These optimizations are not all something that can be done in all assembly language, but because the way my compiler generates code it does not expect value to stay in register.
For example normally `movq $4, %rax; movq %rax, %rcx` cannot be optimized to `movq $4, %rcx` since this assumes rax is not used afterwards. In my compiler this kind of assumptions are not done, so this is a valid optimization.
Depending on the code that is to be compiled, optimizations might not do anything meaningful, so this is left as a optional flag.
### Usage of the program
```
usage: main.py [-h] [-d] (--who | -f FILE) [-o OUTPUT] [-a ASSEMBLY] [-O] [-r]
options:
-h, --help show this help message and exit
-d, --debug debug?
--who print out student IDs and NAMEs of authors
-f FILE, --file FILE filename to process
-o OUTPUT, --output OUTPUT
output filename for compiled code. default (a.out)
-a ASSEMBLY, --assembly ASSEMBLY
output filename for generated assembly code
-O, --optimize run simple optimization steps on the generated assembly code
-r, --run run the compiled code after compilation
```
By default when the program is run, it compiles file pointed by `-f` into binary called _a.out_. If you want the binary to be named something different, you can use the `-o` flag to specify output file for the generated binary.
If you are interested in the generated assembly, you can use the `-a` flag. This flag names a file where the generated assembly is dumped. Without this flag, no assembly is written to any file.
If you want to run the optimization pass, you can specify the `-O` flag. This flag takes no arguments. If both this flag and the `-a` flag are specified, the dumped assembly will the optimized verision.
To run the program automatically after compilation, you can specify the `-r` flag. This flag calls the output binary specified by `-o` flag and executes it.
### Requirements
For this program to function properly, you must have GCC installed and the required PLY library for python. Without GCC compilation step will fail. Assembly will be dumped even if there is no GCC available, as that happens before the invocation of GCC.

View File

@ -0,0 +1,16 @@
print 1 + 1,
print 10 - 5,
print 5 - 10,
print -5 + 2,
print -5 - 5,
print -5 - -5,
print 2 * 3,
print 2 * -3,
print -2 * 3,
print -2 * -3,
print 10 / 2,
print 10 / -2,
print -10 / 2,
print -10 / -2,
print 2024-04-27 - 5,
print 2024-04-27 + 5

View File

@ -0,0 +1,19 @@
var date = Today()
print date,
print date'day,
print date'month,
print date'year,
print date'weekday,
print date'weeknum,
date.day = 2,
date.month = 3,
date.year = 2'000,
print date,
print date'day,
print date'month,
print date'year,
print date'weekday,
print date'weeknum

View File

@ -0,0 +1 @@
print 1+1+1+1+1+1+1+1+1+1

View File

@ -0,0 +1 @@
print 1 + 2 - 3 + 5 - 7 + 11 - 13 + 17 - 19

View File

@ -0,0 +1,18 @@
var nn = 1'000'000'000
var count = 0
var sum = 0
function Func{ second_number[int] } return int
is
(second_number + 3) * 5
end function
(% Calculate sum with a loop %)
do
do
sum = sum + Func(count),
count = count + 1
until nn < count
unless nn = 0 done,
print "Sum from 0 to" & nn & "is" & sum

View File

@ -0,0 +1,6 @@
var num = -10
do
print num,
num = num + 1
until num = 11

View File

@ -0,0 +1,4 @@
print "hello" & "world",
print 10,
print 2024-04-27,
print Today() - 365*100

View File

@ -0,0 +1,6 @@
procedure PROC{ arg1[int], arg2[int], arg3[int], arg4[int], arg5[int] }
is
print arg1 & arg2 & arg3 & arg4 & arg5
end procedure
PROC(1, 2, 3, 4, 5)

View File

@ -0,0 +1,11 @@
procedure PROC{ }
var var1 = 1
var var2 = 2
var var3 = 3
var var4 = 4
var var5 = 5
is
print var1 & var2 & var3 & var4 & var5
end procedure
PROC()

View File

@ -0,0 +1,9 @@
var foo = 20
var bar = 10
print foo + bar,
print foo - bar,
print foo * bar,
print foo / bar,
print (foo + (foo + (foo + (foo + bar))))

View File

@ -335,16 +335,22 @@ class CompileData:
i = 0
# Remove redundant movq instructions
# movq %rax, %rax
while i < len(instructions):
if instructions[i].opcode != 'movq' or instructions[i].operands[0] != instructions[i].operands[1]:
i += 1
continue
instructions.pop(i)
changed = True
if changed: continue
i = 0
# Optimize movq to register followed by pushq register
while i < len(instructions):
# movq $1, %rax
# pushq %rax
# becomes
# pushq $1
while i < len(instructions) - 1:
if instructions[i].opcode != 'movq' or instructions[i + 1].opcode != 'pushq':
i += 1
continue
@ -355,10 +361,15 @@ class CompileData:
instructions.pop(i + 1)
i -= 1
changed = True
if changed: continue
i = 0
# Optimize movq to rax followed by movq from rax
while i < len(instructions):
# movq $1, %rax
# movq %rax, %rcx
# becomes
# movq $1, %rcx
while i < len(instructions) - 1:
if instructions[i].opcode != 'movq' or instructions[i + 1].opcode != 'movq':
i += 1
continue
@ -375,6 +386,119 @@ class CompileData:
instructions.pop(i + 1)
i -= 1
changed = True
if changed: continue
i = 0
# Replace negative immediate in addq/subq with positive immediate
# This is not a real optimization, but it makes the code easier to optimize
# subq $-1, %rax
# becomes
# addq $1, %rax
while i < len(instructions):
if instructions[i].opcode not in ['addq', 'subq']:
i += 1
continue
if instructions[i].operands[1][0] != '$':
i += 1
continue
value = int(instructions[i].operands[1][1:])
if value >= 0:
i += 1
continue
new_opcode = 'subq' if instructions[i].opcode == 'addq' else 'addq'
instructions[i] = Instruction(new_opcode, [instructions[i].operands[0], f'${-value}'])
changed = True
if changed: continue
i = 0
# Optimize repeated addq/subq instructions
# addq $1, %rax
# addq $2, %rax
# becomes
# addq $3, %rax
while i < len(instructions) - 1:
if instructions[i].opcode not in ['addq', 'subq']:
i += 1
continue
if instructions[i].operands[1][0] != '$' or instructions[i + 1].operands[1][0] != '$':
i += 1
continue
if instructions[i].operands[1] != instructions[i + 1].operands[1]:
i += 1
continue
lhs = int(instructions[i].operands[0][1:])
if instructions[i].opcode == 'subq': lhs = -lhs
rhs = int(instructions[i + 1].operands[0][1:])
if instructions[i + 1].opcode == 'subq': rhs = -rhs
new_value = lhs + rhs
if abs(new_value) > 0xFFFFFFFF:
i += 1
continue
new_opcode = 'addq' if new_value >= 0 else 'subq'
instructions[i] = Instruction(new_opcode, [f'${abs(new_value)}', instructions[i].operands[1]])
instructions.pop(i + 1)
i -= 1
changed = True
if changed: continue
i = 0
# Optimize movq immediate to register followed addq/subq with immediate
# movq $1, %rax
# addq $2, %rax
# becomes
# movq $3, %rax
while i < len(instructions) - 1:
if instructions[i].opcode != 'movq' or instructions[i + 1].opcode not in ['addq', 'subq']:
i += 1
continue
if instructions[i].operands[1] != instructions[i + 1].operands[1]:
i += 1
continue
if instructions[i].operands[0][0] != '$' or instructions[i + 1].operands[0][0] != '$':
i += 1
continue
lhs = int(instructions[i].operands[0][1:])
rhs = int(instructions[i + 1].operands[0][1:])
if instructions[i + 1].opcode == 'subq': rhs = -rhs
new_value = lhs + rhs
instructions[i] = Instruction('movq', [f'${new_value}', instructions[i].operands[1]])
instructions.pop(i + 1)
i -= 1
changed = True
if changed: continue
i = 0
# Optimize addq/subq for immediate 1
# addq $1, %rax
# becomes
# incq %rax
while i < len(instructions):
if instructions[i].opcode not in ['addq', 'subq']:
i += 1
continue
if instructions[i].operands[0] != '$1':
i += 1
continue
new_opcode = 'incq' if instructions[i].opcode == 'addq' else 'decq'
instructions[i] = Instruction(new_opcode, [instructions[i].operands[1]])
changed = True
if changed: continue
i = 0
# Optimize zeroing of register
# movq $0, %rax
# becomes
# xorq %rax, %rax
while i < len(instructions):
if instructions[i].opcode != 'movq':
i += 1
continue
if instructions[i].operands[0] != '$0' or instructions[i].operands[1][0] != '%':
i += 1
continue
instructions[i] = Instruction('xorq', [instructions[i].operands[1], instructions[i].operands[1]])
changed = True
if changed: continue
def add_builtin_functions(self) -> None:
today = []
@ -462,9 +586,23 @@ class CompileData:
break
saved_registers = list(saved_registers)
# Stack frame is needed for builtin functions, functions with local variables
# and functions that call other functions
needs_stack_frame = False
if name.startswith('__builtin'):
needs_stack_frame = True
elif name in self.sem_data.callables and len(self.sem_data.callables[name].children_formals) > 0:
needs_stack_frame = True
else:
for instruction in code:
if instruction.opcode == 'call':
needs_stack_frame = True
break
code_str += name + ':\n'
code_str += ' pushq %rbp\n'
code_str += ' movq %rsp, %rbp\n'
if needs_stack_frame:
code_str += ' pushq %rbp\n'
code_str += ' movq %rsp, %rbp\n'
for reg in saved_registers:
code_str += f' pushq {reg}\n'
if len(saved_registers) % 2 != 0:
@ -478,7 +616,8 @@ class CompileData:
code_str += ' addq $8, %rsp\n'
for reg in reversed(saved_registers):
code_str += f' popq {reg}\n'
code_str += ' leave\n'
if needs_stack_frame:
code_str += ' leave\n'
code_str += ' ret\n'
code_str += '\n'
@ -574,18 +713,27 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
compile_data.code = old_code
# check if we can use temporary registers instead of stack
# If RHS is an 32 bit integer literal, we can use it as an immediate value
register = None
if node.child_rhs.nodetype == 'int_literal' and node.child_rhs.value <= 0x7FFFFFFF:
if node.child_lhs.type != 'date':
register = f'${node.child_rhs.value}'
elif node.child_rhs.value * 86400 <= 0x7FFFFFFF:
register = f'${node.child_rhs.value * 86400}'
# Otherwise, we need to use a register
usable_registers = ['%r8', '%r9', '%r10', '%r11']
register = '%rcx'
for reg in usable_registers:
valid = True
for instruction in lhs_code:
if reg in instruction.operands:
valid = False
if register is None:
register = '%rcx'
for reg in usable_registers:
valid = True
for instruction in lhs_code:
if reg in instruction.operands:
valid = False
break
if valid:
register = reg
break
if valid:
register = reg
break
# check if lhs uses call, this determines whether we need to align stack
align_stack = False
@ -595,18 +743,21 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
break
# Add code for RHS calculation
compile_data.code += rhs_code
if register != '%rcx':
compile_data.code.append(Instruction('movq', ['%rax', register]))
elif not align_stack:
compile_data.code.append(Instruction('pushq', ['%rax']))
if register[0] == '$':
pass
else:
compile_data.code.append(Instruction('subq', ['$16', '%rsp']))
compile_data.code.append(Instruction('movq', ['%rax', '0(%rsp)']))
compile_data.code += rhs_code
if register != '%rcx':
compile_data.code.append(Instruction('movq', ['%rax', register]))
elif not align_stack:
compile_data.code.append(Instruction('pushq', ['%rax']))
else:
compile_data.code.append(Instruction('subq', ['$16', '%rsp']))
compile_data.code.append(Instruction('movq', ['%rax', '0(%rsp)']))
# Add code for LHS calculation
compile_data.code += lhs_code
if register != '%rcx':
if register[0] == '$' or register != '%rcx':
pass
elif not align_stack:
compile_data.code.append(Instruction('popq', [register]))
@ -615,10 +766,11 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
compile_data.code.append(Instruction('addq', ['$16', '%rsp']))
# If we are adding or subtracting dates with integers, multiply the integer by number of seconds in a day
if node.child_lhs.type == 'date' and node.child_rhs.type == 'int':
# If register is immediate, this has already been done
if register[0] != '$' and node.child_lhs.type == 'date' and node.child_rhs.type == 'int':
compile_data.code.append(Instruction('imulq', ['$86400', register]))
# perform operation
# Perform operation
if node.value == '+':
compile_data.code.append(Instruction('addq', [register, '%rax']))
elif node.value == '-':
@ -626,19 +778,21 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
elif node.value == '*':
compile_data.code.append(Instruction('imulq', [register, '%rax']))
elif node.value == '/':
# Division by immediate is not possible
if register[0] == '$':
compile_data.code.append(Instruction('movq', [register, '%rcx']))
register = '%rcx'
compile_data.code.append(Instruction('cqo'))
compile_data.code.append(Instruction('idivq', [register]))
elif node.value == '<':
compile_data.code.append(Instruction('cmpq', [register, '%rax']))
compile_data.code.append(Instruction('setl', ['%al']))
compile_data.code.append(Instruction('movzbq', ['%al', '%rax']))
elif node.value == '=':
compile_data.code.append(Instruction('cmpq', [register, '%rax']))
compile_data.code.append(Instruction('sete', ['%al']))
compile_data.code.append(Instruction('movzbq', ['%al', '%rax']))
else: assert False
# if both operands are dates, divide result by number of seconds in a day
# If both operands are dates, divide result by number of seconds in a day
if node.child_lhs.type == 'date' and node.child_rhs.type == 'date':
assert node.value == '-'
compile_data.code.append(Instruction('movq', ['$86400', register]))
@ -692,7 +846,7 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
# compile condition
compile_ast(node.child_condition, compile_data)
compile_data.code.append(Instruction('testq', ['%rax', '%rax']))
compile_data.code.append(Instruction('testb', ['%al', '%al']))
compile_data.code.append(Instruction('jz', [label_loop]))
case 'do_unless' | 'unless_expression':
label_true = compile_data.get_label()
@ -700,7 +854,7 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
# compile condition
compile_ast(node.child_condition, compile_data)
compile_data.code.append(Instruction('testq', ['%rax', '%rax']))
compile_data.code.append(Instruction('testb', ['%al', '%al']))
compile_data.code.append(Instruction('jnz', [label_true]))
# compile false statements
@ -762,6 +916,7 @@ if __name__ == '__main__':
group.add_argument('-f', '--file', help='filename to process')
parser.add_argument('-o', '--output', help='output filename for compiled code. default (a.out)', default='a.out')
parser.add_argument('-a', '--assembly', help='output filename for generated assembly code')
parser.add_argument('-O', '--optimize', action='store_true', help='run simple optimization steps on the generated assembly code')
parser.add_argument('-r', '--run', action='store_true', help='run the compiled code after compilation')
args = parser.parse_args()
@ -782,7 +937,9 @@ if __name__ == '__main__':
compile_data = CompileData(sem_data)
compile_ast(ast, compile_data)
compile_data.optimize_assembly()
if args.optimize:
compile_data.optimize_assembly()
assembly = compile_data.get_full_code()
if args.assembly is not None:
@ -792,4 +949,7 @@ if __name__ == '__main__':
subprocess.run(['gcc', '-x', 'assembler', '-o', args.output, '-static', '-'], input=assembly, encoding='utf-8')
if args.run:
subprocess.run([f'./{args.output}'])
if args.output.startswith('/'):
subprocess.run([args.output])
else:
subprocess.run([f'./{args.output}'])