04_semantics_and_running/README.md

@@ -1,132 +0,0 @@
-## Phase 4
-
-I implemented a basic compiler for the phase 4. This compiler translates the given program file into GNU assembly that is then compiled using GCC and linked against system libc. Libc is used for printing values and time manipulation.
-
-
-### Semantic checking
-
-Before the compilation check, full semantic check is performed on the code. All types are statically checked to be valid.
-
-#### Levels of interpretation
-
-1. All arithmetic expressions and print statements are implemented
-  - Arithmetic expressions are checked during semantic checking.
-	- Valid addition expressions
-		- [int] + [int] -> [int]
-		- [date] + [int] -> [date]
-	- Valid subtraction expression
-		- [int] - [int] -> [int]
-		- [date] - [int] -> [date]
-		- [date] - [date] -> [int]
-	- Valid multiplication expressions
-		- [int] * [int] -> [int]
-	- Valid division expressions
-		- [int] / [int] -> [int]
-	- Valid comparisons
-		- [int] = [int] -> [bool]
-		- [date] = [date] -> [bool]
-		- [int] < [int] -> [bool]
-		- [date] < [date] -> [bool]
-  - Print statement can take a list of [int], [date] or [string] types
-2. Global variables can be defined and they are accessible from anywhere in the code
-3. For [date] variables, reading a field is possible. Writing is not because of limitations of assembly language.
-  - Readable fields include
-    - [date]'day -> [int], day number in month (1-31)
-	- [date]'month -> [int], month number in year (1-12)
-	- [date]'year -> [int], year number
-	- [date]'weekday -> [int], number of the day in week (1-7)
-	- [date]'weeknum -> [int], number of the week in year (1-53)
-  - Writable fields include
-    - [date].day -> [int], day number in month (1-31)
-	- [date].month -> [int], month number in year (1-12)
-	- [date].year -> [int], year number
-  - Writing to date will pass semantic checking but fail during compilation
-4. Unless-statement, unless-expression and until-loop all work as expected
-5. Function and procedure definitions and calls are properly handled and work as expected
-  - Function and procedure return types are validated during semantic check
-  - Auto keyword works. In this case the return type is determined using the type of returned value.
-6. There is no runtime checking done as my compiler has static typing. All variables and functions are typed, so there is no need for doing it during runtime.
-7. Recursion and proper local variables are implemented. This happens pretty much automatically when local variables and function arguments are kept on stack.
-
-#### Test cases
-
-All tests in _public\_examples/04\_semantics\_and\_running/running\_examples_ pass semantic checking and output the correct values.
-
-All tests in _public\_examples/04\_semantics\_and\_running/semantic\_error\_examples_ fail during semantic checking.
-
-
-### Compiler ABI and generated code
-
-Some insights on how different data types are handled and how the compiler works in general.
-- Integer literals are stored as 64 bit values and arithmetic is done using basic assembly instructions
-- Date literals are stored as an Unix timestamp (seconds since 1970). To add or subtract days, the right operand is multiplied by 86400 (seconds in 24 hours) for the timestamp to work expecedly. Subtracting dates is done using basic integer arithmetic and dividing the result by 86400 which gives number of days as the difference.
-- String literals are stored in the .data section with automatically generated labels.
-- Printing
-  - Integers are printed using `printf("%lld", value)`, so `printf` takes care of the formatting
-  - Strings are printed using `printf("%s", label)`, where label is the strings read-only label in .data section
-  - Dates are printed by first calling `localtime(timestamp)` to get `struct tm*` pointer containing broken down time. Then dates are formatted into a fixed buffer in .bss section using `strftime(date_buffer, date_buffer_size, "%Y-%m-%d", tm)`, to format the string into the wanted form. After the date buffer has formatted string, it can printed using `printf("%s", date_buffer)`.
-- `Today()` builtin function calls `__builtin_today` which calls `time(NULL)` from libc. This function returns current unix timestamp, so it is really suitable for this usage.
-- Reading attributes from date objects is kind of hacky. It is mostly done as printing date objects. First date is converted into `struct tm*` with `localtime(timestamp)`, and formatted to wanted attribute (day -> "%d", month -> "%m", ...). This is then formatted into the date buffer using `strftime`. After formatting date buffer should contain only the asked part of the date. This can be converted into integer using the `atoi()` function.
-- Writing date attributes is not implemented as that would require modifying `struct tm*` which is not feasible in pure assembly as the actual structure of it is unknown. There doesn't seem to be easy way do determine size and offset of a field in structure in GNU assembly.
-- Conditional statements are done comparing the low byte of rax. After which a conditional jump happens to a label defined in the code.
-- Calling convension of generated assembly code is as follows
-  - Arguments are pushed to stack before function call
-  - Local variables are initialized to proper values in the stack after as the first step in function call
-  - Rest of the code is generated
-  - Using either arguments or local variables happens through offset to rbp which contains the current stack frame.
-  - Return value of each operation of pythons `compile_ast()` is stored in rax register
-  - Stack is always aligned to 16-byte boundary according to _System V ABI_. This is because called libc functions might use sse-instructions that depend on this alignment.
-  - Global variables are stored in .bss section. The first thing main function does, is initialize global variables to proper values.
-
-### Optimization
-
-There is also an optional `-O` flag for the "compiler". This does a simple optimization pass over the generated assembly and optimizes some obvious instructions. These are the current optimizations implemented
-- Moving from a register to the same register is removed
-  - `movq %rax, %rax` => `nop`
-- Moving into rax and then pusing into rax is optimized to a single push instruction
-  - `movq $1, %rax; pushq %rax` => `pushq $1`
-- Moving into rax ant then moving out of rax is optimized to single move instruction without accessing rax
-  - `movq $1, %rax; movq %rax, %rcx` => `movq $1, %rcx`
-- Repeated addition and subtraction instruction with immediate values on the same register are combined into a signle instruction
-  - `addq $5, %rax; subq $2, %rax` => `addq $3, %rax`
-- Moving immediate into register followed by add or subtract on the same register is optimized to single move
-  - `movq $5, %rax; subq $2, %rax` => `movq $3, %rax`
-- Adding or subtracting immediate 1 is optimized to corresponding inc or dec instruction
-  - `addq $1, %rax` => `incq %rax`
-- Moving immediate zero to register is optimized to xor instruction
-  - `movq $0, %rax` => `xorq %rax, %rax`
-
-These optimizations are not all something that can be done in all assembly language, but because the way my compiler generates code it does not expect value to stay in register.
-For example normally `movq $4, %rax; movq %rax, %rcx` cannot be optimized to `movq $4, %rcx` since this assumes rax is not used afterwards. In my compiler this kind of assumptions are not done, so this is a valid optimization.
-
-Depending on the code that is to be compiled, optimizations might not do anything meaningful, so this is left as a optional flag.
-
-### Usage of the program
-
-```
-usage: main.py [-h] [-d] (--who | -f FILE) [-o OUTPUT] [-a ASSEMBLY] [-O] [-r]
-
-options:
-  -h, --help            show this help message and exit
-  -d, --debug           debug?
-  --who                 print out student IDs and NAMEs of authors
-  -f FILE, --file FILE  filename to process
-  -o OUTPUT, --output OUTPUT
-                        output filename for compiled code. default (a.out)
-  -a ASSEMBLY, --assembly ASSEMBLY
-                        output filename for generated assembly code
-  -O, --optimize        run simple optimization steps on the generated assembly code
-  -r, --run             run the compiled code after compilation
-```
-
-By default when the program is run, it compiles file pointed by `-f` into binary called _a.out_. If you want the binary to be named something different, you can use the `-o` flag to specify output file for the generated binary.
-
-If you are interested in the generated assembly, you can use the `-a` flag. This flag names a file where the generated assembly is dumped. Without this flag, no assembly is written to any file.
-
-If you want to run the optimization pass, you can specify the `-O` flag. This flag takes no arguments. If both this flag and the `-a` flag are specified, the dumped assembly will the optimized verision.
-
-To run the program automatically after compilation, you can specify the `-r` flag. This flag calls the output binary specified by `-o` flag and executes it.
-
-### Requirements
-
-For this program to function properly, you must have GCC installed and the required PLY library for python. Without GCC compilation step will fail. Assembly will be dumped even if there is no GCC available, as that happens before the invocation of GCC.

04_semantics_and_running/examples/arithmetic.ph

@@ -1,16 +0,0 @@
-print 1 + 1,
-print 10 - 5,
-print 5 - 10,
-print -5 + 2,
-print -5 - 5,
-print -5 - -5,
-print 2 * 3,
-print 2 * -3,
-print -2 * 3,
-print -2 * -3,
-print 10 / 2,
-print 10 / -2,
-print -10 / 2,
-print -10 / -2,
-print 2024-04-27 - 5,
-print 2024-04-27 + 5

04_semantics_and_running/examples/date-attr.ph

@@ -1,19 +0,0 @@
-var date = Today()
-
-print date,
-print date'day,
-print date'month,
-print date'year,
-print date'weekday,
-print date'weeknum,
-
-date.day = 2,
-date.month = 3,
-date.year = 2'000,
-
-print date,
-print date'day,
-print date'month,
-print date'year,
-print date'weekday,
-print date'weeknum

04_semantics_and_running/examples/long-add.ph

@@ -1 +0,0 @@
-print 1+1+1+1+1+1+1+1+1+1

04_semantics_and_running/examples/long-arith.ph

@@ -1 +0,0 @@
-print 1 + 2 - 3 + 5 - 7 + 11 - 13 + 17 - 19

04_semantics_and_running/examples/loop-sum.ph

@@ -1,18 +0,0 @@
-var nn = 1'000'000'000
-var count = 0
-var sum = 0
-
-function Func{ second_number[int] } return int
-is
-  (second_number + 3) * 5
-end function
-
-(% Calculate sum with a loop %)
-do
-  do
-    sum = sum + Func(count),
-    count = count + 1
-  until nn < count
-unless nn = 0 done,
-
-print "Sum from 0 to" & nn & "is" & sum

04_semantics_and_running/examples/loop.ph

@@ -1,6 +0,0 @@
-var num = -10
-
-do
-    print num,
-    num = num + 1
-until num = 11

04_semantics_and_running/examples/print.ph

@@ -1,4 +0,0 @@
-print "hello" & "world",
-print 10,
-print 2024-04-27,
-print Today() - 365*100

04_semantics_and_running/examples/proc-arg.ph

@@ -1,6 +0,0 @@
-procedure PROC{ arg1[int], arg2[int], arg3[int], arg4[int], arg5[int] }
-is
-	print arg1 & arg2 & arg3 & arg4 & arg5
-end procedure
-
-PROC(1, 2, 3, 4, 5)

04_semantics_and_running/examples/proc-var.ph

@@ -1,11 +0,0 @@
-procedure PROC{ }
-var var1 = 1
-var var2 = 2
-var var3 = 3
-var var4 = 4
-var var5 = 5
-is
-	print var1 & var2 & var3 & var4 & var5
-end procedure
-
-PROC()

04_semantics_and_running/examples/var.ph

@@ -1,9 +0,0 @@
-var foo = 20
-var bar = 10
-
-print foo + bar,
-print foo - bar,
-print foo * bar,
-print foo / bar,
-
-print (foo + (foo + (foo + (foo + bar))))

04_semantics_and_running/main.py

@@ -335,22 +335,16 @@ class CompileData:
 
                 i = 0
                 # Remove redundant movq instructions
-                #   movq %rax, %rax
                 while i < len(instructions):
                     if instructions[i].opcode != 'movq' or instructions[i].operands[0] != instructions[i].operands[1]:
                         i += 1
                         continue
                     instructions.pop(i)
                     changed = True
-                if changed: continue
 
                 i = 0
                 # Optimize movq to register followed by pushq register
-                #   movq $1, %rax
+                while i < len(instructions):
-                #   pushq %rax
-                # becomes
-                #   pushq $1
-                while i < len(instructions) - 1:
                     if instructions[i].opcode != 'movq' or instructions[i + 1].opcode != 'pushq':
                         i += 1
                         continue
@@ -361,15 +355,10 @@ class CompileData:
                     instructions.pop(i + 1)
                     i -= 1
                     changed = True
-                if changed: continue
 
                 i = 0
                 # Optimize movq to rax followed by movq from rax
-                #   movq $1, %rax
+                while i < len(instructions):
-                #   movq %rax, %rcx
-                # becomes
-                #   movq $1, %rcx
-                while i < len(instructions) - 1:
                     if instructions[i].opcode != 'movq' or instructions[i + 1].opcode != 'movq':
                         i += 1
                         continue
@@ -386,119 +375,6 @@ class CompileData:
                     instructions.pop(i + 1)
                     i -= 1
                     changed = True
-                if changed: continue
-
-                i = 0
-                # Replace negative immediate in addq/subq with positive immediate
-                # This is not a real optimization, but it makes the code easier to optimize
-                #   subq $-1, %rax
-                # becomes
-                #   addq $1, %rax
-                while i < len(instructions):
-                    if instructions[i].opcode not in ['addq', 'subq']:
-                        i += 1
-                        continue
-                    if instructions[i].operands[1][0] != '$':
-                        i += 1
-                        continue
-                    value = int(instructions[i].operands[1][1:])
-                    if value >= 0:
-                        i += 1
-                        continue
-                    new_opcode = 'subq' if instructions[i].opcode == 'addq' else 'addq'
-                    instructions[i] = Instruction(new_opcode, [instructions[i].operands[0], f'${-value}'])
-                    changed = True
-                if changed: continue
-
-                i = 0
-                # Optimize repeated addq/subq instructions
-                #   addq $1, %rax
-                #   addq $2, %rax
-                # becomes
-                #   addq $3, %rax
-                while i < len(instructions) - 1:
-                    if instructions[i].opcode not in ['addq', 'subq']:
-                        i += 1
-                        continue
-                    if instructions[i].operands[1][0] != '$' or instructions[i + 1].operands[1][0] != '$':
-                        i += 1
-                        continue
-                    if instructions[i].operands[1] != instructions[i + 1].operands[1]:
-                        i += 1
-                        continue
-                    lhs = int(instructions[i].operands[0][1:])
-                    if instructions[i].opcode == 'subq': lhs = -lhs
-                    rhs = int(instructions[i + 1].operands[0][1:])
-                    if instructions[i + 1].opcode == 'subq': rhs = -rhs
-                    new_value = lhs + rhs
-                    if abs(new_value) > 0xFFFFFFFF:
-                        i += 1
-                        continue
-                    new_opcode = 'addq' if new_value >= 0 else 'subq'
-                    instructions[i] = Instruction(new_opcode, [f'${abs(new_value)}', instructions[i].operands[1]])
-                    instructions.pop(i + 1)
-                    i -= 1
-                    changed = True
-                if changed: continue
-
-                i = 0
-                # Optimize movq immediate to register followed addq/subq with immediate
-                #   movq $1, %rax
-                #   addq $2, %rax
-                # becomes
-                #   movq $3, %rax
-                while i < len(instructions) - 1:
-                    if instructions[i].opcode != 'movq' or instructions[i + 1].opcode not in ['addq', 'subq']:
-                        i += 1
-                        continue
-                    if instructions[i].operands[1] != instructions[i + 1].operands[1]:
-                        i += 1
-                        continue
-                    if instructions[i].operands[0][0] != '$' or instructions[i + 1].operands[0][0] != '$':
-                        i += 1
-                        continue
-                    lhs = int(instructions[i].operands[0][1:])
-                    rhs = int(instructions[i + 1].operands[0][1:])
-                    if instructions[i + 1].opcode == 'subq': rhs = -rhs
-                    new_value = lhs + rhs
-                    instructions[i] = Instruction('movq', [f'${new_value}', instructions[i].operands[1]])
-                    instructions.pop(i + 1)
-                    i -= 1
-                    changed = True
-                if changed: continue
-
-                i = 0
-                # Optimize addq/subq for immediate 1
-                #   addq $1, %rax
-                # becomes
-                #   incq %rax
-                while i < len(instructions):
-                    if instructions[i].opcode not in ['addq', 'subq']:
-                        i += 1
-                        continue
-                    if instructions[i].operands[0] != '$1':
-                        i += 1
-                        continue
-                    new_opcode = 'incq' if instructions[i].opcode == 'addq' else 'decq'
-                    instructions[i] = Instruction(new_opcode, [instructions[i].operands[1]])
-                    changed = True
-                if changed: continue
-
-                i = 0
-                # Optimize zeroing of register
-                #   movq $0, %rax
-                # becomes
-                #   xorq %rax, %rax
-                while i < len(instructions):
-                    if instructions[i].opcode != 'movq':
-                        i += 1
-                        continue
-                    if instructions[i].operands[0] != '$0' or instructions[i].operands[1][0] != '%':
-                        i += 1
-                        continue
-                    instructions[i] = Instruction('xorq', [instructions[i].operands[1], instructions[i].operands[1]])
-                    changed = True
-                if changed: continue
 
     def add_builtin_functions(self) -> None:
         today = []
@@ -586,21 +462,7 @@ class CompileData:
                             break
             saved_registers = list(saved_registers)
 
-            # Stack frame is needed for builtin functions, functions with local variables
-            # and functions that call other functions
-            needs_stack_frame = False
-            if name.startswith('__builtin'):
-                needs_stack_frame = True
-            elif name in self.sem_data.callables and len(self.sem_data.callables[name].children_formals) > 0:
-                needs_stack_frame = True
-            else:
-                for instruction in code:
-                    if instruction.opcode == 'call':
-                        needs_stack_frame = True
-                        break
-
             code_str += name + ':\n'
-            if needs_stack_frame:
             code_str += '  pushq %rbp\n'
             code_str += '  movq %rsp, %rbp\n'
             for reg in saved_registers:
@@ -616,7 +478,6 @@ class CompileData:
                 code_str += '  addq $8, %rsp\n'
             for reg in reversed(saved_registers):
                 code_str += f'  popq {reg}\n'
-            if needs_stack_frame:
             code_str += '  leave\n'
             code_str += '  ret\n'
             code_str += '\n'
@@ -713,17 +574,8 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
 
             compile_data.code = old_code
 
-            # If RHS is an 32 bit integer literal, we can use it as an immediate value
+            # check if we can use temporary registers instead of stack
-            register = None
-            if node.child_rhs.nodetype == 'int_literal' and node.child_rhs.value <= 0x7FFFFFFF:
-                if node.child_lhs.type != 'date':
-                    register = f'${node.child_rhs.value}'
-                elif node.child_rhs.value * 86400 <= 0x7FFFFFFF:
-                    register = f'${node.child_rhs.value * 86400}'
-
-            # Otherwise, we need to use a register
             usable_registers = ['%r8', '%r9', '%r10', '%r11']
-            if register is None:
             register = '%rcx'
             for reg in usable_registers:
                 valid = True
@@ -743,9 +595,6 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
                     break
 
             # Add code for RHS calculation
-            if register[0] == '$':
-                pass
-            else:
             compile_data.code += rhs_code
             if register != '%rcx':
                 compile_data.code.append(Instruction('movq', ['%rax', register]))
@@ -757,7 +606,7 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
 
             # Add code for LHS calculation
             compile_data.code += lhs_code
-            if register[0] == '$' or register != '%rcx':
+            if register != '%rcx':
                 pass
             elif not align_stack:
                 compile_data.code.append(Instruction('popq', [register]))
@@ -766,11 +615,10 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
                 compile_data.code.append(Instruction('addq', ['$16', '%rsp']))
 
             # If we are adding or subtracting dates with integers, multiply the integer by number of seconds in a day
-            # If register is immediate, this has already been done
+            if node.child_lhs.type == 'date' and node.child_rhs.type == 'int':
-            if register[0] != '$' and node.child_lhs.type == 'date' and node.child_rhs.type == 'int':
                 compile_data.code.append(Instruction('imulq', ['$86400', register]))
 
-            # Perform operation
+            # perform operation
             if node.value == '+':
                 compile_data.code.append(Instruction('addq', [register, '%rax']))
             elif node.value == '-':
@@ -778,21 +626,19 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
             elif node.value == '*':
                 compile_data.code.append(Instruction('imulq', [register, '%rax']))
             elif node.value == '/':
-                # Division by immediate is not possible
-                if register[0] == '$':
-                    compile_data.code.append(Instruction('movq', [register, '%rcx']))
-                    register = '%rcx'
                 compile_data.code.append(Instruction('cqo'))
                 compile_data.code.append(Instruction('idivq', [register]))
             elif node.value == '<':
                 compile_data.code.append(Instruction('cmpq', [register, '%rax']))
                 compile_data.code.append(Instruction('setl', ['%al']))
+                compile_data.code.append(Instruction('movzbq', ['%al', '%rax']))
             elif node.value == '=':
                 compile_data.code.append(Instruction('cmpq', [register, '%rax']))
                 compile_data.code.append(Instruction('sete', ['%al']))
+                compile_data.code.append(Instruction('movzbq', ['%al', '%rax']))
             else: assert False
 
-            # If both operands are dates, divide result by number of seconds in a day
+            # if both operands are dates, divide result by number of seconds in a day
             if node.child_lhs.type == 'date' and node.child_rhs.type == 'date':
                 assert node.value == '-'
                 compile_data.code.append(Instruction('movq', ['$86400', register]))
@@ -846,7 +692,7 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
 
             # compile condition
             compile_ast(node.child_condition, compile_data)
-            compile_data.code.append(Instruction('testb', ['%al', '%al']))
+            compile_data.code.append(Instruction('testq', ['%rax', '%rax']))
             compile_data.code.append(Instruction('jz', [label_loop]))
         case 'do_unless' | 'unless_expression':
             label_true = compile_data.get_label()
@@ -854,7 +700,7 @@ def compile_ast(node: ASTnode, compile_data: CompileData) -> None:
 
             # compile condition
             compile_ast(node.child_condition, compile_data)
-            compile_data.code.append(Instruction('testb', ['%al', '%al']))
+            compile_data.code.append(Instruction('testq', ['%rax', '%rax']))
             compile_data.code.append(Instruction('jnz', [label_true]))
 
             # compile false statements
@@ -916,7 +762,6 @@ if __name__ == '__main__':
     group.add_argument('-f', '--file', help='filename to process')
     parser.add_argument('-o', '--output', help='output filename for compiled code. default (a.out)', default='a.out')
     parser.add_argument('-a', '--assembly', help='output filename for generated assembly code')
-    parser.add_argument('-O', '--optimize', action='store_true', help='run simple optimization steps on the generated assembly code')
     parser.add_argument('-r', '--run', action='store_true', help='run the compiled code after compilation')
 
     args = parser.parse_args()
@@ -937,9 +782,7 @@ if __name__ == '__main__':
         compile_data = CompileData(sem_data)
         compile_ast(ast, compile_data)
 
-        if args.optimize:
         compile_data.optimize_assembly()
-
         assembly = compile_data.get_full_code()
 
         if args.assembly is not None:
@@ -949,7 +792,4 @@ if __name__ == '__main__':
         subprocess.run(['gcc', '-x', 'assembler', '-o', args.output, '-static', '-'], input=assembly, encoding='utf-8')
 
         if args.run:
-            if args.output.startswith('/'):
-                subprocess.run([args.output])
-            else:
             subprocess.run([f'./{args.output}'])