Sinipelto
/
banan-os
forked from Bananymous/banan-os
1
0
Fork 0
Code Pull Requests Activity

update main #1

Merged
Sinipelto merged 240 commits from Bananymous/banan-os:main into main 2023-11-20 13:20:51 +02:00
Conversation 0 Commits 240 Files Changed 263 +14912 -157609
Sinipelto commented 2023-11-20 13:20:09 +02:00
Owner
Copy Link
No description provided.
Sinipelto added 240 commits 2023-11-20 13:20:10 +02:00
b0c22b61ec Kernel: Writes to disk are not synchronous anymore 
Implement "proper" DiskCache syncing
5fae3cec2a Kernel: Implement SYS_SYNC and add sync executable to userspace 
You can (and have to) manually sync disk after writes to it.
14a608effd 1000th COMMIT: Kernel: Add basic E1000 driver 
This driver is only capable to read mac address and enable and read
link status
7774f56ab6 Kernel: PCI devices can now create region for BAR 
This creates either MEM or IO region for read/write access to PCI
device.
c9e09b840e Kernel: Add LAI as a dependency 
I did not feel like implementing AML interpreter now, and wanted
everything AML has to offer. I will be writing my own AML interperter
at some point.
fee3677fb9 Kernel/LibC: add mmap for private anonymous mappings 
This will be used by the userspace to get more memory. Currently
kernel handles all allocations, which is not preferable.
f662aa6da2 Kernel/LibC: userspace malloc now uses mmap to get memory 
We could remove syscalls to allocate more memory. This was not
something the kernel should have done.
fe2dca16f0 Kernel/LibC: add flag to enable/disable sse support 
SSE support is very experimental and causes GP. I decided to make
SSE not default until I get to fixing it :)
fc953df281 Kernel/LibC: remove PATH resoltion from kernel 
I have no idea why I had made PATH environment variable parsing
to be part of the kernel. Now the shell does the parsing and
environment syscall is no longer needed.
c0a89e8951 Kernel: Fully remove sys_alloc and sys_free 
I could delete the whole FixedWidthAllocator as it was now obsolete.
GeneralAllocator is still used by kmalloc. Kmalloc cannot actually
use it since, GeneralAllocator depends on SpinLock and kmalloc runs
without interrupts.
2ef496a24a Kernel: all mapped ranges are now stored in one container 
We just now have a flag if a mapping is unmappable
8b2bb95b81 Kernel: VirtualRange doesn't store physical addresses of pages 
This was unnecessarry allocation, since the page table allready
contains virtual address -> physical address mappings.
18d582c6ce Kernel: Hacky kmalloc quick fix 
Remove GeneralAllocator from kmalloc as it is not CriticalScope safe.
This requires increasing kmalloc memory.
3ea707c0e7 BuildSystem: Optimize image creation 
We now use truncate to create disk image, since it doesn't require
writing zeroes to full disk. I also removed creation of third
partition as this was not currently used.
7bdb428938 Kernel: Fix ext2 block allocation 
Redo ext2 block allocation. This is now much "cleaner" although
I'm not too fond of the macros.
8caba1e774 Kernel: Ext2 filesystem now holds 10 preallocated block buffers 
Inodes can query blocks from this buffer. This allows allocation of
blocks to not fail during normal operations. Also less stress on
kmalloc.
b62186441b BAN: Implement basic WeakPtr 
This can be constructed from classes that inherit from Weakable
55d30a7cc3 Kernel: Ext2 inodes are now stored in cache 
This allows faster inode access and ensures working inodes when
opened in multiple places.
1d470fb5ba Kernel: All syscalls now validate users pointers 
We now validate pointers passed by the user, to forbid arbitary
memory read/write. Now the user is only allowed to pass in pointers
in their own mapped memory space (or null).
09c1aa44d8 Kernel: Allow creationg of empty processes and manual registration 
You can now create kernel processes without any threads, add the
needed threads and only then register the process and its threads
to the scheduler.
b924c85669 Kernel: DiskCache now requires sync to be called from kernel thread 
This disables the scenario where user interrupts sync operation
possibly leaving the syncing in invalid state.
05e57801e7 Kernel: SYS_SYNC now schedules sync to happen soon 
You can pass non-zero argument to the syscall to block until the
sync has finished.
6cb8bda6e1 LibC: add syncsync() to unistd.h 
This is my own WELL NAMED (:D) function that takes a paramemeter
to make the sync operation synchronous.
ec2baeb276 Sync: Add some argument parsing to sync(1) 
You can specify --block to make the program wait until sync is
complete.
f9b347f9d9 BuildSystem: Rework calling qemu 
I change always manually the serial/graphical. When running cmake
you can define variable QEMU_ACCEL that will be used as accelerator.

Also ninja has the following targets for running qemu
  1. qemu:            Run graphical qemu environment
  2. qemu-nographic:  Run qemu without graphical screen. You should
                      select 'serial only' from grub menu.
  3. qemu-debug:      Run qemu without accelerator and interrupt
                      debugger.
8d5369fafe Kernel: Add some functionality to disable TTY input/output 
Userspace programs can call tty_ctrl() to disable/enable tty from
handling input and displaying output.

This API is probably going to change in the future to ioctl calls
but I'm not sure how ioctl is used and what functionality should it
have. I decided to create whole new function and syscall for now.

Next I will expose framebuffer in /dev/fb0 and then I can start work
on graphical environment! :D
27adb9486b BAN: Update Endiannes API 
Add functions to swap endiannes or convert host to big/little endian

This code should be very compiler friendly and should be optimized to
single bswap instruction on x86.
c84b66d078 Shell: String leading and trailing whitespace from commands 
This fixes a bug of inserting empty argument if command had trailing
whitespace
f7097398ca Kernel: Make tty overload correct has_data() function 
This allows snake game to work again :)
79851394b3 Kernel/LibC/Userspace: Add SYS_POWEROFF + cli tool 
You can now shutdown/reboot banan-os with the poweroff cli tool.

Reboot doesn't seem to work on qemu.
547eabb403 Kernel: Reboot will now always succeed 
If acpi reset fails, we forcefully trigger a triple fault to restart
the system.
49d941ad65 LibC: Fix a bug in malloc 
You could not allocate with size equal to one of the pool sizes.
b51d2f5295 Kernel: mmap regions are now demand paged 
mmap will not actually take any memory unless you use the given
memory.
be13120554 LibELF: Implement new ELF structure 
This structure is used for demand pagable execution. It handles all
memory allocation and file reading.
c11e84b248 Kernel: Use the new on demand ELF structure 
All executable files are now read from disk and paged on demand.
This was a big rewrite of the old ELF library but in the end
everything seems much cleaner, since all the old functionality was
not actually needed for execution.

I have to do some measurements, but I feel like memory usage dropped
quite a bit after this change.
7e9e4c47ae LibELF: Optimize LoadableELF::clone() memory usage 
We only clone mapped pages that have been marked as writeble.
Read/execute only pages will be exactly as in the file itself and
can be demand paged also :D
4a92f44cf6 Kernel: Implement new abstract MemoryRegion 
MemoryBackedRegion now inherits from this and is used for private
anonymous mappigs. This will make shared mappings and file backed
mappings much easier to implement.
db5d6a7f80 Kernel: Implement MAP_PRIVATE file mappings 
mmap() now supports mapping files with MAP_PRIVATE.
d54c6b7f6b LibC: Fix mmap() 
mmap() did not pass fildes to the syscall structure.
7a5bb6a56b Userspace: Implement cat-mmap 
This behaves exactly as cat, but uses mmap to read the file.
f953f3d3ff Kernel: Implement MAP_SHARED for regular files 
Every inode holds a weak pointer to shared file data. This contains
physical addresses of pages for inode file data. Physical addresses
are allocated and read on demand.

When last shared mapping is unmapped. The inodes shared data is freed
and written to the inode.
94ce2c97be Shell: Quick fix to not freeze for multiple seconds 
When sync is writing to disk, it reserves whole disk to itself.
This commit makes Shell to read username only once from getpwuid().
We used to get username every time prompt was printed.
eb5c6cf736 BAN: Remove endianness functions from Math 
There is now a Endianness.h for these. The functions were super slow.
e949e8550c Kernel: Rework RamInode API 
RamInode is now a general RamInode with no data. RamFileInode is now
a inode for regular files. This is much cleaner and more intuitive
since there is no reason for most non-regular inodes to hold data
Vector.
dedb2a2399 Kernel: RamInode verifies that you have not specified mode type 
This is kinda weird behaviour, but it ensures the user cannot
create e.g. CharacterDevice with mode having IFLNK.

The Inode overrider is the only one setting the mode.
38320018dc LibC: Implement stpcpy since gcc seems to need it 
gcc seems to optimize some calls to strcpy to stpcpy
7ce8e2d57b cat: Use write() instead of puts to print file contents 
This allows printing files that contain null bytes behave more like
you would expect
bcf62c5f2e Kernel: Rework interrupt mechanism 
All interruptrable classes now inherit from Interruptable which
has methdo handle_irq which is called on a interrupt.
27364f64a6 Kernel: Rework whole ATA driver structure 
Make ATA driver more compatible when we are adding SATA support
f071240b33 Kernel: Fix PCI BarRegion offsets 
Calculations accidentally assumed bar registers are 8 byte instead
of 4.
03d2bf4002 Kernel: Rework physical memory allocation 
PhysicalRange is now much simpler bitmap. This makes expanding
PhysicalRange API much easier.
8a9816d6e0 Kernel: Add API for getting contiguous physcial pages 
This will be used to create DMA regions.
211cad03ff Kernel: Implement bare boness DMA Region 
This does nothing but allocate contiguous physical and virtual memory
and map it as CacheDisable. Also memory is automatically freed RAII style.
400db176d1 Kernel: fix some math in physical ranges 
I allocated 1 bitmap page per 8 data pages. Bitmap page can actually
store 8*PAGE_SIZE data pages.

Also properly set last bits in bitmap. I did not care about endianness
but now we set the bits on unsigned long longs instead of bytes.
d2cfc843e4 BAN: Optional can now be constructed from another Optional 
Also fix bug in release_value() where we did not call the
destructor.
a2b5e71654 Kernel: Implement AHCI driver 
SATA drives can now be used with banan-os. This allows much faster
disk access (writing 10 MiB from 30s to 1.5s). This can definitely
be optimized but the main slow down is probably the whole disk
structure in the os.

AHCI drive is now the default when running qemu.
773dcdd3a2 Kernel: Check whether ELF address space can be loaded 
Before reserving address space in SYS_EXEC verify that ELF address
space is actually loadable. For example when trying to execute the
kernel binary in userspace, binarys address space would overlap with
current kernel address space. Now kernel won't crash anymore and
will just send SIGKILL to the process calling exec*().
39be6ab099 Kernel: Add temporary terminal output before controlling terminal 
Starting work on getting this boot on real hardware.
5630f64175 Kernel: Add 16 more irq handlers 
IDT will now panic if trying to assing handler for non supported
irq.
6b180da4e8 Kernel: Separate scheduler execution and stack loading 
Not sure if this is actually needed, but this allows actual
executing function to be in clean environment
dafc016293 Kernel: Clear TTY when setting as current 
Actually this should replace from old buffer, but this works
for now.
6b1b3d333c BuildSystem: add cmake variable UEFI_BOOT 
If this variable is defined in cmake, image will be build with esp
and booted with uefi.
5977341610 Kernel: PCI checks if ethernet device is E1000 before initialization 
I used to treat all ethernet deivices as E1000 but now it is actually
verified before initialization
31aa157201 Kernel: Don't require framebuffer 
Initializes virtual tty only if framebuffer is initialized
6f8fce94a0 Kernel: Fix PCI bugs 
IO BarRegion used vaddr instead of the correct paddr. Add API for
memory region iobase query.
b767317a7a Kernel: Fix ATADevice naming 
ATADevice now stores its name instead of using static buffer. Old
static buffer was changing on every name query. I just hadn't noticed
since virtual machine disks were always sda.
48980b56ab Kernel: ATABuses are but to compatibility mode if possible 
I don't support native mode ata bus (irq sharing) so ata buses are
but to compatibility mode if possible.
e01928d186 Kernel: Fix device identification with all bits as ones 
If device identification sends all ones, don't initialize the device.
be3efb0b92 Kernel: Start using multiboot2 instead of multiboot 
This allows better compatibility with (U)EFI and gives RSDP location
instead of me having to scan ram to find it.
e2e5c31d54 Kernel: Map multiboot2 memory in PageTable initialization 
It cannot be assumed that multiboot data lies between kernel_end
and 2 GiB mark, so I properly allocate virtual address space for it.
db5c24b2a5 BAN: Implement ByteSpan 
This is a span over exisiting containers/data types. I'm not too
happy with the constructors and assignment operators, but they will
work for now
5bfeb9f3ca Kernel: Rewrite all read/write functions to use BAN::ByteSpan 
This allows us to not work with raw pointers and use sized containers
for reading and writing.
df69612bb1 BuildSystem: Rewrite whole build system structure 
Now you have to use script/build.sh for building and running banan-os
00ee86920a Kernel: Add timeout to ACHI commands 
ACHI commands can now fail from timeouts.
e8d20bc653 BuildSystem: Fix bugs in new build system 
I had not tested the new build system with clean toolchain build
but it seems to work now.
adbbdf73c4 meminfo: fix g++ warning for oob write 
g++ doesn't realize that read can only return -1
988a4e1cd8 BAN: Fix bug of size of splice after slice() 
I have no idea what was I doing before :D
1e2c2fb973 Shell: Set get old termios earlier 
I sourced the config file before getting old termios. Sourcing
updated the termios so old_termios was always in non canonical, non
echoing mode.
e8890062d6 Userspace: Implement basic cp 
This does not support any meaningful command line arguments but
is a good start.
8bb47aee02 Kernel/LibC/Userspace: Implement mkdir and creat 
Touch now uses creat insteadd of open with O_CREAT flag
fda4a4ad24 BAN: ByteSpan can be sliced without specified size 
This will give span with all remaining size after offset
091c5b6a66 BAN: Implement Ext2 file unlinking 
Ext2 inodes can now be unlinked from directories and after last
inode closes (destructor gets called) we check if link count is 0
and cleanup the inode from filesystem
d09310f388 Kernel: Fix ext2 inode deletion 
fsck now reports clean filesystem even after deleting files
99bde9aa49 Kernel: Fix ext2 inode deletion cleanup 
I now cleanup all blocks (including direct) in i_block array
3bffbe330d BAN: Update ByteSpan API 
Add ByteSpan::as_span<T> and const versions of as() and as_span()
require T to be const.
0757834176 Kernel: Rewrite a lot of ext2 code 
This commit consists of multiple big changes

1. blocks for inodes are now allocated on demand
  - reading from non allocated block will just return zeroes
  - writing to non allocated block allocates it

2. code doesn't really use raw pointers anymore
  - all casts to uint32_t or structures are now replaced with
    spans. either as<T> or as_span<T> which both are bounds
	checked

3. code doesn't depend on random macros for accessing indirect blocks
  - i added some recursive functions which take care of this :)
09fcc613c7 BAN: Add variant to ForwardList 
I should be using the forward list more
1af3ca19ab BAN: Rewrite String with small string optimizations 
String now holds a 15 byte sso buffer. I'm not sure what the size
should actually be but 15 will work for now. Maybe the sso buffer
should be contained in an union with one bit flag in size instead of
variant that uses extra 8 bytes for type index.

This patch buffs sizeof(String) from 24 bytes to 32 bytes on 64 bit.
I assume this is much better version than the old which had to make
allocation even for empty strings :D.
f312c3a4d7 Kernel: Fix ACPI DSDT address 
Read x_dsdt address only if fadt's length contains it. Bochs seems
to have version 1 fadt without the x_* fields.
ea5ed3001e Toolchain: Clone GCC and Binutils from git 
This feels much cleaner than just downloading tar balls from
pregiven urls. Also patching works much better like this!

I added --disable-initfini-array since global constructors were
not called.
f05b9a6877 Kernel/LibC: Add crt* files to LibC and remove crt0 from kernel 
There was no reason for libc get crt0 from kernel.
bc1d1bf919 LibC: implement and call __cxa_finalize() on exit() 
This allows global destructors to be actually called
f72fdeeb59 BAN: String now uses union for its sso storage 
This allows String to shrink by 8 bytes since Variant's 8 index is
no longer stored in here.

This required me to make Strings max size one bit less, but that
should still be fine. There should never be strings with size of
over half of the computer's address space.
3d899d2e44 Kernel/LibELF: Map pages always as RW so kernel can write to them 
Kernel doesn't seem to require W permissions to a page without UEFI
so this had not been found earlier.
5d34cebeca Kernel: Fix stack OOB detection 
I now check both interrupt and normal stack to detect OOB. Processes
are killed if they encouner stack over/under flow.
0405461742 BAN: Implement better ASSERT macros 
Implement macros for all basic binary ops. These macros print failed
values when the fail.
1c5985148c Kernel: Allow offsetof with packed fields 
This is not standard C++ but should be fine with my toolchain.
3bac19e518 Kernel: Add fast page to page table 
Add "fast page" to KERNEL_OFFSET. This is always present in page
tables and only requires changing the page table entry to map. This
requires no interrupts since it should only be for very operations
like memcpy.

I used to map all temporary mappings to vaddr 0, but this is much
better. C++ standard always says that nullptr access is undefined
and this gets rid of it.

Fixed some bugs I found along the way
a44482639d Kernel: Temporarily force FileBackedRegion mappings writable 
Now that write-protect bit is enabled this is neccessary.
120f7329b1 BAN: Update ASSERT api 
its now much harder to mix < with <= and > with >=
6d899aa6ce BuildSystem: using sysroot doesn't need root privileges anymore! 
Sysroot is now created with fakeroot. This allows root access to be
only needed for disk image creation, since it uses loopback devices.
ab4f033385 Kernel: Cleanup TmpFS code and block access doesn't require allocs 
TmpFS blocks are now accessed with a simple wrapper
cec04a2858 Kernel: Remove now obsolete RamFS 
Everything is using now the better TmpFS which uses physical pages
for page allocation instead of the static kmalloc memory.
430a006acf Toolchain: Fix typo when setting make flags 
I defaultet MAKEFLAGS to -j which will launch processes in parallel
without any limit.
c47f6a78bc Bootloader: Start work on bootloader 
I wrote a fast first stage bootloader and a installer to put it into
a disk image.
a3a287f5ca Bootloader: Continue work on bootloader 
Bootloader can now get the memory map and read cmdline from user.

Now 'just' video mode query, ext2 and ELF parsing are needed :D
1a75262b04 BuildSystem: add bootloader target 
Use this target to run banan-os with custom bootloader
055b1a2a1a Bootloader move bootloader code from arch directory 
The os itself only supports x86 so this won't matter. x86_64 and i386
use the same bootloader assembly.
b0b39c56ba Bootloader: Split bootloader into multiple files 
This cleans up the code since bootloader is starting to near 1k lines
a9412aa741 Bootloader: Implement basic ext2 filesystem 
This can search for files in an ext2 filesystem. Only 12 blocks
are currently supported.

Now only ELF loading is missing for loading the actual kernel!
ec56e9c6f1 Kernel: Don't use multiboot2 explicitly. Parse it to common structure 
This allows support of multiple different bootloaders
3daf3d53a3 Kernel: Serial now uses random size for some serial ports 
If the serial port doesn't repond with a size, just use a random
one. There is no reason to ditch the whole output if you cannot
determine its size.
5f4d81a502 Bootloader: Clear screen, better memcpy 
Clear screen before jumping to kernel. Memcpy now uses ebx as offset
register, so only one register has to updated every loop
f0d2a211ea Bootloader add temporary initial command line 
This will probably be read from some config file at some point
a312d75bb2 Bootloader: Implement VESA video mode query and pass it to kernel 
Kernel now gets framebuffer from bootloader. Framebuffer dimensions
and bpp are hardcoded in bootloader, but will probably be read from
config file at some point.
5293ae070d Kernel: ProcFS inodes reflect processes ruid/rgid 
setgid/setuid did not change the permissions of procfs inodes. This
made Shell launched by init not appear in meminfo.
e2515c1109 Buildsystem: default bootloader is not my custom one 
You can set BANAN_BOOTLOADER=GRUB to use grub instead. Image creation
does not convert disk image now automatically between bootloaders and
calling ./bos image-full is now required.
f2397b775c BuildSystem: Remove old bootloader target 
And creating image now builds the bootloader
Sinipelto merged commit a6a5c00763 into main 2023-11-20 13:20:51 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Sinipelto/banan-os#1
No description provided.
Delete Branch "Bananymous/banan-os:main"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?