update main #1

Merged
Sinipelto merged 240 commits from Bananymous/banan-os:main into main 2023-11-20 11:20:51 +00:00
Owner
No description provided.
Sinipelto added 240 commits 2023-11-20 11:20:10 +00:00
Implement "proper" DiskCache syncing
You can (and have to) manually sync disk after writes to it.
This driver is only capable to read mac address and enable and read
link status
This creates either MEM or IO region for read/write access to PCI
device.
I did not feel like implementing AML interpreter now, and wanted
everything AML has to offer. I will be writing my own AML interperter
at some point.
This will be used by the userspace to get more memory. Currently
kernel handles all allocations, which is not preferable.
We could remove syscalls to allocate more memory. This was not
something the kernel should have done.
SSE support is very experimental and causes GP. I decided to make
SSE not default until I get to fixing it :)
I have no idea why I had made PATH environment variable parsing
to be part of the kernel. Now the shell does the parsing and
environment syscall is no longer needed.
I could delete the whole FixedWidthAllocator as it was now obsolete.
GeneralAllocator is still used by kmalloc. Kmalloc cannot actually
use it since, GeneralAllocator depends on SpinLock and kmalloc runs
without interrupts.
We just now have a flag if a mapping is unmappable
This was unnecessarry allocation, since the page table allready
contains virtual address -> physical address mappings.
Remove GeneralAllocator from kmalloc as it is not CriticalScope safe.
This requires increasing kmalloc memory.
We now use truncate to create disk image, since it doesn't require
writing zeroes to full disk. I also removed creation of third
partition as this was not currently used.
Redo ext2 block allocation. This is now much "cleaner" although
I'm not too fond of the macros.
Inodes can query blocks from this buffer. This allows allocation of
blocks to not fail during normal operations. Also less stress on
kmalloc.
This can be constructed from classes that inherit from Weakable
This allows faster inode access and ensures working inodes when
opened in multiple places.
We now validate pointers passed by the user, to forbid arbitary
memory read/write. Now the user is only allowed to pass in pointers
in their own mapped memory space (or null).
You can now create kernel processes without any threads, add the
needed threads and only then register the process and its threads
to the scheduler.
This disables the scenario where user interrupts sync operation
possibly leaving the syncing in invalid state.
You can pass non-zero argument to the syscall to block until the
sync has finished.
This is my own WELL NAMED (:D) function that takes a paramemeter
to make the sync operation synchronous.
You can specify --block to make the program wait until sync is
complete.
I change always manually the serial/graphical. When running cmake
you can define variable QEMU_ACCEL that will be used as accelerator.

Also ninja has the following targets for running qemu
  1. qemu:            Run graphical qemu environment
  2. qemu-nographic:  Run qemu without graphical screen. You should
                      select 'serial only' from grub menu.
  3. qemu-debug:      Run qemu without accelerator and interrupt
                      debugger.
Userspace programs can call tty_ctrl() to disable/enable tty from
handling input and displaying output.

This API is probably going to change in the future to ioctl calls
but I'm not sure how ioctl is used and what functionality should it
have. I decided to create whole new function and syscall for now.

Next I will expose framebuffer in /dev/fb0 and then I can start work
on graphical environment! :D
Add functions to swap endiannes or convert host to big/little endian

This code should be very compiler friendly and should be optimized to
single bswap instruction on x86.
This fixes a bug of inserting empty argument if command had trailing
whitespace
This allows snake game to work again :)
You can now shutdown/reboot banan-os with the poweroff cli tool.

Reboot doesn't seem to work on qemu.
If acpi reset fails, we forcefully trigger a triple fault to restart
the system.
You could not allocate with size equal to one of the pool sizes.
mmap will not actually take any memory unless you use the given
memory.
This structure is used for demand pagable execution. It handles all
memory allocation and file reading.
All executable files are now read from disk and paged on demand.
This was a big rewrite of the old ELF library but in the end
everything seems much cleaner, since all the old functionality was
not actually needed for execution.

I have to do some measurements, but I feel like memory usage dropped
quite a bit after this change.
We only clone mapped pages that have been marked as writeble.
Read/execute only pages will be exactly as in the file itself and
can be demand paged also :D
MemoryBackedRegion now inherits from this and is used for private
anonymous mappigs. This will make shared mappings and file backed
mappings much easier to implement.
mmap() now supports mapping files with MAP_PRIVATE.
mmap() did not pass fildes to the syscall structure.
This behaves exactly as cat, but uses mmap to read the file.
Every inode holds a weak pointer to shared file data. This contains
physical addresses of pages for inode file data. Physical addresses
are allocated and read on demand.

When last shared mapping is unmapped. The inodes shared data is freed
and written to the inode.
When sync is writing to disk, it reserves whole disk to itself.
This commit makes Shell to read username only once from getpwuid().
We used to get username every time prompt was printed.
There is now a Endianness.h for these. The functions were super slow.
RamInode is now a general RamInode with no data. RamFileInode is now
a inode for regular files. This is much cleaner and more intuitive
since there is no reason for most non-regular inodes to hold data
Vector.
This is kinda weird behaviour, but it ensures the user cannot
create e.g. CharacterDevice with mode having IFLNK.

The Inode overrider is the only one setting the mode.
gcc seems to optimize some calls to strcpy to stpcpy
This allows printing files that contain null bytes behave more like
you would expect
All interruptrable classes now inherit from Interruptable which
has methdo handle_irq which is called on a interrupt.
Make ATA driver more compatible when we are adding SATA support
Calculations accidentally assumed bar registers are 8 byte instead
of 4.
PhysicalRange is now much simpler bitmap. This makes expanding
PhysicalRange API much easier.
This will be used to create DMA regions.
This does nothing but allocate contiguous physical and virtual memory
and map it as CacheDisable. Also memory is automatically freed RAII style.
I allocated 1 bitmap page per 8 data pages. Bitmap page can actually
store 8*PAGE_SIZE data pages.

Also properly set last bits in bitmap. I did not care about endianness
but now we set the bits on unsigned long longs instead of bytes.
Also fix bug in release_value() where we did not call the
destructor.
SATA drives can now be used with banan-os. This allows much faster
disk access (writing 10 MiB from 30s to 1.5s). This can definitely
be optimized but the main slow down is probably the whole disk
structure in the os.

AHCI drive is now the default when running qemu.
Before reserving address space in SYS_EXEC verify that ELF address
space is actually loadable. For example when trying to execute the
kernel binary in userspace, binarys address space would overlap with
current kernel address space. Now kernel won't crash anymore and
will just send SIGKILL to the process calling exec*().
Starting work on getting this boot on real hardware.
IDT will now panic if trying to assing handler for non supported
irq.
Not sure if this is actually needed, but this allows actual
executing function to be in clean environment
Also remove spammy debug printing
Actually this should replace from old buffer, but this works
for now.
If this variable is defined in cmake, image will be build with esp
and booted with uefi.
I used to treat all ethernet deivices as E1000 but now it is actually
verified before initialization
Initializes virtual tty only if framebuffer is initialized
IO BarRegion used vaddr instead of the correct paddr. Add API for
memory region iobase query.
ATADevice now stores its name instead of using static buffer. Old
static buffer was changing on every name query. I just hadn't noticed
since virtual machine disks were always sda.
I don't support native mode ata bus (irq sharing) so ata buses are
but to compatibility mode if possible.
If device identification sends all ones, don't initialize the device.
This allows better compatibility with (U)EFI and gives RSDP location
instead of me having to scan ram to find it.
It cannot be assumed that multiboot data lies between kernel_end
and 2 GiB mark, so I properly allocate virtual address space for it.
This is a span over exisiting containers/data types. I'm not too
happy with the constructors and assignment operators, but they will
work for now
This allows us to not work with raw pointers and use sized containers
for reading and writing.
Now you have to use script/build.sh for building and running banan-os
ACHI commands can now fail from timeouts.
I had not tested the new build system with clean toolchain build
but it seems to work now.
g++ doesn't realize that read can only return -1
I have no idea what was I doing before :D
I sourced the config file before getting old termios. Sourcing
updated the termios so old_termios was always in non canonical, non
echoing mode.
This does not support any meaningful command line arguments but
is a good start.
Touch now uses creat insteadd of open with O_CREAT flag
This will give span with all remaining size after offset
Ext2 inodes can now be unlinked from directories and after last
inode closes (destructor gets called) we check if link count is 0
and cleanup the inode from filesystem
fsck now reports clean filesystem even after deleting files
I now cleanup all blocks (including direct) in i_block array
Add ByteSpan::as_span<T> and const versions of as() and as_span()
require T to be const.
This commit consists of multiple big changes

1. blocks for inodes are now allocated on demand
  - reading from non allocated block will just return zeroes
  - writing to non allocated block allocates it

2. code doesn't really use raw pointers anymore
  - all casts to uint32_t or structures are now replaced with
    spans. either as<T> or as_span<T> which both are bounds
	checked

3. code doesn't depend on random macros for accessing indirect blocks
  - i added some recursive functions which take care of this :)
I should be using the forward list more
String now holds a 15 byte sso buffer. I'm not sure what the size
should actually be but 15 will work for now. Maybe the sso buffer
should be contained in an union with one bit flag in size instead of
variant that uses extra 8 bytes for type index.

This patch buffs sizeof(String) from 24 bytes to 32 bytes on 64 bit.
I assume this is much better version than the old which had to make
allocation even for empty strings :D.
Read x_dsdt address only if fadt's length contains it. Bochs seems
to have version 1 fadt without the x_* fields.
This feels much cleaner than just downloading tar balls from
pregiven urls. Also patching works much better like this!

I added --disable-initfini-array since global constructors were
not called.
There was no reason for libc get crt0 from kernel.
This allows global destructors to be actually called
This allows String to shrink by 8 bytes since Variant's 8 index is
no longer stored in here.

This required me to make Strings max size one bit less, but that
should still be fine. There should never be strings with size of
over half of the computer's address space.
Kernel doesn't seem to require W permissions to a page without UEFI
so this had not been found earlier.
I now check both interrupt and normal stack to detect OOB. Processes
are killed if they encouner stack over/under flow.
Implement macros for all basic binary ops. These macros print failed
values when the fail.
Also fix deallocation bug
This is not standard C++ but should be fine with my toolchain.
Add "fast page" to KERNEL_OFFSET. This is always present in page
tables and only requires changing the page table entry to map. This
requires no interrupts since it should only be for very operations
like memcpy.

I used to map all temporary mappings to vaddr 0, but this is much
better. C++ standard always says that nullptr access is undefined
and this gets rid of it.

Fixed some bugs I found along the way
Now that write-protect bit is enabled this is neccessary.
its now much harder to mix < with <= and > with >=
Sysroot is now created with fakeroot. This allows root access to be
only needed for disk image creation, since it uses loopback devices.
TmpFS blocks are now accessed with a simple wrapper
Everything is using now the better TmpFS which uses physical pages
for page allocation instead of the static kmalloc memory.
I defaultet MAKEFLAGS to -j which will launch processes in parallel
without any limit.
I wrote a fast first stage bootloader and a installer to put it into
a disk image.
Bootloader can now get the memory map and read cmdline from user.

Now 'just' video mode query, ext2 and ELF parsing are needed :D
Use this target to run banan-os with custom bootloader
The os itself only supports x86 so this won't matter. x86_64 and i386
use the same bootloader assembly.
This cleans up the code since bootloader is starting to near 1k lines
This can search for files in an ext2 filesystem. Only 12 blocks
are currently supported.

Now only ELF loading is missing for loading the actual kernel!
This allows support of multiple different bootloaders
If the serial port doesn't repond with a size, just use a random
one. There is no reason to ditch the whole output if you cannot
determine its size.
Clear screen before jumping to kernel. Memcpy now uses ebx as offset
register, so only one register has to updated every loop
This will probably be read from some config file at some point
Kernel now gets framebuffer from bootloader. Framebuffer dimensions
and bpp are hardcoded in bootloader, but will probably be read from
config file at some point.
setgid/setuid did not change the permissions of procfs inodes. This
made Shell launched by init not appear in meminfo.
You can set BANAN_BOOTLOADER=GRUB to use grub instead. Image creation
does not convert disk image now automatically between bootloaders and
calling ./bos image-full is now required.
And creating image now builds the bootloader
Sinipelto merged commit a6a5c00763 into main 2023-11-20 11:20:51 +00:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Sinipelto/banan-os#1