From 24e89ff20a4749f9d1eb67ee75427b186e78c841 Mon Sep 17 00:00:00 2001 From: sagi Date: Fri, 19 Jun 2026 18:46:31 +0300 Subject: [PATCH 1/3] Changed the names of the chapters and added redirection --- book.toml | 4 + ...3-implementing-the-bitfields-proc-macro.md | 854 ------------------ src/ch03-00-printing-to-screen.md | 136 --- 3 files changed, 4 insertions(+), 990 deletions(-) delete mode 100644 src/ch02-03-implementing-the-bitfields-proc-macro.md delete mode 100644 src/ch03-00-printing-to-screen.md diff --git a/book.toml b/book.toml index db1b170..365976b 100644 --- a/book.toml +++ b/book.toml @@ -45,3 +45,7 @@ mathjax-support = true [output.html.playground] runnable = true editable = true + +[output.html.redirect] +"/ch02-03-implementing-the-bitfields-proc-macro.html" = "/ch02-05-implementing-the-bitfields-proc-macro.html" +"/ch03-00-printing-to-screen.html" = "/ch04-00-printing-to-screen.html" diff --git a/src/ch02-03-implementing-the-bitfields-proc-macro.md b/src/ch02-03-implementing-the-bitfields-proc-macro.md deleted file mode 100644 index b6dcaad..0000000 --- a/src/ch02-03-implementing-the-bitfields-proc-macro.md +++ /dev/null @@ -1,854 +0,0 @@ -# Writing the Bitflags Macro - -_"In Lisp, you don't just write your program down toward the language, you also build the language up toward your program." - Paul Graham_ - ---- - -#![repository_card] - -As you may recall from the previous chapter, we used a proc-macro that was called `bitfields`. In this chapter, we are going to learn about Rust's procedural macros and even implement one ourselves. - -> Another great resource for this subject is the great video [Comprehending Proc Macros](https://youtu.be/SMCRQj9Hbx8?si=p-JUX0rLronBG_Nz) by Logan Smith - - -_If you are familiar with procedural macros, `syn` and `quote`, and want to go straight to the macro implementation, click [here](#defining-our-macro)_ - -## A Little Introduction to Procedural Macros - -Macros are not a new idea in programming languages, and most languages have them in some form. But what even is a macro? - -If you ask Wikipedia, we get the following definition. -### [_Macro_](https://en.wikipedia.org/wiki/Macro_(computer_science)) -
- -_A macro is a rule or pattern that specifies how a certain input should be mapped to a replacement output._ - -
- -When I read this definition, the first thing that comes to mind is that it really sounds like a function. After all, a function maps the input arguments to the output arguments, which is exactly what a macro does. And that is exactly right. Rust, macros (specifically procedural macros), are indeed a specific type of function, but let's not get ahead of ourselves. - -The key differences between macros and regular functions are that macros _replace_ the inputs and the outputs, and that is not always true with functions. Secondly, macros operate on our source code instead of variables in our program. - -Rust takes this definition very literally, and the definition for a proc-macro function looks like this: - -```rust -#![function!("snippets/src/lib.rs", custom_proc_macro)] -``` - -As you can see in this function, the input is Rust's `TokenStream`, which is literally our source code, and the output is also a `TokenStream,` which means it expects us to return also source code, which could be the same (Like the example above), but most of the time it is not. - -But what is this `TokenStream`? Why not just use strings of the source code? - -Well, the main reason we are even discussing this is that we want to manipulate the initial code in some way. Tokenizing the source code allows us to manipulate the code at a higher level, which is easier to reason about. This `TokenStream` is the most basic tokenization unit that we are going to work with, and it contains a sequence of `TokenTree` nodes that represent the source code. - -```rust -#![enum!("/lib/rustlib/src/rust/library/proc_macro/src/lib.rs", TokenTree)] -``` - -To see this more visibly, we can print our TokenStream, because it implements the `Debug` trait. Which, for a simple struct, would look like this: - -```text -TokenStream [ - Ident { - ident: "struct", - span: #0 bytes(43..49), - }, - Ident { - ident: "Example", - span: #0 bytes(50..57), - }, - Group { - delimiter: Brace, - stream: TokenStream [ - Ident { - ident: "a", - span: #0 bytes(64..65), - }, - Punct { - ch: ':', - spacing: Alone, - span: #0 bytes(65..66), - }, - Ident { - ident: "i32", - span: #0 bytes(67..70), - }, - Punct { - ch: ',', - spacing: Alone, - span: #0 bytes(70..71), - }, - ], - span: #0 bytes(58..73), - }, -] -``` - -_Can you understand the name of the struct and its fields?_ - -## How Macros are Executed - -As you may have noticed, macros do not behave exactly like regular functions. Another difference that they have is that they are evaluated at compile time. - -This thinking can also be used on regular functions, but not from our point of view, but from the compiler's point of view. For the compiler, regular functions are also a mapping, from some target language (in our case, Rust) to some other target language (in most cases, ASM[^1]). - -For example, this function: - -```rust -#![function!("snippets/src/book/ch02_03/general.rs", square)] -``` - -would map to the following ASM code: - -> [!TIP] -> Look it yourself at [compiler explorer](https://godbolt.org/z/7vTzbs6e9) - -```x86asm,icon=@https://icons.veryicon.com/png/o/business/vscode-program-item-icon/assembly-7.png -square: - mov eax, edi - imul eax, edi - ret -``` - -From this point of view, macros are not so different, but instead of a target language, they are mapped to the same language. -So this macro: - -```rust -#![source_file!("snippets/src/book/ch02_03/general.rs", 8:12)] - -#![function!("snippets/src/book/ch02_03/general.rs", foo)] -``` - -Would map to this literal Rust code: - -```rust -#![function!("snippets/src/book/ch02_03/general.rs", foo_expanded)] -``` - -The fact that macros operate on our source code means that we can abstract certain logics that regular functions cannot. For example, take a look at this macro: - -```rust -#![source_file!("snippets/src/book/ch02_03/general.rs", 24:31)] -#![function!("snippets/src/book/ch02_03/general.rs", main)] -``` - -It works because it injects the `break` expression into the code at the call site, which is something that a function just can't do. - -```rust -#![function!("snippets/src/book/ch02_03/general.rs", unwrap_or_break)] -``` - -At this time, I hope you understand the great power of macros, and the great [code generation](https://en.wikipedia.org/wiki/Code_generation) capabilities that they enable. But, you might think rightfully think that in the examples above, we didn't have the option to insert 'coding' logic into the macro expansion. This is where procedural macros come in. - -[^1]: This is actually a simplified view; compilers have intermediate representations. These representations are really useful but out of the scope of this book. If you are like me, and this really interests you, I will drop a great blog post that gives an example of why the intermediate representations are useful. [From Rust to Reality: The Hidden Journey of fetch_max](https://questdb.com/blog/rust-fetch-max-compiler-journey/) - -## Macro Types - -Just before we dive into procedural macros, let's cover the type of macro that we already used in the examples above. - -> All the syntax information about how macros are structured is taken directly from the [Official Rust Reference](https://doc.rust-lang.org/reference/macros.html). - -### Declarative Macros - -Declarative macros are the simplest type of macro, and they are the ones that we used in the examples above. They are mainly used to generate simple syntax extensions, which are commonly called "macros by example". - -Each macro is defined by a set of rules that specify how the macro should expand. Each rule looks a bit like a function signature that can get certain `Metavariables`. These `Metavariables` are placeholders for certain Rust syntax that are replaced with actual values when the macro is expanded. - -Let's analyze the syntax of a declarative macro rule from the earlier examples. - -```rust -#![source_file!("snippets/src/book/ch02_03/general.rs", 51:65)] -``` - -We will go a bit deeper than necessary on the common types of metavariables that are available. This is because later in this chapter, we are going to talk about the `syn` library, which will parse Rust's syntax into similar structures. - -Each metavariable starts with a `$` followed by the name of the metavariable, which is used to refer to it. Then it is followed by a colon and the type of the metavariable. - -The common types of metavariables are: - -1. **Idents ($i:ident)** => These can be function names, variable names, type names, etc. They also include keywords like `fn`, `let`, `struct`, etc. -2. **Expressions ($e:expr)** => Expressions are things that are evaluated to a value, like `1 + 2` or `foo.bar()`. -3. **Items ($i:item)** => Items are the components of a module, for example the entire definition of a function or a struct. -4. **Statements ($s:stmt)** => Statements are the individual lines of code that make up a function or block. For example, `let x = 42;` is a statement. -5. **Blocks ($b:block)** => Blocks are groups of statements that are executed on the same scope. For example, `{ let y = 33; let x = 7 + y; x }` is a block. - -_For a full list of available metavariable types, see the [reference](https://doc.rust-lang.org/reference/macros-by-example.html#r-macro.decl.meta.specifier)_ - - -### Procedural Macros - -Now for the real deal. Procedural macros give us the ability to go beyond simple syntax extensions and allow us to write custom Rust code that will run at compile time on the macro input to consume and produce new Rust syntax (Depending on the macro type, the returned syntax will replace the input syntax or will be added to it). - -Because procedural macros are another piece of code that will run at compile time, they cannot be defined in the same crate as the code that uses them. This is because the Rust compiler must initially compile the code of the macro so it will be able to run it during the compilation process. In addition, each proc macro crate must add the following configuration to its `Cargo.toml` file, which will tell Cargo that this is a proc macro crate. - -```toml -[lib] -proc-macro = true -``` - -Like all functions, these macro functions can also fail, although these functions are allowed to panic. They are encouraged to use the `compile_error!` macro to return a compile-time error instead, which is the compiler form of `panic!` - -To gain the `Tokenstream` type and the attributes that will be used on the macro functions, we will use the `proc_macro` crate, which is automatically linked to our crate if it is a proc macro crate. - -### `function_like!()` - -Function like macros are very similar to declarative macros. They are invoked like a regular function and take a `TokenStream` as input and return a `TokenStream` as output. - -This type of macro can be called anywhere in our code, even in the global scope, and is defined using the following syntax: - -```rust -#![function!("snippets/src/lib.rs", foo)] -``` - -Then it can be called like a regular function, which will create a function that is called `bar` which could be used in our code. - -```rust -#![source_file!("snippets/src/book/ch02_03/invoke.rs", 1:99)] -``` - -_This type of macro replaces the macro invocation with the generated code, so the macro invocation is effectively replaced with the generated code._ - -### `#[derive(CustomDerive)]` - -Derive macros are used on Rust items to generate code automatically. They are invoked using the `#[derive]` attribute and take the item they are applied to as input. Most of the time, derive macros are used to implement traits such as `Debug`, `Clone`, `PartialEq`, etc. - -Derives may also include helper attributes, which are used to customize the generated code. - -This type of macro can be called only from structs, enums, or unions. - -```rust -#![function!("snippets/src/lib.rs", derive_with_helper_attr)] -``` - -And it is used on a structure like this: - -```rust -#![struct!("snippets/src/book/ch02_03/general.rs", Foo)] -``` - -_This type of macro does not replace the macro invocation or the input item with the generated code, and the generated TokenStream is appended to the input TokenStream._ - -### `#[attribute(macros)]` - -Attributes are used to annotate items. They are placed before the item they are applied to and are used to customize the behavior of the item. - -Attributes may also include input variables, which can be used to pass 'configuration' to the macro. - -```rust -#![function!("snippets/src/lib.rs", return_as_is)] -``` - -And it is used on a structure like this: - -```rust -#![struct!("snippets/src/book/ch02_03/general.rs", Bar)] -#![function!("snippets/src/book/ch02_03/general.rs", bar)] -``` - -_This type of macro replaces the macro invocation and the input item with the generated code._ - -## Introduction to Syn and Quote - -Remembering our goal to write the `bitfield` macro from the earlier chapter, you can already guess that we want to write an `attribute` macro. But, parsing the TokenStream we saw above is really hard, because it will require us to understand Rust's syntax tree, which can be quite complex. - -Luckily for us, the `syn` crate, written by `David Tolnay` provides a way to parse Rust syntax tree into a structured AST (Abstract Syntax Tree), which makes it easier to work with Rust source code. - -### What are Abstract Syntax Trees - -As the name suggests, this is a tree like structure that represents the syntax of a certain programming language (in our case, Rust). -Before diving right into the implementation of `syn` on Rust syntax, let's first understand what an AST is. - -We will look at a really simple program that is written in Python. - -```python -current = 0 -for item in items: - if item > current: - current = item -``` - -A simplified syntax tree for a simple program like this might look like this: - -
- -
Figure 3-1: simplified syntax tree
-
- -As you can see, in a tree like this, we can have types that help us represent the syntax in our language. For example, the `Assign` statement, which contains a `left` and `right` side. Or the `For` loop, which contains the item that is being iterated over, the collection name, and the body of the loop. Then, when we want to operate on the syntax itself, for example, create the same if statement, but change the name of the item. We can simply copy the type and change the item ident to a new one. - -As you may have guessed, `syn` does the exact same thing we did with our small program, but with all the complexity of a real language. So let's see what types it offers. - -_There are a lot of types on the `syn` crate, and we will only cover some of them. Once you get the hang of it, all the other will be easy to understand._ - -The top level type for the AST is `syn::File`, which represents a complete Rust source file. - -```rust -#![source_file!("/syn-2.0.118/src/file.rs", 6:86)] -``` - -Ok, we can see that `syn::File` is made out of a list of `syn::Attribute` and `syn::Item`. But this doesn't tell us much, so let's also explore them. - -```rust -#![source_file!("/syn-2.0.118/src/attr.rs", 23:183)] -``` - -So we can see an attribute, like `#[derive(Debug)]`, is represented by `syn::Attribute`. Currently, we will not dive deeper into `Attribute`, but we will cover more of it when we will use it in our macro implementation. - -Now let's see `syn::Item`. - -```rust -#![source_file!("/syn-2.0.118/src/item.rs", 22:101)] -``` - -As you can see, we have a lot of items, and I hope that you can start and recognize some of them. As an example, let's cover `ItemConst`. - -```rust -#![source_file!("/syn-2.0.118/src/item.rs", 103:118)] -``` - -If you have noticed closely, the order of the fields in the struct definition is the same as the order in the source code. This makes it really easy to map the AST back to the source code. - -Also, as a side note, keywords like `const`, `struct` and punctuation like `:` and `=` does have types, but `syn` also provides a `Token!` macro that maps the literal token to its corresponding type. - -The last type that we are going to cover is `syn::Expr`, which represents an expression from the source code. Because most of Rust's syntax is represented as expressions, `syn::Expr` is a very large type. - -```rust -#![source_file!("/syn-2.0.118/src/expr.rs", 37:269)] -``` - -These types are very powerful and help us express language in a structured way. As a quick example, let's see how `syn::ItemStruct` is represented in the AST. In this example, we have the exact same struct that we showed its `TokenStream` representation. - -``` -ItemStruct { - attrs: [], - vis: Visibility::Inherited, - struct_token: Struct, - ident: Ident { - ident: "Example", - span: #0 bytes(50..57), - }, - generics: Generics { - lt_token: None, - params: [], - gt_token: None, - where_clause: None, - }, - fields: Fields::Named { - brace_token: Brace, - named: [ - Field { - attrs: [], - vis: Visibility::Inherited, - mutability: FieldMutability::None, - ident: Some( - Ident { - ident: "a", - span: #0 bytes(64..65), - }, - ), - colon_token: Some( - Colon, - ), - ty: Type::Path { - qself: None, - path: Path { - leading_colon: None, - segments: [ - PathSegment { - ident: Ident { - ident: "i32", - span: #0 bytes(67..70), - }, - arguments: PathArguments::None, - }, - ], - }, - }, - }, - Comma, - ], - }, - semi_token: None, -} -``` -_Can you see the name of the struct, and the type of the field in the AST?_ - -As you can see, what was before a list of punctuations and idents has now become a structured representation that is easier to work with. - -The most important thing about Syn is that we can use the types that it offers to create new, custom types that are not bound to the language's AST. - -But how would Syn know to parse our custom syntax into the AST types it offers? This is where the `Parse` trait comes in. When Syn wants to parse our custom syntax, it will call the `parse` method from the `Parse` trait and pass in the token stream to parse. - -```rust -#![trait!("/syn-2.0.118/src/parse.rs", Parse)] -``` - -We will go deeper into this when we create our own custom `Parse` implementation. One important thing to understand is that all of syn's types implement `Parse` themselves, so most of the time, implementing `Parse` for types that are built from syn's AST types is easy. - -Up until now, we have learned how to parse our source code into a meaningful AST representation. This representation will help us to work with the syntax and to implement our macro's logic. But, after we parsed the source code and processed it to our needs, we needed to return it to a `TokenStream`. This is where the `quote` crate comes in. - -Quoting is a term that is borrowed from Lisp, and it means that we write things that look like code, but they will actually convert into data under the hood, or in our case, the `TokenStream` type. - -The `quote` crate provides a `quote!` macro that allows us to write quoted expressions. - -For example, let's define a simple quoted expression that represents a struct definition: - -```rust -#![source_file!("snippets/src/book/ch02_03/general.rs", 81:90)] -``` - -As you can see, it seems like we write Rust code, but actually, under the hood, it is converted into a `TokenStream`. - -Another great quality that this macro has is that it supports entering variables into the quoted expression. Let's look at an example where we change the name of a function inside an attribute macro. - -```rust -#![function!("snippets/src/lib.rs", change_name)] -``` - -As you can see, we parsed the input with `syn` into a function item. Then, we changed the name of the function and transferred it to the `quote!` macro with the `#` so that it would convert the variable into a `TokenStream`. - -But how quote know to convert the variable into a `TokenStream`? This is where the `ToTokens` trait comes in. - -```rust -#![trait!("/quote-1.0.45/src/to_tokens.rs", ToTokens)] -``` - -In this trait, the `to_tokens` method is defined, which gets a `&mut TokenStream` and appends the tokenized representation of the variable to it. - -The types that are defined in `syn` already implement this trait, so they can be used with `quote!` without any additional work, like in the example above, where the `ItemFn` became a function definition. - -## Defining our Macro - -In my opinion, the most important thing to do before we even start to code (even not specifically for macros) is to define what we want from our program. - -The main thing we wanted in the first place was to represent a number, e.g., u8, u16, u32, etc., as flags. To see a clear example, look at the drawing below. - -
- -
Figure 3-1: bitflag struct example
-
- - -In this drawing, we can see six different flags. Each takes a different part inside our u16 number. -1. Flag A is between bits 00-02 -2. Flag B is between bits 02-07 -3. Flag C is between bits 07-10 -4. Flag D is between bits 10-12 -5. Flag E is between bits 12-15 -6. Flag F is between bits 15-16 - -For each flag, we would like to have multiple functions. - -- A getter, which returns the value of the flag. -- A setter, which sets the value of the flag. -- Clear function, which writes a clear value if defined directly to the flag. (Will be necessary in the future) - -Because we need multiple functions defined, the best Rust item suited for the job is a `struct`. Also, because a struct will wrap the entire definition, the macro will have in its context all of the definitions of all the flags, which means we could also implement the `Debug` trait on it to print all of the flags. - -Some flags will need different functions, and may also have `types`. For example, think about the protection level field in the previous section. While we can just leave it as a number, most of the time, it is more convenient to have an enum that represents the valid values. Also, some flags may not need all the functionalities of get, set, and clear. For that, we want to have the ability to control which functions will be generated. - -And for the last caveat, some flags will be written as absolute values on their setter function, and will return absolute values on their getter. What does that mean? Take, as an example, flag `E` on the example above. The span of this flag is between bits 12-15. In most of our flag cases, we would want to write numbers between 0 and 7 to this value, because it is 3 bits wide. When we set the don't shift attribute, we would want an absolute value for this flag, which means the lowest value (besides 0) will be `1 << 12` (The first bit of the flag), and the highest value will be `1 << 14,` where the jumps between each value will be `1 << 12.` - -This design for this macro, with inspiration from [Proceadural Macro Workshop](https://github.com/dtolnay/proc-macro-workshop#attribute-macro-bitfield), will be a regular Rust struct, with helper attributes. - -For example, this struct will represent the flags in the example above (with example helper attributes). - -```rust -#![struct!("snippets/src/book/ch02_03/general.rs", MyFlags)] -``` - -## Implementing the Macro - -### Sketching the Idea - -The first thing that I like to do when creating a macro is to create a simple input for the macro and generate the key functions output by hand. -This way, I could have a mental model of what it is supposed to do, and I can generalize on that. - -So, for starters, let's create a really simple input and output for our macro. - -```rust -#![struct!("snippets/src/book/ch02_03/general.rs", SimpleFlags)] -``` - -Just before we create our functions, what will our struct type be? In this case, we have a two bit field and a one bit field, but there is no type that is three bits wide. Instead, we are going to pick the closest uint type that is large enough to hold our fields. In this case, a u8. - -```rust -#![struct!("snippets/src/book/ch02_03/general.rs", SimpleFlagsType)] -``` - -Now for our functions. The problem that we need to solve is how to get and set the value of the bits that are stored in the underlying `u8` field. - -_This part assumes familiarity with bitwise operations like right and left shifts, and simple gates like AND, OR, and NOT. For those of you who are not familiar with these operations, I really recommend seeing this [video](https://www.youtube.com/watch?v=vqpfrSIyojo) by BitLemon._. - -
- -
figure 3-2: simpleflags layout
-
- -We will start with reading the value for the `b` flag. There are multiple combinations of bitwise operations that can achieve this. The one that we will use is to first zero out the entire content of the `u8` except for our `b` flag, and then shift it to the right and read it. - -So first, let's think about how we can zero out the entire content of the `u8` except for our `b` flag. We can do this by using the `&` operator to perform a bitwise AND operation between our `u8` value and a mask[^2] that has all bits set to 0 except for our `b` flag, which will be all 1s. By hand, this mask will look like this `0b00000100`. But this, of course, does not help us much, because we need to automatically generate this mask for each bitfield, and it may also have multiple 1 bits, and not only one, like in this case. - -To generate this mask, we will think of a much simpler case: how can we put a sequence of ones at the start of our mask? Before I give the answer, let's think about what a sequence of ones means. A sequence of ones is always a number, which, when we add 1 to it, will always become a perfect power of 2 on the bit after the sequence. For example, `0b00000111` (7) will become `0b00001000` (8) when we add 1 to it. - -You may have also noticed that the number of bits that were set to 1 before we added 1 is equal to the power of 2 of the number after we added 1. For example, `0b00000111` (7) has 3 bits set to 1, and 8 is exactly `2^3`. - -> [!TIP] -> If I were you, I wouldn't accept this fact. Go try it for yourself with more examples to see that it is true - -To generally create a mask with the first `n` bits set, we can use our formula: `2^n - 1`. Because we are speaking only of powers of two, we will use `(1 << n) - 1` to create the mask. Which is the same thing. - -[^2]: The sequence of bits that will be used along with our value in a logic gate. - -```rust,playground -#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_1)] - -#![function!("snippets/src/book/ch02_03/mask.rs", main)] -``` - -If you played with this example in the demo, you may have found that in one particular case, this formula does not work as expected. (If you didn't find it, I urge you to try it yourself. - -When our `(1 << n) - 1` will result in an all 1, it means that `1 << n` was bigger than our underlying type. For example, `(1 << 8) - 1,` which should generate the `0b11111111` mask, will instead generate `0`, because `1 << 8` is `256`, which is bigger than `u8` can hold. While we can use bigger types, for the maximum size type, it will not work. - -The alternative method that we are going to use is instead of increasing the number of 1 bits in our mask each time, and starting from 0, we are going to start with an all 1 mask, and reduce the number of 1 bits each time. This won't have the gap at all 1 mask, because it is the starting value. - -To achieve it, we are going to start with our type maximum mask, and then shift it to the right by the total number of bits in our type, minus our width. For example, if our type is `u8`, and our width is `3`, our mask will be `0b11111111 >> (8 - 3) = 0b00000111`. - -```rust,playground -#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_2)] - -#![function!("snippets/src/book/ch02_03/mask.rs", main)] -``` - - -The next thing that we are going to do is to relocate the position of the bits in our mask to the flag position in our u8. - -This could easily be done using the left shift operator `<<` with the offset of our flag. For example, if the starting bit of our flag is at position 2, we can shift our mask to the left by 2 bits: `mask << 2`. Which makes our final mask generation function look like this: - -```rust,playground -#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] - -#![function!("snippets/src/book/ch02_03/mask.rs", main)] -``` - -To read the value, we just need to apply an AND gate with the mask, and then shift the result to the right by the offset to normalize it. - -```rust,playground -#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] - -#![function!("snippets/src/book/ch02_03/read.rs", read_flag)] - -#![function!("snippets/src/book/ch02_03/read.rs", main)] -``` - -To write to our value, you may be tempted to use the left shift operation on the `new_value` to shift to the correct position and then OR it with the original value. While your intuition is good, this approach will not work. This is because the OR gate only changes bits from 0 to 1, but cannot change bits from 1 to 0. So our approach will be to first clear the bits we want to change, and then OR it with the new value. - -To clear the flag, we can use an AND gate, where all the flag bits are set to 1, and the rest are 0. This will leave the flag bits unchanged, and the rest will be cleared. So our clear mask for flag `b` looks like this `0b11111011`. - -You may have noticed that this is the exact inverse of the mask we used to read the flag. So we will use the same approach to generate it, and use the NOT gate with the `!` operator to invert all the bits. After that, we can OR it with the new value shifted to the correct position. - -```rust,playground -#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] - -#![function!("snippets/src/book/ch02_03/write.rs", write_flag)] - -#![function!("snippets/src/book/ch02_03/write.rs", main)] -``` - -### Struct Definition - -When starting to implement any piece of code, it is always a good idea to first sketch out the types that we are going to use. - -Borrowing the definition of our macro again, these are the types that come to mind. - -``` -#[bitfields] -struct MyFlags { - #[flag(r)] - a: B2, - b: B5, - #[flag(rwc(30))] - c: B3, - #[flag(flag_type = ProtectionLevel)] - d: B2, - #[flag(r, dont_shift)] - e: B3, - f: B1, -} -``` - -- BitFields - - FlagAttribute (i.e `#[flag(r, dont_shift, flag_type = ProtectionLevel)]`) - - Permissions - - FlagType - - DontShift - - Single Bitfield (i.e., `a: B2`) - - FlagMeta (i.e., `width: 2, type: u8`) - -### FlagAttributes - -Our `FlagAttribute` struct will simply store the permissions, flag type, and `dont_shift` flag of a flag. - -```rust -#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagAttribute)] -``` - -#### Permissions - -For our permission attribute, we want to store the read, write, or clear permissions and the clear value. You may be tempted to use one number here and encode it in bits to have good performance on it. But, this will not be a good idea because macros are expanded at compile time, and the expansion between compilations is cached. So the performance increase doesn't really matter. - -```rust -#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagPermission)] -#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Default for FlagPermission)] -``` - -#### DontShift - -For some of our flags, and especially the `dont_shift` flag, we want to parse custom idents, in this example, the literal `dont_shift` keyword. - -Instead of parsing ident's by our own logic, `syn` provides a cool `custom_keyword!` macro that allows us to parse custom idents easily. - -```rust -#![source_file!("crates/macros/src/bitfields/flag_attr.rs", 6:11)] -``` -#### FlagType - -For our final type on the attribute, we want to parse the sequence `flag_type = some_type`. To represent this, we will use the following struct. - -For our type, we will use syn's [`TypePath`](https://docs.rs/syn/latest/syn/struct.TypePath.html) struct, which represents a path to a type, such as `std::ffi::CString`. - -```rust -#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagType)] -``` - -### Single Bitfield - -For a single field, we would want to include the attribute we just defined, the comments, visibility, and name of the field to use on the generated functions, and the size and offset of the field for our read and write functions. - -```rust -#![struct!("crates/macros/src/bitfields/bitfield.rs", BitField)] -``` - -> [!NOTE] -> We use references on some of the fields because this structure will be created from the [`syn::Field`](https://docs.rs/syn/latest/syn/struct.Field.html) struct, so instead of cloning the values, we use references to avoid unnecessary allocations. - - -#### FlagMeta - -For our `FlagMeta` struct, we will want to store the width of the field, but also the type that will represent it. So `B3` would have width `3` and type `u8`. - -```rust -#![struct!("crates/macros/src/bitfields/utils.rs", FlagMeta)] -``` - -### Parsing the Attribute - -We will start off easy by parsing the `FlagType` attribute. Because every element in this attribute already implements the `Parse` trait, we can call its parse function in the correct order. - -```rust -#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagType)] -``` - -If you were wondering why we are calling the `parse` function on the input instead of on the type itself. It is because the `ParseStream` implements this very convenient `parse` function that allows us to parse a single token from the stream at a time. - -```rust -#![impl_method!("/syn-2.0.118/src/parse.rs", ParseBuffer::parse)] -``` - -Next, let's parse something that takes a little more effort, our `FlagPermission`. - -```rust -#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagPermission)] -``` - -For the `dont_shift` keyword, the parsing is implemented automatically because we used the `custom_keyword!` macro to define it. - -And now for the parsing of the entire attribute. While we can define a strict order for the attribute, and then call parse on each field. We will not do that because it will be annoying to use the macro. Instead, we will `fork` the stream and try to parse it as each of our fields. If the parsing succeeds, we will save the parsed item and keep forking and parsing until we reach the end of the stream. - -But what is forking? And why do we need it? - -Imagine our stream as a really large linked list that contains all of our tokens. When we parse the stream, we are moving our position through the list by `consuming` the tokens as we parse them. The problem with what we are trying to do is that if we start parsing an item, and it fails, by the time we have already parsed some of the tokens of the item, we have no way of coming back to the exact position we were at before the failure. - -This is where forking comes in. When we fork the stream, we create another pointer to the position on the list, which is independent of the original position we had. Then, we can try to parse the stream with the fork. If it fails, we can simply discard the fork, and if it succeeds, we can advance the original position to the fork's position. - -
- -
figure 3-3: parse stream with fork
-
- -The last thing we want to keep in mind is that we want to avoid duplicates in our attributes. So we will keep for each attribute a variable that stores whether we have already seen that attribute before. - -```rust -#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagAttribute)] -``` - -And now for the `try_parse` function, where we basically want to fork the stream and try to parse the item, if we succeed, we advance the original position to the fork's position, we discard the fork, and increment the error_counter. - -```rust -#![function!("crates/macros/src/bitfields/flag_attr.rs", try_parse)] -``` - -Although we implemented `Parse` for `FlagAttribute`, we are not going to create it from raw tokens. Because, if you noticed, we only parsed the inside of the attribute, but not the `#[flag()]` part. -For that, we are going to use the `Meta` part of our `syn::Attribute`. - -```rust -#![source_file!("/syn-2.0.118/src/attr.rs", 455:486)] -``` - -In our case, we are going to have a `Meta::List`, which contains a `TokenStream` of the attribute's contents, hence the implementation of the `Parse` trait. - -```rust -#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", TryFrom for FlagAttribute)] -``` - -### Parsing the Struct - -Instead of parsing the struct directly, we will instead parse a regular `syn::ItemStruct` and then implement the `TryFrom` trait to convert between types in the regular `syn::ItemStruct` and our custom `BitFields` type. - -Again, we will start off easy, by converting the type of the struct field into our custom `FlagMeta` type. - -```rust -#![trait_impl!("crates/macros/src/bitfields/utils.rs", TryFrom for FlagMeta)] -``` - -To turn the width number of the type into the type that will represent it, we will use the following function. - -```rust -#![function!("crates/macros/src/bitfields/utils.rs", type_from_size)] -``` - -For each field on our struct, we are going to initially extract all the attributes on it and divide them into document attributes and our flag attributes. - -This can be easily done, because Rust doesn't store our comments as a string starting with `///` but as a `#[doc(some_comment)]` attribute. This makes our comments actually a `syn::Attribute` token, which we already know how to work with. - -```rust -#![function!("crates/macros/src/bitfields/bitfield.rs", extract_attributes)] -``` - -After that, we are going to create our field from the `syn::Field` token. But, the field itself is not enough, because from it we can't know the offset of the field in the struct. We are going to give as a parameter in the `new` function that will create our `BitField` instance. We can do that because when we create our fields one by one, we will add each time their size to an offset, which will, of course, start at 0. - -```rust -#![impl_method!("crates/macros/src/bitfields/bitfield.rs", BitField::new)] -``` - -Finally, after all that parsing, we can turn the parsed `syn::ItemStruct` into our `BitFields` instance. - -```rust -#![trait_impl!("crates/macros/src/bitfields.rs", TryFrom for BitFields)] -``` - -## Generating the Code - -With all of our types set up, we can now generate the code for our functions from them. - -Our first function will be a utility function that will provide us with some checks on our input value. This check will be used to check that the input value is within the valid range for the field, and it will be guarded by a `debug_assert!` macro, so in release builds it will be optimized out. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::checks)] -``` - -For each of our functions, we are going to use three main types. The first is the type of the variable that we are getting, the second is going to be the type that represents the type of the variable we are getting, and the third is the type of the entire struct. For example, we might have a field `#[flag(flag_type = Bar)] foo: B6`. The type of our variable in this case will be `Bar`, the type that represents the field is `u8` because it is only 6 bits wide, and the type of the entire struct depends also on the other fields and their sizes, but it will also follow the rules of the `type_from_size` function. - -To store all of these types, we are going to use a struct. - -```rust -#![struct!("crates/macros/src/bitfields.rs", FieldTypes)] -``` - -Then, to create it from our field, so it can be used in other functions, we are going to use the following function. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::field_types)] -``` - -Without further diving into our utility functions, let's look at our read function. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_read)] -``` - -As explained above, the first thing we do, is to extract the types that we are going to use inside this function. Then, we are going to get the function name, for the rest of our functions, we are not going to have a function like this, but for the read function, I personally wanted, for my convenience, that if the type of the item was `bool` so that it would change to `is_` instead of `get_` - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::read_fn_name)] -``` - -For our `read_shift` function, we need to know if to shift the value or not per the `dont_shift` attribute, and in which direction. For write operations, we need a left shift to change from the absolute value we get to the relative value on the flag, and for read operations, we need a right shift to convert from the relative value on the flag to its absolute value. When the `dont_shift` attribute is present, we don't need any of this, because the values are always absolute. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::read_shift, write_shift)] -``` - - -> [!IMPORTANT] -> You may have noticed that when we use functions from the `core` library, I am referring to them as `::core` with a leading `::`. And that, for example, when I use the `try_from` method from the `TryFrom` trait, I call the trait function with the object instead of `object.try_from(T)`. -> -> When writing a macro, we don't want to insert a `use` statement into the codebase of the person that is using our macros, and we can't assume (although most of the time unlikely) that he or she didn't implement functions with similar names as in our example `try_into`, that are doing an entirely different thing. -> -> Because of that, the safest way to call functions from libraries and trait methods is to use their fully qualified name. So we use `::` before core, to reference the compiler's core library (in case they have a core.rs module) and the fully qualified trait name to call its functions. - -When implementing our write and clear functions, we are going to use almost the exact same code for the writing logic. Because of that, we are going to extract this piece of code to a general `volatile_write` function. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::volatile_write)] -``` - -Which makes this our write function. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_write)] -``` - -Our `v_to_repr` function will be used to convert our value `v` from the type of the flag to its representation type by using the `try_from` function. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::v_to_repr)] -``` - -Therefore, we can use this value `v` that it defines in the `volatile_write` function, because it can `as` cast into the struct type. - -For the final function, which is our clear function, we are going to use the same logic as the write function, but instead of operating on a value `v`, we are going to operate on the clear value, which is already a usize. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_clear)] -``` - -As a little bonus function, that is mainly used for convenience, we will create a build function that is meant to operate on an empty struct, and define multiple flags on its creation (e.g., `let flags = Flags::new().flag1(2).flag2(3)`). - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_build)] -``` - -Because we know we are operating on an empty flag, instead of clearing the flag and then writing, we can simply use the OR gate to write our value, because we know nothing is set in the flag yet. - -The last things that we are going to generate is the `Debug` trait implementation, and the `From` trait from the flags into the struct repr, and from the struct repr into the flags. - -The latter is really easy; to translate into the underlying type, we just return the inner type. And to construct from the inner type, we simply call the constructor. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::conversion_impls)] -``` - -To implement the `Debug` trait, we need to first create the formatter debug struct builder and then add each of our fields to it. -Because we can have multiple fields, we need to insert some sort of repetition. Luckily, the `quote!` macro provides us a way to do it. -If we have a vector, or an iterator, of things that implement `ToTokens`, we can insert them all using the `#(#..)` syntax. - -```rust -#![impl_method!("crates/macros/src/bitfields.rs", BitFields::debug_impl)] -``` - -> [!WARNING] -> The debug trait implementation makes our binary larger and adds additional compilation time. In the correct version of the macro, the `Debug` trait implementation is not guarded by a feature, but in the future, it will be generated only if the main struct includes `#[derive(Debug)]`. -> As a cool exercise, you can try to add that feature to the macro yourself. - -And for the grand finale, the implementation of the `ToTokens` trait for our macro. - -```rust -#![trait_impl!("crates/macros/src/bitfields.rs", ToTokens for BitFields)] -``` - -And for the macro itself, we need to parse a struct from the input and convert it into a `BitFields` struct. Then just turn it into tokens or raise an error in parsing, depending on the result. - -```rust -#![function!("crates/macros/src/lib.rs", bitfields)] -``` diff --git a/src/ch03-00-printing-to-screen.md b/src/ch03-00-printing-to-screen.md deleted file mode 100644 index 2ea3f26..0000000 --- a/src/ch03-00-printing-to-screen.md +++ /dev/null @@ -1,136 +0,0 @@ -# Printing To Screen - -_"The most effective debugging tool is still and careful thought, coupled with judiciously placed print statements." - Brian Kernighan_ - ---- - -#![repository_card] - -Printing is an important aspect of an operating system, especially in early development because it is our way to gain a visual output from our operating system. This will massively improve the interaction with our OS, and not only will it give us a huge advantage in debugging, but it will also grant us the ability to display a shell, which we will do in the upcoming chapters. - -## Why didn't we print until now? - -If you remember the [example code](./ch01-02-booting-our-binary.md#hello-world) in the first bootable code we wrote, we did print to screen during that code. -This print utilized the `Video (int 10h)` interrupt on BIOS with the `Print Char (0xE)` function to print character by character the string 'Hello, World!' - -This was our only way to print we were on `real mode`. And while I developed the code, I actually did use it to print single characters as errors code, So I could understand what was my program doing. - -On `protected mode`, we couldn't use the BIOS anymore, so printing was much harder. Additionally, we only turned on paging, so debugging with [QEMU monitor](https://qemu-project.gitlab.io/qemu/system/monitor.html) was much easier. - -While we could have written a simple printer for each stage, it was not necessary, and it would have bloated our binary, which in the first stage had only 512 bytes, and had almost no use in the second stage. But now, on the kernel init stage, it would become really handy! - -## How to print without BIOS? - -We are gonna print using the Video Graphics Array or VGA for short. This protocol as the name suggests, puts an array in memory which will represent our screen. When we want to print, we simply write the content to the array, and it will automatically refresh on certain interval display to newly provided content. - -## The VGA Protocol - -VGA has primarily two modes, the first one is called `graphic mode`, which is used to write raw pixels to the screen. The second mode is called `text mode` and it is used to write text to the screen. In this chapter we are going to focus on the `text mode` because we mostly want to provide messages and text on the screen. - -_Maybe on later chapters we will implement UI, so we will a more graphic mode, but then we actually might not use VGA_ - -### Printing with Text Mode - -To print with text mode, we need to write to the screen buffer a special character that is 2 bytes long. This special character encodes the actual ASCII character that we are going to print, the background color of the text, and the foreground color of the text. - -> The screen buffer of the `graphic mode` starts at address 0xA0000 and the screen buffer of the `text mode` starts at address 0xB8000. - -The first byte encodes the ASCII character, and it is not special. The second byte will encode our color, the first 4 bits will be the foreground color, and the next 4 bits will be the background color. - -There are multiple color palettes that VGA uses, the one our mode uses, is the 4 bit color palette and it includes the following colors. - -```rust -#![enum!("crates/common/src/enums/vga.rs", Color)] -``` - -```rust -#![struct!("crates/drivers/vga-display/src/color_code.rs", ColorCode)] - -#![impl!("crates/drivers/vga-display/src/color_code.rs", ColorCode)] - -#![trait_impl!("crates/drivers/vga-display/src/color_code.rs", Default for ColorCode)] -``` - -Then the encoding of each `Screen Character` will look like this. - - -```rust -#![struct!("crates/drivers/vga-display/src/screen_char.rs", ScreenChar)] - -#![impl!("crates/drivers/vga-display/src/screen_char.rs", ScreenChar)] - -#![trait_impl!("crates/drivers/vga-display/src/screen_char.rs", Default for ScreenChar)] -``` - -At this point, we are ready to write to the screen whatever we want, we just need to write a `ScreenChar` to the screen. But, this is not exactly what we want, because it is hard to print strings this way. - -## Creating a Custom Writer - -As always, rust has amazing features, and one of them is built in formatting on the core library. - -> For those who are unfamiliar with the subject, formatting is turning a variable or a struct into a printable string. -> -> For example, if we have a variable `x` which holds the number `100`, how do we know how to print it? because it is not a string, formatting helps us with this 'type change'. -> -> You might be familiar with the [`printf`](https://en.wikipedia.org/wiki/Printf) function is C (Print Formatted), Rust offers us the [`fmt::Display`](https://doc.rust-lang.org/core/fmt/trait.Display.html) and [`fmt::Debug`](https://doc.rust-lang.org/core/fmt/trait.Debug.html) traits to handle formatting - -But what does it mean for us? It means that if we implement our custom writer (which just needs to print regular ASCII strings), we freely get the ability to print variables in the code, and complex structs, since they can be easily derived by the Debug trait. - -To create our custom writer we just need to implement the [`fmt::Writer`](https://doc.rust-lang.org/core/fmt/trait.Write.html) trait on a custom struct. Our simple writer, will just include place we currently are on the screen, the color the print has, and, and a reference to the screen buffer. - -```rust -#![struct!("crates/drivers/vga-display/src/writer.rs", Writer)] - -#![trait_impl!("crates/drivers/vga-display/src/writer.rs", Default for Writer)] -``` - -Then, we need to handle the following functionalities: - -1. If a character is in ASCII range, write it to the buffer at cursor position, and advance the cursor. - -2. If the `\n` character was entered, don't print anything, but put the cursor at the start of the next line. - -3. If `Backspace` or `Delete` character were entered, move the cursor back one position, and fill that position with the default character. - -4. If we are at the end of the screen, we need to scroll down a line, which means to copy the entire buffer one line to the left[^1]. - -[^1]: Our buffer represents a 2D grid of `ScreenChar` elements, but it is actually just one big 1D buffer. So copying the entire buffer one line up is equivalent to shifting all the characters one line to the left. - -5. Function to clear the screen entirely - -Now that we have all the functionality in mind, we can go right into the implementation! - -```rust -#![impl_method!("crates/drivers/vga-display/src/writer.rs", Writer::write_char, scroll_down, new_line, backspace, clear)] -``` - -> For now, the `change_cursor_position_on_screen` function is not relevant, and it uses I/O instruction to change the cursor position. This will be covered in future chapters. - -With this, we are ready to implement the `fmt::Writer` trait on our struct. Because it only requires us to implement the `write_str` function, which is easy to implement because we have our `write_char` function. - -```rust -#![trait_impl!("crates/drivers/vga-display/src/writer.rs", Write for Writer)] -``` - -The only thing that is missing is to initialize the writer, and write a function that will also print with a custom color, this function is relatively straight forward, and it will just change the color, print the message, and restore the color back to default. - -```rust -#![static!("crates/drivers/vga-display/src/lib.rs", WRITER)] -#![function!("crates/drivers/vga-display/src/lib.rs", vga_print)] -``` - -An example usage, could be an OK message of what we already initialized! - -```rust -#![function!("snippets/src/book/ch03_00/print_example.rs", _start)] -``` - -
- -## Exercise - -1. The standard library has a `print!` and `println!` macros, we are really close for one, implement it! - -2. Implement the `okprintln!` and `eprintln!` that we used above. - -Answers can be found at [here](https://github.com/sagi21805/LearnixOS/blob/master/crates/drivers/vga-display/src/lib.rs#37) From 3289123348b5e4d26c6d8cb4aafb4947b2b69e8d Mon Sep 17 00:00:00 2001 From: sagi Date: Fri, 19 Jun 2026 18:46:52 +0300 Subject: [PATCH 2/3] Inserted a new synchronization chapter. --- src/SUMMARY.md | 47 +- ...5-implementing-the-bitfields-proc-macro.md | 854 ++++++++++++++++++ src/ch03-00-synchronization-primitives.md | 1 + ...-is-achieved-on-a-multiprocessor-system.md | 1 + ...enting-basic-synchronization-primitives.md | 1 + src/ch04-00-printing-to-screen.md | 136 +++ ...gement.md => ch05-00-memory-management.md} | 0 ...s.md => ch05-01-memory-allocator-types.md} | 0 ...ap.md => ch05-02-implementing-a-bitmap.md} | 0 ...5-03-writing-a-physical-page-allocator.md} | 0 ...d => ch06-00-interrupts-and-exceptions.md} | 0 ...he-idt.md => ch06-01-utilizing-the-idt.md} | 0 ...6-02-programmable-interrupt-controller.md} | 0 ...d => ch06-03-writing-a-keyboard-driver.md} | 0 ... ch07-00-file-systems-and-disk-drivers.md} | 0 ...isk-drivers.md => ch07-01-disk-drivers.md} | 0 ... => ch07-02-implementing-a-file-system.md} | 0 ...md => ch08-00-processes-and-scheduling.md} | 0 ...ch08-01-thinking-in-terms-of-processes.md} | 0 ...08-02-implementing-a-process-scheduler.md} | 0 ...-a-shell.md => ch09-00-writing-a-shell.md} | 0 21 files changed, 1018 insertions(+), 22 deletions(-) create mode 100644 src/ch02-05-implementing-the-bitfields-proc-macro.md create mode 100644 src/ch03-00-synchronization-primitives.md create mode 100644 src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md create mode 100644 src/ch03-02-implementing-basic-synchronization-primitives.md create mode 100644 src/ch04-00-printing-to-screen.md rename src/{ch04-00-memory-management.md => ch05-00-memory-management.md} (100%) rename src/{ch04-01-memory-allocator-types.md => ch05-01-memory-allocator-types.md} (100%) rename src/{ch04-02-implementing-a-bitmap.md => ch05-02-implementing-a-bitmap.md} (100%) rename src/{ch04-03-writing-a-physical-page-allocator.md => ch05-03-writing-a-physical-page-allocator.md} (100%) rename src/{ch05-00-interrupts-and-exceptions.md => ch06-00-interrupts-and-exceptions.md} (100%) rename src/{ch05-01-utilizing-the-idt.md => ch06-01-utilizing-the-idt.md} (100%) rename src/{ch05-02-programmable-interrupt-controller.md => ch06-02-programmable-interrupt-controller.md} (100%) rename src/{ch05-03-writing-a-keyboard-driver.md => ch06-03-writing-a-keyboard-driver.md} (100%) rename src/{ch06-00-file-systems-and-disk-drivers.md => ch07-00-file-systems-and-disk-drivers.md} (100%) rename src/{ch06-01-disk-drivers.md => ch07-01-disk-drivers.md} (100%) rename src/{ch06-02-implementing-a-file-system.md => ch07-02-implementing-a-file-system.md} (100%) rename src/{ch07-00-processes-and-scheduling.md => ch08-00-processes-and-scheduling.md} (100%) rename src/{ch07-01-thinking-in-terms-of-processes.md => ch08-01-thinking-in-terms-of-processes.md} (100%) rename src/{ch07-02-implementing-a-process-scheduler.md => ch08-02-implementing-a-process-scheduler.md} (100%) rename src/{ch08-00-writing-a-shell.md => ch09-00-writing-a-shell.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4ef040c..39b8151 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -14,27 +14,30 @@ - [A Minimal Bootloader](./ch02-00-a-minimal-bootloader.md) - [Legacy Legacy Legacy [OS]](./ch02-01-legacy-legacy-legacy.md) - [Entering Protected Mode [OS]](./ch02-02-entering-protected-mode.md) - - [Implementing the Bitfields Proc-Macro [RUST]](./ch02-03-implementing-the-bitfields-proc-macro.md) - [What is Memory Paging? [OS]](./ch02-03-what-is-memory-paging.md) - [Booting the Kernel [OS]](./ch02-04-booting-the-kernel.md) - -- [Printing To Screen [OS]](./ch03-00-printing-to-screen.md) -- [Memory Management [OS]](./ch04-00-memory-management.md) - - [Memory Allocator Types [OS]](./ch04-01-memory-allocator-types.md) - - [Implementing a Bitmap [OS]](./ch04-02-implementing-a-bitmap.md) - - [Writing a Physical Page Allocator [OS]](./ch04-03-writing-a-physical-page-allocator.md) - -- [Interrupts and Exceptions [OS]](./ch05-00-interrupts-and-exceptions.md) - - [Utilizing the Interrupt Descriptor Table [OS]](./ch05-01-utilizing-the-idt.md) - - [The Programmable Interrupt Controller [OS]](./ch05-02-programmable-interrupt-controller.md) - - [Writing a Keyboard driver [OS]](./ch05-03-writing-a-keyboard-driver.md) - -- [File Systems and Disk Drivers [OS]](./ch06-00-file-systems-and-disk-drivers.md) - - [Disk Drivers [OS]](./ch06-01-disk-drivers.md) - - [Implementing a File System [OS]](./ch06-02-implementing-a-file-system.md) - -- [Processes and Scheduling [OS]](./ch07-00-processes-and-scheduling.md) - - [Thinking in Terms of Processes [OS]](./ch07-01-thinking-in-terms-of-processes.md) - - [Implementing a Process Scheduler [OS]](./ch07-02-implementing-a-process-scheduler.md) - -- [Writing a Shell [OS]](./ch08-00-writing-a-shell.md) + - [Implementing the Bitfields Proc-Macro [RUST]](./ch02-05-implementing-the-bitfields-proc-macro.md) + +- [Synchronization](./ch03-00-synchronization-primitives.md) + - [How Synchronization is Achieved on a Multiprocessor System [RUST]](./ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md) + - [Implementing Basic Synchronization Primitives [OS]](./ch03-02-implementing-basic-synchronization-primitives.md) +- [Printing To Screen [OS]](./ch04-00-printing-to-screen.md) +- [Memory Management [OS]](./ch05-00-memory-management.md) + - [Memory Allocator Types [OS]](./ch05-01-memory-allocator-types.md) + - [Implementing a Bitmap [OS]](./ch05-02-implementing-a-bitmap.md) + - [Writing a Physical Page Allocator [OS]](./ch05-03-writing-a-physical-page-allocator.md) + +- [Interrupts and Exceptions [OS]](./ch06-00-interrupts-and-exceptions.md) + - [Utilizing the Interrupt Descriptor Table [OS]](./ch06-01-utilizing-the-idt.md) + - [The Programmable Interrupt Controller [OS]](./ch06-02-programmable-interrupt-controller.md) + - [Writing a Keyboard driver [OS]](./ch06-03-writing-a-keyboard-driver.md) + +- [File Systems and Disk Drivers [OS]](./ch07-00-file-systems-and-disk-drivers.md) + - [Disk Drivers [OS]](./ch07-01-disk-drivers.md) + - [Implementing a File System [OS]](./ch07-02-implementing-a-file-system.md) + +- [Processes and Scheduling [OS]](./ch08-00-processes-and-scheduling.md) + - [Thinking in Terms of Processes [OS]](./ch08-01-thinking-in-terms-of-processes.md) + - [Implementing a Process Scheduler [OS]](./ch08-02-implementing-a-process-scheduler.md) + +- [Writing a Shell [OS]](./ch09-00-writing-a-shell.md) diff --git a/src/ch02-05-implementing-the-bitfields-proc-macro.md b/src/ch02-05-implementing-the-bitfields-proc-macro.md new file mode 100644 index 0000000..b6dcaad --- /dev/null +++ b/src/ch02-05-implementing-the-bitfields-proc-macro.md @@ -0,0 +1,854 @@ +# Writing the Bitflags Macro + +_"In Lisp, you don't just write your program down toward the language, you also build the language up toward your program." - Paul Graham_ + +--- + +#![repository_card] + +As you may recall from the previous chapter, we used a proc-macro that was called `bitfields`. In this chapter, we are going to learn about Rust's procedural macros and even implement one ourselves. + +> Another great resource for this subject is the great video [Comprehending Proc Macros](https://youtu.be/SMCRQj9Hbx8?si=p-JUX0rLronBG_Nz) by Logan Smith + + +_If you are familiar with procedural macros, `syn` and `quote`, and want to go straight to the macro implementation, click [here](#defining-our-macro)_ + +## A Little Introduction to Procedural Macros + +Macros are not a new idea in programming languages, and most languages have them in some form. But what even is a macro? + +If you ask Wikipedia, we get the following definition. +### [_Macro_](https://en.wikipedia.org/wiki/Macro_(computer_science)) +
+ +_A macro is a rule or pattern that specifies how a certain input should be mapped to a replacement output._ + +
+ +When I read this definition, the first thing that comes to mind is that it really sounds like a function. After all, a function maps the input arguments to the output arguments, which is exactly what a macro does. And that is exactly right. Rust, macros (specifically procedural macros), are indeed a specific type of function, but let's not get ahead of ourselves. + +The key differences between macros and regular functions are that macros _replace_ the inputs and the outputs, and that is not always true with functions. Secondly, macros operate on our source code instead of variables in our program. + +Rust takes this definition very literally, and the definition for a proc-macro function looks like this: + +```rust +#![function!("snippets/src/lib.rs", custom_proc_macro)] +``` + +As you can see in this function, the input is Rust's `TokenStream`, which is literally our source code, and the output is also a `TokenStream,` which means it expects us to return also source code, which could be the same (Like the example above), but most of the time it is not. + +But what is this `TokenStream`? Why not just use strings of the source code? + +Well, the main reason we are even discussing this is that we want to manipulate the initial code in some way. Tokenizing the source code allows us to manipulate the code at a higher level, which is easier to reason about. This `TokenStream` is the most basic tokenization unit that we are going to work with, and it contains a sequence of `TokenTree` nodes that represent the source code. + +```rust +#![enum!("/lib/rustlib/src/rust/library/proc_macro/src/lib.rs", TokenTree)] +``` + +To see this more visibly, we can print our TokenStream, because it implements the `Debug` trait. Which, for a simple struct, would look like this: + +```text +TokenStream [ + Ident { + ident: "struct", + span: #0 bytes(43..49), + }, + Ident { + ident: "Example", + span: #0 bytes(50..57), + }, + Group { + delimiter: Brace, + stream: TokenStream [ + Ident { + ident: "a", + span: #0 bytes(64..65), + }, + Punct { + ch: ':', + spacing: Alone, + span: #0 bytes(65..66), + }, + Ident { + ident: "i32", + span: #0 bytes(67..70), + }, + Punct { + ch: ',', + spacing: Alone, + span: #0 bytes(70..71), + }, + ], + span: #0 bytes(58..73), + }, +] +``` + +_Can you understand the name of the struct and its fields?_ + +## How Macros are Executed + +As you may have noticed, macros do not behave exactly like regular functions. Another difference that they have is that they are evaluated at compile time. + +This thinking can also be used on regular functions, but not from our point of view, but from the compiler's point of view. For the compiler, regular functions are also a mapping, from some target language (in our case, Rust) to some other target language (in most cases, ASM[^1]). + +For example, this function: + +```rust +#![function!("snippets/src/book/ch02_03/general.rs", square)] +``` + +would map to the following ASM code: + +> [!TIP] +> Look it yourself at [compiler explorer](https://godbolt.org/z/7vTzbs6e9) + +```x86asm,icon=@https://icons.veryicon.com/png/o/business/vscode-program-item-icon/assembly-7.png +square: + mov eax, edi + imul eax, edi + ret +``` + +From this point of view, macros are not so different, but instead of a target language, they are mapped to the same language. +So this macro: + +```rust +#![source_file!("snippets/src/book/ch02_03/general.rs", 8:12)] + +#![function!("snippets/src/book/ch02_03/general.rs", foo)] +``` + +Would map to this literal Rust code: + +```rust +#![function!("snippets/src/book/ch02_03/general.rs", foo_expanded)] +``` + +The fact that macros operate on our source code means that we can abstract certain logics that regular functions cannot. For example, take a look at this macro: + +```rust +#![source_file!("snippets/src/book/ch02_03/general.rs", 24:31)] +#![function!("snippets/src/book/ch02_03/general.rs", main)] +``` + +It works because it injects the `break` expression into the code at the call site, which is something that a function just can't do. + +```rust +#![function!("snippets/src/book/ch02_03/general.rs", unwrap_or_break)] +``` + +At this time, I hope you understand the great power of macros, and the great [code generation](https://en.wikipedia.org/wiki/Code_generation) capabilities that they enable. But, you might think rightfully think that in the examples above, we didn't have the option to insert 'coding' logic into the macro expansion. This is where procedural macros come in. + +[^1]: This is actually a simplified view; compilers have intermediate representations. These representations are really useful but out of the scope of this book. If you are like me, and this really interests you, I will drop a great blog post that gives an example of why the intermediate representations are useful. [From Rust to Reality: The Hidden Journey of fetch_max](https://questdb.com/blog/rust-fetch-max-compiler-journey/) + +## Macro Types + +Just before we dive into procedural macros, let's cover the type of macro that we already used in the examples above. + +> All the syntax information about how macros are structured is taken directly from the [Official Rust Reference](https://doc.rust-lang.org/reference/macros.html). + +### Declarative Macros + +Declarative macros are the simplest type of macro, and they are the ones that we used in the examples above. They are mainly used to generate simple syntax extensions, which are commonly called "macros by example". + +Each macro is defined by a set of rules that specify how the macro should expand. Each rule looks a bit like a function signature that can get certain `Metavariables`. These `Metavariables` are placeholders for certain Rust syntax that are replaced with actual values when the macro is expanded. + +Let's analyze the syntax of a declarative macro rule from the earlier examples. + +```rust +#![source_file!("snippets/src/book/ch02_03/general.rs", 51:65)] +``` + +We will go a bit deeper than necessary on the common types of metavariables that are available. This is because later in this chapter, we are going to talk about the `syn` library, which will parse Rust's syntax into similar structures. + +Each metavariable starts with a `$` followed by the name of the metavariable, which is used to refer to it. Then it is followed by a colon and the type of the metavariable. + +The common types of metavariables are: + +1. **Idents ($i:ident)** => These can be function names, variable names, type names, etc. They also include keywords like `fn`, `let`, `struct`, etc. +2. **Expressions ($e:expr)** => Expressions are things that are evaluated to a value, like `1 + 2` or `foo.bar()`. +3. **Items ($i:item)** => Items are the components of a module, for example the entire definition of a function or a struct. +4. **Statements ($s:stmt)** => Statements are the individual lines of code that make up a function or block. For example, `let x = 42;` is a statement. +5. **Blocks ($b:block)** => Blocks are groups of statements that are executed on the same scope. For example, `{ let y = 33; let x = 7 + y; x }` is a block. + +_For a full list of available metavariable types, see the [reference](https://doc.rust-lang.org/reference/macros-by-example.html#r-macro.decl.meta.specifier)_ + + +### Procedural Macros + +Now for the real deal. Procedural macros give us the ability to go beyond simple syntax extensions and allow us to write custom Rust code that will run at compile time on the macro input to consume and produce new Rust syntax (Depending on the macro type, the returned syntax will replace the input syntax or will be added to it). + +Because procedural macros are another piece of code that will run at compile time, they cannot be defined in the same crate as the code that uses them. This is because the Rust compiler must initially compile the code of the macro so it will be able to run it during the compilation process. In addition, each proc macro crate must add the following configuration to its `Cargo.toml` file, which will tell Cargo that this is a proc macro crate. + +```toml +[lib] +proc-macro = true +``` + +Like all functions, these macro functions can also fail, although these functions are allowed to panic. They are encouraged to use the `compile_error!` macro to return a compile-time error instead, which is the compiler form of `panic!` + +To gain the `Tokenstream` type and the attributes that will be used on the macro functions, we will use the `proc_macro` crate, which is automatically linked to our crate if it is a proc macro crate. + +### `function_like!()` + +Function like macros are very similar to declarative macros. They are invoked like a regular function and take a `TokenStream` as input and return a `TokenStream` as output. + +This type of macro can be called anywhere in our code, even in the global scope, and is defined using the following syntax: + +```rust +#![function!("snippets/src/lib.rs", foo)] +``` + +Then it can be called like a regular function, which will create a function that is called `bar` which could be used in our code. + +```rust +#![source_file!("snippets/src/book/ch02_03/invoke.rs", 1:99)] +``` + +_This type of macro replaces the macro invocation with the generated code, so the macro invocation is effectively replaced with the generated code._ + +### `#[derive(CustomDerive)]` + +Derive macros are used on Rust items to generate code automatically. They are invoked using the `#[derive]` attribute and take the item they are applied to as input. Most of the time, derive macros are used to implement traits such as `Debug`, `Clone`, `PartialEq`, etc. + +Derives may also include helper attributes, which are used to customize the generated code. + +This type of macro can be called only from structs, enums, or unions. + +```rust +#![function!("snippets/src/lib.rs", derive_with_helper_attr)] +``` + +And it is used on a structure like this: + +```rust +#![struct!("snippets/src/book/ch02_03/general.rs", Foo)] +``` + +_This type of macro does not replace the macro invocation or the input item with the generated code, and the generated TokenStream is appended to the input TokenStream._ + +### `#[attribute(macros)]` + +Attributes are used to annotate items. They are placed before the item they are applied to and are used to customize the behavior of the item. + +Attributes may also include input variables, which can be used to pass 'configuration' to the macro. + +```rust +#![function!("snippets/src/lib.rs", return_as_is)] +``` + +And it is used on a structure like this: + +```rust +#![struct!("snippets/src/book/ch02_03/general.rs", Bar)] +#![function!("snippets/src/book/ch02_03/general.rs", bar)] +``` + +_This type of macro replaces the macro invocation and the input item with the generated code._ + +## Introduction to Syn and Quote + +Remembering our goal to write the `bitfield` macro from the earlier chapter, you can already guess that we want to write an `attribute` macro. But, parsing the TokenStream we saw above is really hard, because it will require us to understand Rust's syntax tree, which can be quite complex. + +Luckily for us, the `syn` crate, written by `David Tolnay` provides a way to parse Rust syntax tree into a structured AST (Abstract Syntax Tree), which makes it easier to work with Rust source code. + +### What are Abstract Syntax Trees + +As the name suggests, this is a tree like structure that represents the syntax of a certain programming language (in our case, Rust). +Before diving right into the implementation of `syn` on Rust syntax, let's first understand what an AST is. + +We will look at a really simple program that is written in Python. + +```python +current = 0 +for item in items: + if item > current: + current = item +``` + +A simplified syntax tree for a simple program like this might look like this: + +
+ +
Figure 3-1: simplified syntax tree
+
+ +As you can see, in a tree like this, we can have types that help us represent the syntax in our language. For example, the `Assign` statement, which contains a `left` and `right` side. Or the `For` loop, which contains the item that is being iterated over, the collection name, and the body of the loop. Then, when we want to operate on the syntax itself, for example, create the same if statement, but change the name of the item. We can simply copy the type and change the item ident to a new one. + +As you may have guessed, `syn` does the exact same thing we did with our small program, but with all the complexity of a real language. So let's see what types it offers. + +_There are a lot of types on the `syn` crate, and we will only cover some of them. Once you get the hang of it, all the other will be easy to understand._ + +The top level type for the AST is `syn::File`, which represents a complete Rust source file. + +```rust +#![source_file!("/syn-2.0.118/src/file.rs", 6:86)] +``` + +Ok, we can see that `syn::File` is made out of a list of `syn::Attribute` and `syn::Item`. But this doesn't tell us much, so let's also explore them. + +```rust +#![source_file!("/syn-2.0.118/src/attr.rs", 23:183)] +``` + +So we can see an attribute, like `#[derive(Debug)]`, is represented by `syn::Attribute`. Currently, we will not dive deeper into `Attribute`, but we will cover more of it when we will use it in our macro implementation. + +Now let's see `syn::Item`. + +```rust +#![source_file!("/syn-2.0.118/src/item.rs", 22:101)] +``` + +As you can see, we have a lot of items, and I hope that you can start and recognize some of them. As an example, let's cover `ItemConst`. + +```rust +#![source_file!("/syn-2.0.118/src/item.rs", 103:118)] +``` + +If you have noticed closely, the order of the fields in the struct definition is the same as the order in the source code. This makes it really easy to map the AST back to the source code. + +Also, as a side note, keywords like `const`, `struct` and punctuation like `:` and `=` does have types, but `syn` also provides a `Token!` macro that maps the literal token to its corresponding type. + +The last type that we are going to cover is `syn::Expr`, which represents an expression from the source code. Because most of Rust's syntax is represented as expressions, `syn::Expr` is a very large type. + +```rust +#![source_file!("/syn-2.0.118/src/expr.rs", 37:269)] +``` + +These types are very powerful and help us express language in a structured way. As a quick example, let's see how `syn::ItemStruct` is represented in the AST. In this example, we have the exact same struct that we showed its `TokenStream` representation. + +``` +ItemStruct { + attrs: [], + vis: Visibility::Inherited, + struct_token: Struct, + ident: Ident { + ident: "Example", + span: #0 bytes(50..57), + }, + generics: Generics { + lt_token: None, + params: [], + gt_token: None, + where_clause: None, + }, + fields: Fields::Named { + brace_token: Brace, + named: [ + Field { + attrs: [], + vis: Visibility::Inherited, + mutability: FieldMutability::None, + ident: Some( + Ident { + ident: "a", + span: #0 bytes(64..65), + }, + ), + colon_token: Some( + Colon, + ), + ty: Type::Path { + qself: None, + path: Path { + leading_colon: None, + segments: [ + PathSegment { + ident: Ident { + ident: "i32", + span: #0 bytes(67..70), + }, + arguments: PathArguments::None, + }, + ], + }, + }, + }, + Comma, + ], + }, + semi_token: None, +} +``` +_Can you see the name of the struct, and the type of the field in the AST?_ + +As you can see, what was before a list of punctuations and idents has now become a structured representation that is easier to work with. + +The most important thing about Syn is that we can use the types that it offers to create new, custom types that are not bound to the language's AST. + +But how would Syn know to parse our custom syntax into the AST types it offers? This is where the `Parse` trait comes in. When Syn wants to parse our custom syntax, it will call the `parse` method from the `Parse` trait and pass in the token stream to parse. + +```rust +#![trait!("/syn-2.0.118/src/parse.rs", Parse)] +``` + +We will go deeper into this when we create our own custom `Parse` implementation. One important thing to understand is that all of syn's types implement `Parse` themselves, so most of the time, implementing `Parse` for types that are built from syn's AST types is easy. + +Up until now, we have learned how to parse our source code into a meaningful AST representation. This representation will help us to work with the syntax and to implement our macro's logic. But, after we parsed the source code and processed it to our needs, we needed to return it to a `TokenStream`. This is where the `quote` crate comes in. + +Quoting is a term that is borrowed from Lisp, and it means that we write things that look like code, but they will actually convert into data under the hood, or in our case, the `TokenStream` type. + +The `quote` crate provides a `quote!` macro that allows us to write quoted expressions. + +For example, let's define a simple quoted expression that represents a struct definition: + +```rust +#![source_file!("snippets/src/book/ch02_03/general.rs", 81:90)] +``` + +As you can see, it seems like we write Rust code, but actually, under the hood, it is converted into a `TokenStream`. + +Another great quality that this macro has is that it supports entering variables into the quoted expression. Let's look at an example where we change the name of a function inside an attribute macro. + +```rust +#![function!("snippets/src/lib.rs", change_name)] +``` + +As you can see, we parsed the input with `syn` into a function item. Then, we changed the name of the function and transferred it to the `quote!` macro with the `#` so that it would convert the variable into a `TokenStream`. + +But how quote know to convert the variable into a `TokenStream`? This is where the `ToTokens` trait comes in. + +```rust +#![trait!("/quote-1.0.45/src/to_tokens.rs", ToTokens)] +``` + +In this trait, the `to_tokens` method is defined, which gets a `&mut TokenStream` and appends the tokenized representation of the variable to it. + +The types that are defined in `syn` already implement this trait, so they can be used with `quote!` without any additional work, like in the example above, where the `ItemFn` became a function definition. + +## Defining our Macro + +In my opinion, the most important thing to do before we even start to code (even not specifically for macros) is to define what we want from our program. + +The main thing we wanted in the first place was to represent a number, e.g., u8, u16, u32, etc., as flags. To see a clear example, look at the drawing below. + +
+ +
Figure 3-1: bitflag struct example
+
+ + +In this drawing, we can see six different flags. Each takes a different part inside our u16 number. +1. Flag A is between bits 00-02 +2. Flag B is between bits 02-07 +3. Flag C is between bits 07-10 +4. Flag D is between bits 10-12 +5. Flag E is between bits 12-15 +6. Flag F is between bits 15-16 + +For each flag, we would like to have multiple functions. + +- A getter, which returns the value of the flag. +- A setter, which sets the value of the flag. +- Clear function, which writes a clear value if defined directly to the flag. (Will be necessary in the future) + +Because we need multiple functions defined, the best Rust item suited for the job is a `struct`. Also, because a struct will wrap the entire definition, the macro will have in its context all of the definitions of all the flags, which means we could also implement the `Debug` trait on it to print all of the flags. + +Some flags will need different functions, and may also have `types`. For example, think about the protection level field in the previous section. While we can just leave it as a number, most of the time, it is more convenient to have an enum that represents the valid values. Also, some flags may not need all the functionalities of get, set, and clear. For that, we want to have the ability to control which functions will be generated. + +And for the last caveat, some flags will be written as absolute values on their setter function, and will return absolute values on their getter. What does that mean? Take, as an example, flag `E` on the example above. The span of this flag is between bits 12-15. In most of our flag cases, we would want to write numbers between 0 and 7 to this value, because it is 3 bits wide. When we set the don't shift attribute, we would want an absolute value for this flag, which means the lowest value (besides 0) will be `1 << 12` (The first bit of the flag), and the highest value will be `1 << 14,` where the jumps between each value will be `1 << 12.` + +This design for this macro, with inspiration from [Proceadural Macro Workshop](https://github.com/dtolnay/proc-macro-workshop#attribute-macro-bitfield), will be a regular Rust struct, with helper attributes. + +For example, this struct will represent the flags in the example above (with example helper attributes). + +```rust +#![struct!("snippets/src/book/ch02_03/general.rs", MyFlags)] +``` + +## Implementing the Macro + +### Sketching the Idea + +The first thing that I like to do when creating a macro is to create a simple input for the macro and generate the key functions output by hand. +This way, I could have a mental model of what it is supposed to do, and I can generalize on that. + +So, for starters, let's create a really simple input and output for our macro. + +```rust +#![struct!("snippets/src/book/ch02_03/general.rs", SimpleFlags)] +``` + +Just before we create our functions, what will our struct type be? In this case, we have a two bit field and a one bit field, but there is no type that is three bits wide. Instead, we are going to pick the closest uint type that is large enough to hold our fields. In this case, a u8. + +```rust +#![struct!("snippets/src/book/ch02_03/general.rs", SimpleFlagsType)] +``` + +Now for our functions. The problem that we need to solve is how to get and set the value of the bits that are stored in the underlying `u8` field. + +_This part assumes familiarity with bitwise operations like right and left shifts, and simple gates like AND, OR, and NOT. For those of you who are not familiar with these operations, I really recommend seeing this [video](https://www.youtube.com/watch?v=vqpfrSIyojo) by BitLemon._. + +
+ +
figure 3-2: simpleflags layout
+
+ +We will start with reading the value for the `b` flag. There are multiple combinations of bitwise operations that can achieve this. The one that we will use is to first zero out the entire content of the `u8` except for our `b` flag, and then shift it to the right and read it. + +So first, let's think about how we can zero out the entire content of the `u8` except for our `b` flag. We can do this by using the `&` operator to perform a bitwise AND operation between our `u8` value and a mask[^2] that has all bits set to 0 except for our `b` flag, which will be all 1s. By hand, this mask will look like this `0b00000100`. But this, of course, does not help us much, because we need to automatically generate this mask for each bitfield, and it may also have multiple 1 bits, and not only one, like in this case. + +To generate this mask, we will think of a much simpler case: how can we put a sequence of ones at the start of our mask? Before I give the answer, let's think about what a sequence of ones means. A sequence of ones is always a number, which, when we add 1 to it, will always become a perfect power of 2 on the bit after the sequence. For example, `0b00000111` (7) will become `0b00001000` (8) when we add 1 to it. + +You may have also noticed that the number of bits that were set to 1 before we added 1 is equal to the power of 2 of the number after we added 1. For example, `0b00000111` (7) has 3 bits set to 1, and 8 is exactly `2^3`. + +> [!TIP] +> If I were you, I wouldn't accept this fact. Go try it for yourself with more examples to see that it is true + +To generally create a mask with the first `n` bits set, we can use our formula: `2^n - 1`. Because we are speaking only of powers of two, we will use `(1 << n) - 1` to create the mask. Which is the same thing. + +[^2]: The sequence of bits that will be used along with our value in a logic gate. + +```rust,playground +#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_1)] + +#![function!("snippets/src/book/ch02_03/mask.rs", main)] +``` + +If you played with this example in the demo, you may have found that in one particular case, this formula does not work as expected. (If you didn't find it, I urge you to try it yourself. + +When our `(1 << n) - 1` will result in an all 1, it means that `1 << n` was bigger than our underlying type. For example, `(1 << 8) - 1,` which should generate the `0b11111111` mask, will instead generate `0`, because `1 << 8` is `256`, which is bigger than `u8` can hold. While we can use bigger types, for the maximum size type, it will not work. + +The alternative method that we are going to use is instead of increasing the number of 1 bits in our mask each time, and starting from 0, we are going to start with an all 1 mask, and reduce the number of 1 bits each time. This won't have the gap at all 1 mask, because it is the starting value. + +To achieve it, we are going to start with our type maximum mask, and then shift it to the right by the total number of bits in our type, minus our width. For example, if our type is `u8`, and our width is `3`, our mask will be `0b11111111 >> (8 - 3) = 0b00000111`. + +```rust,playground +#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_2)] + +#![function!("snippets/src/book/ch02_03/mask.rs", main)] +``` + + +The next thing that we are going to do is to relocate the position of the bits in our mask to the flag position in our u8. + +This could easily be done using the left shift operator `<<` with the offset of our flag. For example, if the starting bit of our flag is at position 2, we can shift our mask to the left by 2 bits: `mask << 2`. Which makes our final mask generation function look like this: + +```rust,playground +#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] + +#![function!("snippets/src/book/ch02_03/mask.rs", main)] +``` + +To read the value, we just need to apply an AND gate with the mask, and then shift the result to the right by the offset to normalize it. + +```rust,playground +#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] + +#![function!("snippets/src/book/ch02_03/read.rs", read_flag)] + +#![function!("snippets/src/book/ch02_03/read.rs", main)] +``` + +To write to our value, you may be tempted to use the left shift operation on the `new_value` to shift to the correct position and then OR it with the original value. While your intuition is good, this approach will not work. This is because the OR gate only changes bits from 0 to 1, but cannot change bits from 1 to 0. So our approach will be to first clear the bits we want to change, and then OR it with the new value. + +To clear the flag, we can use an AND gate, where all the flag bits are set to 1, and the rest are 0. This will leave the flag bits unchanged, and the rest will be cleared. So our clear mask for flag `b` looks like this `0b11111011`. + +You may have noticed that this is the exact inverse of the mask we used to read the flag. So we will use the same approach to generate it, and use the NOT gate with the `!` operator to invert all the bits. After that, we can OR it with the new value shifted to the correct position. + +```rust,playground +#![function!("snippets/src/book/ch02_03/mask.rs", generate_mask_3)] + +#![function!("snippets/src/book/ch02_03/write.rs", write_flag)] + +#![function!("snippets/src/book/ch02_03/write.rs", main)] +``` + +### Struct Definition + +When starting to implement any piece of code, it is always a good idea to first sketch out the types that we are going to use. + +Borrowing the definition of our macro again, these are the types that come to mind. + +``` +#[bitfields] +struct MyFlags { + #[flag(r)] + a: B2, + b: B5, + #[flag(rwc(30))] + c: B3, + #[flag(flag_type = ProtectionLevel)] + d: B2, + #[flag(r, dont_shift)] + e: B3, + f: B1, +} +``` + +- BitFields + - FlagAttribute (i.e `#[flag(r, dont_shift, flag_type = ProtectionLevel)]`) + - Permissions + - FlagType + - DontShift + - Single Bitfield (i.e., `a: B2`) + - FlagMeta (i.e., `width: 2, type: u8`) + +### FlagAttributes + +Our `FlagAttribute` struct will simply store the permissions, flag type, and `dont_shift` flag of a flag. + +```rust +#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagAttribute)] +``` + +#### Permissions + +For our permission attribute, we want to store the read, write, or clear permissions and the clear value. You may be tempted to use one number here and encode it in bits to have good performance on it. But, this will not be a good idea because macros are expanded at compile time, and the expansion between compilations is cached. So the performance increase doesn't really matter. + +```rust +#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagPermission)] +#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Default for FlagPermission)] +``` + +#### DontShift + +For some of our flags, and especially the `dont_shift` flag, we want to parse custom idents, in this example, the literal `dont_shift` keyword. + +Instead of parsing ident's by our own logic, `syn` provides a cool `custom_keyword!` macro that allows us to parse custom idents easily. + +```rust +#![source_file!("crates/macros/src/bitfields/flag_attr.rs", 6:11)] +``` +#### FlagType + +For our final type on the attribute, we want to parse the sequence `flag_type = some_type`. To represent this, we will use the following struct. + +For our type, we will use syn's [`TypePath`](https://docs.rs/syn/latest/syn/struct.TypePath.html) struct, which represents a path to a type, such as `std::ffi::CString`. + +```rust +#![struct!("crates/macros/src/bitfields/flag_attr.rs", FlagType)] +``` + +### Single Bitfield + +For a single field, we would want to include the attribute we just defined, the comments, visibility, and name of the field to use on the generated functions, and the size and offset of the field for our read and write functions. + +```rust +#![struct!("crates/macros/src/bitfields/bitfield.rs", BitField)] +``` + +> [!NOTE] +> We use references on some of the fields because this structure will be created from the [`syn::Field`](https://docs.rs/syn/latest/syn/struct.Field.html) struct, so instead of cloning the values, we use references to avoid unnecessary allocations. + + +#### FlagMeta + +For our `FlagMeta` struct, we will want to store the width of the field, but also the type that will represent it. So `B3` would have width `3` and type `u8`. + +```rust +#![struct!("crates/macros/src/bitfields/utils.rs", FlagMeta)] +``` + +### Parsing the Attribute + +We will start off easy by parsing the `FlagType` attribute. Because every element in this attribute already implements the `Parse` trait, we can call its parse function in the correct order. + +```rust +#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagType)] +``` + +If you were wondering why we are calling the `parse` function on the input instead of on the type itself. It is because the `ParseStream` implements this very convenient `parse` function that allows us to parse a single token from the stream at a time. + +```rust +#![impl_method!("/syn-2.0.118/src/parse.rs", ParseBuffer::parse)] +``` + +Next, let's parse something that takes a little more effort, our `FlagPermission`. + +```rust +#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagPermission)] +``` + +For the `dont_shift` keyword, the parsing is implemented automatically because we used the `custom_keyword!` macro to define it. + +And now for the parsing of the entire attribute. While we can define a strict order for the attribute, and then call parse on each field. We will not do that because it will be annoying to use the macro. Instead, we will `fork` the stream and try to parse it as each of our fields. If the parsing succeeds, we will save the parsed item and keep forking and parsing until we reach the end of the stream. + +But what is forking? And why do we need it? + +Imagine our stream as a really large linked list that contains all of our tokens. When we parse the stream, we are moving our position through the list by `consuming` the tokens as we parse them. The problem with what we are trying to do is that if we start parsing an item, and it fails, by the time we have already parsed some of the tokens of the item, we have no way of coming back to the exact position we were at before the failure. + +This is where forking comes in. When we fork the stream, we create another pointer to the position on the list, which is independent of the original position we had. Then, we can try to parse the stream with the fork. If it fails, we can simply discard the fork, and if it succeeds, we can advance the original position to the fork's position. + +
+ +
figure 3-3: parse stream with fork
+
+ +The last thing we want to keep in mind is that we want to avoid duplicates in our attributes. So we will keep for each attribute a variable that stores whether we have already seen that attribute before. + +```rust +#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", Parse for FlagAttribute)] +``` + +And now for the `try_parse` function, where we basically want to fork the stream and try to parse the item, if we succeed, we advance the original position to the fork's position, we discard the fork, and increment the error_counter. + +```rust +#![function!("crates/macros/src/bitfields/flag_attr.rs", try_parse)] +``` + +Although we implemented `Parse` for `FlagAttribute`, we are not going to create it from raw tokens. Because, if you noticed, we only parsed the inside of the attribute, but not the `#[flag()]` part. +For that, we are going to use the `Meta` part of our `syn::Attribute`. + +```rust +#![source_file!("/syn-2.0.118/src/attr.rs", 455:486)] +``` + +In our case, we are going to have a `Meta::List`, which contains a `TokenStream` of the attribute's contents, hence the implementation of the `Parse` trait. + +```rust +#![trait_impl!("crates/macros/src/bitfields/flag_attr.rs", TryFrom for FlagAttribute)] +``` + +### Parsing the Struct + +Instead of parsing the struct directly, we will instead parse a regular `syn::ItemStruct` and then implement the `TryFrom` trait to convert between types in the regular `syn::ItemStruct` and our custom `BitFields` type. + +Again, we will start off easy, by converting the type of the struct field into our custom `FlagMeta` type. + +```rust +#![trait_impl!("crates/macros/src/bitfields/utils.rs", TryFrom for FlagMeta)] +``` + +To turn the width number of the type into the type that will represent it, we will use the following function. + +```rust +#![function!("crates/macros/src/bitfields/utils.rs", type_from_size)] +``` + +For each field on our struct, we are going to initially extract all the attributes on it and divide them into document attributes and our flag attributes. + +This can be easily done, because Rust doesn't store our comments as a string starting with `///` but as a `#[doc(some_comment)]` attribute. This makes our comments actually a `syn::Attribute` token, which we already know how to work with. + +```rust +#![function!("crates/macros/src/bitfields/bitfield.rs", extract_attributes)] +``` + +After that, we are going to create our field from the `syn::Field` token. But, the field itself is not enough, because from it we can't know the offset of the field in the struct. We are going to give as a parameter in the `new` function that will create our `BitField` instance. We can do that because when we create our fields one by one, we will add each time their size to an offset, which will, of course, start at 0. + +```rust +#![impl_method!("crates/macros/src/bitfields/bitfield.rs", BitField::new)] +``` + +Finally, after all that parsing, we can turn the parsed `syn::ItemStruct` into our `BitFields` instance. + +```rust +#![trait_impl!("crates/macros/src/bitfields.rs", TryFrom for BitFields)] +``` + +## Generating the Code + +With all of our types set up, we can now generate the code for our functions from them. + +Our first function will be a utility function that will provide us with some checks on our input value. This check will be used to check that the input value is within the valid range for the field, and it will be guarded by a `debug_assert!` macro, so in release builds it will be optimized out. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::checks)] +``` + +For each of our functions, we are going to use three main types. The first is the type of the variable that we are getting, the second is going to be the type that represents the type of the variable we are getting, and the third is the type of the entire struct. For example, we might have a field `#[flag(flag_type = Bar)] foo: B6`. The type of our variable in this case will be `Bar`, the type that represents the field is `u8` because it is only 6 bits wide, and the type of the entire struct depends also on the other fields and their sizes, but it will also follow the rules of the `type_from_size` function. + +To store all of these types, we are going to use a struct. + +```rust +#![struct!("crates/macros/src/bitfields.rs", FieldTypes)] +``` + +Then, to create it from our field, so it can be used in other functions, we are going to use the following function. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::field_types)] +``` + +Without further diving into our utility functions, let's look at our read function. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_read)] +``` + +As explained above, the first thing we do, is to extract the types that we are going to use inside this function. Then, we are going to get the function name, for the rest of our functions, we are not going to have a function like this, but for the read function, I personally wanted, for my convenience, that if the type of the item was `bool` so that it would change to `is_` instead of `get_` + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::read_fn_name)] +``` + +For our `read_shift` function, we need to know if to shift the value or not per the `dont_shift` attribute, and in which direction. For write operations, we need a left shift to change from the absolute value we get to the relative value on the flag, and for read operations, we need a right shift to convert from the relative value on the flag to its absolute value. When the `dont_shift` attribute is present, we don't need any of this, because the values are always absolute. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::read_shift, write_shift)] +``` + + +> [!IMPORTANT] +> You may have noticed that when we use functions from the `core` library, I am referring to them as `::core` with a leading `::`. And that, for example, when I use the `try_from` method from the `TryFrom` trait, I call the trait function with the object instead of `object.try_from(T)`. +> +> When writing a macro, we don't want to insert a `use` statement into the codebase of the person that is using our macros, and we can't assume (although most of the time unlikely) that he or she didn't implement functions with similar names as in our example `try_into`, that are doing an entirely different thing. +> +> Because of that, the safest way to call functions from libraries and trait methods is to use their fully qualified name. So we use `::` before core, to reference the compiler's core library (in case they have a core.rs module) and the fully qualified trait name to call its functions. + +When implementing our write and clear functions, we are going to use almost the exact same code for the writing logic. Because of that, we are going to extract this piece of code to a general `volatile_write` function. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::volatile_write)] +``` + +Which makes this our write function. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_write)] +``` + +Our `v_to_repr` function will be used to convert our value `v` from the type of the flag to its representation type by using the `try_from` function. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::v_to_repr)] +``` + +Therefore, we can use this value `v` that it defines in the `volatile_write` function, because it can `as` cast into the struct type. + +For the final function, which is our clear function, we are going to use the same logic as the write function, but instead of operating on a value `v`, we are going to operate on the clear value, which is already a usize. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_clear)] +``` + +As a little bonus function, that is mainly used for convenience, we will create a build function that is meant to operate on an empty struct, and define multiple flags on its creation (e.g., `let flags = Flags::new().flag1(2).flag2(3)`). + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::fn_build)] +``` + +Because we know we are operating on an empty flag, instead of clearing the flag and then writing, we can simply use the OR gate to write our value, because we know nothing is set in the flag yet. + +The last things that we are going to generate is the `Debug` trait implementation, and the `From` trait from the flags into the struct repr, and from the struct repr into the flags. + +The latter is really easy; to translate into the underlying type, we just return the inner type. And to construct from the inner type, we simply call the constructor. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::conversion_impls)] +``` + +To implement the `Debug` trait, we need to first create the formatter debug struct builder and then add each of our fields to it. +Because we can have multiple fields, we need to insert some sort of repetition. Luckily, the `quote!` macro provides us a way to do it. +If we have a vector, or an iterator, of things that implement `ToTokens`, we can insert them all using the `#(#..)` syntax. + +```rust +#![impl_method!("crates/macros/src/bitfields.rs", BitFields::debug_impl)] +``` + +> [!WARNING] +> The debug trait implementation makes our binary larger and adds additional compilation time. In the correct version of the macro, the `Debug` trait implementation is not guarded by a feature, but in the future, it will be generated only if the main struct includes `#[derive(Debug)]`. +> As a cool exercise, you can try to add that feature to the macro yourself. + +And for the grand finale, the implementation of the `ToTokens` trait for our macro. + +```rust +#![trait_impl!("crates/macros/src/bitfields.rs", ToTokens for BitFields)] +``` + +And for the macro itself, we need to parse a struct from the input and convert it into a `BitFields` struct. Then just turn it into tokens or raise an error in parsing, depending on the result. + +```rust +#![function!("crates/macros/src/lib.rs", bitfields)] +``` diff --git a/src/ch03-00-synchronization-primitives.md b/src/ch03-00-synchronization-primitives.md new file mode 100644 index 0000000..ef0d0ee --- /dev/null +++ b/src/ch03-00-synchronization-primitives.md @@ -0,0 +1 @@ +# Synchronization diff --git a/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md b/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md new file mode 100644 index 0000000..5080ac2 --- /dev/null +++ b/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md @@ -0,0 +1 @@ +# How Synchronization is Achieved on a Multiprocessor System diff --git a/src/ch03-02-implementing-basic-synchronization-primitives.md b/src/ch03-02-implementing-basic-synchronization-primitives.md new file mode 100644 index 0000000..cb0fb2b --- /dev/null +++ b/src/ch03-02-implementing-basic-synchronization-primitives.md @@ -0,0 +1 @@ +# Implementing Basic Synchronization Primitives diff --git a/src/ch04-00-printing-to-screen.md b/src/ch04-00-printing-to-screen.md new file mode 100644 index 0000000..2ea3f26 --- /dev/null +++ b/src/ch04-00-printing-to-screen.md @@ -0,0 +1,136 @@ +# Printing To Screen + +_"The most effective debugging tool is still and careful thought, coupled with judiciously placed print statements." - Brian Kernighan_ + +--- + +#![repository_card] + +Printing is an important aspect of an operating system, especially in early development because it is our way to gain a visual output from our operating system. This will massively improve the interaction with our OS, and not only will it give us a huge advantage in debugging, but it will also grant us the ability to display a shell, which we will do in the upcoming chapters. + +## Why didn't we print until now? + +If you remember the [example code](./ch01-02-booting-our-binary.md#hello-world) in the first bootable code we wrote, we did print to screen during that code. +This print utilized the `Video (int 10h)` interrupt on BIOS with the `Print Char (0xE)` function to print character by character the string 'Hello, World!' + +This was our only way to print we were on `real mode`. And while I developed the code, I actually did use it to print single characters as errors code, So I could understand what was my program doing. + +On `protected mode`, we couldn't use the BIOS anymore, so printing was much harder. Additionally, we only turned on paging, so debugging with [QEMU monitor](https://qemu-project.gitlab.io/qemu/system/monitor.html) was much easier. + +While we could have written a simple printer for each stage, it was not necessary, and it would have bloated our binary, which in the first stage had only 512 bytes, and had almost no use in the second stage. But now, on the kernel init stage, it would become really handy! + +## How to print without BIOS? + +We are gonna print using the Video Graphics Array or VGA for short. This protocol as the name suggests, puts an array in memory which will represent our screen. When we want to print, we simply write the content to the array, and it will automatically refresh on certain interval display to newly provided content. + +## The VGA Protocol + +VGA has primarily two modes, the first one is called `graphic mode`, which is used to write raw pixels to the screen. The second mode is called `text mode` and it is used to write text to the screen. In this chapter we are going to focus on the `text mode` because we mostly want to provide messages and text on the screen. + +_Maybe on later chapters we will implement UI, so we will a more graphic mode, but then we actually might not use VGA_ + +### Printing with Text Mode + +To print with text mode, we need to write to the screen buffer a special character that is 2 bytes long. This special character encodes the actual ASCII character that we are going to print, the background color of the text, and the foreground color of the text. + +> The screen buffer of the `graphic mode` starts at address 0xA0000 and the screen buffer of the `text mode` starts at address 0xB8000. + +The first byte encodes the ASCII character, and it is not special. The second byte will encode our color, the first 4 bits will be the foreground color, and the next 4 bits will be the background color. + +There are multiple color palettes that VGA uses, the one our mode uses, is the 4 bit color palette and it includes the following colors. + +```rust +#![enum!("crates/common/src/enums/vga.rs", Color)] +``` + +```rust +#![struct!("crates/drivers/vga-display/src/color_code.rs", ColorCode)] + +#![impl!("crates/drivers/vga-display/src/color_code.rs", ColorCode)] + +#![trait_impl!("crates/drivers/vga-display/src/color_code.rs", Default for ColorCode)] +``` + +Then the encoding of each `Screen Character` will look like this. + + +```rust +#![struct!("crates/drivers/vga-display/src/screen_char.rs", ScreenChar)] + +#![impl!("crates/drivers/vga-display/src/screen_char.rs", ScreenChar)] + +#![trait_impl!("crates/drivers/vga-display/src/screen_char.rs", Default for ScreenChar)] +``` + +At this point, we are ready to write to the screen whatever we want, we just need to write a `ScreenChar` to the screen. But, this is not exactly what we want, because it is hard to print strings this way. + +## Creating a Custom Writer + +As always, rust has amazing features, and one of them is built in formatting on the core library. + +> For those who are unfamiliar with the subject, formatting is turning a variable or a struct into a printable string. +> +> For example, if we have a variable `x` which holds the number `100`, how do we know how to print it? because it is not a string, formatting helps us with this 'type change'. +> +> You might be familiar with the [`printf`](https://en.wikipedia.org/wiki/Printf) function is C (Print Formatted), Rust offers us the [`fmt::Display`](https://doc.rust-lang.org/core/fmt/trait.Display.html) and [`fmt::Debug`](https://doc.rust-lang.org/core/fmt/trait.Debug.html) traits to handle formatting + +But what does it mean for us? It means that if we implement our custom writer (which just needs to print regular ASCII strings), we freely get the ability to print variables in the code, and complex structs, since they can be easily derived by the Debug trait. + +To create our custom writer we just need to implement the [`fmt::Writer`](https://doc.rust-lang.org/core/fmt/trait.Write.html) trait on a custom struct. Our simple writer, will just include place we currently are on the screen, the color the print has, and, and a reference to the screen buffer. + +```rust +#![struct!("crates/drivers/vga-display/src/writer.rs", Writer)] + +#![trait_impl!("crates/drivers/vga-display/src/writer.rs", Default for Writer)] +``` + +Then, we need to handle the following functionalities: + +1. If a character is in ASCII range, write it to the buffer at cursor position, and advance the cursor. + +2. If the `\n` character was entered, don't print anything, but put the cursor at the start of the next line. + +3. If `Backspace` or `Delete` character were entered, move the cursor back one position, and fill that position with the default character. + +4. If we are at the end of the screen, we need to scroll down a line, which means to copy the entire buffer one line to the left[^1]. + +[^1]: Our buffer represents a 2D grid of `ScreenChar` elements, but it is actually just one big 1D buffer. So copying the entire buffer one line up is equivalent to shifting all the characters one line to the left. + +5. Function to clear the screen entirely + +Now that we have all the functionality in mind, we can go right into the implementation! + +```rust +#![impl_method!("crates/drivers/vga-display/src/writer.rs", Writer::write_char, scroll_down, new_line, backspace, clear)] +``` + +> For now, the `change_cursor_position_on_screen` function is not relevant, and it uses I/O instruction to change the cursor position. This will be covered in future chapters. + +With this, we are ready to implement the `fmt::Writer` trait on our struct. Because it only requires us to implement the `write_str` function, which is easy to implement because we have our `write_char` function. + +```rust +#![trait_impl!("crates/drivers/vga-display/src/writer.rs", Write for Writer)] +``` + +The only thing that is missing is to initialize the writer, and write a function that will also print with a custom color, this function is relatively straight forward, and it will just change the color, print the message, and restore the color back to default. + +```rust +#![static!("crates/drivers/vga-display/src/lib.rs", WRITER)] +#![function!("crates/drivers/vga-display/src/lib.rs", vga_print)] +``` + +An example usage, could be an OK message of what we already initialized! + +```rust +#![function!("snippets/src/book/ch03_00/print_example.rs", _start)] +``` + +
+ +## Exercise + +1. The standard library has a `print!` and `println!` macros, we are really close for one, implement it! + +2. Implement the `okprintln!` and `eprintln!` that we used above. + +Answers can be found at [here](https://github.com/sagi21805/LearnixOS/blob/master/crates/drivers/vga-display/src/lib.rs#37) diff --git a/src/ch04-00-memory-management.md b/src/ch05-00-memory-management.md similarity index 100% rename from src/ch04-00-memory-management.md rename to src/ch05-00-memory-management.md diff --git a/src/ch04-01-memory-allocator-types.md b/src/ch05-01-memory-allocator-types.md similarity index 100% rename from src/ch04-01-memory-allocator-types.md rename to src/ch05-01-memory-allocator-types.md diff --git a/src/ch04-02-implementing-a-bitmap.md b/src/ch05-02-implementing-a-bitmap.md similarity index 100% rename from src/ch04-02-implementing-a-bitmap.md rename to src/ch05-02-implementing-a-bitmap.md diff --git a/src/ch04-03-writing-a-physical-page-allocator.md b/src/ch05-03-writing-a-physical-page-allocator.md similarity index 100% rename from src/ch04-03-writing-a-physical-page-allocator.md rename to src/ch05-03-writing-a-physical-page-allocator.md diff --git a/src/ch05-00-interrupts-and-exceptions.md b/src/ch06-00-interrupts-and-exceptions.md similarity index 100% rename from src/ch05-00-interrupts-and-exceptions.md rename to src/ch06-00-interrupts-and-exceptions.md diff --git a/src/ch05-01-utilizing-the-idt.md b/src/ch06-01-utilizing-the-idt.md similarity index 100% rename from src/ch05-01-utilizing-the-idt.md rename to src/ch06-01-utilizing-the-idt.md diff --git a/src/ch05-02-programmable-interrupt-controller.md b/src/ch06-02-programmable-interrupt-controller.md similarity index 100% rename from src/ch05-02-programmable-interrupt-controller.md rename to src/ch06-02-programmable-interrupt-controller.md diff --git a/src/ch05-03-writing-a-keyboard-driver.md b/src/ch06-03-writing-a-keyboard-driver.md similarity index 100% rename from src/ch05-03-writing-a-keyboard-driver.md rename to src/ch06-03-writing-a-keyboard-driver.md diff --git a/src/ch06-00-file-systems-and-disk-drivers.md b/src/ch07-00-file-systems-and-disk-drivers.md similarity index 100% rename from src/ch06-00-file-systems-and-disk-drivers.md rename to src/ch07-00-file-systems-and-disk-drivers.md diff --git a/src/ch06-01-disk-drivers.md b/src/ch07-01-disk-drivers.md similarity index 100% rename from src/ch06-01-disk-drivers.md rename to src/ch07-01-disk-drivers.md diff --git a/src/ch06-02-implementing-a-file-system.md b/src/ch07-02-implementing-a-file-system.md similarity index 100% rename from src/ch06-02-implementing-a-file-system.md rename to src/ch07-02-implementing-a-file-system.md diff --git a/src/ch07-00-processes-and-scheduling.md b/src/ch08-00-processes-and-scheduling.md similarity index 100% rename from src/ch07-00-processes-and-scheduling.md rename to src/ch08-00-processes-and-scheduling.md diff --git a/src/ch07-01-thinking-in-terms-of-processes.md b/src/ch08-01-thinking-in-terms-of-processes.md similarity index 100% rename from src/ch07-01-thinking-in-terms-of-processes.md rename to src/ch08-01-thinking-in-terms-of-processes.md diff --git a/src/ch07-02-implementing-a-process-scheduler.md b/src/ch08-02-implementing-a-process-scheduler.md similarity index 100% rename from src/ch07-02-implementing-a-process-scheduler.md rename to src/ch08-02-implementing-a-process-scheduler.md diff --git a/src/ch08-00-writing-a-shell.md b/src/ch09-00-writing-a-shell.md similarity index 100% rename from src/ch08-00-writing-a-shell.md rename to src/ch09-00-writing-a-shell.md From 2ef9d36017a692fb02f01281ebe23c40c911e9ed Mon Sep 17 00:00:00 2001 From: sagi Date: Fri, 19 Jun 2026 19:19:19 +0300 Subject: [PATCH 3/3] Add base chapter content, which includes a quote and the repo card --- src/ch03-00-synchronization-primitives.md | 8 ++++++++ ...chronization-is-achieved-on-a-multiprocessor-system.md | 8 ++++++++ ...03-02-implementing-basic-synchronization-primitives.md | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/src/ch03-00-synchronization-primitives.md b/src/ch03-00-synchronization-primitives.md index ef0d0ee..0409c57 100644 --- a/src/ch03-00-synchronization-primitives.md +++ b/src/ch03-00-synchronization-primitives.md @@ -1 +1,9 @@ # Synchronization + +_"Shared mutable state is the root of all evil."_ + + --- + + #![repository_card] + +Coming soon! diff --git a/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md b/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md index 5080ac2..60618e7 100644 --- a/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md +++ b/src/ch03-01-how-synchronization-is-achieved-on-a-multiprocessor-system.md @@ -1 +1,9 @@ # How Synchronization is Achieved on a Multiprocessor System + +_"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare_ + +--- + +#![repository_card] + +Coming soon! diff --git a/src/ch03-02-implementing-basic-synchronization-primitives.md b/src/ch03-02-implementing-basic-synchronization-primitives.md index cb0fb2b..02af2ad 100644 --- a/src/ch03-02-implementing-basic-synchronization-primitives.md +++ b/src/ch03-02-implementing-basic-synchronization-primitives.md @@ -1 +1,9 @@ # Implementing Basic Synchronization Primitives + +_"I was motivated by fear of the nondeterminism of concurrency and wanted to prove that it was tamed in my programs." — Edsger W. Dijkstra_ + +--- + +#![repository_card] + +Coming soon!