Katashift

Katashift https://maguire.tech Vaguely directed ramblings Zola en Sat, 18 Oct 2025 00:00:00 +0000 Shove Sat, 18 Oct 2025 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/shove/ https://maguire.tech/posts/shove/ <p>Say hello to <a href="https://github.com/BurntNail/shove">shove</a>, my brand new HTTP file server and content manager!</p> <p>Shove is an S3-backed HTTP file server that handles live-reloads, partial updates and even basic HTTP auth and it has now replaced caddy as the HTTP server that serves this very blog<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>! This blog post will be part explanation, part advertisement and part me being excited about my latest project!</p> <h2 id="current-functionality">Current Functionality</h2> <p>Currently, there’s three main commands inside <code>shove</code> - <code>protect</code>, <code>upload</code> and <code>serve</code>, which I’ll briefly explain in that order. I’ll then go over some of the interesting things I did with each.</p> <h3 id="protect">Protect</h3> <p><code>shove protect</code> deals with the HTTP Basic Auth that <code>shove</code> has - internally it’s modelled as a bunch of users (each of which has a username, a password and a uuid), and a list of realms (each of which has a pattern to protect and a list of uuids that can access it).</p> <p>You run <code>shove protect</code> from wherever you’re uploading from (to get easy access to environment variables), add the users and add the realms. The default behaviour for paths which don’t match any patterns is just to allow anyone and everyone access to those files - for example, my blog currently has no access control in place.</p> <h3 id="protect-1">Protect</h3> <p><code>shove upload</code> uploads the given directory to the given S3 bucket - as far as the user cares, that’s all it does.</p> <p>The complexity comes from the fact that it only uploads new files and deletes old files - to achieve this there’s a file in the root directory that stores hashes of each file. <code>shove protect</code> first reads in all the files, then gets that file, checks the hashes and only uploads the files that have changed.</p> <p>That’s why <code>shove protect</code> can’t upload a ‘current directory’ - it has to have somewhere to put all the files in the bucket where there’s a guarantee the data file (and the auth file) won’t clash.</p> <h3 id="serve">Serve</h3> <p>By far however, the most complicated part is <code>shove serve</code> - this is the general lifecycle:</p> <ol> <li>Read in the data file from S3</li> <li>Find all the files</li> <li>Read them in from S3 and put them into a cache</li> </ol> <p>Then, when we need to serve a file it does this:</p> <ol> <li>Check for access control - if that fails, then send a <a href="https://http.cat/status/401">401</a>.</li> <li>Check if that file is in the cache - if so, send it</li> <li>If it isn’t: grab it from S3, put that in the cache and serve it</li> <li>If we couldn’t find it in S3, serve a <a href="https://http.cat/status/404">404</a> page.</li> </ol> <p>That’s all <span id="animated-918806491"></span> <script> let personToThankForThisSickAnimation="https://codepen.io/zachkrall/pen/MWWGMPx" let delay = 250; let outerspan = document.getElementById("animated-918806491"); outerspan.innerHTML = "relatively" .split("") .map(letter => { return `<span>` + letter + `</span>`; }) .join(""); Array.from(outerspan.children).forEach((span, index) => { setTimeout(() => { span.classList.add("wavy"); }, index * 60 + delay); }); </script> simple<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">2</a></sup>. The complicated part is the live reloading which goes a little something like this:</p> <ol> <li>Check if we need to reload - Tigris has a webhook, or we can just re-fetch the upload data every 60s and see if that file has changed.</li> <li>Work out which files have changed, and add their new versions to our cache.</li> <li>Work out which files have gone missing, and evict them from our cache.</li> <li>Send a notice to all the websockets connected to reload their pages.</li> </ol> <h2 id="fun-tricks-stories">Fun Tricks & Stories</h2> <h3 id="protect-2">Protect</h3> <p>HTTP Basic Auth was certainly interesting to get working - I wanted to have some form of authentication, and got 90% the way through implementing SCRAM-SHA-256<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">3</a></sup>, before I realised that this is supposed to be for server->server authentication and no web browser lets you just put in a password for that.</p> <p>I then redirected that work into my HTTP Basic Auth implementation. I’d originally planned to more closely copy Caddy (which this project is designed to replace for me), but I kinda ended up going ham on the access control. I’d always been annoyed that I had to SSH-in, use <code>caddy hash-password</code>, update the caddyfile and restart caddy to change the auth<sup class="footnote-reference" id="fr-20-1"><a href="#fn-20">4</a></sup> but <code>shove</code> auth just updates an encrypted file in the bucket that gets regularly checked for updates!</p> <p>On that front, the auth benefits from the livereloading as well - <code>shove serve</code> never changes the auth, so we know that if it’s been changed then we need to update. Technically, if multiple people were to start changing the protections and finishing their work in weird orders, there could be a race condition where work would be lost, but I’m not concerned about this at the moment<sup class="footnote-reference" id="fr-30-1"><a href="#fn-30">5</a></sup>.</p> <h3 id="upload">Upload</h3> <p><code>upload</code> was (<strong>comparatively</strong>) simple - we just read in the provided directory, calculate hashes, read in the upload data from S3, and deal with the S3 bits. The only vaguely interesting parts are that I’ve used things like <code>FuturesUnordered</code> to get a load of uploads going at once.</p> <h3 id="serve-1">Serve</h3> <p><code>serve</code> is by far the most complicated, because it handles so damn much. This section will be more an explanation of how stuff works, rather than just tricks and stories. With <a href="https://lib.rs/crates/axum"><code>axum</code></a>/<a href="https://lib.rs/crates/hyper"><code>hyper</code></a>/<a href="https://lib.rs/crates/tower"><code>tower</code></a> projects, I always find a fun way to gauge the complexity is to have a look at the <code>State</code> that the services use, and this is mine:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Clone)] </span><span style="color:#b48ead;">pub struct </span><span>State { </span><span> </span><span style="color:#bf616a;">bucket</span><span>: Box<Bucket>, </span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">tigris_token</span><span>: Option<Arc<</span><span style="color:#b48ead;">str</span><span>>>, </span><span> </span><span style="color:#bf616a;">pages</span><span>: Pages, </span><span> </span><span style="color:#bf616a;">live_reloader</span><span>: LiveReloader, </span><span> </span><span style="color:#bf616a;">auth</span><span>: AuthChecker, </span><span>} </span></code></pre> <p>Yeah, that’s not a lot but still considerable, especially considering three of them are custom structs with lots of other functionality. They also happen to be relatively neat lines to draw for explanations and stories about <code>serve</code> so I’ll give a general explanation, and then dive into those three parts (the live reloader, the auth and the pages).</p> <h4 id="uploaddata"><code>UploadData</code></h4> <p>The <code>UploadData</code> is the struct that holds all of the hashes for all of the files in S3 and was where this whole project started. Whenever the reload is triggered<sup class="footnote-reference" id="fr-40-1"><a href="#fn-40">6</a></sup>, we check whether any of the upload data has changed and if so, change it. Once I got hashing working, this was relatively simple.</p> <p>It doesn’t store any of the actual data, just a map which links a path to a hash, and the name of the root directory we’re serving from.</p> <h4 id="livereloader"><code>LiveReloader</code></h4> <p>To use the livereloader, sites must include a small snippet in their javascript that creates a websocket connection to the server and reloads the page when it receives a <code>reload</code> message. Technically, they can open a websocket connection to <em>any</em> path as the HTTP server only checks whether the requests are for websocket upgrades and not what path they’re looking for. This works here and for now because I know there won’t be any other reasons to open websockets<sup class="footnote-reference" id="fr-50-1"><a href="#fn-50">7</a></sup>.</p> <p>But how does that get handled in the server? So, whenever we receive an upgrade request<sup class="footnote-reference" id="fr-60-1"><a href="#fn-60">8</a></sup>, we send back an upgrade response (kindfully handled by <a href="https://lib.rs/crates/soketto"><code>soketto</code></a>), and then create a <code>Sender</code>/<code>Receiver</code> pair for messages. Then, whenever we need to reload the pages (handled by a channel<sup class="footnote-reference" id="fr-70-1"><a href="#fn-70">9</a></sup>), we just send a <code>reload</code> message to all of the senders!</p> <p>If you’ve had a peek over the source code though, it might look like <code>livereload.rs</code> is doing a fair bit more than that, and it is because we have to deal with closed sockets. That gets dealt with by another thread that pings all of the sockets every 60 seconds to make sure that they’re still alive and stops keeping track of the dead ones. It could be argued that this is excessive, but it’s only a server-side load and I’m happy with it. If you’ve got a better solution, feel free to submit a PR! The code is also slightly complicated because it uses a <a href="https://docs.rs/futures/latest/futures/prelude/stream/struct.FuturesUnordered.html"><code>FuturesUnordered</code></a> to deal with dead senders as their pings finish, rather than needing to wait for all of them.</p> <p>There’s also some code there for closing sockets when the server finishes.</p> <h4 id="authchecker"><code>AuthChecker</code></h4> <p>Shove also has ways of dealing with HTTP basic auth. That’s when you open a webpage and the browser asks for a username and password, rather than the webpage. On the plus side, it’s relatively easy to implement from the server, but also it involves sending the entered password over the wire in plaintext which means that this <strong>must only</strong> be used with HTTPS deployments. In addition, you have to implement things like ratelimiting and it’s a faff.</p> <p>How do I deal with it? I’ll firstly explain how the auth works, and then how it applies itself. So, when we load a page we firstly get a list of every user that has access to that page, and the hash of their password<sup class="footnote-reference" id="fr-80-1"><a href="#fn-80">10</a></sup>. That function deliberately returns an <code>Option<HashMap></code> so that we can encode three states:</p> <ul> <li>A <code>None</code> signifies that there’s no auth needed - in that case we just return the page as normal!</li> <li>A <code>Some</code> with an empty <code>HashMap</code> signifies that nobody has access. This could be useful for pages that are only necessary for a specific job and that job isn’t always filled.</li> <li>A <code>Some</code> with a filled <code>HashMap</code> contains the users we need to auth against.</li> </ul> <p>Once we’ve got the list of users and confirmed that the page needs auth, we check that user’s IP against a ratelimiter. At any point past here, if we fail then we return a response that tells the browser that the user is unauthenticated and needs to provide the correct password. If they pass that check, we then check the <code>Authorization</code> header in their request - if it isn’t there then we fail them and their browser prompts them for a username and password. We then try to find their user in the list we retreieved earlier. If we couldn’t find it then we fail them, but not before running a hash on a fake password - this ensures that they can’t use the round-trip-time to work out which users are valid and invalid. If we could find their password hash, then we hash what they provided and compare the two. If they match, then we serve the page and if not we fail them.</p> <p>But how do we know when to apply the auth? As I mentioned earlier, I am not necessarily a fan of the way Caddy does things and I wanted to have some fun here - there’s a whole system of Users and Realms. There’s a list of Users which are just uuid-name-hash combinations, and of Realms which can match on paths (there’s exact match, regex, etc) and then we can link together users and realms for authentication. For me, this system just makes sense.</p> <h2 id="conclusion">Conclusion</h2> <p>This project was a blast to work on - just large enough to have some fun complicated parts, but not so large that it becomes painful to work on. It’s also legitimately useful for me which is a nice side effect. This was also my first project deployed with <code>fly.io</code>, which has been incredible to work with - speedy support, great documentation, reasonable prices, and the CLI works very very well.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>I honestly don’t understand why more people don’t <a href="https://en.wikipedia.org/wiki/Eating_your_own_dog_food">dogfood</a> their stuff - it’s been so incredible for finding bugs and new features. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-5"> <p>Yes I spent far too long getting wavy text working - even with the help of <a href="https://codepen.io/zachkrall/pen/MWWGMPx">a sick codepen</a>, it was certainly interesting learning more about Hugo & Zola and how to get a version that was repeatable on dynamic pages working. <a href="#fr-5-1">↩</a></p> </li> <li id="fn-10"> <p>From the RFC alone, I might add. Definitely because I wanted the challenge, not because there were’t any rust crates I could find to do it for me ;). And yes, I am aware of how horrific an idea rolling my own cryptography is. <a href="#fr-10-1">↩</a></p> </li> <li id="fn-20"> <p>And I just updated the files through <code>rclone</code> for reference. <a href="#fr-20-1">↩</a></p> </li> <li id="fn-30"> <p>But if you are, feel free to open a PR! My first thought would be some kind of lockfile in the bucket which is the first thing created and the last thing destroyed in the <code>protect</code> flow. <a href="#fr-30-1">↩</a></p> </li> <li id="fn-40"> <p>Either by a webhook which lives on <code>/reload</code> or on a timer every 60s. <a href="#fr-40-1">↩</a></p> </li> <li id="fn-50"> <p>If I ever need some kind of control/management plane, I’ll probably just do it with a raw TCP socket. <a href="#fr-50-1">↩</a></p> </li> <li id="fn-60"> <p>A special kind of request that signals to a server: <em>Heyo! I’d love to open a new websocket connection, can I?</em> <a href="#fr-60-1">↩</a></p> </li> <li id="fn-70"> <p>That channel gets set off by either a 60s timer that checks for a different content hash in S3, or a Tigris webhook that lives on <code>/reload</code>. <a href="#fr-70-1">↩</a></p> </li> <li id="fn-80"> <p>For persistence, this data is stored in an encrypted file in S3. <a href="#fr-80-1">↩</a></p> </li> </ol> </section> Exploring Async Runtimes by Building our Own Wed, 20 Nov 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/exploring-async-runtimes/ https://maguire.tech/posts/exploring-async-runtimes/ <p>For most people, asynchronous rust is as simple as <code>cargo add tokio -F full</code>, and then adding <code>#[tokio::main]</code> to your main function. Then, wham bam alakazam, you’re running rust asynchronously - you’ve got tasks and polling and all the other wonders of async rust.</p> <p>If you’ve ever felt particularly curious, you might’ve chosen to <code>cargo expand</code>, and you’d find something like this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> body = async { </span><span> { </span><span> ::std::io::_print(format_args!("</span><span style="color:#a3be8c;">Hello, world!</span><span style="color:#96b5b4;">\n</span><span>")); </span><span> }; </span><span> }; </span><span> #[</span><span style="color:#bf616a;">allow</span><span>(clippy::expect_used, clippy::diverging_sub_expression)] </span><span> { </span><span> </span><span style="color:#b48ead;">return </span><span>tokio::runtime::Builder::new_multi_thread() </span><span> .</span><span style="color:#96b5b4;">enable_all</span><span>() </span><span> .</span><span style="color:#96b5b4;">build</span><span>() </span><span> .</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">Failed building the Runtime</span><span>") </span><span> .</span><span style="color:#96b5b4;">block_on</span><span>(body); </span><span> } </span><span>} </span></code></pre> <p>Here, we can see that it just starts a runtime with some custom options - if you add options to <code>tokio::main</code>, then it’ll change stuff here, but it doesn’t really tell us anything interesting.</p> <p>If you then try to find the source code, it doesn’t really explain all that much - you can dig into the tokio source code, and you’ll eventually find <a href="https://docs.rs/tokio/latest/src/tokio/runtime/runtime.rs.html#328-355"><code>Runtime::block_on</code></a> which’ll eventually lead to <a href="https://docs.rs/tokio/latest/src/tokio/runtime/scheduler/current_thread/mod.rs.html"><code>CurrentThread</code></a> or one of the other executors. The problem is that they’re not the easiest to read for a beginner<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>?</p> <p>My goal for this essay is to try to explain how an executor works by building the most basic, least performant executor you will have ever seen but we’ll build it step-by-step and hopefully you’ll be able to understand other executors better afterwards.</p> <p>For reading this, I’d like a intermediate understanding of Rust, and a pretty strong understanding of what a future is - if you have the former but not the latter, I can enthusiastically Amos’ <a href="https://fasterthanli.me/articles/understanding-rust-futures-by-going-way-too-deep">article on futures</a> because it really gets to the core of the matter in a way I aim to replicate.</p> <h2 id="what-does-an-async-executor-do-anyways">What does an async executor do, anyways?</h2> <p>I’m going to start this essay much the same as how I started my <a href="https://maguire.tech/posts/allocators/">allocators essay</a> - that is that I’ll try to find a standard library trait (or traits) which describe the required behaviour.</p> <p>The easiest place to start is to look at <a href="https://doc.rust-lang.org/std/future/trait.Future.html"><code>std::future::Future</code></a> - we aren’t necessarily going to be writing any, but I know that we’ll definitely be calling them:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub trait </span><span>Future { </span><span> </span><span style="color:#b48ead;">type </span><span>Output; </span><span> </span><span> </span><span style="color:#65737e;">// Required method </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">poll</span><span>(</span><span style="color:#bf616a;">self</span><span>: Pin<&</span><span style="color:#b48ead;">mut Self</span><span>>, </span><span style="color:#bf616a;">cx</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Context<'_>) -> Poll<</span><span style="color:#b48ead;">Self::</span><span>Output>; </span><span>} </span></code></pre> <p>Here, we can see that when we call the <code>poll</code> method, we need to provide a <a href="https://doc.rust-lang.org/std/task/struct.Context.html"><code>std::task::Context</code></a> with an anonymous lifetime - if we go to check that, we can see that it just provides access to a <a href="https://doc.rust-lang.org/std/task/struct.Waker.html"><code>std::task::Waker</code></a> which it gets from the constructor.</p> <p>A <code>Waker</code> allows a task to wake itself back up and encapsulates a <a href="https://doc.rust-lang.org/std/task/struct.RawWaker.html"><code>std::task::RawWaker</code></a> which holds two pointers:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>RawWaker { </span><span> </span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>(), </span><span> </span><span style="color:#bf616a;">vtable</span><span>: &</span><span style="color:#b48ead;">'static</span><span> RawWakerVTable, </span><span>} </span></code></pre> <p>The <code>data</code> pointer holds pretty much any data that the <code>executor</code> needs to hold - it’s expected to be used as an equivalent of <a href="https://doc.rust-lang.org/std/any/trait.Any.html"><code>std::any::Any</code></a><sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">2</a></sup>.</p> <p><a href="https://doc.rust-lang.org/std/task/struct.RawWakerVTable.html"><code>std::task::RawWakerVTable</code></a> holds a list of function pointers, and if we check the documentation we can see that each function takes in a <code>*const ()</code> which always uses data from the <code>RawWaker</code>, and then each function has the following purpose:</p> <table><thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody> <tr><td><code>clone</code></td><td>This function gets used to create a new <code>RawWaker</code> for use when we clone a <code>Waker</code>.</td></tr> <tr><td><code>wake</code></td><td>This function is called when <code>wake</code> is called on the <code>Waker</code> and should wake up the relevant task. This also should consume the <code>Waker</code> and so needs to deal with any relevant cleanup (eg. freeing memory).</td></tr> <tr><td><code>wake_by_ref</code></td><td>This function is the same as <code>wake</code> but doesn’t consume the <code>Waker</code>, so doesn’t need to deal with cleanup.</td></tr> <tr><td><code>drop</code></td><td>This is called when the <code>Waker</code> is dropped and has to <code>Drop</code> the contents of the <code>WakerData</code>.</td></tr> </tbody></table> <p>So, we now know roughly what our <code>Waker</code> needs, so let’s think about what else an async runtime could need.</p> <h3 id="what-else">What else?</h3> <p>We probably need a list of tasks to poll and a list of results. I think it’ll be easiest to look at these as we go, so we’ll dive straight in.</p> <h2 id="executor-time">Executor Time!</h2> <p>I haven’t built an async executor before, so this essay is more of the <em>come explore with me</em> variety, rather than the <em>guided tour of my thesis</em> variety. We’ll start with a relatively simple single-threaded executor and then see what happens from there.</p> <p>First, we’ll begin a new cargo project with <code>cargo new --bin basic_async_executor</code>, and then open that project with whatever editor you think is best.</p> <p>We can then start by laying out our definition for our executor.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::future::Future; </span><span style="color:#b48ead;">use </span><span>std::sync::mpsc::{Receiver, Sender, channel}; </span><span style="color:#b48ead;">use </span><span>std::pin::Pin; </span><span> </span><span style="color:#b48ead;">pub type </span><span>BoxedFuture<T> = Pin<Box<dyn Future<Output = T>>>; </span><span> </span><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">tasks_to_poll</span><span>: Vec<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>, </span><span> </span><span style="color:#bf616a;">tasks_receiver</span><span>: Receiver<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>, </span><span> </span><span style="color:#bf616a;">tasks_sender</span><span>: Sender<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>, </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>Default </span><span style="color:#b48ead;">for </span><span>Executor { </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">default</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let </span><span>(tx, rx) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> tasks_to_poll: vec![], </span><span> tasks_sender: tx, </span><span> tasks_receiver: rx, </span><span> } </span><span> } </span><span>} </span></code></pre> <p>Here, you can see that we start with some imports - <code>Future</code>, <a href="https://doc.rust-lang.org/std/pin/struct.Pin.html"><code>std::pin::Pin</code></a> and channels. Even though this is all single-threaded we need to use the synchronisation primitives because of the method signatures in <a href="https://doc.rust-lang.org/std/task/struct.RawWakerVTable.html"><code>std::task::RawWakerVTable</code></a> - they all have the data as a non-mutable object, so we have to use interior mutability which is easiest with the synchronisation primitives we have in the standard library.</p> <p>We then also define a type alias to not have to write out the whole future definition every time, which just makes life a bit easier. For our boxed future, we also need to have wrapped it in <code>Pin</code>. This is because of the way that Futures work, and I could try to explain it but I think I’d probably do it poorly and so will defer to <a href="https://fasterthanli.me/articles/pin-and-suffering">Amos’ Article</a>.</p> <p>We then define a struct for our actual executor - a list of tasks that we need to poll, and a receiver for tasks to add themselves to the list of tasks to be polled. They both have the concrete type of <code>u32</code> because that can communicate a fair few distinct values and it makes it easier not to have to deal with multiple types yet. As I mentioned earlier, we’re not trying to make a good one, we’re trying to make one that works.</p> <p>Our basic event loop will go something like this:</p> <ol> <li>Check the receiver for tasks to poll.</li> <li>Loop through the tasks we need to poll.</li> <li>For each task, poll it.</li> <li>For each task, if it is pending, continue on.</li> <li>For each task, if it is ready, then just print out the result.</li> <li>Repeat ad infinitum.</li> </ol> <p>Since that should be relatively simple, we can also write out that loop:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">block_on</span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">future</span><span>: BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">push</span><span>(future); </span><span> </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> task in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">push</span><span>(task); </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">for mut</span><span> task in std::mem::take(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll) { </span><span> </span><span style="color:#b48ead;">let</span><span> cx = todo!(); </span><span> </span><span style="color:#b48ead;">match</span><span> task.</span><span style="color:#96b5b4;">as_mut</span><span>().</span><span style="color:#96b5b4;">poll</span><span>(&</span><span style="color:#b48ead;">mut</span><span> cx) { </span><span> Poll::Ready(res) => println!("</span><span style="color:#a3be8c;">Received Result: </span><span style="color:#d08770;">{res}</span><span>"), </span><span> Poll::Pending => {} </span><span> } </span><span> } </span><span> } </span><span> } </span><span>} </span></code></pre> <p>Here, we’ve now got a skeleton for our basic <code>block_on</code> function, with the only missing bit being our context, which will probably be the more difficult part.</p> <h3 id="context">Context</h3> <p>First, I think the best place to start would be to create the data for each waker that lives inside the <code>*const ()</code>.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Clone)] </span><span style="color:#b48ead;">pub struct </span><span>WakerData; </span></code></pre> <p>As you can see, it’s currently empty because it isn’t yet clear what data we’ll need.</p> <p>Now we need to do the functions for our VTable. We need to pay pretty careful attention here because it’s unsafe to call the functions that make the VTable, where the invariant is that all of these functions need to be thread-safe. Here, we can follow that by making sure to use the synchronisation primitives.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::task::RawWakerVTable; </span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">VTABLE</span><span>: RawWakerVTable = RawWakerVTable::new(clone, wake, wake_by_ref, drop); </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">clone </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) -> RawWaker { </span><span> todo!() </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">wake </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> todo!() </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">wake_by_ref </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> todo!() </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">drop </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> todo!() </span><span>} </span></code></pre> <p>Remember when I mentioned that I was learning at the same time? Well we’ve come across our first problem here - we need to be able to wake up a task using only the data in our <code>WakerData</code>. The first thing that comes to mind is the idea that we could have each <code>WakerData</code> store an index into the list of tasks, and our sender sends that index rather than the entire future. This is because we can’t send a partially completed future from inside the future, where we can’t access it.</p> <p>Following Rust best practices, we’ll use an enum rather than a boolean for by ref versus owned - I’m not normally a shill for mindlessly following best practices but this one makes 100% sense to me. It makes it far easier to see in github diffs and to know what the code does at a glance.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::task::Poll; </span><span> </span><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">tasks_to_poll</span><span>: Vec<Option<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>>, </span><span> </span><span style="color:#bf616a;">tasks_receiver</span><span>: Receiver<</span><span style="color:#b48ead;">usize</span><span>>, </span><span> </span><span style="color:#bf616a;">tasks_sender</span><span>: Sender<</span><span style="color:#b48ead;">usize</span><span>>, </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>Default </span><span style="color:#b48ead;">for </span><span>Executor { </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">default </span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let </span><span>(tasks_sender, tasks_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> tasks_to_poll: Vec::new(), </span><span> tasks_sender, </span><span> tasks_receiver </span><span> } </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">block_on</span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">future</span><span>: BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">push</span><span>(Some(future)); </span><span> </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> index in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span> </span><span style="color:#b48ead;">if</span><span> index >= </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>() { </span><span> panic!("</span><span style="color:#a3be8c;">index out of bounds</span><span>"); </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> cx = todo!(); </span><span> </span><span> </span><span style="color:#b48ead;">if let </span><span>Some(task) = &</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] { </span><span> </span><span style="color:#b48ead;">match</span><span> task.</span><span style="color:#96b5b4;">as_mut</span><span>().</span><span style="color:#96b5b4;">poll</span><span>(&</span><span style="color:#b48ead;">mut</span><span> cx) { </span><span> Poll::Ready(res) => { </span><span> println!("</span><span style="color:#a3be8c;">[executor] got: </span><span style="color:#d08770;">{res}</span><span>"); </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] = None; </span><span> }, </span><span> Poll::Pending => {} </span><span> } </span><span> } </span><span> } </span><span> } </span><span> } </span><span>} </span></code></pre> <p>Here you can see that I’ve moved to a list of Options where each can hold a future, and that future gets removed when the task finishes. We technically don’t need to do this, but it allows the program to free the memory used by those tasks and ensures we don’t poll a task after completion.</p> <p>We’ll also need to add the index to the waker data, as well as a use statement for the <code>RawWaker</code> parts we’ll be doing next.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::task::RawWaker; </span><span> </span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Clone)] </span><span style="color:#b48ead;">pub struct </span><span>WakerData { </span><span> </span><span style="color:#bf616a;">tasks_sender</span><span>: Sender<</span><span style="color:#b48ead;">usize</span><span>>, </span><span> </span><span style="color:#bf616a;">id</span><span>: </span><span style="color:#b48ead;">usize</span><span>, </span><span>} </span></code></pre> <p>Now, we can get back to our VTable.</p> <h4 id="clone">Clone</h4> <p>Clone should be relatively easy - we can just make a new RawWaker relatively easily. However, we do need to be careful on a few fronts to try to avoid undefined behaviour:</p> <ol> <li>We need to make sure that we create a new <code>Sender</code> - this avoids freeing the same <code>Sender</code> multiple times.</li> <li>We need to ensure that the <code>WakerData</code> inside <code>data</code> does not get dropped as <code>clone</code> should only read the original, not drop it.</li> <li>We need to make sure that the new data gets sent to the heap to make sure it will not just be dropped at the end of the function.</li> </ol> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">clone </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) -> RawWaker { </span><span> </span><span style="color:#b48ead;">let</span><span> old_data = std::ptr::read(data as </span><span style="color:#b48ead;">*const</span><span> WakerData); </span><span> </span><span style="color:#b48ead;">let</span><span> new_data = old_data.</span><span style="color:#96b5b4;">clone</span><span>(); </span><span style="color:#65737e;">//cloning clones all the interior contents, creating a new `Sender` </span><span> std::mem::forget(old_data); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> boxed = Box::new(new_data); </span><span> </span><span style="color:#b48ead;">let</span><span> raw_ptr = Box::into_raw(boxed); </span><span> </span><span> RawWaker::new(raw_ptr as </span><span style="color:#b48ead;">*const </span><span>(), &</span><span style="color:#d08770;">VTABLE</span><span>) </span><span>} </span></code></pre> <h4 id="wake-wake-by-ref">Wake & Wake By Ref</h4> <p>These are pretty easy - we just need to read in the pointer and send the task id in the sender. However, we also need to remember to <code>std::mem::forget</code> the data to ensure that it doesn’t drop the contents of the pointer and make it invalid. For our implementation, there isn’t really any difference between <code>wake</code> and <code>wake_by_ref</code>.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">wake</span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> </span><span style="color:#b48ead;">let</span><span> data = std::ptr::read(data as </span><span style="color:#b48ead;">*const</span><span> WakerData); </span><span> data.tasks_sender </span><span> .</span><span style="color:#96b5b4;">send</span><span>(data.id) </span><span> .</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">unable to send task id to executor</span><span>"); </span><span> std::mem::forget(data); </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">wake_by_ref</span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> </span><span style="color:#b48ead;">let</span><span> data = std::ptr::read(data as </span><span style="color:#b48ead;">*const</span><span> WakerData); </span><span> data.tasks_sender </span><span> .</span><span style="color:#96b5b4;">send</span><span>(data.id) </span><span> .</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">unable to send task id to executor</span><span>"); </span><span> std::mem::forget(data); </span><span>} </span></code></pre> <h4 id="drop">Drop</h4> <p>For drop, we need to ensure that both the <code>WakerData</code> is properly dropped <strong>and</strong> the <code>Box</code> is properly deallocated. If we use <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.from_raw"><code>Box::from_raw</code></a>, we can get our pointer back into the <code>Box</code>, and then when the function finishes, the contents will be both dropped and deallocated. We also know that we fit the safety requirements as we know this method will only be called once per pointer!</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">drop </span><span>(</span><span style="color:#bf616a;">data</span><span>: </span><span style="color:#b48ead;">*const </span><span>()) { </span><span> </span><span style="color:#b48ead;">let </span><span>_ = Box::from_raw(data as </span><span style="color:#b48ead;">*mut</span><span> WakerData); </span><span>} </span></code></pre> <h4 id="virtual-function-table">Virtual Function Table</h4> <p>We can finally then create the virtual function table. Here, I’m using a <code>const</code> variable as the constructor for <code>RawWakerVTable</code> is <code>const</code> and it just contains function pointers and so this makes it easiest to use.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub const </span><span style="color:#d08770;">VTABLE</span><span>: RawWakerVTable = RawWakerVTable::new(clone, wake, wake_by_ref, drop); </span></code></pre> <h3 id="creating-the-context">Creating the Context</h3> <p>However, we still need to create the context to poll with. The best way to do this is to work backwards through the documentation to work out what we need.</p> <ol> <li>We need to finish with a <code>Context</code>, which seems to have a constructor method of <a href="https://doc.rust-lang.org/std/task/struct.Context.html#method.from_waker"><code>Context::from_waker</code></a> which takes in a <code>Waker</code>.</li> <li><code>Waker</code> has <a href="https://doc.rust-lang.org/std/task/struct.Waker.html#method.from_raw"><code>Waker::from_raw</code></a> which takes in a <code>RawWaker</code>. This is an unsafe function, where the invariant is that we need to make sure that our <code>RawWaker</code> and our <code>RawWakerVTable</code> are all sound and follow their invariants.</li> <li><code>RawWaker</code> has a constructor which takes in a pointer to some <code>WakerData</code> and our <code>VTABLE</code>.</li> </ol> <p>So, we can then use this code block inside of our <code>block_on</code> loop to create our <code>Context</code>! When you add this, don’t forget to add an <code>use std::task::{Waker, Context}</code> to the top.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> waker_data = WakerData { </span><span> id: index, </span><span> tasks_sender: </span><span style="color:#bf616a;">self</span><span>.tasks_sender.</span><span style="color:#96b5b4;">clone</span><span>(), </span><span>}; </span><span style="color:#b48ead;">let</span><span> boxed_waker_data = Box::new(waker_data); </span><span style="color:#b48ead;">let</span><span> raw_waker_data = Box::into_raw(boxed_waker_data); </span><span> </span><span style="color:#b48ead;">let</span><span> raw_waker = </span><span> RawWaker::new(raw_waker_data as </span><span style="color:#b48ead;">*const</span><span> WakerData as </span><span style="color:#b48ead;">*const </span><span>(), &</span><span style="color:#d08770;">VTABLE</span><span>); </span><span style="color:#b48ead;">let</span><span> waker = </span><span style="color:#b48ead;">unsafe </span><span>{ Waker::from_raw(raw_waker) }; </span><span style="color:#b48ead;">let mut</span><span> cx = Context::from_waker(&waker); </span></code></pre> <p>Similar to the <code>clone</code> method, we need to make sure to put the <code>WakerData</code> onto the heap to make sure it doesn’t get double-freed.</p> <h3 id="creating-a-future">Creating a Future</h3> <p>We can then create a very basic future that immediately returns to test our executor:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">struct </span><span>BasicFuture; </span><span> </span><span style="color:#b48ead;">impl </span><span>Future </span><span style="color:#b48ead;">for </span><span>BasicFuture { </span><span> </span><span style="color:#b48ead;">type </span><span>Output = </span><span style="color:#b48ead;">u32</span><span>; </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">poll</span><span>(</span><span style="color:#bf616a;">self</span><span>: Pin<&</span><span style="color:#b48ead;">mut Self</span><span>>, </span><span style="color:#bf616a;">cx</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Context<'_>) -> Poll<</span><span style="color:#b48ead;">Self::</span><span>Output> { </span><span> Poll::Ready(</span><span style="color:#d08770;">123</span><span>) </span><span> } </span><span>} </span></code></pre> <p>And create a main method to run it:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> fut = BasicFuture; </span><span> </span><span style="color:#b48ead;">let</span><span> exec = Executor::default(); </span><span> exec.</span><span style="color:#96b5b4;">block_on</span><span>(Box::pin(fut)); </span><span>} </span></code></pre> <p>We then get this output: <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>[executor] Received Result: 123 </span></code></pre> </div> </p> <p>And the program hangs forever. This however, we can easily fix! Since we know that once our main future (the one we called <code>block_on</code> with) is finished, all tasks are finished, we can just break the loop when that one finishes!</p> <p>Just put this in our <code>Poll::Ready</code> branch, and make sure to label the loop.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">if</span><span> index == </span><span style="color:#d08770;">0 </span><span>{ </span><span> </span><span style="color:#b48ead;">break 'main</span><span>; </span><span>} </span></code></pre> <p>Here, I’ve used the <code>'main</code> label, but you could use anything - this is to make sure that it breaks the right loop, as <code>break</code> breaks the inner-most loop by default.</p> <h3 id="creating-a-more-complicated-future">Creating a more complicated future</h3> <p>Whilst that future works, we can also create more complicated futures to actually test whether things are actually running concurrently. For a basic timer future, we can use this one.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">struct </span><span>TimerFuture { </span><span> </span><span style="color:#bf616a;">start</span><span>: Option<Instant>, </span><span> </span><span style="color:#bf616a;">time</span><span>: Duration, </span><span> </span><span style="color:#bf616a;">timeout_ms</span><span>: </span><span style="color:#b48ead;">u32 </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>TimerFuture { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">new </span><span>(</span><span style="color:#bf616a;">timeout_ms</span><span>: </span><span style="color:#b48ead;">u32</span><span>) -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> start: None, </span><span> timeout_ms, </span><span> time: Duration::from_millis(timeout_ms as </span><span style="color:#b48ead;">u64</span><span>) </span><span> } </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>Future </span><span style="color:#b48ead;">for </span><span>TimerFuture { </span><span> </span><span style="color:#b48ead;">type </span><span>Output = </span><span style="color:#b48ead;">u32</span><span>; </span><span> </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">poll</span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>: Pin<&</span><span style="color:#b48ead;">mut Self</span><span>>, </span><span style="color:#bf616a;">cx</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Context<'_>) -> Poll<</span><span style="color:#b48ead;">Self::</span><span>Output> { </span><span> </span><span style="color:#b48ead;">match </span><span style="color:#bf616a;">self</span><span>.start { </span><span> None => { </span><span> </span><span style="color:#bf616a;">self</span><span>.start = Some(Instant::now()); </span><span> }, </span><span> Some(x) => </span><span style="color:#b48ead;">if</span><span> x.</span><span style="color:#96b5b4;">elapsed</span><span>() >= </span><span style="color:#bf616a;">self</span><span>.time { </span><span> </span><span style="color:#b48ead;">return </span><span>Poll::Ready(</span><span style="color:#bf616a;">self</span><span>.timeout_ms); </span><span> } </span><span> } </span><span> </span><span> cx.</span><span style="color:#96b5b4;">waker</span><span>().</span><span style="color:#96b5b4;">wake_by_ref</span><span>(); </span><span> Poll::Pending </span><span> } </span><span>} </span></code></pre> <p>We can also then add the <a href="https://lib.rs/crates/futures"><code>futures</code></a> crate and this snippet which creates three of our futures, and then prints how long it takes for them all to run at the same time. If we’re doing actual concurrency, they should hopefully only take roughly as long as the longest timeout.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::time::{Instant, Duration}; </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> Executor::default().</span><span style="color:#96b5b4;">block_on</span><span>(Box::pin(async </span><span style="color:#b48ead;">move </span><span>{ </span><span> println!("</span><span style="color:#a3be8c;">[main] Creating futures</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> one = TimerFuture::new(</span><span style="color:#d08770;">250</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> two = TimerFuture::new(</span><span style="color:#d08770;">500</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> three = TimerFuture::new(</span><span style="color:#d08770;">750</span><span>); </span><span> </span><span> println!("</span><span style="color:#a3be8c;">[main] Created all</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> t = Instant::now(); </span><span> </span><span style="color:#b48ead;">let</span><span> res = futures::join!(one, two, three); </span><span> </span><span style="color:#b48ead;">let</span><span> elapsed = t.</span><span style="color:#96b5b4;">elapsed</span><span>(); </span><span> println!("</span><span style="color:#a3be8c;">[main] Got </span><span style="color:#d08770;">{res:?}</span><span style="color:#a3be8c;"> in </span><span style="color:#d08770;">{elapsed:?}</span><span>"); </span><span> </span><span> </span><span style="color:#d08770;">0 </span><span> })); </span><span>} </span></code></pre> <p>And if we run that, we then get:</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>[main] Creating futures </span><span>[main] Created all </span><span>[main] Got (250, 500, 750) in 750.380037ms </span><span>[executor] Received Result: 0 </span></code></pre> </div> <p>Yes! We now have a basic executor that works!</p> <h2 id="but-is-it-safe">But is it safe?</h2> <p>So, all of the code looks relatively safe but we still have to check. My tool for that is <code>miri</code>, so let’s <code>cargo +nightly miri run</code> and we get the exact same result without any complaints!</p> <h2 id="running-multiple-tasks-at-once">Running multiple tasks at once</h2> <p>Right now, we’re only running our <code>main</code> task, which works well, but also means that we have to use the <code>futures</code> machinery to run multiple futures at once. Right now, all of our work with keeping track of multiple tasks is also all going to waste!</p> <p>So, we need to fix that.</p> <h3 id="what-will-we-need">What will we need?</h3> <ol> <li>We need to move the task execution to a new thread - this way we can add more tasks whilst tasks are running.</li> <li>We need a way to add tasks to the executor.</li> </ol> <h3 id="moving-tasks-to-a-new-thread">Moving tasks to a new thread</h3> <p>For this, we basically just need to move most of the Executor <code>block_on</code> logic to a new thread, so let’s change that method. For now, we’ll also change it to not take in a task and we can call it a <code>start</code> method.</p> <p>For reference, this is our current code:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">block_on</span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">future</span><span>: BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">push</span><span>(Some(future)); </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_sender.</span><span style="color:#96b5b4;">send</span><span>(</span><span style="color:#d08770;">0</span><span>).</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span> </span><span> 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> index in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span> </span><span style="color:#b48ead;">if</span><span> index >= </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>() { </span><span> panic!("</span><span style="color:#a3be8c;">index out of bounds</span><span>"); </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> waker_data = WakerData::new(</span><span style="color:#bf616a;">self</span><span>.tasks_sender.</span><span style="color:#96b5b4;">clone</span><span>(), index); </span><span> </span><span style="color:#b48ead;">let</span><span> boxed_waker_data = Box::new(waker_data); </span><span> </span><span style="color:#b48ead;">let</span><span> raw_waker_data = Box::into_raw(boxed_waker_data); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> raw_waker = </span><span> RawWaker::new(raw_waker_data as </span><span style="color:#b48ead;">*const</span><span> WakerData as </span><span style="color:#b48ead;">*const </span><span>(), &</span><span style="color:#d08770;">VTABLE</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> waker = </span><span style="color:#b48ead;">unsafe </span><span>{ Waker::from_raw(raw_waker) }; </span><span> </span><span style="color:#b48ead;">let mut</span><span> cx = Context::from_waker(&waker); </span><span> </span><span> </span><span style="color:#b48ead;">if let </span><span>Some(task) = &</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] { </span><span> </span><span style="color:#b48ead;">if let </span><span>Poll::Ready(res) = task.</span><span style="color:#96b5b4;">as_mut</span><span>().</span><span style="color:#96b5b4;">poll</span><span>(&</span><span style="color:#b48ead;">mut</span><span> cx) { </span><span> println!("</span><span style="color:#a3be8c;">[executor] Received Result: </span><span style="color:#d08770;">{res}</span><span>"); </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] = None; </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> index == </span><span style="color:#d08770;">0 </span><span>{ </span><span> </span><span style="color:#b48ead;">break 'main</span><span>; </span><span> } </span><span> </span><span> } </span><span> } </span><span> } </span><span> } </span><span>} </span></code></pre> <p>And our first try at moving stuff around gets us this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start</span><span>(&</span><span style="color:#bf616a;">self</span><span>) { </span><span> std::thread::spawn(|| { </span><span> 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> index in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span> </span><span style="color:#b48ead;">if</span><span> index >= </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>() { </span><span> panic!("</span><span style="color:#a3be8c;">index out of bounds</span><span>"); </span><span> } </span><span> println!("</span><span style="color:#a3be8c;">[executor] polling </span><span style="color:#d08770;">{index}</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> waker_data = WakerData::new(</span><span style="color:#bf616a;">self</span><span>.tasks_sender.</span><span style="color:#96b5b4;">clone</span><span>(), index); </span><span> </span><span style="color:#b48ead;">let</span><span> boxed_waker_data = Box::new(waker_data); </span><span> </span><span style="color:#b48ead;">let</span><span> raw_waker_data = Box::into_raw(boxed_waker_data); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> raw_waker = </span><span> RawWaker::new(raw_waker_data as </span><span style="color:#b48ead;">*const</span><span> WakerData as </span><span style="color:#b48ead;">*const </span><span>(), &</span><span style="color:#d08770;">VTABLE</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> waker = </span><span style="color:#b48ead;">unsafe </span><span>{ Waker::from_raw(raw_waker) }; </span><span> </span><span style="color:#b48ead;">let mut</span><span> cx = Context::from_waker(&waker); </span><span> </span><span> </span><span style="color:#b48ead;">if let </span><span>Some(task) = &</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] { </span><span> </span><span style="color:#b48ead;">if let </span><span>Poll::Ready(res) = task.</span><span style="color:#96b5b4;">as_mut</span><span>().</span><span style="color:#96b5b4;">poll</span><span>(&</span><span style="color:#b48ead;">mut</span><span> cx) { </span><span> println!("</span><span style="color:#a3be8c;">[executor] Received Result: </span><span style="color:#d08770;">{res}</span><span>"); </span><span> </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll[index] = None; </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> index == </span><span style="color:#d08770;">0 </span><span>{ </span><span> </span><span style="color:#b48ead;">break 'main</span><span>; </span><span> } </span><span> } </span><span> } </span><span> } </span><span> } </span><span> }); </span><span>} </span></code></pre> <p>However, we immediately encounter problems. Firstly - we get told that our futures cannot be sent across threads:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>error[</span><span style="color:#d08770;">E0277</span><span>]: `(dyn futures::Future<Output = </span><span style="color:#b48ead;">u32</span><span>> + </span><span style="color:#b48ead;">'static</span><span>)` cannot be sent between threads safely </span><span> --> src/main.rs:</span><span style="color:#d08770;">41</span><span>:</span><span style="color:#d08770;">28 </span><span> | </span><span style="color:#d08770;">41 </span><span>| std::thread::spawn(|| { </span><span> | _________------------------_^ </span><span> | | | </span><span> | | required by a bound introduced by this call </span><span style="color:#d08770;">42 </span><span>| | 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span style="color:#d08770;">43 </span><span>| | </span><span style="color:#b48ead;">for</span><span> index in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span style="color:#d08770;">44 </span><span>| | </span><span style="color:#b48ead;">if</span><span> index >= </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>() { </span><span>... | </span><span style="color:#d08770;">71 </span><span>| | } </span><span style="color:#d08770;">72 </span><span>| | }); </span><span> | |</span><span style="color:#d08770;">_________</span><span>^ `(dyn futures::Future<Output = </span><span style="color:#b48ead;">u32</span><span>> + </span><span style="color:#b48ead;">'static</span><span>)` cannot be sent between threads safely </span><span> | </span><span> = help: the </span><span style="color:#b48ead;">trait</span><span> `std::marker::Send` is not implemented </span><span style="color:#b48ead;">for</span><span> `(dyn futures::Future<Output = </span><span style="color:#b48ead;">u32</span><span>> + </span><span style="color:#b48ead;">'static</span><span>)`, which is required by `{closure@src/main.rs:</span><span style="color:#d08770;">41</span><span>:</span><span style="color:#d08770;">28</span><span>: </span><span style="color:#d08770;">41</span><span>:</span><span style="color:#d08770;">30</span><span>}: std::marker::Send` </span></code></pre> <p>This is because we are trying to move all of our tasks from one thread to another, and to do that they need to implement <a href="https://doc.rust-lang.org/std/marker/trait.Send.html"><code>std::marker::Send</code></a>. That one is relatively easy to fix - we just need to change our <code>BoxedFuture</code> to have a <code>Send</code> bound, like this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub type </span><span>BoxedFuture<T> = Pin<Box<dyn Future<Output = T> + Send>>; </span></code></pre> <p>Our next error is then:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>error[</span><span style="color:#d08770;">E0521</span><span>]: borrowed data escapes outside of method </span><span> --> src/main.rs:</span><span style="color:#d08770;">41</span><span>:</span><span style="color:#d08770;">9 </span><span> | </span><span style="color:#d08770;">40 </span><span>| </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start</span><span>(&</span><span style="color:#bf616a;">self</span><span>) { </span><span> | ----- </span><span> | | </span><span> | `</span><span style="color:#bf616a;">self</span><span>` is a reference that is only valid in the method body </span><span> | </span><span style="color:#b48ead;">let's</span><span> call the lifetime of this reference `'</span><span style="color:#d08770;">1</span><span>` </span><span style="color:#d08770;">41 </span><span>| / std::thread::spawn(|| { </span><span style="color:#d08770;">42 </span><span>| | 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span style="color:#d08770;">43 </span><span>| | </span><span style="color:#b48ead;">for</span><span> index in </span><span style="color:#bf616a;">self</span><span>.tasks_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span style="color:#d08770;">44 </span><span>| | </span><span style="color:#b48ead;">if</span><span> index >= </span><span style="color:#bf616a;">self</span><span>.tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>() { </span><span>... | </span><span style="color:#d08770;">71 </span><span>| | } </span><span style="color:#d08770;">72 </span><span>| | }); </span><span> | | ^ </span><span> | | | </span><span> | |</span><span style="color:#d08770;">__________</span><span>`</span><span style="color:#bf616a;">self</span><span>` escapes the method body here </span><span> | argument requires that `'</span><span style="color:#d08770;">1</span><span>` must outlive `</span><span style="color:#b48ead;">'static</span><span>` </span></code></pre> <p>This one is that we borrow things from <code>self</code>, but that the thread must have a <code>'static</code> lifetime.</p> <p>The easiest way to fix this one<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">3</a></sup> is to change the way that our start function works - we can make it into a constructor, rather than just a method. That way, we can let the thread own things like <code>tasks_to_poll</code> and the <code>tasks_receiver</code> and we don’t need to worry about borrowing and other issues.</p> <h3 id="making-start-a-constructor">Making <code>start</code> a constructor</h3> <p>Firstly, we need to decide on what our <code>Executor</code> will hold and not hold.</p> <p>Currently, we have this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">tasks_to_poll</span><span>: Vec<Option<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>>, </span><span> </span><span style="color:#bf616a;">tasks_receiver</span><span>: Receiver<</span><span style="color:#b48ead;">usize</span><span>>, </span><span> </span><span style="color:#bf616a;">tasks_sender</span><span>: Sender<</span><span style="color:#b48ead;">usize</span><span>>, </span><span>} </span></code></pre> <p>We already decided to move <code>tasks_to_poll</code>, <code>tasks_receiver</code> and <code>tasks_sender</code> to the thread, and since we’re now using a thread we’ll hold onto a <a href="https://doc.rust-lang.org/std/thread/struct.JoinHandle.html"><code>std::thread::JoinHandle</code></a>.</p> <p>That gives our new definition as:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::thread::JoinHandle; </span><span> </span><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">thread</span><span>: JoinHandle<()> </span><span>} </span></code></pre> <p>We can then finish off our new constructor:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> thread = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#b48ead;">let mut</span><span> tasks_to_poll: Vec<Option<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>> = vec![]; </span><span> </span><span style="color:#b48ead;">let </span><span>(tasks_sender, tasks_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span> </span><span style="color:#65737e;">//snip: executor loop, with all `self.`s removed </span><span> }); </span><span> </span><span> </span><span style="color:#b48ead;">Self </span><span>{ thread } </span><span> } </span></code></pre> <p>One slight issue now - we can’t actually add any new tasks! For that, we’ll need another <code>Sender</code>/<code>Receiver</code> pair.</p> <h3 id="sending-tasks">Sending tasks</h3> <p>Firstly, we’ll add the <code>Sender</code> to our struct definition:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">thread</span><span>: JoinHandle<()>, </span><span> </span><span style="color:#bf616a;">new_task_sender</span><span>: Sender<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>> </span><span>} </span></code></pre> <p>Then add a function for sending tasks when we get them:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">add_task </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">f</span><span>: impl Future<Output = </span><span style="color:#b48ead;">u32</span><span>> + Send + </span><span style="color:#b48ead;">'static</span><span>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.new_task_sender.</span><span style="color:#96b5b4;">send</span><span>(Box::pin(f)).</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">unable to send new task to executor</span><span>"); </span><span> } </span><span>} </span></code></pre> <p>We can even use some a fun <code>impl Trait</code> to allow users to pass in any future to be used and then we deal with <code>Box</code>ing/<code>Pin</code>ning it. As with <code>BoxedFuture</code>, we need to add the <code>Send</code> bound and here we also need to add the <code>'static</code> bound to make sure that the task can live forever as we’ll be sending it to a different thread with no idea when we’ll be executing it.</p> <p>We then just need to add some more logic to our <code>start</code> function to add any new tasks. This logic also needs to send those tasks to the <code>tasks_sender</code> to ensure they get polled!</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let </span><span>(new_task_sender, new_task_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> thread = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#b48ead;">let mut</span><span> tasks_to_poll: Vec<Option<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>> = vec![]; </span><span> </span><span style="color:#b48ead;">let </span><span>(tasks_sender, tasks_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span> 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> task in new_task_receiver.</span><span style="color:#96b5b4;">try_iter</span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> index = tasks_to_poll.</span><span style="color:#96b5b4;">len</span><span>(); </span><span> tasks_to_poll.</span><span style="color:#96b5b4;">push</span><span>(Some(task)); </span><span> tasks_sender.</span><span style="color:#96b5b4;">send</span><span>(index).</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span> } </span><span> </span><span> </span><span style="color:#65737e;">//snip: polling loop </span><span> } </span><span> }); </span><span> </span><span> </span><span style="color:#b48ead;">Self </span><span>{ thread, new_task_sender } </span><span> } </span><span>} </span></code></pre> <h3 id="time-to-test">Time to test!</h3> <p>If we then add a few simple futures to our <code>main</code> function, let’s see what happens!</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> executor = Executor::start(); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">0</span><span>}); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">1</span><span>}); </span><span>} </span></code></pre> <p>These futures don’t do anything vaguely interesting, but we should see some print statements with their results if this works.</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo run --quiet </span><span># snip: cargo warnings </span></code></pre> </div> <p>Nothing!</p> <h3 id="nothing">Nothing?</h3> <p>First step when things don’t seem to work? Print statements! Let’s add one at each ‘stage’ of our <code>main</code> function to see how far it gets.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> println!("</span><span style="color:#a3be8c;">[main] start</span><span>"); </span><span> </span><span style="color:#b48ead;">let</span><span> executor = Executor::start(); </span><span> println!("</span><span style="color:#a3be8c;">[main] started executor</span><span>"); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">0</span><span>}); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">1</span><span>}); </span><span> println!("</span><span style="color:#a3be8c;">[main] added tasks</span><span>"); </span><span>} </span></code></pre> <p>And then if we run it: <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo run --quiet </span><span># snip: warnings from cargo </span><span>[main] start </span><span>[main] started executor </span><span>[main] added tasks </span></code></pre> </div> </p> <p>So it appears that our main function is all running, but we aren’t getting anything from our executor. I’ve got a hunch, so I’m going to add some print statements to our thread:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let </span><span>(new_task_sender, new_task_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> thread = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> println!("</span><span style="color:#a3be8c;">[executor] thread started</span><span>"); </span><span> </span><span style="color:#b48ead;">let mut</span><span> tasks_to_poll: Vec<Option<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>> = vec![]; </span><span> </span><span style="color:#b48ead;">let </span><span>(tasks_sender, tasks_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> println!("</span><span style="color:#a3be8c;">[executor] channels created</span><span>"); </span><span> </span><span> 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span> println!("</span><span style="color:#a3be8c;">[executor] loop</span><span>"); </span><span> </span><span style="color:#65737e;">//snip: adding tasks from new_tasks_receiver </span><span> println!("</span><span style="color:#a3be8c;">[executor] got new tasks</span><span>"); </span><span> </span><span style="color:#65737e;">//snip: polling </span><span> } </span><span> }); </span><span> </span><span> </span><span style="color:#b48ead;">Self </span><span>{ thread, new_task_sender } </span><span> } </span><span>} </span></code></pre> <p>And then:</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo run --quiet </span><span># snip: warnings from cargo </span><span>[main] start </span><span>[main] started executor </span><span>[main] added tasks </span></code></pre> </div> <p>It looks like our thread is never actually running. If we check the <a href="https://doc.rust-lang.org/std/thread/fn.spawn.html">docs</a> for <code>thread::spawn</code>, it looks like our thread has become <em>detached</em>. This means that our program starts the thread, and runs all of the rest of the logic before our thread can do anything and then exits.</p> <p>That means that we need to <code>join</code> the thread at the end of the program. We unfortunately cannot use <code>Drop</code> as <a href="https://doc.rust-lang.org/std/thread/struct.JoinHandle.html#method.join"><code>JoinHandle::join</code></a> takes <code>self</code> on the thread, but <code>Drop</code> only has <code>&mut self</code>.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">join </span><span>(</span><span style="color:#bf616a;">self</span><span>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.thread.</span><span style="color:#96b5b4;">join</span><span>().</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">error joining thread</span><span>"); </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> println!("</span><span style="color:#a3be8c;">[main] start</span><span>"); </span><span> </span><span style="color:#b48ead;">let</span><span> executor = Executor::start(); </span><span> println!("</span><span style="color:#a3be8c;">[main] started executor</span><span>"); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">0</span><span>}); </span><span> executor.</span><span style="color:#96b5b4;">add_task</span><span>(async {</span><span style="color:#d08770;">1</span><span>}); </span><span> println!("</span><span style="color:#a3be8c;">[main] added tasks</span><span>"); </span><span> executor.</span><span style="color:#96b5b4;">join</span><span>(); </span><span>} </span></code></pre> <p>And then? <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo run --quiet </span><span>[main] start </span><span>[main] started executor </span><span>[main] added tasks </span><span>[executor] thread started </span><span>[executor] channels created </span><span>[executor] loop </span><span>[executor] got new tasks </span><span>[executor] polling 0 </span><span>[executor] Received Result: 0 </span></code></pre> </div> </p> <p>Nearly? It looks like only the first task is running…. Which actually makes sense, because when the first task added finishes, we stop the executor 🤦.</p> <h3 id="stopping-the-executor-at-the-right-time">Stopping the Executor at the right time</h3> <p>My initial instinct is just to remove the check and break, but then the issue becomes that we then can’t stop the executor.</p> <p>The next possibility is changing it out for a check whether all the tasks are done. This would work, but it would also mean that if we got through a single cycle of the loop before any tasks were added, it would finish straight away.</p> <p>I propose we add an atomic variable to <em>allow</em> the executor to stop, and then if that variable is true and all the tasks are finished, we can stop the loop. We can have that variable default to false, and then set it to true when we call <code>Executor::join</code>.</p> <p>Firstly, the struct definition:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::sync::{Arc, atomic::AtomicBool}; </span><span> </span><span style="color:#b48ead;">pub struct </span><span>Executor { </span><span> </span><span style="color:#bf616a;">thread</span><span>: JoinHandle<()>, </span><span> </span><span style="color:#bf616a;">new_task_sender</span><span>: Sender<BoxedFuture<</span><span style="color:#b48ead;">u32</span><span>>>, </span><span> </span><span style="color:#bf616a;">can_stop</span><span>: Arc<AtomicBool> </span><span>} </span></code></pre> <p>Then the <code>join</code> method:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::sync::atomic::Ordering; </span><span> </span><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">join </span><span>(</span><span style="color:#bf616a;">self</span><span>) { </span><span> </span><span style="color:#bf616a;">self</span><span>.can_stop.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">true</span><span>, Ordering::Relaxed); </span><span> </span><span style="color:#bf616a;">self</span><span>.thread.</span><span style="color:#96b5b4;">join</span><span>().</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">error joining thread</span><span>"); </span><span> } </span><span>} </span></code></pre> <p>And finally, a few changes to the constructor:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Executor { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">start </span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let </span><span>(new_task_sender, new_task_receiver) = </span><span style="color:#96b5b4;">channel</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> can_stop = Arc::new(AtomicBool::new(</span><span style="color:#d08770;">false</span><span>)); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> executor_can_stop = can_stop.</span><span style="color:#96b5b4;">clone</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> thread = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#65737e;">//snip: variable construction </span><span> 'main: </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#65737e;">//snip: get tasks </span><span> </span><span style="color:#65737e;">//snip: poll tasks. NB: we removed the check for breaking the loop </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> executor_can_stop.</span><span style="color:#96b5b4;">load</span><span>(Ordering::Relaxed) { </span><span> </span><span style="color:#b48ead;">if</span><span> tasks_to_poll.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">all</span><span>(Option::is_none) { </span><span> </span><span style="color:#b48ead;">break 'main</span><span>; </span><span> } </span><span> } </span><span> } </span><span> }); </span><span> </span><span> </span><span style="color:#b48ead;">Self </span><span>{ thread, new_task_sender, can_stop } </span><span> } </span><span>} </span></code></pre> <h3 id="does-it-work">Does it work?</h3> <p>If we run our same main function from earlier, we get:</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>[main] start </span><span>[main] started executor </span><span>[main] added tasks </span><span>[executor] thread started </span><span>[executor] channels created </span><span>[executor] loop </span><span>[executor] got new tasks </span><span>[executor] polling 0 </span><span>[executor] Received Result: 0 </span><span>[executor] polling 1 </span><span>[executor] Received Result: 1 </span></code></pre> </div> <p>Which looks all correct to me!</p> <p>However, I want to test some more complicated futures. I’ve removed some of the print statements and created the following main function to test lots of futures at once.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> executor = Executor::start(); </span><span> </span><span> </span><span style="color:#b48ead;">let </span><span style="color:#8fa1b3;">create_task </span><span>= |</span><span style="color:#bf616a;">time</span><span>, </span><span style="color:#bf616a;">id</span><span>| async </span><span style="color:#b48ead;">move </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> fut = TimerFuture::new(time); </span><span> println!("</span><span style="color:#a3be8c;">[task </span><span style="color:#d08770;">{id}</span><span style="color:#a3be8c;">] created future</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> start = Instant::now(); </span><span> </span><span style="color:#b48ead;">let</span><span> res = fut.await; </span><span> </span><span style="color:#b48ead;">let</span><span> el = start.</span><span style="color:#96b5b4;">elapsed</span><span>(); </span><span> </span><span> println!("</span><span style="color:#a3be8c;">[task </span><span style="color:#d08770;">{id}</span><span style="color:#a3be8c;">] awaited future, got </span><span style="color:#d08770;">{res:?}</span><span style="color:#a3be8c;"> in </span><span style="color:#d08770;">{el:?}</span><span>"); </span><span> </span><span> res </span><span> }; </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> task_1 = </span><span style="color:#96b5b4;">create_task</span><span>(</span><span style="color:#d08770;">150</span><span>, </span><span style="color:#d08770;">1</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> task_2 = </span><span style="color:#96b5b4;">create_task</span><span>(</span><span style="color:#d08770;">150</span><span>, </span><span style="color:#d08770;">2</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> task_3 = </span><span style="color:#96b5b4;">create_task</span><span>(</span><span style="color:#d08770;">50</span><span>, </span><span style="color:#d08770;">3</span><span>); </span><span> </span><span style="color:#b48ead;">let</span><span> task_4 = </span><span style="color:#96b5b4;">create_task</span><span>(</span><span style="color:#d08770;">200</span><span>, </span><span style="color:#d08770;">4</span><span>); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> start = Instant::now(); </span><span> executor.</span><span style="color:#96b5b4;">run</span><span>(task_1); </span><span> executor.</span><span style="color:#96b5b4;">run</span><span>(task_2); </span><span> executor.</span><span style="color:#96b5b4;">run</span><span>(task_3); </span><span> executor.</span><span style="color:#96b5b4;">run</span><span>(task_4); </span><span> </span><span> println!("</span><span style="color:#a3be8c;">[main] created all tasks, joining executor</span><span>"); </span><span> executor.</span><span style="color:#96b5b4;">join</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> el = start.</span><span style="color:#96b5b4;">elapsed</span><span>(); </span><span> println!("</span><span style="color:#a3be8c;">[main] joined, took </span><span style="color:#d08770;">{el:?}</span><span>"); </span><span>} </span></code></pre> <p>This creates a number of different tasks, some with similar times, some with different times. Ideally, they should all print out and say that they took as long as we would have expected, and they’ll finish in the order 3,1/2,1/2,4. Then, the whole thing should only take about as long as the longest task.</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>[main] created all tasks, joining executor </span><span>[executor] adding new task @ 0 </span><span>[executor] adding new task @ 1 </span><span>[executor] adding new task @ 2 </span><span>[executor] adding new task @ 3 </span><span>[task 1] created future </span><span>[task 2] created future </span><span>[task 3] created future </span><span>[task 4] created future </span><span>[task 3] awaited future, got 50 in 50.005528ms </span><span>[executor] Received 50 from 2 </span><span>[task 1] awaited future, got 150 in 150.008273ms </span><span>[executor] Received 150 from 0 </span><span>[task 2] awaited future, got 150 in 150.023708ms </span><span>[executor] Received 150 from 1 </span><span>[task 4] awaited future, got 200 in 200.004303ms </span><span>[executor] Received 200 from 3 </span><span>[main] joined, took 200.377115ms </span></code></pre> </div> <p>And that’s what we got! Sick!</p> <p>We can even run it through <code>miri</code>, and whilst it might take longer, it should still work? We can also add the <code>--release</code> flag. This won’t make it any faster (as we’re still in miri), but it will remove some checks like integer overflows that can cause issues if you aren’t being careful.</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo +nightly miri run --release </span><span>[main] created all tasks, joining executor </span><span>[executor] adding new task @ 0 </span><span>[executor] adding new task @ 1 </span><span>[executor] adding new task @ 2 </span><span>[executor] adding new task @ 3 </span><span>[task 1] created future </span><span>[task 2] created future </span><span>[task 3] created future </span><span>[task 4] created future </span><span>[task 3] awaited future, got 50 in 70.405ms </span><span>[executor] Received 50 from 2 </span><span>[task 1] awaited future, got 150 in 290.625ms </span><span>[executor] Received 150 from 0 </span><span>[task 2] awaited future, got 150 in 346.535ms </span><span>[executor] Received 150 from 1 </span><span>[task 4] awaited future, got 200 in 391.34ms </span><span>[executor] Received 200 from 3 </span><span>[main] joined, took 763.05ms </span></code></pre> </div> <p>It took longer, but no task executed faster than we would have expected and <em>crucially</em> we don’t have any memory issues!</p> <h2 id="conclusion">Conclusion</h2> <p>Bam! We’ve just finished making our async executor and it actually works. If I can work it out, next time will be running the tasks on multiple threads using a pool of workers.</p> <p>The final code from this is <a href="https://gist.github.com/BurntNail/36a5543738d208305d59cecbd2fb5c9e">here</a>.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>Read <em>noob</em> - the Tokio developers have a ton of features and are working with performance as a goal, rather than being easy to read for someone without any async background experience. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-1"> <p>Or in <code>C</code>, a <code>void*</code>. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-10"> <p>Or at least the way I’ve done it. <a href="#fr-10-1">↩</a></p> </li> </ol> </section> Integer Compression Wed, 30 Oct 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/integercmp/ https://maguire.tech/posts/integercmp/ <p>Recently, I’ve been working on a compression engine (<a href="https://github.com/BurntNail/Souris">Souris</a>), and I’m really pretty proud of some of my compression tricks, so I thought I’d go through a few of them here!</p> <p>We’ll start with one of the fundamental ones - integers. The main point of this compression is to get integers to take as few bytes as possible - if I’m encoding the length of a list, the chances are that it might only be 10 or a 100, which could probably fit in one byte, but normally takes 8<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>.</p> <p>For reference, whenever I mention <code>serialising</code>, I’m talking about turning an in-memory structure into a bunch of bytes that can be written, and I’m using <code>deserialising</code> to refer to turning the bytes back into an in-memory structure.</p> <h2 id="getting-started">Getting started</h2> <p>The way I think is best to do this is to go through two main stages - we first take in a number and turn it into an intermediary form that’s easy to serialise and deserialise as well as being easy to get from/into a normal number type like <code>u32</code> or <code>usize</code>.</p> <p>For now, we’ll only be dealing with unsigned integers, as signed integers make this quite messy.</p> <h2 id="usize-integer"><code>usize</code> -> <code>Integer</code></h2> <p>We’ll first create our intermediary struct <code>Integer</code>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">const </span><span style="color:#d08770;">MAX_BYTES</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#b48ead;">usize</span><span>::</span><span style="color:#d08770;">BITS </span><span>as </span><span style="color:#b48ead;">usize </span><span>/ </span><span style="color:#d08770;">8</span><span>; </span><span> </span><span style="color:#b48ead;">struct </span><span>Integer { </span><span> </span><span style="color:#bf616a;">content</span><span>: [</span><span style="color:#b48ead;">u8</span><span>; MAX_BYTES], </span><span> </span><span style="color:#bf616a;">bytes_used</span><span>: </span><span style="color:#b48ead;">usize </span><span>} </span></code></pre> <p>We can see that we’ve got space to store all the bytes for anything up to a <code>u128</code>, the current largest integer, as well as a variable to store how many bytes we’ve used in this.</p> <p>For now, let’s start by making the conversion from <code>usize</code>, and then we can work out how to generalise it. In Rust, when converting infallibly<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">2</a></sup>, we typically implement <code>From</code>, so let’s add a skeleton for that.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>From<</span><span style="color:#b48ead;">usize</span><span>> </span><span style="color:#b48ead;">for </span><span>Integer { </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">from </span><span>(</span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">usize</span><span>) -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> todo!("</span><span style="color:#a3be8c;">AGRRGGHHH!!</span><span>") </span><span> } </span><span>} </span></code></pre> <p>We need to fill out those bytes, and luckily <code>usize</code> has a method called <a href="https://doc.rust-lang.org/std/primitive.usize.html#method.to_le_bytes"><code>usize::to_le_bytes</code></a>, which converts our <code>usize</code> into an array of little-endian bytes. Little-endian bytes are the standard on modern machines, so it is easier for the CPU to get it in that form, so we can use that.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let mut</span><span> content = [</span><span style="color:#d08770;">0_</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">MAX_BYTES</span><span>]; </span><span style="color:#b48ead;">for </span><span>(i, b) in n.</span><span style="color:#96b5b4;">to_le_bytes</span><span>().</span><span style="color:#96b5b4;">into_iter</span><span>().</span><span style="color:#96b5b4;">enumerate</span><span>() { </span><span> content[i] = b; </span><span>} </span></code></pre> <p>That all works! The next part of our scaffold is working out how many bytes we actually used here. There’s two ways we could do this:</p> <ol> <li>We could use <a href="https://doc.rust-lang.org/std/primitive.usize.html#method.ilog2"><code>usize::ilog2</code></a> and some maths to work out how many bytes we used.</li> <li>We could find the last full byte and work out how many bytes are empty, and use that to work out the number we used.</li> </ol> <p>Having done both, they’re both equally good solutions. However, I’ve encountered some issues with the first<sup class="footnote-reference" id="fr-20-1"><a href="#fn-20">3</a></sup> as the maths isn’t 100% obvious on reading. It’s the eternal dilemma of code - I love the sick one-liners and cool tricks, but I spend far more time reading the code than writing it so I’m going to go with the simpler solution here.</p> <p>To go with the second, we just need to find the index of the last byte that isn’t empty, and then add 1 to get the number of bytes we used.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let mut</span><span> content = [</span><span style="color:#d08770;">0_</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">MAX_BYTES</span><span>]; </span><span style="color:#b48ead;">let mut</span><span> last_non_zero_byte = </span><span style="color:#d08770;">0</span><span>; </span><span style="color:#b48ead;">for </span><span>(i, b) in n.</span><span style="color:#96b5b4;">to_le_bytes</span><span>().</span><span style="color:#96b5b4;">into_iter</span><span>().</span><span style="color:#96b5b4;">enumerate</span><span>() { </span><span> content[i] = b; </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> b != </span><span style="color:#d08770;">0 </span><span>{ </span><span> last_non_zero_byte = i; </span><span> } </span><span>} </span><span> </span><span>Integer { </span><span> content, </span><span> bytes_used: last_non_zero_byte + </span><span style="color:#d08770;">1 </span><span>} </span></code></pre> <p>If we add a quick <code>#[derive(Debug)]</code> to the <code>Integer</code> declaration, and a quick main function, we get:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> int: Integer = </span><span style="color:#d08770;">1000.</span><span style="color:#96b5b4;">into</span><span>(); </span><span> println!("</span><span style="color:#d08770;">{int:?}</span><span>"); </span><span>} </span></code></pre> <p>Which then outputs:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>Integer { content: [</span><span style="color:#d08770;">232</span><span>, </span><span style="color:#d08770;">3</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>], bytes_used: </span><span style="color:#d08770;">2 </span><span>} </span></code></pre> <p>We used two bytes, and those numbers look alright to me so let’s keep on going!</p> <h2 id="integer-vec-u8"><code>Integer</code> -> <code>Vec<u8></code></h2> <p>Theoretically here we just need to serialise the number of bytes used (which we can do in just one byte, as <code>MAX_BYTES</code> is equal to 16 which is less than <a href="https://doc.rust-lang.org/std/primitive.u8.html#associatedconstant.MAX"><code>u8::MAX</code></a>), and then add all of the relevant bytes!</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Integer { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">ser </span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> Vec<</span><span style="color:#b48ead;">u8</span><span>> { </span><span> </span><span style="color:#b48ead;">let mut</span><span> output = vec![]; </span><span> output.</span><span style="color:#96b5b4;">push</span><span>(</span><span style="color:#bf616a;">self</span><span>.bytes_used as </span><span style="color:#b48ead;">u8</span><span>); </span><span> output.</span><span style="color:#96b5b4;">extend</span><span>(&</span><span style="color:#bf616a;">self</span><span>.content[</span><span style="color:#d08770;">0</span><span>..</span><span style="color:#bf616a;">self</span><span>.bytes_used]); </span><span> output </span><span> } </span><span>} </span></code></pre> <p>Theoretically here we could even skip serialising the length! However, that trick only works if you are only dealing with Integers alone in arrays, as otherwise it isn’t clear how much to read. If you do serialise the length, you could convert our next <code>deser</code> method to use something like <a href="https://docs.rs/sourisdb/latest/sourisdb/utilities/cursor/struct.Cursor.html"><code>sourisdb::Cursor</code></a> to only take as many bytes as needed from a longer series of bytes.</p> <p>If we then modify our <code>main</code> method to print out the serialised version<sup class="footnote-reference" id="fr-30-1"><a href="#fn-30">4</a></sup>, we get this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>[</span><span style="color:#d08770;">2</span><span>, </span><span style="color:#d08770;">232</span><span>, </span><span style="color:#d08770;">3</span><span>] </span></code></pre> <p>That makes sense! We get the two from the two bytes used, then the same bytes from earlier. Congrats! We’ve just managed to turn an 8-byte <code>usize</code> into a 3-byte <code>Vec<u8></code>!</p> <h2 id="vec-u8-integer"><code>Vec<u8></code> -> <code>Integer</code></h2> <p>But what about the other way around?</p> <p>This is even easier! We just need to read in the length, then read in those bytes and add them to an <code>Integer</code>.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Integer { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">deser </span><span>(</span><span style="color:#bf616a;">bytes</span><span>: &[</span><span style="color:#b48ead;">u8</span><span>]) -> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> len = bytes[</span><span style="color:#d08770;">0</span><span>] as </span><span style="color:#b48ead;">usize</span><span>; </span><span> </span><span style="color:#b48ead;">let mut</span><span> content = [</span><span style="color:#d08770;">0_</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">MAX_BYTES</span><span>]; </span><span> </span><span style="color:#b48ead;">for </span><span>(i, b) in (&bytes[</span><span style="color:#d08770;">1</span><span>..(len + </span><span style="color:#d08770;">1</span><span>)]).</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">copied</span><span>().</span><span style="color:#96b5b4;">enumerate</span><span>() { </span><span> content[i] = b; </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> content, </span><span> bytes_used: len </span><span> } </span><span> } </span><span>} </span></code></pre> <p>If we go back to our main method and deserialise the serialised bytes, we get this out:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>Integer { content: [</span><span style="color:#d08770;">232</span><span>, </span><span style="color:#d08770;">3</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>], bytes_used: </span><span style="color:#d08770;">2 </span><span>} </span></code></pre> <p>Which is the same as earlier!</p> <h2 id="integer-usize"><code>Integer</code> -> <code>usize</code></h2> <p>Our last step is to get it back into a <code>usize</code>. This is the first step where we could fail - we need to make sure that our destination type has enough space. For example, we wouldn’t be able to store our current test value inside of a <code>u8</code>.</p> <p>Let’s start with a skeleton.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>TryFrom<Integer> </span><span style="color:#b48ead;">for </span><span>usize { </span><span> </span><span style="color:#b48ead;">type </span><span>Error = (); </span><span> </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">try_from </span><span>(</span><span style="color:#bf616a;">n</span><span>: Integer) -> Result<</span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#b48ead;">Self::</span><span>Error> { </span><span> todo!("</span><span style="color:#a3be8c;">AARRGGHH</span><span>"); </span><span> } </span><span>} </span></code></pre> <p>Because we’re only dealing with one error type right now, and this is just a proof-of-concept, I just used <code>()</code> as the error type.</p> <p>So, earlier we used <code>usize::to_le_bytes</code> to get our <code>usize</code> into some bytes, so we should probably use <a href="https://doc.rust-lang.org/std/primitive.usize.html#method.from_le_bytes"><code>usize::from_le_bytes</code></a> to turn some bytes into a <code>usize</code>.</p> <p>As with always, we can just create enough bytes to fill the method signature, then just fill them in!</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">const </span><span style="color:#d08770;">T_BYTES</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#b48ead;">usize</span><span>::</span><span style="color:#d08770;">BITS </span><span>as </span><span style="color:#b48ead;">usize </span><span>/ </span><span style="color:#d08770;">8</span><span>; </span><span style="color:#b48ead;">if</span><span> n.bytes_used > </span><span style="color:#d08770;">T_BYTES </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>Err(()); </span><span>} </span><span> </span><span style="color:#b48ead;">let mut</span><span> output = [</span><span style="color:#d08770;">0_</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">T_BYTES</span><span>]; </span><span style="color:#b48ead;">for </span><span>(i, b) in n.content.</span><span style="color:#96b5b4;">into_iter</span><span>().</span><span style="color:#96b5b4;">take</span><span>(n.bytes_used).</span><span style="color:#96b5b4;">enumerate</span><span>() { </span><span> output[i] = b; </span><span>} </span><span> </span><span>Ok(</span><span style="color:#b48ead;">usize</span><span>::from_le_bytes(output)) </span></code></pre> <p>If we try that in this main method:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> int: Integer = </span><span style="color:#d08770;">1000.</span><span style="color:#96b5b4;">into</span><span>(); </span><span style="color:#b48ead;">let</span><span> sered = int.</span><span style="color:#96b5b4;">ser</span><span>(); </span><span style="color:#b48ead;">let</span><span> deser = Integer::deser(&sered); </span><span style="color:#b48ead;">let</span><span> back_out: </span><span style="color:#b48ead;">usize </span><span>= deser.</span><span style="color:#96b5b4;">try_into</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span>println!("</span><span style="color:#d08770;">{back_out}</span><span>"); </span></code></pre> <p>We get $1000$ out, and since $1000 \equiv 1000$, we should be all good?</p> <h2 id="let-s-check">Let’s check</h2> <p>This is a great opportunity to introduce <a href="https://lib.rs/crates/proptest"><code>proptest</code></a>, which we’ll be using to work out whether our algorithm actually works, or only works on certain inputs.</p> <p>We’re pretty sure it works on 1000, but we haven’t checked any other number. We could write a test to check every number, but that would take far too long<sup class="footnote-reference" id="fr-40-1"><a href="#fn-40">5</a></sup> to be reasonable to be running as frequently as tests should run.</p> <p><code>proptest</code> is designed to check a random set of inputs from a set of all possible inputs, and when it finds an error, it’ll try to ‘reduce’ it down to the simplest input that results in an error. It isn’t guaranteed to find all the edge cases, but it can be incredibly useful for checking lots of cases without having to basically re-write the test harness on every run.</p> <p>We can add <code>proptest</code> to our <code>Cargo.toml</code> following their instructions, and then get started with some tests!</p> <p>Our main four use-cases are the four stages we went through earlier, so we can have these two tests to test all of that:</p> <ol> <li>A test to ensure that it can go from <code>usize</code> -> <code>Integer</code> -> <code>usize</code> correctly.</li> <li>A test to go from <code>usize</code> -> <code>Integer</code> -> <code>Vec<u8></code> -> <code>Integer</code> -> <code>usize</code>.</li> </ol> <p>If these tests fail, we can also add more tests to diagnose exactly where the error comes from.</p> <p>This is what I’d write for that:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">cfg</span><span>(test)] </span><span style="color:#b48ead;">mod </span><span>tests { </span><span> </span><span style="color:#b48ead;">use super</span><span>::*; </span><span> </span><span style="color:#b48ead;">use </span><span>proptest::prelude::*; </span><span> </span><span> proptest!{ </span><span> #[</span><span style="color:#bf616a;">test</span><span>] </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">check_in_memory </span><span>(</span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">usize</span><span>) { </span><span> </span><span style="color:#b48ead;">let</span><span> int = Integer::from(n); </span><span> </span><span style="color:#b48ead;">let</span><span> back = </span><span style="color:#b48ead;">usize</span><span>::try_from(int).</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span> prop_assert_eq!(back, n); </span><span> } </span><span> </span><span> #[</span><span style="color:#bf616a;">test</span><span>] </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">check_serialised </span><span>(</span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">usize</span><span>) { </span><span> </span><span style="color:#b48ead;">let</span><span> int = Integer::from(n); </span><span> </span><span style="color:#b48ead;">let</span><span> sered = int.</span><span style="color:#96b5b4;">ser</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> back_int = Integer::deser(&sered); </span><span> </span><span style="color:#b48ead;">let</span><span> back_n = </span><span style="color:#b48ead;">usize</span><span>::try_from(back_int).</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span> </span><span> prop_assert_eq!(back_n, n); </span><span> } </span><span> } </span><span>} </span></code></pre> <p>If we then <code>cargo test</code> or even <a href="https://nexte.st/"><code>cargo nextest run</code></a>, we see that all of our tests pass! Since we’re checking all of the functionality here<sup class="footnote-reference" id="fr-50-1"><a href="#fn-50">6</a></sup>, I’m confident that we can move on!</p> <h2 id="small-numbers">Small Numbers</h2> <p>There’s one thing that’s annoying me about this right now. If we were to try to expand this to <code>u8</code>, most numbers would have to take two bytes - one for the length (which is always one) and one for the content.</p> <p>However, there’s actually a trick we can modify from the <a href="https://lib.rs/crates/bincode"><code>bincode</code></a> <a href="https://github.com/bincode-org/bincode/blob/trunk/docs/spec.md">specification</a>.</p> <p>We always know that our integers can be no more than 8 bytes long, so our first byte will always be in the range <code>0..=8</code>. There’s 247 numbers we aren’t using there, so we can say that, for the first byte, any number in the range <code>0..=247</code> represents that number, but any number greater than that represents $247 + x$ where $x$ is the length.</p> <p>For convenience, we can also add a constant to the top of our file that holds <code>247</code> - this also makes it clearer to future programmers why the constant is the way it is.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">const </span><span style="color:#d08770;">ONE_BYTE_MAX_SIZE</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#b48ead;">u8</span><span>::</span><span style="color:#d08770;">MAX </span><span>as </span><span style="color:#b48ead;">usize </span><span>- </span><span style="color:#d08770;">MAX_BYTES</span><span>; </span></code></pre> <p>To implement this, we can just add this if statement to the top of our <code>ser</code> method, and change the length serialisation below:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">if </span><span style="color:#bf616a;">self</span><span>.bytes_used <= </span><span style="color:#d08770;">1 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> first_byte = </span><span style="color:#bf616a;">self</span><span>.content[</span><span style="color:#d08770;">0</span><span>]; </span><span> </span><span style="color:#b48ead;">if</span><span> first_byte as </span><span style="color:#b48ead;">usize </span><span><= </span><span style="color:#d08770;">ONE_BYTE_MAX_SIZE </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>vec![first_byte]; </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">let mut</span><span> output = vec![]; </span><span>output.</span><span style="color:#96b5b4;">push</span><span>((</span><span style="color:#bf616a;">self</span><span>.bytes_used + </span><span style="color:#d08770;">ONE_BYTE_MAX_SIZE</span><span>) as </span><span style="color:#b48ead;">u8</span><span>); </span><span>output.</span><span style="color:#96b5b4;">extend</span><span>(&</span><span style="color:#bf616a;">self</span><span>.content[</span><span style="color:#d08770;">0</span><span>..</span><span style="color:#bf616a;">self</span><span>.bytes_used]); </span><span>output </span></code></pre> <p>And change our <code>deser</code> method to do some checks to the <code>len</code> and fix it if it is representing a length:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> first_byte = bytes[</span><span style="color:#d08770;">0</span><span>]; </span><span style="color:#b48ead;">let mut</span><span> content = [</span><span style="color:#d08770;">0_</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">MAX_BYTES</span><span>]; </span><span> </span><span style="color:#b48ead;">if</span><span> first_byte as </span><span style="color:#b48ead;">usize </span><span><= </span><span style="color:#d08770;">ONE_BYTE_MAX_SIZE </span><span>{ </span><span> content[</span><span style="color:#d08770;">0</span><span>] = first_byte; </span><span> </span><span style="color:#b48ead;">let</span><span> bytes_used = </span><span style="color:#b48ead;">usize</span><span>::from(content[</span><span style="color:#d08770;">0</span><span>] > </span><span style="color:#d08770;">0</span><span>); </span><span> </span><span style="color:#b48ead;">return Self </span><span>{ </span><span> content, bytes_used </span><span> }; </span><span>} </span><span> </span><span style="color:#b48ead;">let</span><span> len = first_byte as </span><span style="color:#b48ead;">usize </span><span>- </span><span style="color:#d08770;">ONE_BYTE_MAX_SIZE</span><span>; </span><span> </span><span style="color:#b48ead;">for </span><span>(i, b) in (&bytes[</span><span style="color:#d08770;">1</span><span>..(len + </span><span style="color:#d08770;">1</span><span>)]).</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">copied</span><span>().</span><span style="color:#96b5b4;">enumerate</span><span>() { </span><span> content[i] = b; </span><span>} </span><span> </span><span style="color:#b48ead;">Self </span><span>{ </span><span> content, </span><span> bytes_used: len </span><span>} </span></code></pre> <p>We can then re-run our tests, and it all works!</p> <h2 id="left-to-the-reader">Left to the Reader</h2> <p>Whilst <a href="https://gist.github.com/BurntNail/1c5f33afd4b10b4bc10bb89a387dce94">this code</a> all works, there are several things that would need to be done before I’d use it in a production environment:</p> <h3 id="error-handling">Error Handling</h3> <p>Right now almost every method is infallible, but there are a ton of methods that have failure modes. For example, what if the <code>bytes</code> in <code>deser</code> don’t have enough bytes?</p> <h3 id="utility-traits">Utility Traits</h3> <p>This code is pretty hard to use in a larger codebase right now - ideally you would implement a number of traits like <code>FromStr</code>, <code>Display</code>, <code>Debug</code>, and maybe even some of the <code>std::ops</code> traits.</p> <h3 id="more-types">More types</h3> <p>Currently, this code only supports dealing with <code>usize</code>s<sup class="footnote-reference" id="fr-60-1"><a href="#fn-60">7</a></sup> - what about <code>u16</code>s or <code>u8</code>s or even <code>u128</code>s? Ideally, you’d implement some kind of macro to convert the code - it shouldn’t be too hard, as you just need to change some of the types as the code isn’t really that dependent on it being a <code>usize</code>.</p> <h3 id="negative-numbers">Negative Numbers</h3> <p>Negative numbers are <em>difficult</em>, but possible - if this post gets much interaction I’ll share how I managed to get them working. If you decide to have a go at it, don’t underestimate it - that algorithm took me 4 iterations to actually get something that worked and that I understood. If you want to see a hint of my pain, check the <a href="https://github.com/BurntNail/Souris/blame/8d607c37efec6cee6f75c550512ebd5890013f0a/crates/sourisdb/src/types/integer.rs">blame</a> for that file and weep.</p> <h2 id="my-production-code">My Production code</h2> <p>As of writing, <a href="https://github.com/BurntNail/Souris/blob/8d607c37efec6cee6f75c550512ebd5890013f0a/crates/sourisdb/src/types/integer.rs">this</a> is my latest iteration - it includes all of the above, and has functioned incredibly effectively as a backbone for all of <code>sourisdb</code>’s integer serialisation needs. As you can see, once you implement the extra traits it <strong>balloons</strong> to almost a thousand lines of code (including docs/tests).</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>Assuming a 64-bit computer, and <a href="https://stackoverflow.com/questions/2098149/what-platforms-have-something-other-than-8-bit-char">8-bit bytes</a>. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-10"> <p>In such a way that we cannot fail. <a href="#fr-10-1">↩</a></p> </li> <li id="fn-20"> <p>Skill issue? 100% yes. <a href="#fr-20-1">↩</a></p> </li> <li id="fn-30"> <p>Left as an exercise to the reader. <a href="#fr-30-1">↩</a></p> </li> <li id="fn-40"> <p>I ran a quick test on my laptop, and it took 1962s to reach $3378000000000$. Assuming it stays at a constant rate (which it likely won’t, because as the numbers get bigger, more bytes will have to be copied), it’ll take around 340 years to reach <code>usize::MAX</code> on my laptop. <a href="#fr-40-1">↩</a></p> </li> <li id="fn-50"> <p>And I’ve already written this algorithm and tested it far more thoroughly in my own time. <a href="#fr-50-1">↩</a></p> </li> <li id="fn-60"> <p>And yes <code>u64</code>s on 64-bit platforms and <code>u32</code>s on 32-bit platforms. <a href="#fr-60-1">↩</a></p> </li> </ol> </section> Cameras??? Sat, 24 Feb 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/cameras/ https://maguire.tech/posts/cameras/ <p>Cameras have 4 main settings that affect the look of a photo - but how do they work and what do they mean?</p> <p>Over Christmas, I got a new Camera - an EOS R10 with a pair of lenses, and I’ll be using it to explain what the different settings do. The sensor is really good and it spits out incredibly high quality images. For the purposes of demonstration, I’ll be using my cat on a beanbag as the <em>purr</em>fect subject. Unless otherwise specified, I’ll be using a 50mm lens.</p> <h2 id="what-are-the-main-settings">What are the main settings?</h2> <p>Here, I’ll only be talking about the main mechanical settings - nothing really artistic like profiles, composition or how to get your control ring to control ISO.</p> <p>In my opinion, there are 4 main mechanical settings:</p> <ul> <li>Aperture</li> <li>Shutter Speed</li> <li>ISO</li> <li>Focal Length</li> </ul> <p>However, to understand the effects of each of those changes (in my humble opinion) you need to actually know how a camera takes a photo, even if only at a very basic level.</p> <h3 id="so-how-do-we-go-from-photons-to-pixels">So how do we go from photons to pixels?</h3> <p>Firstly - here, I’ll just be talking about digital cameras - although the main difference is that for film you’d swap ‘sensor’ for ‘film’ and add steps for development. ISO is also called Speed for film and comes factory set in the particular roll of film.</p> <p>Light is emitted from sources in the form of photons - each photon can act like a wave, and here we’ll be focusing on that part<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>. Every wave has a <em>wavelength</em>, which is a measure of the distance between two peaks of that wave. When light hits the rods and cones in your eyes, the colours come from the different wavelengths interacting differently with your eyes - for example we see red at about 700nm, green at about 525nm and blue at about 425nm<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">2</a></sup>. The only principle difference between radio waves, microwaves and visible light is the wavelength - our eyes only see what we call visible light, but some animals see far more than we can.</p> <p>Most light sources emit a variety of wavelengths (white is an equal mix of the whole visible spectrum), and as light bounces around some of those surfaces some of the wavelengths get absorbed. Eventually, only the wavelengths that didn’t get absorbed (or just spread elsewhere) reach our eyes and that influences what colour we see. When I see the green cover for my pad of paper, that tells me that the ink absorbs every wavelength apart from the ones that correspond to green.</p> <p>The same way that light can hit your eyes, it can also hit a camera sensor - they use complicated electronics to detect the different wavelengths of light and their different quantities to produce binary data that can then be displayed on our screens as pixels in photos.</p> <p>However, you never see anyone ever take photos with sensors alone and they always use a lens on top - why? So, it’s partially to protect the sensor which is the most crucial part of the camera - if I scratch the dial to change modes, nobody cares but a broken sensor will ruin a camera body. The other reason is to help control the light and ensure it enters the sensor perpendicularly. Normally, the light entering a sensor might come from all different kinds of angles and the lens just makes them all straight using glass. The lens also controls Aperture, but we’ll get there when we get there.</p> <p>To actually take a photo, a digital camera just resets a few counters and then waits for a specified amount of time for the sensor to accumulate data and then it spits it all out. Simple!</p> <h2 id="the-settings-themselves">The Settings Themselves</h2> <p>Now, after that fun physics detour, time for the settings themselves.</p> <h3 id="shutter-speed">Shutter Speed</h3> <p>The Shutter Speed is probably the simplest to understand - the specified amount of time that the sensor waits is the shutter speed. So, technically it should be the Shutter Time or the Exposure Time or something because it doesn’t measure distance, but I don’t make the rules. Being serious, this might have come from film, where you could qualify the distance as the width of the film, but I’m not a photography etymologist.</p> <p>Shutter Speed is usually referred to as 1/x where x is some nice number, but for longer exposures it’ll say the number of seconds. Obviously, if you have a longer shutter speed then you’ll get more light but there’s a cost! If you set the shutter speed to be too high, then you have to start worrying about whether your camera will move during the exposure. If we take this back to the photons, in a shorter exposure all of the photons hitting the specific spots on the sensor will have taken similar paths, but if you’re taking a photo of usain bolt over a second, then the photons hitting the same spots on the sensor are taking very different paths which leads to absorption of different wavelengths and so different colours. It’s blurry, to put it simply.</p> <p>Here, I’ve got two photos of my cat (ISO 2000, F1.8) - one taken at 1/20 and one at 1/80. <figure> <img class="no-hover" src="/images/camera_workings/lowshutter.JPG"/> <figcaption>1/80</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/highshutter.JPG"/> <figcaption>1/20</figcaption> </figure></p> <p>If we assume that my hand stayed relatively still, we can see that the second photo is far blurrier than the second. However, in the second photo we can actually see the cat which is a plus.</p> <p>Clearly, we need another way to change the light.</p> <h3 id="aperture">Aperture</h3> <p>Inside the lens, there are ‘blades’ which form rotate in to the centre to form a circle. The Aperture measures how close they are together. Counter-intuitively, a larger aperture means a smaller number.</p> <p>Here, I’ve got 3 photos of my lens taken at varying apertures from my phone. <figure> <img class="no-hover" src="/images/camera_workings/small_ap.jpeg"/> <figcaption>Large Aperture</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/med_ap.jpeg"/> <figcaption>Medium Aperture</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/huge_ap.jpeg"/> <figcaption>Small Aperture</figcaption> </figure></p> <p>Common sense dictates that the a bigger hole would let in more light and here common sense is 100% correct. I have another two photos below (1/80, ISO 8000): <figure> <img class="no-hover" src="/images/camera_workings/highap.JPG"/> <figcaption>F4</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/lowap.JPG"/> <figcaption>F1.8</figcaption> </figure></p> <p>There’s also another effect at play here with the aperture which is far more subtle. A wide aperture makes for a far shallower focal plane. I can hear you already - <code>Too much jargon Jack!</code>, <code>What does that mean?</code>, <code>Is that related to my glasses?</code>. The answer to the third is actually yes! The focal plane represents where is in-focus for the image - if you’re in the focal plane then you’ll be lovely and sharp and if not then you’ll be blurred. A varifocal lens for a pair of glasses would just mean that you can have multiple focal planes with one lens<sup class="footnote-reference" id="fr-15-1"><a href="#fn-15">3</a></sup>. A shallower focal plane means that more will be in blur, which is incredible for portraits but maybe less good for taking a big group photo. The farther away you are from the subject, the more this diminishes (eg. feel free to use F1.8 for landscapes), but for close subjects this can be an issue (above, you can see my cat’s back out of focus).</p> <p>To provide an example of why I almost always stick to F1.8 for things like portraits, I’ll put a photo of myself below that a friend took: <figure> <img class="no-hover" src="/images/camera_workings/mug.jpg"/> <figcaption>Background Blur ❤️, Seeing my face 😑</figcaption> </figure></p> <p>Without that wide aperture we couldn’t have got that background blur<sup class="footnote-reference" id="fr-20-1"><a href="#fn-20">4</a></sup>. The extra light also allowed me to put a relatively low ISO (leading to less digital noise, as you’ll learn in a second) and a lower shutter speed (to remove blur from shaky hands).</p> <p>So to conclude, the tradeoff for increasing aperture is more light in exchange for a shallower focal plane.</p> <h3 id="iso">ISO</h3> <p>The nerdier amongst you may have recognised this acronym as being for the International Standards Organisation and quickly dismissed it for being unrelated to just photography - I’m here to tell you that this is the correct acronym expansion<sup class="footnote-reference" id="fr-30-1"><a href="#fn-30">5</a></sup>. Why?? I have not the foggiest clue.</p> <p>The ISO represents the sensitivity of the sensor - the higher the ISO, the brighter the photo! There isn’t anything mechanical about this - it’s just the sensor. See more photos (F4, 1/80): <figure> <img class="no-hover" src="/images/camera_workings/lowiso.JPG"/> <figcaption>Low ISO</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/highiso.JPG"/> <figcaption>High ISO</figcaption> </figure></p> <p>At first glance, this seems to have no real effect but when you look closer at the high ISO photo, it seems <em>off</em>. If you zoom in sufficiently, you’ll also start to see that the pixels average out to a nicer colour, but individually are a mess. We call that digital noise and that is the tradeoff for ISO - more light but more noise at higher ISO.</p> <h3 id="focal-length">Focal Length</h3> <p>The focal length is like the zoom, but instead of cropping, we just choose to focus on different light rays. It is measured in mm for millimetres which approximately measures the distance between the front of the lens and the sensor<sup class="footnote-reference" id="fr-40-1"><a href="#fn-40">6</a></sup>. A longer lens means more zoomed in.</p> <p>Here are two example photos taken on my other lens (F6.3, 1/50, ISO 16000): <figure> <img class="no-hover" src="/images/camera_workings/18mm.JPG"/> <figcaption>18mm</figcaption> </figure> <figure> <img class="no-hover" src="/images/camera_workings/150mm.JPG"/> <figcaption>150mm</figcaption> </figure></p> <p>My other lens is the lens that came with my camera and goes from 15 to 180mm. As you can see, the light isn’t vastly changed by the focal length (except for an exception I’ll get into), but the main thing that is clearly changed is what photographers call Compression. In the 150mm photo, the cat and the beanbag look to be at about the same depth and that makes total sense - I took that photo from 3 or 4 metres away, and the cat is centimetres away from the beanbag. However, I took the other photo only centimetres from the cat’s nose so you can really clearly see the depth in the photo which makes his face pop.</p> <p>Because of that compression, we also change the focal plane, which is how lots of people choose to simulate the same effects you get on a lower aperture. The reason you need to simulate those effects is the same reason that I almost never take my 18-150mm with me unless I’m going outside - you lose light with a cheap lens <em>really</em> easily. To buy both new, my 50mm and 18-150mm lenses are about the same price (if not, the zoom lens being more expensive) but my 50mm lens has a maximum aperture of F1.8 and the 18-150mm has a <strong>floating</strong> maximum aperture of F3.6 to F6.3<sup class="footnote-reference" id="fr-50-1"><a href="#fn-50">7</a></sup>. Whilst that doesn’t seem like a lot, aperture doesn’t increase as a continuous variable - it goes in stops which jump larger gaps as you get higher, which means that F3.6 is closer to F6.3 than to F1.8. This means that even if I can get more people in a photo with an 18mm lens, it’ll be a better quality photo (as I can use a higher aperture and so a lower ISO) with my 50mm lens. Since I normally shoot in manual mode, this also means that as I zoom in, the light available to the sensor decreases which means that I have to adjust two or three settings at once - not ideal, especially if I need to get a photo quickly in a crowded touristy area.</p> <p>My prime lens is far cheaper and has a far higher maximum aperture because it is far easier for manufacturers to make prime lenses with fast (fast glass means a higher maximum aperture and vice versa), and so fast variable lenses are <em>incredibly expensive</em>.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>I freaking love my camera - not only has it allowed me to capture some really precious moments, even in the time just since Christmas, it’s given me the perfect excuse to learn more about optics and physics and to deep dive into a field I knew I wanted to look at but had never had the opportunity to before.</p> <p>Learning about all of the different settings has also allowed me to make interesting artistic choices, where I’d always just used the Automatic mode on cameras I’d borrowed before to make the photos <em>perfect</em>.</p> <p>If you’re looking to get into photography, I can recommend the following youtube channels most:</p> <ul> <li><a href="https://www.youtube.com/@PetaPixel">PetaPixel</a> does a <em>ton</em> of reviews and roundups, and the host is really charismatic.</li> <li><a href="https://www.youtube.com/@simon_dentremont">Simon d’Entremont</a> does a lot of more educational content, and is always incredibly concise and clear.</li> <li><a href="https://www.youtube.com/@froknowsphoto">Jared Polin</a> does more week-to-week news as well as reviews as well as expeditions and things adjacent to photography. He can be a bit modern-youtube-shouty-and-clickbaity<sup class="footnote-reference" id="fr-60-1"><a href="#fn-60">8</a></sup>, but I’d still 100% recommend.</li> </ul> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>This is intro to photography, not intro to Quantum Physics - we’ll leave it here for now, and probably for the foreseeable future. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-5"> <p>This is why some people call purple a <em>fake</em> colour - you get purple when you mix red and blue light, but technically the middle of red and blue is green in the electromagnetic spectrum. As far as I believe, purple has just as much a right as any other colour, because the part that actually matters to people is what we see. ALso - by that logic, white is a fake colour. <a href="#fr-5-1">↩</a></p> </li> <li id="fn-15"> <p>The glass inside the camera shifts using motors to focus at different distances in case you were wondering - if you here some whirring as you focus the image, that’s what it is. <a href="#fr-15-1">↩</a></p> </li> <li id="fn-20"> <p>Except maybe without focal length shenanigans - and I do mean shenanigans (put a pin in this for now though). <a href="#fr-20-1">↩</a></p> </li> <li id="fn-30"> <p>English speaking peoples will be forever cursed with the limits of 3-letter acronyms - 17576 sounds like a lot of combinations until you realise it’s really not (just ask the <a href="https://youtu.be/jfOUVYQnuhw">ICAO 🤣</a>). <a href="#fr-30-1">↩</a></p> </li> <li id="fn-40"> <p>Approximately because of things like <a href="https://en.wikipedia.org/wiki/Crop_factor">Crop Factor</a>, internal zooming lenses and advances in lens technology from when the convention started. <a href="#fr-40-1">↩</a></p> </li> <li id="fn-50"> <p>So here I took both photos at F6.3 to keep the light as similar as possible, whilst keeping the compression effects. <a href="#fr-50-1">↩</a></p> </li> <li id="fn-60"> <p>I also can’t blame him at all for that - that’s the way the platform is going right now, and he’s the one who can afford to take $60k of photography gear on an expedition in Kenya. <a href="#fr-60-1">↩</a></p> </li> </ol> </section> London My Beloved Sat, 27 Jan 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/london-my-beloved/ https://maguire.tech/ramblings/london-my-beloved/ <p>I’ve lived in London for a pretty long while and I’ve done a lot of what there is to be done. Here, I’ll be detailing my opinions on stuff I’ve done. I’ll try to vaguely organise it into the different parts of London, but there’s public transport so feel free to hop, jump and skip around to whatever interests you.</p> <p>If you are visiting, I’d also definitely have a gander over my previous article about <a href="https://maguire.tech/ramblings/tfl-for-dummies/">the public transport here</a> because it’s the best way to get around.</p> <h2 id="west-end">West End</h2> <p>A visit to the West End gets you to a lovely part of Central London. You’ve got most of the London theatres and you can see everything from Six to Hamilton. I’ve been to see a fair few productions, so here are my general opinions on some of the ones that made an impact on me (but you’ll be happy with pretty much anything).</p> <p>If you’re after snacks, I’d recommend grabbing something outside of the theatre because it’s always insanely expensive inside. The same goes for the cinemas if that’s your poison - there’s corner shops everywhere.</p> <h3 id="six">Six</h3> <p>Desc: <em>(historical - stories told from the perspectives of each of Henry VIII’s wives)</em></p> <p><a href="https://www.sixthemusical.com/london">Six</a> is incredible, and I would highly recommend going to watch. When I went, there were incredible vibes all around and you’ll definitely come out smiling.</p> <p>Interestingly enough, this was started at <a href="https://en.wikipedia.org/wiki/Edinburgh_Festival_Fringe">Fringe</a> - a huge performing arts festival in Scotland. Anyone can perform (assuming you can book a venue & accommodation 🤣), so you’ll often see lots of university companies. You also hear about productions starting at Fringe, getting noticed and then getting West End & Broadway Runs (with Six being one of the most common examples of a Fringe production doing incredibly well).</p> <p>It was also fun for me to see some new parts to Henry VIII’s wives which I didn’t know about before. The <a href="https://open.spotify.com/album/5jTDaLFNQovRyjNcWe4cZh?si=agbVb4i0SRajFvxRLhxqUg">songs</a> were also great and stays on my playlist to this day<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>.</p> <h3 id="the-ocean-at-the-end-of-the-lane">The Ocean at the End of the Lane</h3> <p>Desc: <em>(late 20th century - a man going for a funeral and remembering the events of the past. adaption of a Neil Gaiman novel)</em></p> <p>This was an incredible piece of <a href="https://www.nationaltheatre.org.uk/productions/the-ocean-at-the-end-of-the-lane/">theatre</a> <em>annnnnnnddddddd</em> we got another <a href="https://open.spotify.com/album/7p6TvPsvX6Xzab9KNPFaEz?si=myTtMbDOT9WFTMv5-4t8qw"><em>banging</em> soundtrack</a> which has also remained on my playlist. The actors all were incredible at their roles (although the actors might have changed by now - I saw it in 2022) and the technical effects were top-notch. It also really made me feel something at the end, which I suppose lends credence to my acclaim of the story and actors’ skill.</p> <p>I also really enjoyed <a href="https://maguire.tech/ramblings/technical-theatre-experience/#assistant-stage-manager-asm">assistant stage managers</a> - they were really effectively utilised and incredibly skilful.</p> <h3 id="harry-potter-the-cursed-child">Harry Potter & The Cursed Child</h3> <p><em>(modern - a Harry Potter story told a long time after the main series)</em> - I’m aware that the <a href="https://www.vox.com/culture/23622610/jk-rowling-transphobic-statements-timeline-history-controversy">writer</a> isn’t the best, but there are better people to speak about that.</p> <p>The story was really good and the pacing was phenomenal (which is a feat for a play this length - see the next paragraph). They also had some really interesting technical effects which I’ve not a clue how they were pulled off<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">2</a></sup>.</p> <p>Somewhat famously, this is shown as 2 2.5hr plays with one interval in each mini-play and a larger interval between both of them. There’s Wednesday, Friday & Saturday showings that start the first at 14:00 and the second play at 19:00 and then a Sunday showing which is an hour earlier. I think you can also book individual parts, but check <a href="https://uk.harrypottertheplay.com/purchase-tickets/#/">the website</a> for latest details. I’m glad I got to see the West End version, as I’ve heard that the Broadway version has some parts cut out to make it all fit into one showing. I loved the double-feature nature because I got to grab dinner in between and digest<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">3</a></sup> what I’d seen so far with my family.</p> <h3 id="the-play-that-goes-wrong">The Play that Goes Wrong</h3> <p>Desc: <em>(modern - a play that goes terribly terribly wrong whilst the actors attempt not to break character)</em></p> <p>I went to <a href="https://theduchesstheatre.co.uk/tickets/the-play-that-goes-wrong/">this one</a> with a friend in the summer and when I left my face hurt from the constant smiling. What happens is in the name, but I won’t spoil the degree to how far it goes. The actors all had perfect comedic timing and I also had the fortune of being a part of a really good audience.</p> <h2 id="convent-garden-soho">Convent Garden & Soho</h2> <p>Convent Garden isn’t too far away from the West End and is really lovely. There’s a fair few nice cafes and ice cream parlours for your culinary enjoyment, or just to hang around in. There’s also the <a href="https://www.ltmuseum.co.uk/">London Transport Museum</a> which I’ve always enjoyed.</p> <p>Then not too far from there is Soho which is also lovely to walk around<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">4</a></sup>. Whenever I’m around there, I’ll also always pop into <a href="https://forbiddenplanet.com/stores/london-megastore/">Forbidden Planet</a> which is the only place I’ve found in London that has stock of all of the various Dungeons & Dragons books.</p> <h2 id="south-kensington">South Kensington</h2> <p>South Kensington is also a lovely area of London which hosts some of the best museums in London (in my humble opinion). There’s also <a href="https://www.allinlondon.co.uk/regions/south-kensington/directory/embassies-consulates">lots of embassies</a> and some very expensive real estate.</p> <p>All of London’s museums are incredible, and they’re mostly free to enter! I’d highly recommend visiting a few, and as with the theatres these are just the ones that have made a particular impact on me.</p> <p>Also - for accessing the ones in South Kensington, you don’t even need to cross any roads! There’s an underground tunnel<sup class="footnote-reference" id="fr-20-1"><a href="#fn-20">5</a></sup> which leads to an underground entrance to the V&A and comes up onto the street next to the Science and Natural History museums<sup class="footnote-reference" id="fr-25-1"><a href="#fn-25">6</a></sup>.</p> <h3 id="the-science-museum">The Science Museum</h3> <p>I could pretty much describe the entire <a href="https://www.sciencemuseum.org.uk/">museum</a> from top to bottom at this point and navigate mostly blindfolded. This museum is my home away from home<sup class="footnote-reference" id="fr-30-1"><a href="#fn-30">7</a></sup> in Central London and I don’t think I’ll ever be able to truly give thanks to all the incredible curators<sup class="footnote-reference" id="fr-26-1"><a href="#fn-26">8</a></sup>.</p> <p>My usual route is as follows - I walk straight through the entrance (on the opposite side to the shop to avoid temptation) where they’ve got some huge steam engines. Then I’ll enjoy the space section that comes afterwards - I’ve over 150 hours in Kerbal Space Program<sup class="footnote-reference" id="fr-40-1"><a href="#fn-40">9</a></sup>, but I’ve never had anything that’s shown me how much rocket is needed to get such a small payload into orbit<sup class="footnote-reference" id="fr-50-1"><a href="#fn-50">10</a></sup>. I then always enjoy all the cars & trains, as well as the smaller pieces off to the side. There’s then the back section with the rotating exhibition but I can never stay too long there because I just feel super tired after a while. I then usually work my way up the exhibits at the back end - one about genetics and people on the second floor which is good, and then one at the 3rd floor about logistics? There isn’t really a concrete theme other than a large number of sofas and interactive games to play - I used to enjoy it, but now it feels a bit shallow.</p> <p>After that, I’ll work my way back to the front of the museum - there used to be an area here with some climate science stuff (it might still be there, but the map says that it’s currently under refurbishment), but this then takes you to the back of the Flight exhibit. After that, I’ll go to the 360° simulator<sup class="footnote-reference" id="fr-60-1"><a href="#fn-60">11</a></sup> if I feel like it. I’ll now go on a brief tangent to mourn the Launchpad.</p> <p>Both of my parents used to work full-time (leave the house at 7:15 and get home at 11 on a good day full-time) - because of this they hired a nanny to look after us in the day and deal with things like the school run and meals. During the holidays, we spent most of the time at museums<sup class="footnote-reference" id="fr-70-1"><a href="#fn-70">12</a></sup> wandering around which is part of the reason I’ve always known the Science Museum like the back of my hand. Out of the time in the Science Museum, my favourite part was always the Launchpad - my memories are a little foggy because it was so long ago but I remember running around and seeing all of the different parts for hours. However, in 2015 it went down for renovations to then be relaunched as the Wonderlab Gallery in 2016. I can see the effort that’s been put in, but it gets a no from me for the following reasons:</p> <ul> <li>It’s paid! The old gallery was free to enter but this new<sup class="footnote-reference" id="fr-80-1"><a href="#fn-80">13</a></sup> one is paid entry for £9 (in fairness, they do also offer a year pass for £14). I get why it’s now paid but wasn’t before - they run lots of shows and the employees do most definitely need to be paid<sup class="footnote-reference" id="fr-90-1"><a href="#fn-90">14</a></sup> but it still annoys me that something that was once free is no longer such.</li> <li>The shows vary in quality - I adore the one that is around something that could be likened to a bar, but the other two aren’t as good.</li> <li>The activities are much smaller in scale - in the old launchpad there was a pretty big variety in activities sizes, with a few that could be explored for a good while without getting boring. The new ones are much smaller and less interesting. However, there’s one factor that probably trumps all of the others:</li> <li>I’m old now, and my desire to go back to the launchpad probably partially reflects a desire to go back to when I could spend all week in museums without any worries for exams or homework or other things. I’ve also grown up a bit and exhibits that once would’ve amazed me don’t quite do it the same any more because I understand why these things happen.</li> </ul> <p>After mourning the launchpad, I’ll head downstairs to the second level and wander through the <code>Information Age</code>. Usually after that I’m pretty happy and so I’ll head out, possibly stopping by the gift shop even if I almost never actually buy anything.</p> <p>Overall - would highly highly highly recommend.</p> <p> <figure> <img class="no-hover" src="/images/london/enema.jpg"/> <figcaption>This was a contraption I saw in the medical gallery for giving a tobacco enema. I wonder what our equivalent will be for future generations?</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/engine.jpg"/> <figcaption>I'm pretty pleased with the bokeh I managed to get on this turbojet from the flight gallery.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/plane1.jpg"/> <figcaption>This was also from the Flight gallery.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/protein.jpg"/> <figcaption>Downstairs, I saw this protein model which was *insane*. I don't even want to think about how long it must have taken to build.</figcaption> </figure></p> <h3 id="the-natural-history-museum">The Natural History Museum</h3> <p>Fortunately for the reading time, I don’t have quite as long a spiel on the <a href="https://www.nhm.ac.uk/">NHM</a> as I do on the Science Museum.</p> <p>The NHM appears much more prominently in popular media than the Science Museum, so if you’ve heard of one of them then it’s probably this one. The most recent depiction I can recall was probably in the Paddington films. They also advertise a fair bit more<sup class="footnote-reference" id="fr-100-1"><a href="#fn-100">15</a></sup>.</p> <p>They tend to spend much more time on life sciences and history - there’s galleries filled wall to wall with specimens of various animals and insects. The main exhibits I can recall are those on dinosaurs, photography and one about the human body and genetics which is far better than the equivalent in the Science Museum.</p> <p>The NHM also seems to have much more of a focus on research and preservation than the Science Museum but I can’t really speak to that because I’m not a postgrad.</p> <p>They also run an ice rink in December, but I can’t speak for the quality of the ice because I’ve never been to use it.</p> <p> <figure> <img class="no-hover" src="/images/london/nhm_ext.jpg"/> <figcaption>I'm pretty happy with this shot I got of the exterior.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/gem1.jpg"/> <figcaption>This a nightmare to focus onto (and is still blurry 😓), and I found it in the Treasures of the Earth gallery.</figcaption> </figure></p> <h3 id="imperial-college-london">Imperial College London</h3> <p>But Jack, that’s not a museum, that’s a university! Most of the time you’d be 100% correct, but every summer they run a <a href="https://www.imperial.ac.uk/events/160698/great-exhibition-road-festival-2023/">weekend festival</a> as a part of the <a href="https://www.greatexhibitionroadfestival.co.uk/about-us/about-festival/">Exhibition Road festival</a>. There’s all kind of stations you can visit, talks you can watch and it gives a similar vibe to something like New Scientist Live with less people trying to sell you stuff. There’s also food everywhere which has been pretty good when I’ve gone.</p> <h3 id="victoria-albert">Victoria & Albert</h3> <p>This is the only arts & history museum on Exhibition Road - I haven’t spent nearly as much time there as in the other two, but whenever I’ve been in, I’ve wanted to spend more time there<sup class="footnote-reference" id="fr-105-1"><a href="#fn-105">16</a></sup>.</p> <h2 id="knightsbridge">Knightsbridge</h2> <p>Knightsbridge is <em><strong>posh</strong></em>, which means that there’s an abundance of fancy shops if you’re feeling like seeing how the other side lives, and nice coffee shops for when you’re done.</p> <p>I’d especially recommend having a look through Harrods - there’s so much insane stuff in there that it’s fun just to wonder through and play guess-the-price with a friend. The food court (and the rest of the food) all looks incredible but I’ve never actually eaten any due to prices that I haven’t ever seen surpassed. If I recall correctly, on my last visit I saw a £17 croissant and a £32 carbonara. <em>insane</em></p> <p>There’s also a huge variety of different shops inside - parts feel more like a retail outlet<sup class="footnote-reference" id="fr-110-1"><a href="#fn-110">17</a></sup> with a labyrinth containing shops from the different brands and parts are the Harrods own-brand stuff. The latter is always funniest to me - there’s time-share and full purchase opportunities for waterside villas on beaches thousands of miles away from here, and the tech section never fails to impress. The idea of a wealthy Harrods customer has always made me giggle - pop in in the morning to grab a new coat that costs more than the median person in the UK makes in a month, stay for lunch with no natural light paying more than a properly fancy night out, and then finishing the day considering entire houses in the Bahamas.</p> <h2 id="riverside">Riverside</h2> <p>Whilst, technically I think Riverside refers to a <a href="https://riversidequarter.com/">specific place</a>, I’m using this to chat about all the lovely bits on the river.</p> <h3 id="the-tate-modern">The Tate Modern</h3> <p>You couldn’t have possibly thought that we were done with museums? The Tate Modern is a lovely space with lots of modern art all housed inside a beautiful building. Before the renovations at Battersea Power Station, this was also pretty unique in that it was built in an old power station - that’s why the two halves are called Turbine Halls A & B.</p> <p>Whenever I walk past, I always have think about the nearby residents - if you go to the top floor then there’s a balcony looking away from the river directly (about 10-20m away) into some nearby apartments. The residents got annoyed at people looking at them from the gallery and brought a <a href="https://www.theguardian.com/artanddesign/2019/feb/12/tate-modern-wins-privacy-case-brought-by-owners-of-4m-flats">suit</a> against Tate Modern and lost whereafter the judge told them to install blinds.</p> <p>I don’t even think I’ve ever paid to enter one of the galleries there as there’s tons of free stuff to go see. You can also easily get to it from Waterloo station via Wobbly bridge.</p> <figure> <img class="no-hover" src="/images/london/tate.jpg"/> <figcaption>I managed to get this from the opposite side of the river - the contrast between the Tate Modern & The Globe is pretty funny.</figcaption> </figure><h3 id="wobbly-bridge">Wobbly Bridge</h3> <p>Wobbly bridge (technically called The Millennium Bridge) is a pretty sick bridge which is worth walking over - the views are great, there aren’t any cars and there’s some nice photo ops. If you do end up walking over it, I’d make sure to look at the floor for a bit - sounds weird but check all of the gum. There’s some dude who doodles all over it, which I’ve never seen anywhere else.</p> <p> <figure> <img class="no-hover" src="/images/london/gum1.jpg"/> <figcaption>See Exhibit A.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/gum2.jpg"/> <figcaption>And Exhibit B.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/paul.jpg"/> <figcaption>You can also get a photo of a lovely sightline to St Pauls.</figcaption> </figure></p> <h3 id="southbank">Southbank</h3> <p>The Southbank is always really lovely - there’s food, a skate park, the national theatre<sup class="footnote-reference" id="fr-120-1"><a href="#fn-120">18</a></sup> <sup class="footnote-reference" id="fr-125-1"><a href="#fn-125">19</a></sup> and some incredible views. You’ve also got the London Eye and 3 pretty decent attractions nearby - see below.</p> <p>There’s also lots of food vendors (especially around the holidays) and a lovely park with a playground.</p> <h3 id="sea-life-aquarium-shrek-s-adventure-the-london-dungeon">Sea Life Aquarium, Shrek’s Adventure & The London Dungeon</h3> <ul> <li>The first I can wholeheartedly recommend - many fond memories and some of the displays are incredible.</li> <li>The second I remember as being pretty decent, but beware of the expected audience - more suited for 7-8 year olds than 18 year olds.</li> <li>The third I can recommend against - the humour is all pretty juvenile but not in a funny way? The pacing was also pretty slow and from what I’ve heard the original incarnation on Tooley St was better.</li> </ul> <h3 id="tower-bridge-parliament-big-ben">Tower Bridge, Parliament & Big Ben</h3> <p>Tower Bridge is also definitely worth seeing - there can be queues, but the view is really good and the glass part is always fun to walk over. For some fun London knowledge, there was once an American who adored Tower Bridge so much that they wanted to buy it and move it to the US. They <a href="https://www.onthisday.com/articles/why-an-american-bought-london-bridge">accidentally bought London Bridge</a> which is still lovely, but not quite the same - make sure not to make the same mistake and visit the wrong one!</p> <p>Parliament & Big Ben are also worth visiting, but there might be some shenanigans around photo ID and booking far in advance, so if you’re interested I’d make sure to look at <a href="https://www.parliament.uk/visiting/visiting-and-tours/big-ben-tour/">their</a> <a href="https://www.parliament.uk/visiting/visiting-and-tours/">websites</a>. I’ve also got another fun fact - Big Ben is technically only the actual bell, and when most people talk about Big Ben they actually mean Elizabeth Tower.</p> <figure> <img class="no-hover" src="/images/london/tower.jpg"/> <figcaption>It's even better in person when the bridge lifts up - if you're ever trying to time it, check here: <https://www.towerbridge.org.uk/lift-times></figcaption> </figure><h3 id="tower-of-london-hampton-court-other-crown-properties">Tower of London (& Hampton Court & other crown properties)</h3> <p>From what I can remember both are similar, but the Tower of London has more of a focus on crown jewels and execution but Hampton Court has more of a focus on the daily life of the royals. Both are very interesting and lovely to go to on a hot day - in my experience Hampton Court is always a bit more relaxed because it’s more out of the way and also has more outside green spaces to relax in. I’ve spent many a summer’s day here with picnics and friends and I’ve never regretted it.</p> <h3 id="battersea-power-station">Battersea Power Station</h3> <p>Battersea Power Station finished renovations <a href="https://batterseapowerstation.co.uk/news/battersea-power-station-opening-date-announcement/">relatively recently</a>, and it’s pretty nice over there. There’s a ton of restaurants, plenty of options for retail therapy & lots of apartments. There’s also a tube station so it’s relatively easy to get to.</p> <h3 id="oxo-tower">Oxo Tower</h3> <p>Oxo Tower was originally created as a power station, and was then later purchased by the manufacturers of Oxo stock cubes and is now owned by a non-profit. Fun fact - when the OXO letters were put into place on the tower, there was <a href="https://www.artofthestate.co.uk/archive/london_photos-2/oxo_tower/">legal faffery</a> around whether or not it qualified as an advertisement. The owners claimed that it was simply a geometric pattern with a cross between 2 circles and they won. Obviously, advertising legislation has changed a bit since then.</p> <p>I don’t have any experience with the gallery or the restaurant, but I love visiting all of the small independent shops below. My favourite is <a href="https://www.suck.uk.com/">suck uk</a> which sell a variety of unique gifts that rotate around regularly. They’re all designed in-house and all incredible.</p> <h2 id="the-city-of-london">The City of London</h2> <p>I’ve had the good fortune of visiting many cities around the world, but I’ve never been around a central business district that gives the same vibes as the city of London. I personally blame it on the fact that most of my more recent visits have been to the US, where the cities are <em>very</em> planned<sup class="footnote-reference" id="fr-130-1"><a href="#fn-130">20</a></sup>. In contrast, the city of London has such a mish-mash of architectural styles and layouts, such that you never really know where you’re going along the right route unless you look up at landmarks, especially unlike New York or Washington DC which are strictly gridded.</p> <p>The history of the City of London is also especially interesting, but I’d probably just butcher the explanation. If you’re interested, I’d highly recommend <a href="https://www.youtube.com/watch?v=LrObZ_HZZUc">this video</a> by CGP Grey.</p> <p>For the purposes of this section though, I’ll mainly just be talking about Central London without truly making a distinction.</p> <h3 id="the-shard-london-eye">The Shard & London Eye</h3> <p>I only visited the Shard for the first time a few weeks ago, and the view was one of the best I’ve ever seen<sup class="footnote-reference" id="fr-140-1"><a href="#fn-140">21</a></sup> and I’d highly recommend heading up to catch the view. If you aren’t as bothered about being the highest or having food/drinks, I’d head up the London Eye instead. Both are very near to public transport terminals (London Bridge & Waterloo) respectively. You can get tickets there and then for both, but I’d make sure to book in advance because they run on time slots and are also cheaper online.</p> <p> <figure> <img class="no-hover" src="/images/london/shard.jpg"/> <figcaption>Here's one photo of the shard.</figcaption> </figure> <figure> <img class="no-hover" src="/images/london/shard2.jpg"/> <figcaption>And another taken pretty far away from the first - it's such a huge monument that I had fun taking pictures of it on my walk today at huge intervals and seeing how it became harder and harder to fit in frame on my 50mm lens.</figcaption> </figure></p> <h3 id="skygarden">Skygarden</h3> <p>Skygarden is a garden at the top of the Walkie Talkie<sup class="footnote-reference" id="fr-150-1"><a href="#fn-150">22</a></sup>, to which you can obtain free tickets and vibe. There’s a cafe and lovely views - would recommend.</p> <figure> <img class="no-hover" src="/images/london/walkie.jpg"/> <figcaption>You can just about make out the skygarden at the top here, and you can also tell why we call it the Walkie Talkie. You can't see it from this angle, but it's just next to the cheese grater (AKA 122 Leadenhall Street).</figcaption> </figure><h2 id="closing-thoughts">Closing Thoughts</h2> <p>By no means is this a complete list of London Attractions, only the ones that have especially stuck with me - as a resident I don’t go to the attractions all that often. I’ll also probably add more things as I do more things and remember old things. I’ll also try and add some photos, because this article<sup class="footnote-reference" id="fr-155-1"><a href="#fn-155">23</a></sup> is currently <em>severely</em> lacking in that area.</p> <p>This was written during a fairly stressful period regarding my personal future and where I’d be going come September and it’s been really lovely to reminisce. I adore this city and have found no parallel yet - maybe it truly is the best for me, or maybe I have to visit more places and stay there longer and I want to do that. I’ve lived here for all my life, but I want to truly get to know somewhere else - part of the reason that I haven’t applied to any London universities<sup class="footnote-reference" id="fr-160-1"><a href="#fn-160">24</a></sup>.</p> <p>I truly look forward to the day I can write a similar post to this about somewhere else, or the day that I know that I’ve seen enough and am happy to move back home - I guess we’ll all have to wait and see ;)</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>For reference, I think I saw Six single-digit days before the UK went into the first Covid-19 related lockdown which would be March 2020. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-1"> <p>Which is pretty rare for me, because a) most shows don’t go that insane with tech and b) I’ve just a little bit of experience with <a href="https://maguire.tech/ramblings/technical-theatre-experience/">tech</a> <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p><em>heh heh</em> <a href="#fr-2-1">↩</a></p> </li> <li id="fn-10"> <p>Although maybe not with small children or nuns. <a href="#fr-10-1">↩</a></p> </li> <li id="fn-20"> <p>Called, confusingly, the Subway. <a href="#fr-20-1">↩</a></p> </li> <li id="fn-25"> <p>And also to Imperial College - I presume that it’s incredibly useful for the students to avoid rain. <a href="#fr-25-1">↩</a></p> </li> <li id="fn-30"> <p>A STEM student, loving the science museum? Scandalous! <a href="#fr-30-1">↩</a></p> </li> <li id="fn-26"> <p>I can’t lie - if I ever <em>make it make it</em> then the Science Museum is relatively high on the list of places to help out. <a href="#fr-26-1">↩</a></p> </li> <li id="fn-40"> <p>tracked - in reality I probably have > 300-400 hours because it took too long to launch Steam (yes, my rockets were limited in size by frame-rate to not much beyond a Duna return w/ Science lab on board) on my old laptop so I just bookmarked the executable on the desktop. <a href="#fr-40-1">↩</a></p> </li> <li id="fn-50"> <p>I have had the good fortune to visit both the <a href="https://www.nasa.gov/kennedy/">Kennedy</a> and <a href="https://www.nasa.gov/marshall/">Marshall</a> Space Centres, which have plenty of rockets but they go beyond the scale of my comprehension of vehicles into the realm of buildings. They were also all pointed vertically which made it harder. Yes, I also also know that there’s a horizontal Saturn in Marshall (which I’ve seen), but that carried a moon mission. <a href="#fr-50-1">↩</a></p> </li> <li id="fn-60"> <p>I don’t get motion sickness, like at all. I’ve not a clue why because I’ve got decent balance (not sure if they’re linked but I think they’re both inner-ear related - do I have the audience to nerdsnipe a biologist?). This usually doesn’t affect that much, but comes in very handy for VR games and doing non-stop barrel rolls and loops in this simulator. Also - you know how people often don’t realise that they’re unique in minor ways until the situation directly confronts them? I managed to work out that my lack of motion sickness is unusual in this very spot, after I went in with my dad and managed to wipe him out for the rest of the day 😔. <a href="#fr-60-1">↩</a></p> </li> <li id="fn-70"> <p>Especially the Exhibition Road ones, especially especially the Science Museum. <a href="#fr-70-1">↩</a></p> </li> <li id="fn-80"> <p>why am I old now? how is 2016 new? <a href="#fr-80-1">↩</a></p> </li> <li id="fn-90"> <p>Especially in London 😭 <a href="#fr-90-1">↩</a></p> </li> <li id="fn-100"> <p>I’ve never seen actual targeted advertisements for either, but I’ve at least seen more <a href="https://maguire.tech/ramblings/tfl-for-dummies/#tube">tube</a> ads for the NHM. <a href="#fr-100-1">↩</a></p> </li> <li id="fn-105"> <p>even if I just end up going to the Science Museum again next time 😔 <a href="#fr-105-1">↩</a></p> </li> <li id="fn-110"> <p>Except it’s not an outlet, it’s full price. <a href="#fr-110-1">↩</a></p> </li> <li id="fn-120"> <p>Which, for reference, is one of the buildings that I think really <em>gets</em> brutalism right. See an example photo <a href="http://civilianglobal.com/wp-content/uploads/2016/12/National-Theatre-from-the-northeast-Photo-by-Philip-Vile.jpg">here</a>. <a href="#fr-120-1">↩</a></p> </li> <li id="fn-125"> <p>They do a variety of performances here, rotating every so often. They tend to be more <em>theatre</em>-y than most of the West End, which is why you might hear people talking about going to the National Theatre but you’d rarely hear people talking about going to, say the Criterion (no slight against them, just the first to pop into my head). <a href="#fr-125-1">↩</a></p> </li> <li id="fn-130"> <p>Yes, I know there are actual <code>Planned Cities</code>. However, London is <em>insanely</em> old. For example, by the time New Amsterdam (the old name for New York) had around 2000 people, London had been around for about 1600 years (counting Roman <code>Londinium</code> as the start). <a href="#fr-130-1">↩</a></p> </li> <li id="fn-140"> <p>even if we had really bad weather 😢 <a href="#fr-140-1">↩</a></p> </li> <li id="fn-150"> <p>AKA <a href="https://youtu.be/hlRNO8xcrgU?si=dh9Zx8d1_5JvPg06">Walkie</a> <a href="https://youtu.be/c7VEu4ZHSUU?si=KuBGVl4dc9sLZxd3">Scorchy</a> AKA 20 Fenchurch Street. <a href="#fr-150-1">↩</a></p> </li> <li id="fn-155"> <p>And blog as a whole - call it my new years resolution to improve on that :) <a href="#fr-155-1">↩</a></p> </li> <li id="fn-160"> <p>That, and London looks like an <em>awful</em> place to be a student in a myriad of ways (spread out accommodation, accommodation far away from campuses, small campuses, and the fact that London is more designed for people earning an insane amount of money which doesn’t include many students.) <a href="#fr-160-1">↩</a></p> </li> </ol> </section> Building 2 'Simple' Allocators Sat, 13 Jan 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/allocators/ https://maguire.tech/posts/allocators/ <p>I’ve always wondered how allocators work, so over the last week I’ve been working on this.</p> <p>NB: This is a pretty true-to-life recreation of my journey, so there are twists, turns and unexpected last-minute betrayals but I hope that showing my process of fixing stuff helps more than if I’d just cleaned up the whole article to make it look like I’d got everything perfect on the first try.</p> <h2 id="where-do-i-even-start">Where do I even start?</h2> <p>All <code>Rust</code> allocators must start with the trait <a href="https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html"><code>std::alloc::GlobalAlloc</code></a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub unsafe trait </span><span>GlobalAlloc { </span><span> </span><span style="color:#65737e;">// Required methods </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8</span><span>; </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">dealloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">ptr</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout); </span><span>} </span></code></pre> <p>There are two methods that must be provided - one to allocate memory and one to deallocate it. Unlike some other languages, the Rust allocator takes in an alignment and a length for the allocation <em>and</em> the deallocation. Fun fact - it’s also UB to use a memory allocation with one alignment for a purpose with another alignment (like you might if you were, say, trying to <a href="https://maguire.tech/posts/binary-serialisation/#ub">reinterpret a vector’s contents as a non byte aligned struct</a>.</p> <p>Then, once you’ve got an allocator, there are unstable methods to use it piecemeal (like <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.new_in"><code>Box::new_in</code></a> for example), but these aren’t stable so I won’t really be looking at them here. Instead, we’ll be trying to build an allocator that gets used globally for our whole program. To do that, you have to do something like this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>Allocator {</span><span style="color:#65737e;">/* contents */ </span><span>}; </span><span> </span><span>#[</span><span style="color:#bf616a;">global_allocator</span><span>] </span><span style="color:#b48ead;">static </span><span style="color:#d08770;">ALLOCATOR</span><span>: Allocator = Allocator {</span><span style="color:#65737e;">/* contents */</span><span>}; </span></code></pre> <p>Interestingly enough, <code>GlobalAlloc</code> is one of a few unsafe traits, which means that there are extra invariants you must follow. For <code>GlobalAlloc</code>, you must ensure the following:</p> <ul> <li>Allocation and Deallocation cannot panic.</li> <li>You must completely follow the <code>Layout</code>s.</li> <li>You must not rely on allocations not being optimised away.</li> </ul> <h2 id="something-simple-to-start">Something simple to start?</h2> <p>How about we start with just making an <em>‘allocator’</em> that just serves 4 byte alignment and size sections of memory. Why - so I can test it easily and make sure it actually works using <a href="https://github.com/rust-lang/miri"><code>miri</code></a>. Then, I can slowly expand the functionality and improve it whilst keeping it correct.</p> <p>My first best guess for an allocator would just be to pretend to be the heap, and use the stack 😉. I’ve taken some inspiration on that front from the example arena allocator from the <code>GlobalAlloc</code> docs.</p> <p>We’ll start here:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::sync::atomic::AtomicBool; </span><span style="color:#b48ead;">use </span><span>std::cell::UnsafeCell; </span><span style="color:#b48ead;">use </span><span>std::alloc::{GlobalAlloc, Layout}; </span><span> </span><span style="color:#b48ead;">pub struct </span><span>FourAllocator { </span><span> </span><span style="color:#bf616a;">is_allocating</span><span>: AtomicBool, </span><span> </span><span style="color:#bf616a;">tracker</span><span>: UnsafeCell<</span><span style="color:#b48ead;">usize</span><span>>, </span><span> </span><span style="color:#bf616a;">spaces</span><span>: UnsafeCell<[</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#bf616a;">usize</span><span>::</span><span style="color:#bf616a;">BITS as usize </span><span>* </span><span style="color:#d08770;">4</span><span>]> </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe impl </span><span>Sync </span><span style="color:#b48ead;">for </span><span>FourAllocator {} </span><span> </span><span style="color:#b48ead;">unsafe impl </span><span>GlobalAlloc </span><span style="color:#b48ead;">for </span><span>FourAllocator { </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> todo!() </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">dealloc </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">ptr</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) { </span><span> todo!() </span><span> } </span><span>} </span><span> </span><span>#[</span><span style="color:#bf616a;">global_allocator</span><span>] </span><span style="color:#b48ead;">static </span><span style="color:#d08770;">ALLOCATOR</span><span>: FourAllocator = FourAllocator { </span><span> tracker: UnsafeCell::new(</span><span style="color:#d08770;">0</span><span>), </span><span> is_allocating: AtomicBool::new(</span><span style="color:#d08770;">false</span><span>), </span><span> spaces: UnsafeCell::new([</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#b48ead;">usize</span><span>::</span><span style="color:#d08770;">BITS </span><span>as </span><span style="color:#b48ead;">usize </span><span>* </span><span style="color:#d08770;">4</span><span>]) </span><span>}; </span><span> </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> test_ptr = </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">alloc</span><span>(Layout::new::<</span><span style="color:#b48ead;">i32</span><span>>()) }; </span><span>} </span></code></pre> <p>We first define our allocator struct, consisting of an atomic boolean to act as a lock (I’ll come back to this), then a <code>tracker</code> (I’ll also come back to this later), and then an array of spaces held inside an <a href="https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html"><code>UnsafeCell</code></a>. <code>UnsafeCell</code> grants us the magical power of interior mutability - the ability to mutate something without needing <code>&mut</code>. You’ve probably used this before, but it would’ve been shielded by something like <code>Mutex</code>. Here, we’re using it by itself to handle our <em>‘allocations’</em>. The difference here is that you just get a <code>*mut T</code>, and you are responsible for ensuring that it doesn’t get concurrently read or wrote to in different threads. That’s why we’ve got a lock.</p> <p>Then <code>Sync</code> is the marker trait for allowing things to be shared between threads, and it is <code>unsafe</code> to implement because it can break lots of external guarantees. We’re safe here though because we ensure that the same spaces aren’t accessed concurrently and mutably in different threads. <a href="https://doc.rust-lang.org/nomicon/send-and-sync.html">More info on <code>Send</code> and <code>Sync</code> can be found here</a>.</p> <p>We then create our allocator using <code>#[global_allocator]</code>, and create an <code>i32</code> on the Heap using <code>Box</code>. Anyone want to guess what happens if we run the program? We stack overflow as the allocator repeatedly calls itself when it allocates a string to print for the <code>todo!</code>. This would seem bad, but it confirms that we are indeed using the custom allocator. This limitation of not being able to allocate when allocating can make things difficult later, but I’ve got some plans cooking to help with that.</p> <h3 id="maybe-not-so-simple">Maybe not so simple?</h3> <p>The first step should be actually checking for our 4 size and alignment, which can be done with a quick <code>if</code>, returning a null pointer if not - which is the expected behaviour for out-of-memory and other errors in the <a href="https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html#tymethod.alloc">docs</a>. If we keep our <code>todo!()</code> underneath for now, but take it out from <code>dealloc</code>, then we can also test this!</p> <p>If we try to allocate something with the correct size and alignment, we should see a stack overflow (as it reaches the <code>todo!()</code>), and if not then nothing should happen:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">//Allocator: </span><span style="color:#b48ead;">unsafe impl </span><span>GlobalAlloc </span><span style="color:#b48ead;">for </span><span>FourAllocator { </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> layout.</span><span style="color:#96b5b4;">size</span><span>() != </span><span style="color:#d08770;">4 </span><span>|| layout.</span><span style="color:#96b5b4;">align</span><span>() != </span><span style="color:#d08770;">4 </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span> } </span><span> </span><span> todo!() </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">dealloc </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">ptr</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) {} </span><span>} </span><span> </span><span style="color:#65737e;">//Main: </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() { </span><span> </span><span style="color:#65737e;">//run 1: </span><span> </span><span style="color:#b48ead;">let</span><span> layout = Layout::new::<</span><span style="color:#b48ead;">u32</span><span>>(); </span><span style="color:#65737e;">//u32 has layout 4, 4 </span><span> </span><span style="color:#65737e;">//run 2: </span><span> </span><span style="color:#b48ead;">let</span><span> layout = Layout::new::<</span><span style="color:#b48ead;">u8</span><span>>(); </span><span style="color:#65737e;">//u8 has layout 1, 1 </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#b48ead;">unsafe </span><span>{ std::alloc::alloc(layout) }; </span><span>} </span></code></pre> <p>And we then run it?</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>memory allocation of 10 bytes failed </span><span>error: process didn't exit successfully: `target\debug\i32-allocator.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN) </span></code></pre> </div> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ # change to run two </span><span>$ cargo r --quiet </span><span>memory allocation of 10 bytes failed </span><span>error: process didn't exit successfully: `target\debug\i32-allocator.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN) </span></code></pre> </div> <h3 id="10-bytes">10 bytes?</h3> <p>Right, so something’s gone very wrong here - we only tried to allocate either 1 or 4 bytes, not 10. Ideally, the two runs should’ve also looked different.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>error: unsupported operation: can</span><span style="color:#b48ead;">'t</span><span> call (diverging) foreign function: __rust_alloc_error_handler </span><span> --> C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\alloc.rs:</span><span style="color:#d08770;">383</span><span>:</span><span style="color:#d08770;">13 </span><span> | </span><span style="color:#d08770;">383 </span><span>| </span><span style="color:#96b5b4;">__rust_alloc_error_handler</span><span>(layout.</span><span style="color:#96b5b4;">size</span><span>(), layout.</span><span style="color:#96b5b4;">align</span><span>()); </span><span> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can</span><span style="color:#b48ead;">'t</span><span> call (</span><span style="color:#bf616a;">diverging</span><span>) foreign </span><span style="color:#bf616a;">function</span><span>: __rust_alloc_error_handler </span><span> | </span><span> = help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support </span><span> = note: </span><span style="color:#d08770;">BACKTRACE</span><span>: </span><span> = note: inside `std::alloc::handle_alloc_error::rt_error` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\alloc.rs:</span><span style="color:#d08770;">383</span><span>:</span><span style="color:#d08770;">13</span><span>: </span><span style="color:#d08770;">383</span><span>:</span><span style="color:#d08770;">70 </span><span> = note: inside `std::alloc::handle_alloc_error` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\alloc.rs:</span><span style="color:#d08770;">389</span><span>:</span><span style="color:#d08770;">9</span><span>: </span><span style="color:#d08770;">389</span><span>:</span><span style="color:#d08770;">75 </span><span> = note: inside `alloc::raw_vec::RawVec::<</span><span style="color:#b48ead;">u16</span><span>>::allocate_in` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\raw_vec.rs:</span><span style="color:#d08770;">204</span><span>:</span><span style="color:#d08770;">27</span><span>: </span><span style="color:#d08770;">204</span><span>:</span><span style="color:#d08770;">53 </span><span> = note: inside `alloc::raw_vec::RawVec::<</span><span style="color:#b48ead;">u16</span><span>>::with_capacity_in` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\raw_vec.rs:</span><span style="color:#d08770;">145</span><span>:</span><span style="color:#d08770;">9</span><span>: </span><span style="color:#d08770;">145</span><span>:</span><span style="color:#d08770;">69 </span><span> = note: inside `std::vec::Vec::<</span><span style="color:#b48ead;">u16</span><span>>::with_capacity_in` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\vec\</span><span style="color:#b48ead;">mod</span><span>.rs:</span><span style="color:#d08770;">672</span><span>:</span><span style="color:#d08770;">20</span><span>: </span><span style="color:#d08770;">672</span><span>:</span><span style="color:#d08770;">61 </span><span> = note: inside `std::vec::Vec::<</span><span style="color:#b48ead;">u16</span><span>>::with_capacity` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\vec\</span><span style="color:#b48ead;">mod</span><span>.rs:</span><span style="color:#d08770;">481</span><span>:</span><span style="color:#d08770;">9</span><span>: </span><span style="color:#d08770;">481</span><span>:</span><span style="color:#d08770;">49 </span><span> = note: inside `std::sys::windows::to_u16s::inner` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys\windows\</span><span style="color:#b48ead;">mod</span><span>.rs:</span><span style="color:#d08770;">186</span><span>:</span><span style="color:#d08770;">32</span><span>: </span><span style="color:#d08770;">186</span><span>:</span><span style="color:#d08770;">63 </span><span> = note: inside `std::sys::windows::to_u16s::<&</span><span style="color:#b48ead;">str</span><span>>` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys\windows\</span><span style="color:#b48ead;">mod</span><span>.rs:</span><span style="color:#d08770;">198</span><span>:</span><span style="color:#d08770;">5</span><span>: </span><span style="color:#d08770;">198</span><span>:</span><span style="color:#d08770;">22 </span><span> = note: inside `std::sys::windows::thread::Thread::set_name` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys\windows\thread.rs:</span><span style="color:#d08770;">66</span><span>:</span><span style="color:#d08770;">32</span><span>: </span><span style="color:#d08770;">66</span><span>:</span><span style="color:#d08770;">45 </span><span> = note: inside `std::sys::windows::init` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\sys\windows\</span><span style="color:#b48ead;">mod</span><span>.rs:</span><span style="color:#d08770;">66</span><span>:</span><span style="color:#d08770;">5</span><span>: </span><span style="color:#d08770;">66</span><span>:</span><span style="color:#d08770;">39 </span><span> = note: inside `std::rt::init` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:</span><span style="color:#d08770;">97</span><span>:</span><span style="color:#d08770;">9</span><span>: </span><span style="color:#d08770;">97</span><span>:</span><span style="color:#d08770;">39 </span><span> = note: inside closure at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:</span><span style="color:#d08770;">147</span><span>:</span><span style="color:#d08770;">42</span><span>: </span><span style="color:#d08770;">147</span><span>:</span><span style="color:#d08770;">67 </span><span> = note: inside `std::panicking::r#try::do_call::<</span><span style="background-color:#bf616a;color:#2b303b;">{</span><span>closure@std::rt::lang_start_internal::{closure#</span><span style="color:#d08770;">1</span><span>}}, ()>` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:</span><span style="color:#d08770;">552</span><span>:</span><span style="color:#d08770;">40</span><span>: </span><span style="color:#d08770;">552</span><span>:</span><span style="color:#d08770;">43 </span><span> = note: inside `std::panicking::r#try::<(), </span><span style="background-color:#bf616a;color:#2b303b;">{</span><span>closure@std::rt::lang_start_internal::{closure#</span><span style="color:#d08770;">1</span><span>}}>` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panicking.rs:</span><span style="color:#d08770;">516</span><span>:</span><span style="color:#d08770;">19</span><span>: </span><span style="color:#d08770;">516</span><span>:</span><span style="color:#d08770;">81 </span><span> = note: inside `std::panic::catch_unwind::<</span><span style="background-color:#bf616a;color:#2b303b;">{</span><span>closure@std::rt::lang_start_internal::{closure#</span><span style="color:#d08770;">1</span><span>}}, ()>` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\panic.rs:</span><span style="color:#d08770;">142</span><span>:</span><span style="color:#d08770;">14</span><span>: </span><span style="color:#d08770;">142</span><span>:</span><span style="color:#d08770;">33 </span><span> = note: inside `std::rt::lang_start_internal` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:</span><span style="color:#d08770;">147</span><span>:</span><span style="color:#d08770;">5</span><span>: </span><span style="color:#d08770;">147</span><span>:</span><span style="color:#d08770;">70 </span><span> = note: inside `std::rt::lang_start::<()>` at C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\rt.rs:</span><span style="color:#d08770;">165</span><span>:</span><span style="color:#d08770;">17</span><span>: </span><span style="color:#d08770;">170</span><span>:</span><span style="color:#d08770;">6 </span></code></pre> <p>I can’t really make heads or tails of most of that, but if you skim around carefully, you can see lots of references to the rust runtime startup. That then explains our issues - something else is allocating. This also makes sense because the runtime startup probably allocates to do things like setting up the environment variables and command-line arguments in a nice place.</p> <p>If this is happening, it might be easier to just allocate pointers as and when we need them, rather than trying to create a global allocator from the start. This’ll also let us do other useful things like allocating during our <code>alloc</code> function. For now though, we’ll keep using the skeleton of <code>GlobalAlloc</code>, if only for it being good practice.</p> <h3 id="local-allocator">Local Allocator</h3> <p>We’ll start by commenting out <code>#[global_allocator]</code>. Then, we’ll replace <code>std::alloc::alloc</code> with <code>ALLOCATOR.alloc</code>.</p> <p>If we re-run our test from earlier (just to remind you - it should panic on unimplemented the first time, and not do anything on the second time):</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>thread 'main' panicked at src\main.rs:18:9: </span><span>not yet implemented </span><span>note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace </span><span>error: process didn't exit successfully: `target\debug\i32-allocator.exe` (exit code: 101) </span></code></pre> </div> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ # change to run two </span><span>$ cargo r --quiet </span><span># program exits successfully! </span></code></pre> </div> <p>Success!</p> <p>The next step would be to actually store allocations. We need some way to track when a slot is allocated and when it isn’t, and that’s why we’ve got a <code>usize</code> tracker and are storing <code>usize::BITS * 4</code> <code>u8</code>s. We can use each bit in the <code>usize</code> to mark whether or not the relevant slot has been handed over yet.</p> <h3 id="actually-storing-allocations">Actually Storing Allocations?</h3> <p>One of the invariants of <code>UnsafeCell</code> that we need to uphold is to ensure that it isn’t accessed mutably by more than one thread at a time. We also need to hold up our own invariant, which is that the <code>tracker</code> is always either accurate or overzealous. In the Rust where we’re allowed to allocate, the solution is clear cut and obvious - <code>Arc<Mutex<usize>></code>, but that doesn’t hold here because that allocates :/ .<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup></p> <p>Instead, we’ll have to use an <code>AtomicBool</code> for whether we’re ‘holding the lock’, and then an <code>UnsafeCell</code> to store the actual <code>usize</code>. Before you point out that we could just use an <code>AtomicUsize</code>, we need to iterate through the <code>usize</code> a lot to find a space and I’m not sure how to ensure that an <code>AtomicUsize</code> doesn’t get accessed for a portion of time. Because of that, we need to emulate a <code>Mutex</code> and luckily, <code>AtomicBool</code> has a method called <a href="https://doc.rust-lang.org/core/sync/atomic/struct.AtomicBool.html#method.compare_exchange"><code>compare_exchange</code></a> where we can use <a href="https://doc.rust-lang.org/std/sync/atomic/enum.Ordering.html#variant.SeqCst"><code>Ordering::SeqCst</code></a> to keep it all thread-safe and ensure that only one thread will not be waiting at any time. <code>compare_exchange</code> takes two arguments at the start - a <code>current</code> variable and a <code>target</code> variable. It’ll only switch the <code>AtomicBool</code> to be the <code>target</code>, if it is currently the <code>current</code>. Using that, we can then set it to only set it to be allocating when it isn’t already allocating. The other two arguments are about ordering - the first is for what ordering to use when comparing and setting on success, and the second is for how to load <code>current</code> on failure. Here, I’ll just use <code>SeqCst</code> because I’m never 100% sure on Atomic Orderings, and I know this one will work.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">while </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">compare_exchange</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#d08770;">true</span><span>, SeqCst, SeqCst).</span><span style="color:#96b5b4;">is_err</span><span>() {std::thread::yield_now()} </span></code></pre> <p>Since it returns <code>Result::Err</code> whenever it fails to store the new value, we can just wait for it to not do that. To give a small performance boost, we can also tell it to <a href="https://doc.rust-lang.org/std/thread/fn.yield_now.html"><code>yield_now</code></a> which just tells the CPU scheduler to go to another thread. This makes sense because we can’t do anything until we’ve allocated so we may as well go elsewhere.</p> <p>Then, we can get the pointer to our tracker, and copy it to the stack for convenience:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> tracker_ptr = </span><span style="color:#bf616a;">self</span><span>.tracker.</span><span style="color:#96b5b4;">get</span><span>(); </span><span style="color:#b48ead;">let</span><span> tracker_copied = *tracker_ptr; </span></code></pre> <p><code>UnsafeCell::get</code> is <code>unsafe</code>, but we know that our use is sound because we’re the only thread accessing this right now (because of our <code>AtomicBool</code> lock).</p> <p>Then, we need to find the next slot that’s free in <code>tracked_copied</code>.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let mut</span><span> chosen_index = None; </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#b48ead;">usize</span><span>::</span><span style="color:#d08770;">BITS </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> tracker_copied & (</span><span style="color:#d08770;">1 </span><span><< i) == </span><span style="color:#d08770;">0 </span><span>{ </span><span> *tracker_ptr = tracker_copied | (</span><span style="color:#d08770;">1 </span><span><< i); </span><span> chosen_index = Some(i as </span><span style="color:#b48ead;">usize</span><span>); </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span>} </span></code></pre> <p>So, we have a counter for iterating through the bits of the tracker. We’re counting <code>0</code> as free and <code>1</code> as occupied, as we start our tracker as all zeroes. We check each bit by shifting <code>0b1</code> the relevant number of bits, and if we logical and that with the <code>tracker_copied</code> and get <code>0</code> we must know that that slot is free. We then update <code>tracker_copied</code> using a logical or, and set <code>chosen_index</code> to be the index we ended up using. We then break the loop.</p> <p>Now that we’re done with parts that care about the tracker, we can ‘hand back’ our ‘lock’ by storing <code>false</code>. We then also need to upkeep our invariants by ensuring not to use the pointer we hold for the rest of the function.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">false</span><span>, SeqCst); </span></code></pre> <p>Then, we need to extract the our chosen index. We aren’t 100% sure that we actually had a free space, but if we didn’t then that variable never got set to <code>Some</code>, and so we can extract it using <a href="https://doc.rust-lang.org/stable/rust-by-example/flow_control/let_else.html"><code>let-else</code></a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let </span><span>Some(chosen_index) = chosen_index </span><span style="color:#b48ead;">else </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span>}; </span></code></pre> <p>That way, if we didn’t find a index, we can just return a null pointer.</p> <p>Finally, we can actually return a valid index into our array with:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>(</span><span style="color:#bf616a;">self</span><span>.spaces.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>).</span><span style="color:#96b5b4;">add</span><span>(chosen_index * </span><span style="color:#d08770;">4</span><span>) </span></code></pre> <p>Since <code>self.spaces</code> is an array, we know it is all in one space in our stack memory, so we can just cast it to the element in that array and add on <code>chosen_index * 4</code> to give each allocation 4 bytes.</p> <p>So, overall:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> layout.</span><span style="color:#96b5b4;">size</span><span>() != </span><span style="color:#d08770;">4 </span><span>|| layout.</span><span style="color:#96b5b4;">align</span><span>() != </span><span style="color:#d08770;">4 </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span> } </span><span> </span><span> </span><span style="color:#65737e;">//wait until we can start allocating </span><span> </span><span style="color:#b48ead;">while </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">compare_exchange</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#d08770;">true</span><span>, SeqCst, SeqCst).</span><span style="color:#96b5b4;">is_err</span><span>() {std::thread::yield_now()} </span><span> </span><span style="color:#65737e;">//theoretically, now only this thread should be able to allocate </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> tracker_ptr = </span><span style="color:#bf616a;">self</span><span>.tracker.</span><span style="color:#96b5b4;">get</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> tracker_copied: </span><span style="color:#b48ead;">usize </span><span>= *tracker_ptr; </span><span> </span><span style="color:#65737e;">//Sound, as we're the only thread accessing this pointer </span><span> </span><span style="color:#b48ead;">let mut</span><span> chosen_index = None; </span><span> </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#b48ead;">usize</span><span>::</span><span style="color:#d08770;">BITS </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> tracker_copied & (</span><span style="color:#d08770;">1 </span><span><< i) == </span><span style="color:#d08770;">0 </span><span>{ </span><span> *tracker_ptr = tracker_copied | (</span><span style="color:#d08770;">1 </span><span><< i); </span><span> chosen_index = Some(i as </span><span style="color:#b48ead;">usize</span><span>); </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> } </span><span> </span><span> </span><span style="color:#65737e;">//we're done with the tracker, so we can give back our lock </span><span> </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">false</span><span>, SeqCst); </span><span> </span><span> </span><span style="color:#b48ead;">let </span><span>Some(chosen_index) = chosen_index </span><span style="color:#b48ead;">else </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span> }; </span><span> </span><span> (</span><span style="color:#bf616a;">self</span><span>.spaces.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>).</span><span style="color:#96b5b4;">add</span><span>(chosen_index * </span><span style="color:#d08770;">4</span><span>) </span><span>} </span></code></pre> <h3 id="but-does-it-actually-work">But does it actually work?</h3> <p>If we go back into our main function, we can write a quick function to keep on allocating until we get a null pointer (which here, means that we ran out of memory):</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> layout = Layout::new::<</span><span style="color:#b48ead;">u32</span><span>>(); </span><span style="color:#b48ead;">let mut</span><span> i = </span><span style="color:#d08770;">0</span><span>; </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">alloc</span><span>(layout) }; </span><span> println!("</span><span style="color:#a3be8c;">Got </span><span style="color:#d08770;">{ptr:?}</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> ptr.</span><span style="color:#96b5b4;">is_null</span><span>() { </span><span> println!("</span><span style="color:#a3be8c;">Ran out @ attempt </span><span style="color:#d08770;">{}</span><span>", i + </span><span style="color:#d08770;">1</span><span>); </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> </span><span> i += </span><span style="color:#d08770;">1</span><span>; </span><span>} </span></code></pre> <p>Then, if we run that we can see that it successfully manages 64 allocations before getting a null pointer. That makes sense, because on a 64-bit platform (like my machine), addresses are 64-bit and so a <code>usize</code> has 64 bits. We can even run it using <code>cargo +nightly miri run</code> and still have success which is a good sign.</p> <p>There’s still another test though - what about multi-threading? Since we know we’ll have a maximum of 64 allocations, we can spin up 5 threads which will each attempt to make 15 allocations so we can see that multiple threads can access our allocations and all will work:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">for</span><span> t in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">5 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> cloned_layout = layout.</span><span style="color:#96b5b4;">clone</span><span>(); </span><span> std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#b48ead;">for</span><span> r in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">15 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#b48ead;">unsafe </span><span>{</span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">alloc</span><span>(cloned_layout)}; </span><span> println!("</span><span style="color:#a3be8c;">T</span><span style="color:#d08770;">{t}</span><span style="color:#a3be8c;">R</span><span style="color:#d08770;">{r}</span><span style="color:#a3be8c;"> got </span><span style="color:#d08770;">{ptr:?}</span><span>"); </span><span> } </span><span> }); </span><span>} </span></code></pre> <p>And then (with miri for safety):</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo +nightly miri run </span><span># warnings from the compiler about not using variables in `dealloc` </span><span>T0R0 got 0x2da10 </span><span>T1R0 got 0x2da14 </span><span>T2R0 got 0x2da18 </span><span>T3R0 got 0x2da1c </span><span>T4R0 got 0x2da20 </span><span>error: the main thread terminated without waiting for all remaining threads </span></code></pre> </div> <p>It looks like each thread managed to get 1 allocation done before stopping without waiting?</p> <p>Right, so it turns out that I just forgot to tell the main thread to actually wait for the threads to finish, so let’s quickly fix that:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let mut</span><span> threads = vec![]; </span><span> </span><span style="color:#b48ead;">for</span><span> t in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">5 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> cloned_layout = layout.</span><span style="color:#96b5b4;">clone</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> handle = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#65737e;">//do thread stuff </span><span> }); </span><span> threads.</span><span style="color:#96b5b4;">push</span><span>(handle); </span><span>} </span><span> </span><span style="color:#b48ead;">for mut</span><span> handle in threads { </span><span> handle.</span><span style="color:#96b5b4;">join</span><span>().</span><span style="color:#96b5b4;">expect</span><span>("</span><span style="color:#a3be8c;">unable to join handle</span><span>"); </span><span>} </span></code></pre> <p>I’ve deliberately picked 5 threads - too many could lead to nobody actually waiting for the allocator because of the time it takes to start a thread, and too few wouldn’t have a high enough chance of causing interesting race conditions (if there were any).</p> <p>And if we then run it we see a bunch of different memory addresses and 11 null pointers. <code>5 * 15 - 11 = 64</code>, which means that we managed to allocate the right number of addresses, and so theoretically didn’t have race conditions. Yay!</p> <h3 id="what-about-de-allocating">What about de-allocating?</h3> <p>So, right now we’ve got 64 allocations for the entire span of our program. That’s not very much, so it’d be nice to be able to de-allocate, so we become limited to 64 concurrent allocations, with theoretically infinite allocations over the span of the program. The following code samples will be to start filling out <code>fn dealloc (...)</code>.</p> <p>When deallocating, an invariant that is held for us (how nice!) is that that pointer was obtained from this allocator and so we can make a few very handy assumptions. Also - since memory from <code>alloc</code> is regarded as uninitialised (and it is in fact UB to access it without writing to it first), we don’t have to worry about zeroing the memory<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">2</a></sup>.</p> <p>This means that we really only have to wait to use the tracker, and then zero the relevant bit. We can work out the relevant bit because we know how long each section is and where the memory block we have starts.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> offset = (ptr as </span><span style="color:#b48ead;">usize </span><span>- </span><span style="color:#bf616a;">self</span><span>.spaces.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">usize</span><span>) / </span><span style="color:#d08770;">4</span><span>; </span><span style="color:#b48ead;">while </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">compare_exchange</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#d08770;">true</span><span>, SeqCst, SeqCst).</span><span style="color:#96b5b4;">is_err</span><span>() {std::thread::yield_now()} </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#bf616a;">self</span><span>.tracker.</span><span style="color:#96b5b4;">get</span><span>(); </span><span>*ptr = *ptr - (</span><span style="color:#d08770;">1 </span><span><< offset); </span><span> </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">false</span><span>, SeqCst); </span></code></pre> <p>We can then <code>miri</code> a quick single-threaded test:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">for </span><span>_ in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">1000 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#b48ead;">unsafe </span><span>{</span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">alloc</span><span>(layout)}; </span><span> </span><span style="color:#b48ead;">if</span><span> ptr.</span><span style="color:#96b5b4;">is_null</span><span>() { </span><span> eprintln!("</span><span style="color:#a3be8c;">What?</span><span>"); </span><span> } </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">dealloc</span><span>(ptr, layout) }; </span><span>} </span></code></pre> <p>And it exits successfully, which implies that everything worked - yay!</p> <p>If we also tell it to print out the pointers, we see that we only actually used one of our slots because each time it would search for places and just find the first place every time!</p> <p>If we modify it to only deallocate every 5 runs, and to only to do 16 total runs:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let mut</span><span> to_be_freed = Vec::with_capacity(</span><span style="color:#d08770;">10</span><span>); </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">16 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#b48ead;">unsafe </span><span>{</span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">alloc</span><span>(layout)}; </span><span> </span><span> println!("</span><span style="color:#d08770;">{ptr:?}</span><span>"); </span><span> to_be_freed.</span><span style="color:#96b5b4;">push</span><span>(ptr); </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> i % </span><span style="color:#d08770;">5 </span><span>== </span><span style="color:#d08770;">0 </span><span>{ </span><span> </span><span style="color:#b48ead;">for</span><span> ptr in std::mem::take(&</span><span style="color:#b48ead;">mut</span><span> to_be_freed) { </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#d08770;">ALLOCATOR</span><span>.</span><span style="color:#96b5b4;">dealloc</span><span>(ptr, layout) }; </span><span> } </span><span> } </span><span>} </span></code></pre> <p>We can see that it re-uses the same 5 slots:</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo +nightly miri run </span><span>0x24320 //start, clears list & deallocates because 0 % 5 == 0 0 </span><span>0x24320 //starts again at beginning 1 </span><span>0x24324 // 2 </span><span>0x24328 // 3 </span><span>0x2432c // 4 </span><span>0x24330 //5 % 5 == 0, so deallocates 5 </span><span>0x24320 //starts again 6 </span><span>0x24324 // 7 </span><span>0x24328 // 8 </span><span>0x2432c // 9 </span><span>0x24330 //deallocates as 10 % 5 == 0 10 </span><span>0x24320 //starts again 11 </span><span>0x24324 // 12 </span><span>0x24328 // 13 </span><span>0x2432c // 14 </span><span>0x24330 //deallocates as 15 % 5 == 0 15 </span></code></pre> </div> <p>Final code can be seen <a href="https://gist.github.com/BurntNail/ec64fb3c2ec3bfcb94d38ecfe9b39b2c">here</a>.</p> <h3 id="what-now">What now?</h3> <p>We could continue on with this basic structure of a bit-tracker and a bank of <code>u8</code>s held in an <code>UnsafeCell</code>. The problems start when you can’t rely on 1 bit being equal to 1 allocation, as you then have to start searching for long spaces which have enough space. If you want to go beyond 64 slots, you’d also need to start handling multiple <code>usize</code>s (possibly by using something like <a href="https://docs.rs/bitvec/latest/bitvec/array/struct.BitArray.html"><code>bitvec::BitArray</code></a>). Then, you need to have all of your memory stored on the stack which would a) limit you to a relatively small size and b) mean that you have to have all of the memory at the ready for the whole program, regardless of how much you actually need.</p> <p>You might also need to start worrying about heap fragmentation, where the heap has enough free slots for the memory required but they aren’t contiguous, so you then need to make sure that your allocations are in ideal places (eg. small allocations in gaps to leave room for bigger ones to come later).</p> <p>All of that having been said, I’ll be damned if I’m not going to even try to make a really shit one that technically works for any sized allocation.</p> <h2 id="time-to-handle-more-layouts">Time to handle more layouts</h2> <p>We’ll be starting from scratch here, but based off a similar structure to the previous allocator and I’ll start with a few things that would get annoying <strong>quickly</strong> with a multi-sized allocator.</p> <h3 id="tracker-replacement"><code>tracker</code> replacement</h3> <p>Instead of storing one <code>tracker</code> in an <code>UnsafeCell</code>, we’ll store an array of <code>Option<Allocation></code>s. This will lead to a maximum number of allocations (which I’ll control using a constant generic parameter here), but that number will <em>not</em> change based off the size of things allocated. I’ll also control the size of our ‘heap memory’ using a constant generic parameter. I’ll also add a function to quickly get the whole range of an Allocation for convenience.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::alloc::Layout; </span><span style="color:#b48ead;">use </span><span>std::sync::atomic::AtomicBool; </span><span style="color:#b48ead;">use </span><span>std::cell::UnsafeCell; </span><span> </span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Copy, Clone)] </span><span style="color:#b48ead;">struct </span><span>Allocation { </span><span> </span><span style="color:#bf616a;">start</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span> </span><span style="color:#bf616a;">layout</span><span>: Layout, </span><span>} </span><span> </span><span style="color:#b48ead;">impl </span><span>Allocation { </span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">get_range </span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> (</span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#b48ead;">*mut u8</span><span>) { </span><span> </span><span style="color:#65737e;">//we need to minus one because this range is inclusive </span><span> (</span><span style="color:#bf616a;">self</span><span>.start, </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#bf616a;">self</span><span>.start.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#bf616a;">self</span><span>.layout.</span><span style="color:#96b5b4;">size</span><span>() - </span><span style="color:#d08770;">1</span><span>) }) </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">struct </span><span>Allocator<</span><span style="color:#b48ead;">const</span><span> MAX_ALLOCS: </span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#b48ead;">const</span><span> MEMORY_SIZE: </span><span style="color:#b48ead;">usize</span><span>> { </span><span> </span><span style="color:#bf616a;">is_allocating</span><span>: AtomicBool, </span><span> </span><span style="color:#bf616a;">allocations</span><span>: UnsafeCell<[Option<Allocation>; </span><span style="color:#bf616a;">MAX_ALLOCS</span><span>]>, </span><span> </span><span style="color:#bf616a;">memory</span><span>: UnsafeCell<[</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#bf616a;">MEMORY_SIZE</span><span>]> </span><span>} </span></code></pre> <h4 id="finding-a-space-to-allocate-into">Finding a space to allocate into</h4> <p>Then, we’ll start with a convenience function to find the next free space of a set size. One interesting thing to note is that I’ve decided to make it <code>unsafe</code>, with the invariant being that we hold the ‘lock’. Below is just a skeleton which we’ll fill out as we go.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl</span><span><</span><span style="color:#b48ead;">const</span><span> MAX_ALLOCS: </span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#b48ead;">const</span><span> MEMORY_SIZE: </span><span style="color:#b48ead;">usize</span><span>> Allocator<MAX_ALLOCS, MEMORY_SIZE> { </span><span> </span><span style="color:#65737e;">///SAFETY: must hold onto the `is_allocating` lock before calling </span><span> </span><span style="color:#65737e;">/// </span><span> </span><span style="color:#65737e;">///Finds the next space free for an allocation of given length. NB: does not update </span><span> </span><span style="color:#65737e;">///`allocations` </span><span> </span><span style="color:#b48ead;">pub unsafe fn </span><span style="color:#8fa1b3;">find_next_space </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">length</span><span>: </span><span style="color:#b48ead;">usize</span><span>) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> current_allocations = *</span><span style="color:#bf616a;">self</span><span>.allocations.</span><span style="color:#96b5b4;">get</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> end = </span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>().</span><span style="color:#96b5b4;">add</span><span>(length); </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> next_space_to_chk = </span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>(); </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(length) > end { </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> </span><span> } </span><span> </span><span> std::ptr::null_mut() </span><span> } </span><span>} </span></code></pre> <p>We quickly copy our list of allocations and store a convenience variable for the end of our list. We also make a temporary variable for the next space we need to check. Then, at the start of each loop we check if we can allocate that pointer without going past the end of the list, and if we can’t we break. If we break out of the loop, the only thing we can do is to just return a <code>None</code>. Now, we’ll have a look over a naive algorithm to find the next usable block of memory that takes place during the loop, after the out-of-bounds check.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let </span><span>(start_check, end_check) = (next_space_to_chk, next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(length)); </span><span> </span><span style="color:#b48ead;">let mut</span><span> space_found = </span><span style="color:#d08770;">true</span><span>; </span><span style="color:#b48ead;">for</span><span> allocation in current_allocations.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">flatten</span><span>() { </span><span> </span><span style="color:#b48ead;">let </span><span>(start_block, end_block) = allocation.</span><span style="color:#96b5b4;">get_range</span><span>(); </span><span> </span><span> </span><span style="color:#b48ead;">if </span><span>(start_block <= start_check && start_check <= end_block) || (start_block <= end_check && end_check <= end_block) { </span><span> space_found = </span><span style="color:#d08770;">false</span><span>; </span><span> next_space_to_chk = end_block.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#d08770;">1</span><span>); </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">if</span><span> space_found { </span><span> </span><span style="color:#b48ead;">return </span><span>Some(next_space_to_chk); </span><span>} </span></code></pre> <p>Here, we just loop through all of our allocations to check if any part of our desired block for allocation is within another allocation. If it is, then we set the next spot to check to be just after the end of the block we intersected with and break the loop. If it isn’t, then we return that slot. We can then use this to write an actual function to allocate!</p> <h3 id="allocating-function">Allocating Function</h3> <p>Firstly, we’ll quickly implement <code>Sync</code> for <code>Allocator</code>. Like before, this is an <code>unsafe impl</code> but we know that this is sound because we control access to be single-threaded for the <code>UnsafeCell</code>. We can then also quickly tell Rust to use this as the global allocator like we did right at the start. I’ll also add a skeleton <code>GlobalAlloc</code> implementation.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::alloc::GlobalAlloc; </span><span> </span><span style="color:#b48ead;">unsafe impl</span><span><</span><span style="color:#b48ead;">const</span><span> MAX_ALLOCS: </span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#b48ead;">const</span><span> MEMORY_SIZE: </span><span style="color:#b48ead;">usize</span><span>> GlobalAlloc </span><span style="color:#b48ead;">for </span><span>Allocator<MAX_ALLOCS, MEMORY_SIZE> { </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> todo!() </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">dealloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">ptr</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) { </span><span> todo!() </span><span> } </span><span>} </span><span> </span><span style="color:#b48ead;">unsafe impl</span><span><</span><span style="color:#b48ead;">const</span><span> MAX_ALLOCS: </span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#b48ead;">const</span><span> MEMORY_SIZE: </span><span style="color:#b48ead;">usize</span><span>> Sync </span><span style="color:#b48ead;">for </span><span>Allocator<MAX_ALLOCS, MEMORY_SIZE> {} </span><span> </span><span>#[</span><span style="color:#bf616a;">global_allocator</span><span>] </span><span style="color:#b48ead;">static </span><span style="color:#d08770;">ALLOCATOR</span><span>: Allocator<1000, 1_000_000> = Allocator { </span><span> is_allocating: AtomicBool::new(</span><span style="color:#d08770;">false</span><span>), </span><span> allocations: UnsafeCell::new([None; </span><span style="color:#d08770;">1000</span><span>]), </span><span> memory: UnsafeCell::new([</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">1_000_000</span><span>]) </span><span>}; </span></code></pre> <p>Then, we have to build a function to actually allocate. These are the steps we need:</p> <ol> <li>Obtain the lock in order to make it sound to mutate <code>allocations</code>.</li> <li>Find a space to allocate (and if not, early return <code>std::ptr::null_mut</code>).</li> <li>Find a space to add it to our allocations list. If we find a spot, add it to the list (and if not, then <code>std::ptr::null_mut</code>).</li> <li>Unlock the lock.</li> <li>Return our pointer from Step 2.</li> </ol> <p>This is what I’ve managed for that:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">alloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> </span><span style="color:#b48ead;">*mut u8 </span><span>{ </span><span> </span><span style="color:#65737e;">//get the lock </span><span> </span><span style="color:#b48ead;">while </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">compare_exchange</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#d08770;">true</span><span>, SeqCst, SeqCst).</span><span style="color:#96b5b4;">is_err</span><span>() {std::thread::yield_now()} </span><span> </span><span> </span><span style="color:#65737e;">//find the position </span><span> </span><span style="color:#b48ead;">let </span><span>Some(position_to_allocate) = </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">find_next_space</span><span>(layout.</span><span style="color:#96b5b4;">size</span><span>()) </span><span style="color:#b48ead;">else </span><span>{ </span><span> </span><span style="color:#65737e;">//out of memory, return null_mut </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span> }; </span><span> </span><span> </span><span style="color:#65737e;">//add it to our array of allocations </span><span> </span><span style="color:#b48ead;">let</span><span> allocations_ptr = </span><span style="color:#bf616a;">self</span><span>.allocations.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut </span><span>Option<Allocation>; </span><span> </span><span style="color:#b48ead;">let mut</span><span> found_spot = </span><span style="color:#d08770;">false</span><span>; </span><span> </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">MAX_ALLOCS </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> this_spot = allocations_ptr.</span><span style="color:#96b5b4;">add</span><span>(i); </span><span> </span><span style="color:#b48ead;">if </span><span>(*this_spot).</span><span style="color:#96b5b4;">is_none</span><span>() { </span><span> *this_spot = Some(Allocation { start: position_to_allocate, layout }); </span><span> found_spot = </span><span style="color:#d08770;">true</span><span>; </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> } </span><span> </span><span style="color:#b48ead;">if </span><span>!found_spot { </span><span> </span><span style="color:#65737e;">//out of allocations, return null_mut </span><span> </span><span style="color:#b48ead;">return </span><span>std::ptr::null_mut(); </span><span> } </span><span> </span><span> </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">false</span><span>, SeqCst); </span><span> </span><span> position_to_allocate </span><span>} </span></code></pre> <p>We first use the same snippet as earlier to acquire the lock. Then we use <code>let-else</code> to extract our pointer or early return. We then iterate through our array of allocations, and when we find a free slot we add our allocation there. If we didn’t find a free slot, then we early return. We then unlock and return the pointer.</p> <p>We can then add a simple main function that does a few allocations to test this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> println!("</span><span style="color:#a3be8c;">Hello, world!</span><span>"); </span><span> </span><span style="color:#b48ead;">let</span><span> a = </span><span style="color:#d08770;">5_</span><span style="color:#b48ead;">i32</span><span>; </span><span> println!("</span><span style="color:#a3be8c;">Variable a = </span><span style="color:#d08770;">{a}</span><span>"); </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> v = (</span><span style="color:#d08770;">1</span><span>..</span><span style="color:#d08770;">10</span><span>).</span><span style="color:#96b5b4;">into_iter</span><span>().</span><span style="color:#96b5b4;">map</span><span>(|</span><span style="color:#bf616a;">x</span><span>| (x, x * </span><span style="color:#d08770;">5</span><span>)).collect::<Vec<(</span><span style="color:#b48ead;">u8</span><span>, </span><span style="color:#b48ead;">u8</span><span>)>>(); </span><span> </span><span style="color:#b48ead;">for </span><span>(one, five) in v { </span><span> println!("</span><span style="color:#d08770;">{one}</span><span style="color:#a3be8c;"> * 5 = </span><span style="color:#d08770;">{five}</span><span>"); </span><span> } </span><span>} </span></code></pre> <p>And if we check our console, we can see that we get all of our expected output:</p> <pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">$</span><span> cargo run</span><span style="color:#bf616a;"> --quiet </span><span style="color:#bf616a;">Hello,</span><span> world! </span><span style="color:#bf616a;">Variable</span><span> a = 5 </span><span style="color:#bf616a;">1 </span><span>* 5 = 5 </span><span style="color:#bf616a;">2 </span><span>* 5 = 10 </span><span style="color:#bf616a;">3 </span><span>* 5 = 15 </span><span style="color:#bf616a;">4 </span><span>* 5 = 20 </span><span style="color:#bf616a;">5 </span><span>* 5 = 25 </span><span style="color:#bf616a;">6 </span><span>* 5 = 30 </span><span style="color:#bf616a;">7 </span><span>* 5 = 35 </span><span style="color:#bf616a;">8 </span><span>* 5 = 40 </span><span style="color:#bf616a;">9 </span><span>* 5 = 45 </span></code></pre> <p>But that’s not the end of allocation, unfortunately.</p> <h3 id="did-you-really-think-we-d-avoided-ub">Did you really think we’d avoided UB?</h3> <p>If we run it through <code>miri</code> instead, we get this:</p> <pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">$</span><span> cargo +nightly miri run </span><span style="color:#bf616a;">error:</span><span> Undefined Behavior: constructing invalid value: encountered an unaligned reference (required 8 byte alignment but found 1) </span><span> </span><span style="color:#bf616a;">--</span><span>> C:</span><span style="color:#96b5b4;">\U</span><span>sers</span><span style="color:#96b5b4;">\m</span><span>icro</span><span style="color:#96b5b4;">\.</span><span>rustup</span><span style="color:#96b5b4;">\t</span><span>oolchains</span><span style="color:#96b5b4;">\n</span><span>ightly-x86_64-pc-windows-msvc</span><span style="color:#96b5b4;">\l</span><span>ib</span><span style="color:#96b5b4;">\r</span><span>ustlib</span><span style="color:#96b5b4;">\s</span><span>rc</span><span style="color:#96b5b4;">\r</span><span>ust</span><span style="color:#96b5b4;">\l</span><span>ibrary</span><span style="color:#96b5b4;">\a</span><span>lloc</span><span style="color:#96b5b4;">\s</span><span>rc</span><span style="color:#96b5b4;">\s</span><span>ync.rs:1834:24 </span><span> | </span><span style="color:#bf616a;">1834 </span><span>| </span><span style="color:#bf616a;">ptr::write</span><span>(&</span><span style="color:#bf616a;">mut</span><span> (*inner)</span><span style="color:#bf616a;">.strong,</span><span> atomic::AtomicUsize::new(1)); </span><span> | </span><span style="color:#bf616a;">^^^^^^^^^^^^^^^^^^^^</span><span> constructing invalid value: encountered an unaligned reference (required 8 byte alignment but found 1) </span><span> | </span><span style="color:#65737e;"># further output removed - it just says that this is from the language runtime startup </span></code></pre> <p>And we come to an old friend - alignment. We didn’t really have to deal with this earlier (in our 4-4-only allocator), because the length was identical to the alignment and we didn’t just randomly skip bytes to have to deal with alignment. Here, however it is an issue. Luckily, the fix is relatively simple.</p> <p>The first change needed is to make our <code>find_next_space</code> take in a <code>Layout</code> rather than just a length (and then in the callsite, we just remove the <code>.size()</code> call). Then, we need to change our code that changes <code>next_space_to_chk</code>. Previously, we just set it to one after the end of the block that we overlapped with, but now we need to set it to the next aligned pointer.</p> <p>Rust integer division rounds down, so we can find the number of align spaces we need to add to get away from this block using <code>(end_block as usize - start_check as usize) / align + 1</code>. Then, we just multiply that by <code>align</code>. That’s this line: <code>next_space_to_chk = end_block.add(1);</code> going to: <code>next_space_to_chk = next_space_to_chk.add(((end_block as usize - start_check as usize) / align + 1) * align);</code>.</p> <p>If we quickly run this through <code>miri</code>, we see no complaints! Next on the list is deallocation.</p> <h3 id="deallocation">Deallocation</h3> <p>Just like last time, de-allocation is vastly simpler than allocation. We just need to find which allocation struct was dealing with that allocation, and then remove it. We don’t even need to worry about using the layout when de-allocating because we know the pointer and the start of the allocation will be identical (as specified in the <code>GlobalAlloc</code> docs).</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">dealloc</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">ptr</span><span>: </span><span style="color:#b48ead;">*mut u8</span><span>, </span><span style="color:#bf616a;">_layout</span><span>: Layout) { </span><span> </span><span style="color:#b48ead;">while </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">compare_exchange</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#d08770;">true</span><span>, SeqCst, SeqCst).</span><span style="color:#96b5b4;">is_err</span><span>() {std::thread::yield_now()} </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> allocations_ptr = </span><span style="color:#bf616a;">self</span><span>.allocations.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut </span><span>Option<Allocation>; </span><span> </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">MAX_ALLOCS </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> this_spot = allocations_ptr.</span><span style="color:#96b5b4;">add</span><span>(i); </span><span> </span><span style="color:#b48ead;">match </span><span>*this_spot { </span><span> Some(Allocation { start, layout: _ }) </span><span style="color:#b48ead;">if</span><span> start == ptr => { </span><span> *this_spot = None; </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> }, </span><span> _ => {} </span><span> } </span><span> } </span><span> </span><span> </span><span style="color:#bf616a;">self</span><span>.is_allocating.</span><span style="color:#96b5b4;">store</span><span>(</span><span style="color:#d08770;">false</span><span>, SeqCst); </span><span>} </span></code></pre> <p>Let’s just run a multi-threading test for safety - I’ve constructed one that does computations and allocations on multiple threads to really check that we’ve got one that works:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() { </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">factorial </span><span>(</span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">u128</span><span>) -> </span><span style="color:#b48ead;">u128 </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> n <= </span><span style="color:#d08770;">1 </span><span>{ </span><span> </span><span style="color:#b48ead;">return</span><span> n; </span><span> } </span><span> </span><span> n * </span><span style="color:#96b5b4;">factorial</span><span>(n - </span><span style="color:#d08770;">1</span><span>) </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> threads = vec![]; </span><span> </span><span style="color:#b48ead;">for</span><span> block in </span><span style="color:#d08770;">0</span><span>..</span><span style="color:#d08770;">6 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> t = std::thread::spawn(</span><span style="color:#b48ead;">move </span><span>|| { </span><span> </span><span style="color:#b48ead;">let mut</span><span> v = vec![]; </span><span> </span><span style="color:#b48ead;">for</span><span> n in (</span><span style="color:#d08770;">1</span><span>..=</span><span style="color:#d08770;">5</span><span>).</span><span style="color:#96b5b4;">into_iter</span><span>().</span><span style="color:#96b5b4;">map</span><span>(|</span><span style="color:#bf616a;">x</span><span>| block + x * </span><span style="color:#d08770;">5</span><span>) { </span><span> v.</span><span style="color:#96b5b4;">push</span><span>((n, </span><span style="color:#96b5b4;">factorial</span><span>(n))); </span><span> } </span><span> </span><span> println!("</span><span style="color:#d08770;">{v:?}</span><span>"); </span><span> }); </span><span> </span><span> threads.</span><span style="color:#96b5b4;">push</span><span>(t); </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">for</span><span> t in threads { </span><span> t.</span><span style="color:#96b5b4;">join</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>(); </span><span> } </span><span>} </span></code></pre> <p>Theoretically, each thread should have a roughly similar amount of work to do with a naive factorial algorithm. This also works perfectly correctly using the standard allocator.</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo +nightly miri run </span><span>[(5, 120), (10, 3628800), (15, 1307674368000), (20, 2432902008176640000), (25, 15511210043330985984000000)] </span><span>[(6, 720), (11, 39916800), (16, 20922789888000), (21, 51090942171709440000), (26, 403291461126605635584000000)] </span><span>[(7, 5040), (12, 479001600), (17, 355687428096000), (22, 1124000727777607680000), (27, 10888869450418352160768000000)] </span><span>error: Undefined Behavior: Data race detected between (1) non-atomic read on thread `<unnamed>` and (2) non-atomic write on thread `<unnamed>` at alloc65+0x4c8. (2) just happened here </span><span> --> C:\Users\micro\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\vec\mod.rs:1923:13 </span><span> | </span><span>1923 | ptr::write(end, value); </span><span> | ^^^^^^^^^^^^^^^^^^^^^^ Data race detected between (1) non-atomic read on thread `<unnamed>` and (2) non-atomic write on thread `<unnamed>` at alloc65+0x4c8. (2) just happened here </span><span> | </span><span># further output that doesn't help </span></code></pre> </div> <p>And we’ve got a data race. 😔</p> <h3 id="ub-2-electric-boogaloo">UB 2: Electric Boogaloo</h3> <p>To debug this, we should probably work out a way to provide some form of logging. The problem occurs when we realise that just using <code>println</code> would cause infinite recursion because that allocates a string<sup class="footnote-reference" id="fr-20-1"><a href="#fn-20">3</a></sup>. I’ve got one idea on how to do this nicely and then a backup (which will definitely work).</p> <h3 id="printing-without-allocating">Printing without allocating</h3> <p>My first instinct is to go with the standard library, and see what I can do safely - and I can test both of these methods by just putting one at the top of my <code>alloc</code> function.</p> <p>Usually, if I want to write to <code>stdout</code> without using <code>println!</code>, I have to get a handle to <code>Stdout</code> and then just use <code>std::io::Write</code>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> s = "</span><span style="color:#a3be8c;">Allocating!</span><span style="color:#96b5b4;">\n</span><span>"; </span><span>std::io::stdout().</span><span style="color:#96b5b4;">lock</span><span>().</span><span style="color:#96b5b4;">write_all</span><span>(s.</span><span style="color:#96b5b4;">as_bytes</span><span>()).</span><span style="color:#96b5b4;">unwrap</span><span>(); </span></code></pre> <p>Here, I’ve unwrapped the error - this technically breaks the invariant on not panicking but I’m going to ignore that for now because this function won’t be here any longer than about another sentence. If we run it we recurse and so we never even reach the <code>unwrap</code> which means that <code>Stdout::write_all</code><sup class="footnote-reference" id="fr-30-1"><a href="#fn-30">4</a></sup> must allocate somewhere. We could dive deep and bypass it, or we could just go to where the standard library eventually links to - <a href="https://lib.rs/crates/libc"><code>libc</code></a>.</p> <p>A while ago, I read Amos’s piece on <a href="https://fasterthanli.me/articles/small-strings-in-rust">Small Strings in Rust</a>, and I remember that he somehow managed to print to stdout <em>without</em> allocating. That method is to use <code>libc::write</code> to <code>stdout</code> just passing a pointer and length. That does unfortunately<sup class="footnote-reference" id="fr-40-1"><a href="#fn-40">5</a></sup> mark our first dependency - I love doing things myself but I draw the line at syscalls (especially on Windows 😭). He then goes on to use an actually sensible method for getting the allocations to the console, but I’m going for something different<sup class="footnote-reference" id="fr-50-1"><a href="#fn-50">6</a></sup>.</p> <p>To test it to start with we’ll just try to print out a string:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> string = "</span><span style="color:#a3be8c;">Allocating!</span><span style="color:#96b5b4;">\n</span><span>"; </span><span>libc::write(???, string.</span><span style="color:#96b5b4;">as_ptr</span><span>() as _, string.</span><span style="color:#96b5b4;">len</span><span>() as _); </span></code></pre> <p>Most of that call is fine, apart from the <code>???</code> for which we need the file descriptor of the output stream we want to send stuff to. This is in <code>libc</code> for linux users under <code>libc::STDOUT_FILENO</code>, but not for windows 😔. I turned to DuckDuckGo and found <a href="https://stackoverflow.com/questions/13531677/stdin-fileno-undeclared-in-windows">this SO post</a>, which mentions the file descriptor being 0. That did nothing for me, so I tried <code>1</code> and luckily it worked. Examining the post closer, it says <code>0!</code> so I’m just going to assume that they meant <code>factorial(0)</code> rather than using <code>!</code> for exclamation. <em>(correction: u/skynetsatellite013 pointed out that I had actually misread the SO post and it was talking about STDIN_FILENO not STDOUT_FILENO 🤦)</em> I’ll chuck that into a constant, and we then get a function for printing string references:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">const </span><span style="color:#d08770;">STDOUT_FILENO</span><span>: </span><span style="color:#b48ead;">i32 </span><span>= </span><span style="color:#d08770;">1</span><span>; </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_string </span><span>(</span><span style="color:#bf616a;">s</span><span>: &</span><span style="color:#b48ead;">str</span><span>) { </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span> libc::write(</span><span style="color:#d08770;">STDOUT_FILENO</span><span>, s.</span><span style="color:#96b5b4;">as_ptr</span><span>() as _, s.</span><span style="color:#96b5b4;">len</span><span>() as </span><span style="color:#b48ead;">u32</span><span>); </span><span> } </span><span>} </span></code></pre> <p>Quick note: we need to cast the pointer to a <code>*const c_void</code>, but since the Rust compiler knows that we can just use an <code>_</code> to tell it to infer. This just saves an import.</p> <p>Unfortunately, this is where <code>miri</code> forsakes us because we’re making function calls which it cannot decipher. Once we’re finished with our whole <code>print_allocation</code> function, I’ll show a quick method to still be able to use <code>miri</code> with the rest of the program.</p> <p>Then, we can extend that logic to make a function to print numbers. After some quick searching to find how to get the digits of a number (I managed to adapt something from <a href="https://stackoverflow.com/questions/41536479/how-do-i-split-an-integer-into-individual-digits">here</a>) we can then print them out. That code spat out numbers in reverse order for me so I just print them out in reverse.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_number </span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">usize</span><span>) { </span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">NUMBERS</span><span>: &[</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">10</span><span>] = &[</span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">0</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">1</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">2</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">3</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">4</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">5</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">6</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">7</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">8</span><span>', </span><span style="color:#b48ead;">b</span><span>'</span><span style="color:#a3be8c;">9</span><span>']; </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> numbers = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">64</span><span>]; </span><span> </span><span style="color:#b48ead;">let mut</span><span> digit = </span><span style="color:#d08770;">0</span><span>; </span><span> </span><span style="color:#b48ead;">while</span><span> n != </span><span style="color:#d08770;">0 </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> this_digit = (n % </span><span style="color:#d08770;">10</span><span>) as </span><span style="color:#b48ead;">usize</span><span>; </span><span> numbers[digit] = this_digit; </span><span> </span><span> n /= </span><span style="color:#d08770;">10</span><span>; </span><span> digit += </span><span style="color:#d08770;">1</span><span>; </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">while</span><span> digit != </span><span style="color:#d08770;">0 </span><span>{ </span><span> digit -= </span><span style="color:#d08770;">1</span><span>; </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ libc::write(</span><span style="color:#d08770;">STDOUT_FILENO</span><span>, (</span><span style="color:#d08770;">NUMBERS</span><span>.</span><span style="color:#96b5b4;">as_ptr</span><span>() as </span><span style="color:#b48ead;">*const u8</span><span>).</span><span style="color:#96b5b4;">add</span><span>(numbers[digit]) as _, </span><span style="color:#d08770;">1</span><span>) }; </span><span> } </span><span>} </span></code></pre> <p>We can then use both of those functions to build out a <code>print_allocation</code> function:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_allocation </span><span>(</span><span style="color:#bf616a;">layout</span><span>: Layout) { </span><span> </span><span style="color:#96b5b4;">print_string</span><span>("</span><span style="color:#a3be8c;">Thread #</span><span>"); </span><span> </span><span style="color:#96b5b4;">print_number</span><span>(</span><span style="color:#b48ead;">unsafe </span><span>{ winapi::um::processthreadsapi::GetCurrentThreadId() as </span><span style="color:#b48ead;">usize</span><span>}); </span><span> </span><span> </span><span style="color:#96b5b4;">print_string</span><span>("</span><span style="color:#a3be8c;"> Allocating </span><span>"); </span><span> </span><span style="color:#96b5b4;">print_number</span><span>(layout.</span><span style="color:#96b5b4;">size</span><span>()); </span><span> </span><span style="color:#96b5b4;">print_string</span><span>("</span><span style="color:#a3be8c;">-</span><span>"); </span><span> </span><span style="color:#96b5b4;">print_number</span><span>(layout.</span><span style="color:#96b5b4;">align</span><span>()); </span><span> </span><span> </span><span style="color:#96b5b4;">print_string</span><span>(" </span><span style="color:#96b5b4;">\n</span><span>"); </span><span>} </span></code></pre> <p>Because I’m on a windows machine, in order to get the thread ID<sup class="footnote-reference" id="fr-60-1"><a href="#fn-60">7</a></sup>, I have to add another dependency - <code>winapi</code> with the <code>processthreadsapi</code> feature. I can then use <code>winapi::um::processthreadsapi::GetCurrentThreadId()</code>.</p> <p>Then, we can add this to the top of <code>fn alloc(layout: Layout)</code>, just below where we acquire the lock.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">cfg</span><span>(</span><span style="color:#bf616a;">not</span><span>(miri))] </span><span style="color:#96b5b4;">print_allocation</span><span>(layout); </span></code></pre> <p>The conditional compilation statement just makes sure that this function never gets called if we’re running with <code>miri</code>.</p> <p>As we worked out earlier, even if you do nothing, the main function does some allocations, so we can just run <code>fn main () {}</code> and see:</p> <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo r --quiet </span><span>Thread #2024 Allocating 10-2 </span><span>Thread #2024 Allocating 5-1 </span><span>Thread #2024 Allocating 48-8 </span><span>Thread #2024 Allocating 64-8 </span></code></pre> </div> <p>If I run it multiple times, then I get the same allocations displayed, only with different thread IDs.</p> <p>We can then add another function:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_end_of_allocation </span><span>() { </span><span> </span><span style="color:#96b5b4;">print_string</span><span>("</span><span style="color:#96b5b4;">\t</span><span style="color:#a3be8c;">Thread #</span><span>"); </span><span> </span><span style="color:#96b5b4;">print_number</span><span>(</span><span style="color:#b48ead;">unsafe </span><span>{ winapi::um::processthreadsapi::GetCurrentThreadId() as </span><span style="color:#b48ead;">usize</span><span>}); </span><span> </span><span style="color:#96b5b4;">print_string</span><span>("</span><span style="color:#a3be8c;"> finished allocation</span><span style="color:#96b5b4;">\n</span><span>"); </span><span>} </span></code></pre> <p>And use it just before we lose the lock and with the same conditional compilation gate. We then see this as output: <div class="crt scanlines" aria-hidden="true"> <pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cargo run --quiet </span><span>Thread #2072 Allocating 10-2 </span><span> Thread #2072 finished allocation </span><span>Thread #2072 Allocating 5-1 </span><span> Thread #2072 finished allocation </span><span>Thread #2072 Allocating 48-8 </span><span> Thread #2072 finished allocation </span><span>Thread #2072 Allocating 64-8 </span><span> Thread #2072 finished allocation </span><span># more output cut </span></code></pre> </div> </p> <h3 id="what-were-we-doing-again">What were we doing again?</h3> <p>If we were to then run our latest multi-threading test again we’d get a <strong>lot</strong> of output. To avoid this, I’ve just replaced the <code>vec!</code> calls with calls to <code>Vec::with_capacity</code> with capacity to hold the relevant struct. I’ve also added the thread ID to each call to <code>println!</code> inside a thread. That way, we might be able to see if it’s an issue with allocation or deallocation.</p> <p>Unfortunately, the output suggests that only one thread is ever holding the lock as each allocation start is immediately followed by that thread ID saying that it finished that allocation. This suggests that our problem lies elsewhere, which is unfortunate because that might be harder to fix.</p> <h3 id="what-else-could-it-be">What else could it be?</h3> <p>Thanks to the printing, we know at least know that only one thread is concurrently using the allocation function. However, since we’re still getting data races<sup class="footnote-reference" id="fr-70-1"><a href="#fn-70">8</a></sup> there must be concurrent access to pointers. The next step<sup class="footnote-reference" id="fr-80-1"><a href="#fn-80">9</a></sup> is then to look at where we choose our pointer to hand over in <code>find_next_space</code>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub unsafe fn </span><span style="color:#8fa1b3;">find_next_space </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> Option<</span><span style="color:#b48ead;">*mut u8</span><span>> { </span><span> </span><span style="color:#b48ead;">let</span><span> current_allocations = </span><span style="color:#bf616a;">self</span><span>.allocations.</span><span style="color:#96b5b4;">get</span><span>().</span><span style="color:#96b5b4;">read</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> end = (</span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>).</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#d08770;">MEMORY_SIZE</span><span>); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> length = layout.</span><span style="color:#96b5b4;">size</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> align = layout.</span><span style="color:#96b5b4;">align</span><span>(); </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> next_space_to_chk = </span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>; </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">if</span><span> next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(length) > end { </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">let </span><span>(start_check, end_check) = (next_space_to_chk, next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(length)); </span><span> </span><span style="color:#b48ead;">let mut</span><span> space_found = </span><span style="color:#d08770;">true</span><span>; </span><span> </span><span style="color:#b48ead;">for</span><span> allocation in current_allocations.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">flatten</span><span>() { </span><span> </span><span style="color:#b48ead;">let </span><span>(start_block, end_block) = allocation.</span><span style="color:#96b5b4;">get_range</span><span>(); </span><span> </span><span style="color:#b48ead;">if </span><span>(start_block <= start_check && start_check <= end_block) || (start_block <= end_check && end_check <= end_block) { </span><span> space_found = </span><span style="color:#d08770;">false</span><span>; </span><span> next_space_to_chk = next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(((end_block as </span><span style="color:#b48ead;">usize </span><span>- start_check as </span><span style="color:#b48ead;">usize</span><span>) / align) * align); </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> space_found { </span><span> </span><span style="color:#b48ead;">return </span><span>Some(next_space_to_chk); </span><span> } </span><span> } </span><span> </span><span> None </span><span>} </span></code></pre> <ol> <li>We make a copy of our existing allocations for convenience - we’re reading from the pointer, but we know that we won’t get any data races because we’ve locked the lock.</li> <li>We make a convenience variable to store the pointer after what we’re allowed to allocate - we only ever compare to this so we’re fine. This accesses the <code>memory</code> pointer, but never dereferences it and we’re in the lock anyway.</li> <li>We set a variable equal to the start of our memory. This will eventually become our pointer to the memory, but we modify and check it several times so we’ll deal with that in a minute.</li> </ol> <p>Then, we enter a loop:</p> <ol> <li>We check if we’ve gone out of our memory bounds. We can check for <code>> end</code> because <code>next_space_to_chk.add(length)</code> represents the pointer after the final one we use (because of zero-indexing).</li> <li>We then create a convenience variable with the that <code>next_space_to_chk.add(length)</code> as the value.</li> <li>We also declare a temporary variable for a flag, which assumes that we will find a space. We can then invalidate that if we overlap in the loop.</li> </ol> <p>We then enter <em>another</em> loop for every allocation:</p> <ol> <li>We get the <strong>inclusive</strong> range of pointers that that allocation uses.</li> <li>We then check if either our start or end checking pointers are in the range. I’m not 100% sure about this maths, so we’ll check back on it later but it looks right to me right now.</li> <li>If that check works, then we invalidate the assumption that we found a space.</li> <li>We then do some pointer maths to add a multiple of <code>align</code> that should be just after the <code>end_block</code>. If we’re testing this part, we can make the assumption that we have far more memory than we need for this program. From our earlier tests, I didn’t see any allocations bigger than about 100 bytes, so if we just add <code>1000 * align</code>, we’re pretty much guaranteed to be fine<sup class="footnote-reference" id="fr-90-1"><a href="#fn-90">10</a></sup>. If we run this with multi-threading under <code>miri</code> it works! 🥳🥳🥳<sup class="footnote-reference" id="fr-100-1"><a href="#fn-100">11</a></sup></li> </ol> <p>This suggests that there’s an error with my logic that chooses when we’ve overlapped with another allocation. I’m also not the happiest with how this function looks so I’m going to make another pass over it:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub unsafe fn </span><span style="color:#8fa1b3;">find_next_space </span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">layout</span><span>: Layout) -> Option<</span><span style="color:#b48ead;">*mut u8</span><span>> { </span><span> </span><span style="color:#b48ead;">let</span><span> current_allocations = </span><span style="color:#bf616a;">self</span><span>.allocations.</span><span style="color:#96b5b4;">get</span><span>().</span><span style="color:#96b5b4;">read</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> end = (</span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>).</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#d08770;">MEMORY_SIZE</span><span>); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> length = layout.</span><span style="color:#96b5b4;">size</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> align = layout.</span><span style="color:#96b5b4;">align</span><span>(); </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> next_space_to_chk = </span><span style="color:#bf616a;">self</span><span>.memory.</span><span style="color:#96b5b4;">get</span><span>() as </span><span style="color:#b48ead;">*mut u8</span><span>; </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> end_check = next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(length - </span><span style="color:#d08770;">1</span><span>); </span><span> </span><span style="color:#b48ead;">if</span><span> end_check >= end { </span><span> </span><span style="color:#b48ead;">return </span><span>None; </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> space_found = current_allocations.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">flatten</span><span>().</span><span style="color:#96b5b4;">all</span><span>(|</span><span style="color:#bf616a;">allocation</span><span>| { </span><span> </span><span style="color:#b48ead;">let </span><span>(start_block, end_block) = allocation.</span><span style="color:#96b5b4;">get_range</span><span>(); </span><span> (next_space_to_chk < start_block && end_check < start_block) || (next_space_to_chk > end_block && end_check > end_block) </span><span> }); </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> space_found { </span><span> </span><span style="color:#b48ead;">return </span><span>Some(next_space_to_chk); </span><span> } </span><span> next_space_to_chk = next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(align); </span><span> } </span><span>} </span></code></pre> <p>This new function reduces some of the duplication I had earlier like both <code>start_check</code> and <code>next_space_to_chk</code> existing which was confusing. This function also works with no UB which would suggest that it is sound. I’ve also changed the logic for working out whether we overlap - either the whole block needs to be before the start, or after the end which makes more sense to me.</p> <p>The low-hanging fruit for optimisation is to change the <code>all</code> call to a <code>find</code> call so we know which block we overlapped, to reduce the number of iterations.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> overlapping_block = current_allocations.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">flatten</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">allocation</span><span>| { </span><span> </span><span style="color:#b48ead;">let </span><span>(start_block, end_block) = allocation.</span><span style="color:#96b5b4;">get_range</span><span>(); </span><span> !((next_space_to_chk < start_block && end_check < start_block) || (next_space_to_chk > end_block && end_check > end_block)) </span><span>}); </span><span> </span><span style="color:#b48ead;">match</span><span> overlapping_block { </span><span> Some(block) => { </span><span> </span><span style="color:#b48ead;">let</span><span> end = block.</span><span style="color:#96b5b4;">get_range</span><span>().</span><span style="color:#d08770;">1</span><span>; </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> delta = end as </span><span style="color:#b48ead;">usize </span><span>- next_space_to_chk as </span><span style="color:#b48ead;">usize</span><span>; </span><span> </span><span style="color:#b48ead;">let</span><span> aligns = delta / align + </span><span style="color:#d08770;">1</span><span>; </span><span> next_space_to_chk = next_space_to_chk.</span><span style="color:#96b5b4;">add</span><span>(align * aligns); </span><span> }, </span><span> None => </span><span style="color:#b48ead;">return </span><span>Some(next_space_to_chk) </span><span>} </span></code></pre> <p>And that’s us done! Final code can be seen <a href="https://gist.github.com/BurntNail/86463774717f1de59d76693e62e768d2">here</a>.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>I can now technically say that I’ve built an allocator - more than likely one of the worst ones you’ve ever seen, but one that does work <em>just about</em>. There are however a few problems with it right now:</p> <ul> <li>Because I don’t really know how to get more memory on the fly<sup class="footnote-reference" id="fr-110-1"><a href="#fn-110">12</a></sup>, all of the memory in this allocator exists for the entire program which means that you have to know how much memory you are going to need at compile-time which is pretty annoying. That’s why in the <code>gist</code> I just give it a gigabyte<sup class="footnote-reference" id="fr-120-1"><a href="#fn-120">13</a></sup> of RAM to make that a problem that I don’t have to deal with.</li> <li>This allocator just tries to find the first space it can which would result in pretty bad heap fragmentation in a longer running program. This is where the heap has more than enough space to store what you’re trying to allocate but not contiguously because of bad planning. For example, let’s say we have a 10 byte heap. We allocate 2 <code>u32</code>s in the first 8 bytes. We then deallocate the first <code>u32</code>, and allocate a <code>u8</code>. That then goes in the first free space at the very start. We now have 5 bytes allocated out of 10, but nowhere to put another <code>u32</code> because our heap is fragmented.</li> <li>Because we’re not getting a heap in a way that <code>miri</code> understands, <code>miri</code> and other similar tools cannot give us any help with pointers that aren’t freed - if we put a quick <code>let ptr = unsafe { alloc(Layout::new::<u8>()) };</code>, and then never free that we won’t have any way to know except for careful program analysis, which as we all know is <a href="https://www.cvedetails.com/vulnerability-list/opmemc-1/memory-corruption.html">100% effective</a>. It also won’t warn us for use-after-frees. It is still a useful tool for things like concurrent accesses, but not much more than that.</li> </ul> <p>That pretty much sums up my thoughts on the topic. I’ve gotta also say that I’ve loved doing this in Rust - the only runtime error that took me forever to figure out was the UB right at the end. This was also a really interesting project for me, because I got to take a deep look at a thing that I haven’t really even thought about before and that I usually just take for granted.</p> <p>As with my other memory-related post, I’m happy to acknowledge that I’m not an all-knowing Rust God and if I’ll happily add corrections with credit. All I can say is that I’ve tried to ensure that the code is sound.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>Yes, I know that because we’re using our own allocator piecemeal that we are allowed to allocate and deal with allocating stuff. However, firstly this is good practice for later when we actually want to allocate everything later. Also - <a href="https://doc.rust-lang.org/std/sync/struct.Arc.html#method.new"><code>Arc::new</code></a> isn’t <code>const</code>, so we can’t create it in a static context. Yes, we could just create the allocator at the top of main, but then we go back to the first point. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-5"> <p>No guarantees are made about the contents of memory from <code>alloc</code>, but if you need pre-zeroed memory one of the provided functions for <code>GlobalAlloc</code> is <a href="https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html#method.alloc_zeroed"><code>GlobalAlloc::alloc_zeroed</code></a>, which just gets the memory and writes zeroes to it if the pointer wasn’t null. <a href="#fr-5-1">↩</a></p> </li> <li id="fn-20"> <p>How do I know? I was pretty sure it did, and then I added a quick <code>println!</code> to the top of <code>Allocator::alloc</code> to confirm. <a href="#fr-20-1">↩</a></p> </li> <li id="fn-30"> <p><code>Stdout::write</code> also hits a recursion error, so not it’s not that. I just didn’t want to deal with partial writes here. <a href="#fr-30-1">↩</a></p> </li> <li id="fn-40"> <p>Unfortunately for my <a href="https://en.wikipedia.org/wiki/Not_invented_here">NIH</a>, not for any actual reason. For actual projects though, I’m more than happy to add dependencies. <a href="#fr-40-1">↩</a></p> </li> <li id="fn-50"> <p>He creates an array by <code>[0; 1024]</code>, then creates a <code>Cursor</code> holding the array which implements <code>std::io::Write</code>, so he can then use <code>serde_json::to_writer</code>. Then, he can get the end of the cursor to create a slice for all of the characters of the array that actually matter. <a href="#fr-50-1">↩</a></p> </li> <li id="fn-60"> <p>I can’t use <code>std::thread::current().id().as_u64()</code> as that allocates. It’s also unstable right now, and this article is about stable Rust. I also can’t just use <code>libc::pthread_self()</code> because windows. <a href="#fr-60-1">↩</a></p> </li> <li id="fn-70"> <p>Just to clarify - this is when multiple threads try to concurrently store data in the same memory address. The problem is that which thread writes first is non-deterministic outside of <code>Atomic</code> operations, which we aren’t using here (because Rust only has a few built-in <code>Atomic types</code>, which wouldn’t cover a 1 million byte array). <a href="#fr-70-1">↩</a></p> </li> <li id="fn-80"> <p>This should really have been the first step, but I was already looking for an excuse to print without allocating. <a href="#fr-80-1">↩</a></p> </li> <li id="fn-90"> <p>It’s also relatively easy to test this - just quickly remove the multi-threading parts, and it all works fine even under <code>miri</code>. <a href="#fr-90-1">↩</a></p> </li> <li id="fn-100"> <p>You know what they say - premature optimisation is the root of all evil - <a href="https://xkcd.com/1691/">relevant xkcd</a>. <a href="#fr-100-1">↩</a></p> </li> <li id="fn-110"> <p>Apart from just using <a href="https://doc.rust-lang.org/std/alloc/struct.System.html"><code>std::alloc::System</code></a>, but at that point you’re basically just wrapping it and probably doing a worse job. <a href="#fr-110-1">↩</a></p> </li> <li id="fn-120"> <p>Fun fact - a gigabyte is actually 1 million bytes because we’re using the <a href="https://www.nist.gov/pml/owm/metric-si-prefixes">SI prefix</a> of <em>giga</em>. If you want to talk about <code>1024 * 1024</code> bytes, you need to use a gibibyte (aka <code>GiB</code>). Similar binary prefixes also exist for smaller units like mebibyte and kibibyte. The confusion comes where people mess them up (sometimes accidentally, and sometimes <em>accidentally</em>) and you get hard drives seeming smaller than they actually are and other similar problems. <a href="#fr-120-1">↩</a></p> </li> </ol> </section> Adventures in Binary Serialisation Sun, 07 Jan 2024 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/binary-serialisation/ https://maguire.tech/posts/binary-serialisation/ <p>The other day I came across an interesting question<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup> - instead of faffing around with <a href="https://lib.rs/crates/serde"><code>serde</code></a> and dealing with the compile-time hits, can I just write the binary data from RAM to a file? This is just an exploration of that problem. I thought this would be simple, but I ended up finding some interesting UB and digging into how Vec allocates just a little bit.</p> <p>If you find any factual errors with this article - I’ll happily take responsibility, edit it and provide appropriate credit. I’ve tried my best and that’s kinda all I can give. See contact details <a href="https://maguire.tech/">here</a>.</p> <p>Because, at least for now, this is just an experience in having fun I’ll be using whatever structs please me<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">2</a></sup> without any real focus on real-world usage. I’ll also be expecting some intermediate to advanced knowledge on programming because I’ll be skipping over some basic explanations. This will also be specific to Rust and other manual-memory management languages.</p> <h2 id="plain-old-data">Plain-Old-Data</h2> <p>Firstly, I have to work out some data to store, so I’ll start with a simple struct:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">struct </span><span>ExampleData { </span><span> </span><span style="color:#bf616a;">a</span><span>: </span><span style="color:#b48ead;">u8</span><span>, </span><span> </span><span style="color:#bf616a;">b</span><span>: </span><span style="color:#b48ead;">u64</span><span>, </span><span> </span><span style="color:#bf616a;">c</span><span>: [</span><span style="color:#b48ead;">u8</span><span>; 7], </span><span> </span><span style="color:#bf616a;">d</span><span>: </span><span style="color:#b48ead;">u32</span><span>, </span><span>} </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">example_data </span><span>() -> ExampleData { </span><span> ExampleData { </span><span> a: </span><span style="color:#d08770;">0</span><span>, </span><span> b: </span><span style="color:#b48ead;">u64</span><span>::</span><span style="color:#d08770;">MAX</span><span>, </span><span> c: [</span><span style="color:#d08770;">10</span><span>, </span><span style="color:#d08770;">11</span><span>, </span><span style="color:#d08770;">12</span><span>, </span><span style="color:#d08770;">13</span><span>, </span><span style="color:#d08770;">14</span><span>, </span><span style="color:#d08770;">15</span><span>, </span><span style="color:#d08770;">16</span><span>], </span><span> d: </span><span style="color:#d08770;">123456789 </span><span> } </span><span>} </span></code></pre> <p>I’ve carefully chosen the types and their positions (but I’ll get more into that later), but all that matters for now is that they’re all stored on the stack<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">3</a></sup>.</p> <h3 id="serialising">Serialising</h3> <p>Guess what? We’re all done here, because the compiler does all the work! The struct must be somewhere in memory and we just need to work out how to trick the compiler into letting us see the struct as just a series of bytes.</p> <p>After some real soul-searching<sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">4</a></sup>, I managed to find this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">unsafe fn </span><span style="color:#8fa1b3;">any_as_u8_slice</span><span><T: Sized>(</span><span style="color:#bf616a;">p</span><span>: &T) -> &[</span><span style="color:#b48ead;">u8</span><span>] { </span><span> ::core::slice::from_raw_parts( </span><span> (p as </span><span style="color:#b48ead;">*const</span><span> T) as </span><span style="color:#b48ead;">*const u8</span><span>, </span><span> ::core::mem::size_of::<T>(), </span><span> ) </span><span>} </span></code></pre> <p>This takes a reference to a Rust object (which has a finite and known-at-compile-time size), and creates a slice of bytes starting at the address of the reference with the length being the length of the struct. This looks sound to me as we know that the slice will only be valid for as long as the struct (because of lifetimes which are elided here), and we’re only accessing memory that we know belongs to the struct.</p> <p>If we just print this to stdout, we see something really interesting that rust does that lots of other languages don’t:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>[</span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">255</span><span>, </span><span style="color:#d08770;">21</span><span>, </span><span style="color:#d08770;">205</span><span>, </span><span style="color:#d08770;">91</span><span>, </span><span style="color:#d08770;">7</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">10</span><span>, </span><span style="color:#d08770;">11</span><span>, </span><span style="color:#d08770;">12</span><span>, </span><span style="color:#d08770;">13</span><span>, </span><span style="color:#d08770;">14</span><span>, </span><span style="color:#d08770;">15</span><span>, </span><span style="color:#d08770;">16</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>, </span><span style="color:#d08770;">0</span><span>] </span></code></pre> <p>I deliberately picked some quite distinctive values for the example data - we can see the 8 <code>255</code>s for the <code>u64::MAX</code>, the <code>[21, 205, 91, 7]</code> for our <code>u32</code>, the <code>0</code> for the u8, the <code>[10, 11, 12, 13, 14, 15, 16]</code> for the <code>u8</code> array. However, there are two main problems here - firstly, the struct’s been reordered and there’s also an extra 4 <code>0</code>s on the end?</p> <p>The first is a consequence of Rust not having a stable ABI - that is to say that the compiler reserves every right to screw around with the internal representation of anything and everything if it works out it can pack it more efficiently. The second is a consequence of padding - unless we tell it otherwise, the compiler always tries to <em>align</em> the length of the struct to the <del><code>word length</code> of the PC running the code (here, 8 bytes for a 64-byte word)</del> <em>correction: alignment is actually usually determined by the maximum length of variables inside - here, a u64 so 8 bytes</em>. This just allows reads to be far more efficient for reasons I’m not 100% sure on - I just chalk it up to one of those things that we deal with in exchange for <em><strong>lightning rock magic</strong></em>. <em>correction: some CPU instructions also require alignment for being able to work, not just performance reasons</em></p> <p>We can fix the first by adding a little <code>#[repr(C)]</code> to just before our struct declaration which tells the compiler to follow the C style which preserves ordering. However, this then balloons our struct to 32 bytes rather than just 24. The only reason you’d do so would be if you were trying to do some form of <code>FFI</code>, but there are far better solutions.</p> <p>However, if you run this through <a href="https://github.com/rust-lang/miri"><code>miri</code></a>, you’ll see that this function exhibits undefined behaviour. This is because we’re accessing uninitialised memory. At first, you might be confused as to why - it’s the padding. Padding is uninitialised memory and so this function is UB (I’ll discuss UB more later, don’t worry). To fix this, we can instead <code>#[repr(packed)]</code>, which removes all padding at a possible cost to performance. That having been said, for <em>future learning opportunities</em><sup class="footnote-reference" id="fr-4-1"><a href="#fn-4">5</a></sup>, we’ll pretend that the struct isn’t packed for the rest of this article.</p> <p>For now, I’ll write those bytes to a file.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::{io::Write, fs::File}; </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main </span><span>() -> std::io::Result<()> { </span><span> </span><span style="color:#b48ead;">let</span><span> eg = </span><span style="color:#96b5b4;">example_data</span><span>(); </span><span> </span><span style="color:#b48ead;">let</span><span> bytes = </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span style="color:#96b5b4;">any_as_u8_slice</span><span>(&eg) }; </span><span> File::create("</span><span style="color:#a3be8c;">out.bin</span><span>")?.</span><span style="color:#96b5b4;">write_all</span><span>(&bytes)?; </span><span> </span><span> </span><span> Ok(()) </span><span>} </span></code></pre> <h3 id="deserialising">Deserialising</h3> <p>Then, on this end, I can read those bytes back in to prove I’m using different bytes.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::io::Read; </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">read_in </span><span>() -> std::io::Result<ExampleData> { </span><span> </span><span style="color:#b48ead;">let mut</span><span> bytes = vec![]; </span><span> </span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">SIZE</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#d08770;">16</span><span>; </span><span> </span><span style="color:#b48ead;">let mut</span><span> reader = File::open("</span><span style="color:#a3be8c;">out.bin</span><span>")?; </span><span> </span><span style="color:#b48ead;">let mut</span><span> tmp = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">SIZE</span><span>]; </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">match</span><span> reader.</span><span style="color:#96b5b4;">read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> tmp)? { </span><span> </span><span style="color:#d08770;">0 </span><span>=> </span><span style="color:#b48ead;">break</span><span>, </span><span> n => { </span><span> bytes.</span><span style="color:#96b5b4;">extend</span><span>(&tmp[</span><span style="color:#d08770;">0</span><span>..n]); </span><span> } </span><span> } </span><span> } </span><span> </span><span> todo!("</span><span style="color:#a3be8c;">What now?</span><span>") </span><span>} </span></code></pre> <p>I now have a vector full of data which consists of a pointer to the beginning of the data and the length of that data. I now need to do the reverse of what I did before but this is not nearly as easy.</p> <h3 id="raw-pointer-dereference">Raw pointer dereference</h3> <p>My first instinct here is just to dereference the pointer like this:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> ptr = bytes.</span><span style="color:#96b5b4;">as_ptr</span><span>() as </span><span style="color:#b48ead;">*const</span><span> ExampleData; </span><span style="color:#b48ead;">let</span><span> example = </span><span style="color:#b48ead;">unsafe </span><span>{ *ptr }; </span></code></pre> <p>We just have to cast it from a <code>*const u8</code> to a <code>*const ExampleData</code>, which is easy.</p> <p>The first problem comes from the fact that we can’t dereference the pointer because we’d need to copy out the data. We can fix that with a quick <code>#[derive(Copy, Clone)]</code>. This then appears to work, until we run it through <code>miri</code> and see that the alignment is incorrect and this is undefined behaviour - we’ll have the same problem with the next method so I’ll dive deeper into it there.</p> <h3 id="boxing">Boxing</h3> <p>Let’s delete the derive and try again by thinking about how we deal with pointers in Rust.</p> <p>Well, we typically do that with a Box, which holds a pointer to a struct on the heap. We usually make one of these with <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.new"><code>Box::new</code></a> to get it from the stack to the heap, but if the data is already on the heap, we can use <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.from_raw"><code>Box::from_raw</code></a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> boxed: Box<ExampleData> = </span><span style="color:#b48ead;">unsafe </span><span>{ Box::from_raw(bytes.</span><span style="color:#96b5b4;">as_mut_ptr</span><span>() as </span><span style="color:#b48ead;">*mut</span><span> ExampleData) }; </span><span style="color:#b48ead;">return </span><span>Ok(*boxed); </span></code></pre> <p>This is very similar to the way we made a slice earlier, but here we don’t need to specify the size as <code>ExampleData</code> has a constant size. This then also works just fine, because we can dereference the box without needing copy and we’ve got the struct back out from the heap. 🥳🥳</p> <p>However, if you run this just with <code>cargo run</code>, you’ll see a problem. You won’t see a lovely <code>0</code> exit code, and instead we get this: <code>error: process didn't exit successfully: 'target\debug\ser.exe' (exit code: 0xc0000374, STATUS_HEAP_CORRUPTION)</code>. This implies that we’ve somehow corrupted the heap, so let’s have a gander at the documentation.</p> <p>The Rust standard library documentation is incredible for having thorough safety discussions and help for unsafe functions. If we look <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.from_raw">back at the function documentation</a>, we can see that it has this to say:</p> <blockquote> <p>Constructs a box from a raw pointer.</p> <p>After calling this function, the raw pointer is owned by the resulting Box. Specifically, the Box destructor will call the destructor of T and free the > allocated memory. For this to be safe, the memory must have been allocated in accordance with the memory layout used by Box.</p> <p><strong>Safety</strong></p> <p>This function is unsafe because improper use may lead to memory problems. For example, a double-free may occur if the function is called twice on the same > raw pointer.</p> </blockquote> <p>Here, we can now see our error - a classic double-free. When the function exits, <code>bytes</code> will get freed. Since <code>*boxed</code> goes back to <code>main</code>, it’ll get freed at the end there and we’ve now freed the same memory twice which corrupts the heap. Luckily, the fix here is relatively simple - we just need to add a strategic <a href="https://doc.rust-lang.org/std/mem/fn.forget.html"><code>std::mem::forget</code></a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>bytes.</span><span style="color:#96b5b4;">shrink_to_fit</span><span>(); </span><span style="color:#b48ead;">let</span><span> boxed: Box<ExampleData> = </span><span style="color:#b48ead;">unsafe </span><span>{ Box::from_raw(bytes.</span><span style="color:#96b5b4;">as_mut_ptr</span><span>() as </span><span style="color:#b48ead;">*mut</span><span> ExampleData) }; </span><span>std::mem::forget(bytes); </span><span style="color:#b48ead;">return </span><span>Ok(*boxed); </span></code></pre> <p><a href="https://doc.rust-lang.org/std/mem/fn.forget.html"><code>std::mem::forget</code></a> basically just tells rust specifically not to run the destructor of the object you give it, therefore not freeing the memory. Here, we use it to make sure that <code>bytes</code> never gets freed. We don’t need to worry about the memory getting leaked because it then gets freed when <code>boxed</code> gets dropped. I’ve also added a <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html#method.shrink_to_fit"><code>Vec::shrink_to_fit</code></a> call to ensure that the spare memory we likely allocated for the Vec doesn’t get lost.<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">6</a></sup>.</p> <p><em>correction: the Vec’s allocation may be reused which would then be UB. Instead, use <code>bytes.leak()</code></em>.</p> <h3 id="ub">UB?</h3> <p>This all seems to work, but if we run it through <code>miri</code>, we can see that this is UB. Specifically: <code>error: Undefined Behavior: constructing invalid value: encountered an unaligned box (required 8 byte alignment but found 2)</code>, which happens when we create the <code>Box</code> with <code>Box::from_raw</code>. Here, we get to alignment issues. I briefly wrote about alignment earlier, and the part that matters here is that alignment is inherent to a specific struct. In fact, if we look at the <a href="https://doc.rust-lang.org/std/alloc/fn.alloc.html"><code>alloc</code></a> and <a href="https://doc.rust-lang.org/std/alloc/fn.alloc.html"><code>dealloc</code></a> methods they take a <a href="https://doc.rust-lang.org/std/alloc/struct.Layout.html"><code>Layout</code></a> which stores size and alignment, unlike C’s <a href="https://man7.org/linux/man-pages/man3/malloc.3.html"><code>malloc</code> and <code>free</code></a> which just take a size.</p> <p>If we look at what behaviour is considered undefined (helpfully <a href="https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html">linked</a> to me by <code>miri</code>), we can see (in the second item) that accessing a misaligned pointer is undefined behaviour. I’ll quickly explain for people coming from C the danger of undefined behaviour in Rust. In C, there are lots of behaviours that people use that aren’t specifically designated by the specification (like integer over/under-flow) which are undefined behaviour. They’re mostly fine there, but Rust UB is a different beast entirely because of how heavily the compiler tries to optimise code. Generally in Rust, no UB is good UB, and if you wrote UB then the compiler reserves the right (and often will, especially in release builds) to do whatever the hell it wants regardless of your intentions. Here, it kinda works but in a larger program <strong>nobody</strong> can know what the compiler will do.</p> <p><strong>correction</strong>: <em>C UB is also very very bad, and I think I just got confused between undefined behaviour and specification-undefined behaviour. the example I mentioned is also wrong - you can’t rely on signed integer over/under-flow, especially in release builds</em></p> <h4 id="alignment">Alignment</h4> <p>To actually explain the issue, we need to look at the <code>Layout</code>’s of <code>Vec</code> and of <code>ExampleData</code>, which is easy because <code>Layout</code> implements <code>Debug</code> which means we can print it out to get all of its fields.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>std::alloc::Layout; </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_layouts </span><span>() { </span><span> println!("</span><span style="color:#a3be8c;">Vec<u8>: </span><span style="color:#d08770;">{:?}</span><span>", Layout::new<Vec<</span><span style="color:#b48ead;">u8</span><span>>>()); </span><span> println!("</span><span style="color:#a3be8c;">ExampleData: </span><span style="color:#d08770;">{:?}</span><span>", Layout::new::<ExampleData>()); </span><span>} </span></code></pre> <p>Which then outputs that they’re the same?</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>Vec<</span><span style="color:#b48ead;">u8</span><span>>: Layout { size: </span><span style="color:#d08770;">24</span><span>, align: </span><span style="color:#d08770;">8 </span><span>(</span><span style="color:#d08770;">1 </span><span><< </span><span style="color:#d08770;">3</span><span>) } </span><span>ExampleData: Layout { size: </span><span style="color:#d08770;">24</span><span>, align: </span><span style="color:#d08770;">8 </span><span>(</span><span style="color:#d08770;">1 </span><span><< </span><span style="color:#d08770;">3</span><span>) } </span></code></pre> <h4 id="the-same">The Same?</h4> <p>This definitely stumped me for a while until I decided to have a look at the documentation & source for <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html"><code>std::vec::Vec</code></a>, and then by clicking on <em>Source</em> I saw that it holds a <code>RawVec</code> buffer and the current length. I’ve added the definition below, simplified to remove the standard library shenanigans:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>Vec<T, A: Allocator = Global> { </span><span> </span><span style="color:#bf616a;">buf</span><span>: RawVec<T, A>, </span><span> </span><span style="color:#bf616a;">len</span><span>: </span><span style="color:#b48ead;">usize</span><span>, </span><span>} </span></code></pre> <p><code>RawVec</code> is a private class that <code>Vec</code> uses to wrap lots of the unsafe behaviours, and you can’t actually see it in any documentation. However, you can just click around in the source to find it <a href="https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#51">here</a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub</span><span>(</span><span style="color:#b48ead;">crate</span><span>) </span><span style="color:#b48ead;">struct </span><span>RawVec<T, A: Allocator = Global> { </span><span> </span><span style="color:#bf616a;">ptr</span><span>: Unique<T>, </span><span> </span><span style="color:#bf616a;">cap</span><span>: </span><span style="color:#b48ead;">usize</span><span>, </span><span> </span><span style="color:#bf616a;">alloc</span><span>: A, </span><span>} </span></code></pre> <p><code>alloc</code> is a zero-sized type, and then both <code>cap</code> and <code>ptr</code> have size <code>usize</code> which is 8 bytes. If we add up all of the bytes and check our alignments we can then see that yeah it’s 24 bytes with 8 byte-alignment.</p> <p>To see where our alignment problem from earlier comes from, we have to dig into <a href="https://doc.rust-lang.org/src/alloc/vec/mod.rs.html#1880"><code>Vec::push</code></a>:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">push</span><span>(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">value</span><span>: T) { </span><span> </span><span style="color:#b48ead;">if </span><span style="color:#bf616a;">self</span><span>.len == </span><span style="color:#bf616a;">self</span><span>.buf.</span><span style="color:#96b5b4;">capacity</span><span>() { </span><span> </span><span style="color:#bf616a;">self</span><span>.buf.</span><span style="color:#96b5b4;">reserve_for_push</span><span>(</span><span style="color:#bf616a;">self</span><span>.len); </span><span> } </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span> </span><span style="color:#b48ead;">let</span><span> end = </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">as_mut_ptr</span><span>().</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#bf616a;">self</span><span>.len); </span><span> ptr::write(end, value); </span><span> </span><span style="color:#bf616a;">self</span><span>.len += </span><span style="color:#d08770;">1</span><span>; </span><span> } </span><span>} </span></code></pre> <p>We can see that it checks for capacity (and reserves space if we don’t have enough which I’ll get to in just a second), and it then writes to a pointer with the new value. That call for reserving is on <code>self.buf</code>, which is the <code>RawVec</code> from earlier which handles all of the allocation so it’s probably <a href="https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#301">where</a> we want to be looking.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">reserve_for_push</span><span>(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">len</span><span>: </span><span style="color:#b48ead;">usize</span><span>) { </span><span> </span><span style="color:#96b5b4;">handle_reserve</span><span>(</span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">grow_amortized</span><span>(len, </span><span style="color:#d08770;">1</span><span>)); </span><span>} </span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">grow_amortized</span><span>(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">len</span><span>: </span><span style="color:#b48ead;">usize</span><span>, </span><span style="color:#bf616a;">additional</span><span>: </span><span style="color:#b48ead;">usize</span><span>) -> Result<(), TryReserveError> { </span><span> debug_assert!(additional > </span><span style="color:#d08770;">0</span><span>); </span><span> </span><span style="color:#b48ead;">if </span><span>T::</span><span style="color:#d08770;">IS_ZST </span><span>{ </span><span> </span><span style="color:#b48ead;">return </span><span>Err(CapacityOverflow.</span><span style="color:#96b5b4;">into</span><span>()); </span><span> } </span><span> </span><span style="color:#b48ead;">let</span><span> required_cap = len.</span><span style="color:#96b5b4;">checked_add</span><span>(additional).</span><span style="color:#96b5b4;">ok_or</span><span>(CapacityOverflow)?; </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> cap = cmp::max(</span><span style="color:#bf616a;">self</span><span>.cap * </span><span style="color:#d08770;">2</span><span>, required_cap); </span><span> </span><span style="color:#b48ead;">let</span><span> cap = cmp::max(</span><span style="color:#b48ead;">Self</span><span>::</span><span style="color:#d08770;">MIN_NON_ZERO_CAP</span><span>, cap); </span><span> </span><span style="color:#b48ead;">let</span><span> new_layout = Layout::array::<T>(cap); </span><span> </span><span> </span><span style="color:#b48ead;">let</span><span> ptr = </span><span style="color:#96b5b4;">finish_grow</span><span>(new_layout, </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">current_memory</span><span>(), &</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>.alloc)?; </span><span> </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">set_ptr_and_cap</span><span>(ptr, cap); </span><span> Ok(()) </span><span>} </span></code></pre> <p><a href="https://doc.rust-lang.org/src/alloc/raw_vec.rs.html#393"><code>handle_reserve</code></a> just deals with errors, and if we peer through <code>grow_amortized</code> which actually allocates (and strategically ignore the amortisation parts and just look for a <code>Layout</code>), we can see that it actually allocates using <code>Layout::array<T>(cap)</code>. If we then print out that layout (using the number of bytes we need for storing an <code>ExampleData</code> for the array size):</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">print_layouts </span><span>() { </span><span> </span><span style="color:#b48ead;">let</span><span> size = std::mem::size_of::<ExampleData>(); </span><span> println!("</span><span style="color:#a3be8c;">Vec<u8>: </span><span style="color:#d08770;">{:?}</span><span>", Layout::array::<</span><span style="color:#b48ead;">u8</span><span>>(size).</span><span style="color:#96b5b4;">unwrap</span><span>()); </span><span> println!("</span><span style="color:#a3be8c;">ExampleData: </span><span style="color:#d08770;">{:?}</span><span>", Layout::new::<ExampleData>()); </span><span>} </span></code></pre> <p>We get this out the other end - success! The alignment is different which then explains why our <code>Box</code> creation earlier was undefined behaviour.</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>Vec<</span><span style="color:#b48ead;">u8</span><span>>: Layout { size: </span><span style="color:#d08770;">24</span><span>, align: </span><span style="color:#d08770;">1 </span><span>(</span><span style="color:#d08770;">1 </span><span><< </span><span style="color:#d08770;">0</span><span>) } </span><span>ExampleData: Layout { size: </span><span style="color:#d08770;">24</span><span>, align: </span><span style="color:#d08770;">8 </span><span>(</span><span style="color:#d08770;">1 </span><span><< </span><span style="color:#d08770;">3</span><span>) } </span></code></pre> <p>The obvious next question is whether there’s any way to convert between these two without copying (since that’s why we’ve gotten into this whole mess). That having been said, I’ll take a brief aside to explain how to do this if you’re OK with allocating another 24 bytes.</p> <h3 id="copying-again-but-without-copy-needed-nor-any-ub">Copying again <em>(but without <code>Copy</code> needed nor any UB)</em></h3> <p>The following code (according to <code>miri</code> and my own gut feeling<sup class="footnote-reference" id="fr-6-1"><a href="#fn-6">7</a></sup>) is unsafe, but sound (meaning that it doesn’t cause UB):</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">return </span><span>Ok(</span><span style="color:#b48ead;">unsafe </span><span>{ </span><span> </span><span style="color:#b48ead;">let mut</span><span> ptr = std::alloc::alloc(Layout::new::<ExampleData>()); </span><span> </span><span> </span><span style="color:#b48ead;">for</span><span> b in bytes { </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ ptr.</span><span style="color:#96b5b4;">write</span><span>(b) }; </span><span> ptr = ptr.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#d08770;">1</span><span>); </span><span> } </span><span> </span><span> ptr = ptr.</span><span style="color:#96b5b4;">sub</span><span>(std::mem::size_of::<ExampleData>()); </span><span> *Box::from_raw(ptr as </span><span style="color:#b48ead;">*mut</span><span> ExampleData) </span><span>}); </span></code></pre> <p>We create a pointer using <code>alloc</code> with the correct layout. We then write each byte and increment the address of the pointer each time. We then take the pointer back to the start and give it to a <code>Box</code> to be managed. That <code>Box</code> is then responsible for freeing the heap memory we allocated and we also use it to get the data out from the heap back to the stack. Don’t ask me how it does that - I can’t tell for the life of me. There’s probably a better way to do it, but I can at least say that this won’t cause memory leaks, memory corruption or compiler mayhem.</p> <h3 id="is-there-a-no-copy-solution">Is there a no-copy solution?</h3> <p>As far as I can tell, there isn’t a way to tell the Rust compiler that we want to change the alignment of an existing allocation, which then means that (unless the struct has 1-byte alignment) there isn’t a way to do a no-copy solution which annoys me. If you work out a way of doing it - I’d love to hear it.</p> <p>Unlike the first copying method which required the struct to be <code>Copy</code> and was UB unless the struct was packed, this doesn’t need that <strong>and</strong> as far as I can tell, it’s sound.</p> <h2 id="heap-data">Heap Data</h2> <p>That’s all well and good, but what about Heap data like a <code>Vector</code> or a <code>String</code>?</p> <p>By itself, you’d have to know at compile-time the type, but you could probably get away with just writing the contents of the pointer, and carefully reading them back. However, you’d have to be careful around cases where more space was allocated than used which would be annoying.</p> <p>The main complications would come from if that heap data was stored inside a struct. You’d no longer be able to just pretend it was all just a series of bytes, and you’d have to put in lots of manual work to not accidentally just write a memory address that would become invalid the second the program finished<sup class="footnote-reference" id="fr-7-1"><a href="#fn-7">8</a></sup>, and to read back the correct number of bytes. At that point, you’re better off just using something like <a href="https://protobuf.dev/"><code>protobuf</code></a> or <code>serde</code>.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>Whilst yes, this does work I would never recommend it for any production use where you couldn’t control it <em>very</em> finely. The main problem comes from what I briefly mentioned earlier about the Rust ABI not being stable. If the compiler updates, even if you don’t change your code, the in-memory representation of your struct could change and then you’d never have any way of recovering that written data other than downgrading your compiler version to ‘before it broke’. I’d also be careful about manually writing bytes by hand instead of outputting an existing struct - Rust does some really interesting optimisations which could lead to bit patterns being something completely different to what you expected - like <code>Option<bool></code> which only takes 1 byte. It’s also undefined behaviour for certain types to be in invalid bit states (eg. 0x0 and 0x1 are fine for a <code>bool</code>, but 0x3 is invalid) because they’re often used for those optimisations.</p> <p>As to why the UB never actually caused any issues, I’ve not a clue. There’s probably someone who could explain more but I’m not that person. Reading padding is undefined behaviour, and I couldn’t say why it didn’t cause any issues here.</p> <p>Whilst you could make arguments around the use of <code>#[repr(C)]</code> to safeguard against this, there’s another problem - there’s no description of what this data is if anyone’s poking around the filesystem. Most file formats start with something to mark what format they are, but there’s nothing here if you’re trying to read data later. Even if you don’t go as far as to use a self-describing format like <code>JSON</code>, most serialisation methods (as far as I can tell) at least make it clear that they’re storing data. With this, unless you’re a rare programmer who actually writes thorough documentation, I can’t even tell if its a binary or a file.</p> <p>I’ll finish off by saying that this article was also a bit <a href="https://en.wikipedia.org/wiki/Not_invented_here">NIH</a> and if you want to actually do things like this, I’d have a gander at <a href="https://lib.rs/crates/bytemuck"><code>bytemuck</code></a>.</p> <p>Anyways, that was fun to explore and try and get a better knowledge of UB and binary formats. Enjoy the new year!</p> <p>Full final code can be viewed <a href="https://gist.github.com/BurntNail/b46b5ecc65d5fdfb9097de2d01ad1966">here</a>.</p> <h2 id="errata"><em>errata</em></h2> <p>I posted a link to this on the subreddit <a href="https://www.reddit.com/r/rust/comments/1913hsh/some_fun_adventures_ive_had_in_ub_and_internal/">here</a> and received some comments - here are my responses.</p> <h3 id="u-dkopgerpgdolfg">u/dkopgerpgdolfg</h3> <p>This user pointed out one thing which makes me feel like a fool - if we know the length of our struct beforehand (which we do - it’s <code>std::mem::size_of::<ExampleData>()</code>), we can just read straight into the pointer:</p> <pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">read_in_straight_to_pointer </span><span>(</span><span style="color:#bf616a;">file_name</span><span>: &</span><span style="color:#b48ead;">str</span><span>) -> std::io::Result<Option<ExampleData>> { </span><span> </span><span style="color:#b48ead;">unsafe </span><span>{ </span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">STRUCT_SIZE</span><span>: </span><span style="color:#b48ead;">usize </span><span>= std::mem::size_of::<ExampleData>(); </span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">LAYOUT</span><span>: Layout = Layout::new::<ExampleData>(); </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> ptr = std::alloc::alloc(</span><span style="color:#d08770;">LAYOUT</span><span>); </span><span style="color:#65737e;">//create a pointer to the heap struct </span><span> </span><span style="color:#b48ead;">let mut</span><span> bytes_read_total = </span><span style="color:#d08770;">0</span><span>; </span><span style="color:#65737e;">//create a counter for how many bytes we've written - this is like the `len` of a Vec, where `STRUCT_SIZE` is the `cap` </span><span> </span><span> </span><span style="color:#b48ead;">let mut</span><span> reader = File::open(file_name)?; </span><span style="color:#65737e;">//open a file </span><span> </span><span style="color:#b48ead;">let mut</span><span> tmp = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">32</span><span>]; </span><span style="color:#65737e;">//create a stack for temporary reads - very little performance impact </span><span> </span><span> </span><span style="color:#b48ead;">loop </span><span>{ </span><span> </span><span style="color:#b48ead;">match</span><span> reader.</span><span style="color:#96b5b4;">read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> tmp)? { </span><span> </span><span style="color:#d08770;">0 </span><span>=> </span><span style="color:#b48ead;">break</span><span>, </span><span> n => { </span><span> </span><span style="color:#b48ead;">let</span><span> n = n.</span><span style="color:#96b5b4;">min</span><span>(</span><span style="color:#d08770;">STRUCT_SIZE </span><span>- bytes_read_total); </span><span style="color:#65737e;">//ensure we don't write past the memory we allocated </span><span> </span><span style="color:#b48ead;">for</span><span> i in </span><span style="color:#d08770;">0</span><span>..n { </span><span> ptr.</span><span style="color:#96b5b4;">write</span><span>(tmp[i]); </span><span> ptr = ptr.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#d08770;">1</span><span>); </span><span> } </span><span> </span><span> bytes_read_total += n; </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> bytes_read_total == </span><span style="color:#d08770;">STRUCT_SIZE </span><span>{ </span><span style="color:#65737e;">//if there's more to read, we don't care </span><span> </span><span style="color:#b48ead;">break</span><span>; </span><span> } </span><span> } </span><span> } </span><span> } </span><span> </span><span> </span><span style="color:#b48ead;">if</span><span> bytes_read_total < </span><span style="color:#d08770;">STRUCT_SIZE </span><span>{ </span><span> std::alloc::dealloc(ptr, </span><span style="color:#d08770;">LAYOUT</span><span>); </span><span style="color:#65737e;">//if we didn't read enough, then we need to remember to deallocate the block, as Rust won't do it for us </span><span> Ok(None) </span><span> } </span><span style="color:#b48ead;">else </span><span>{ </span><span> ptr = ptr.</span><span style="color:#96b5b4;">sub</span><span>(</span><span style="color:#d08770;">STRUCT_SIZE</span><span>); </span><span> Ok(Some(*Box::from_raw(ptr as </span><span style="color:#b48ead;">*mut</span><span> ExampleData))) </span><span> } </span><span> } </span><span>} </span></code></pre> <p>I’ve also added corrections from them in the rest of the article relating to alignment, leaking the vec and C undefined behaviour.</p> <h3 id="u-hniksic">u/hniksic</h3> <p>I’ve added corrections relating to C undefined behaviour.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>At least, interesting to me ;) <a href="#fr-0-1">↩</a></p> </li> <li id="fn-1"> <p>and some semblance of a structure for you delightful readers. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p>That is, they’re all in memory in one place without any pointers or memory addresses. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>That is, duckduckgo-ing for StackOverflow. <a href="#fr-3-1">↩</a></p> </li> <li id="fn-4"> <p>That is, I didn’t realise this properly until I’d written far more of this article (the future bits are also super interesting). If you’ve read more of the article, this is why: if you pack the struct the alignment becomes 1 byte (because of the <code>u8</code>), which then means that it has the same alignment as the vec which makes the initial solution (using <code>Box::from_raw</code> with the <code>bytes.as_mut_ptr() as *mut ExampleData</code>) not unsound any more. <a href="#fr-4-1">↩</a></p> </li> <li id="fn-5"> <p>When the Rust vector reallocates, it doesn’t just add one to the capacity, it goes up in a pattern which is designed to amortise the allocation cost to be as low as possible. Because the <code>Box</code> only knows about the length, it won’t know to free the memory that’s in the <code>capacity</code> but not the <code>length</code>; <a href="#fr-5-1">↩</a></p> </li> <li id="fn-6"> <p>100% correct 60% of the time <a href="#fr-6-1">↩</a></p> </li> <li id="fn-7"> <p>Technically, you could modify the <code>ExampleData</code> from this post with any number of heap-based structures and it might still work. If the memory allocator didn’t de-allocate between you writing out the data (and dropping the old struct), and reading the memory address back I think it’d work? The problem would come from if you wrote on one run and read on another because that memory that it’s pointing to would no longer exist. <a href="#fr-7-1">↩</a></p> </li> </ol> </section> Comparative Thoughts on University Admissions Mon, 18 Dec 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/us-and-uk-unis/ https://maguire.tech/ramblings/us-and-uk-unis/ <p>As of writing (the 16th of December), I’ve just got back my MIT results - a deferral (which in this case means waiting until Pi Day for a more concrete result<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>). I’m now pretty confident in my thoughts on both systems of admissions, so to wait out the time I’ll be writing out my thoughts on their pros and cons. I might’ve briefly mentioned these in my <a href="https://maguire.tech/ramblings/mit-interview/">other</a> <a href="https://maguire.tech/ramblings/cambridge-admissions/">two</a> blogs on University admissions, but this’ll be one place for what I wrote there in comparison & much more. If you haven’t read the other two and aren’t familiar with the admissions systems I’d recommend doing so - there’ll be things that I’ll be referencing without explaining.</p> <p>As per usual, I’ll also mention that this post has my usual biases - I’ve been applying to a small number of highly selective universities from a position of relative privilege, so you’re experiences might have been different. I’ll try and vaguely organise it on the different stages of the admissions processes, but prepare for tangents 😉. I’ll also add that I probably didn’t perform as well on my admissions tests as I would’ve wanted to, which probably somewhat sours my opinion.</p> <h2 id="essays">Essays</h2> <p>For MIT & Caltech, I’ve had to write a number of different essays that are all relatively short<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">2</a></sup> which have been from a variety of prompts ranging from how I feel about their values to how I’ve helped and been helped by my community to why I want to study Computer Science. The main result of this is that students end up having to write <strong>a lot</strong> of essays, which I’ve heard can be incredibly draining. I’ve been somewhat insulated from this by limiting my US Admissions to only 2 universities, but I don’t really want to imagine what it must be like for the people applying to 20+ universities<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">3</a></sup>.</p> <p>Whereas in the UK System, you only write 1 essay (known as the <em>Personal Statement</em>) about yourself. 4000 characters over 47 lines<sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">4</a></sup> with no prompt. UCAS & <em>The Universities</em> expect you to write about why you want to go for the course you’ve applied for, why you’d be good at that, why you’d be good for them and anything else interesting about you.</p> <p>So here I’m pretty split. The US Essays seemed to care more about me as a person (which fits as you seem to mostly apply for a University rather than a Course) whereas my Personal Statement ended up being full of academics and waffle about how the I enjoyed the subjects I’m doing and how well they would transfer. That having been said, I got across most of what I wanted in my Personal Statement, and more of the struggle was getting it inside the character limit, rather than trying to examine my life with a microscope. I’ll also say that I think my Personal Statement came out better, purely because I could invest more time into it - it went through 11 drafts if I recall correctly because it was going to all of my UK Universities, whereas my MIT essays got far fewer drafts and so will my Caltech essays because there are more of them that go to fewer places.</p> <h2 id="admissions-tests-grade-requirements">Admissions Tests & Grade Requirements</h2> <p>MIT & Caltech both take very different approaches here - MIT is test-mandatory, which means that as well as your recommendations, essays & general performance you need to complete either the <a href="https://www.act.org/content/act/en.html">ACT</a> or the <a href="https://satsuite.collegeboard.org/sat">SAT</a>. The idea here seems to be that they can get a standardised test performance metric for all students applying<sup class="footnote-reference" id="fr-4-1"><a href="#fn-4">5</a></sup>. One interesting difference is that you can take more than one SAT or ACT and <em>‘superscore’</em> them which means to take your best scores from each portion of the tests you’ve completed (eg. I can take a 36 in Science from my September sitting and a 35 in Maths from my November sitting), which I guess allows for if students have bad days? There are two ways of discouraging students from taking endless tests - the fact that they’re insanely expensive (my last ACT test was $171.50)<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">6</a></sup> & the fact that the universities see all the tests that you submit, which makes it rather obvious. I ended up having the luxury to take two. Caltech is test-blind, which just means that they won’t even look at tests if you have submitted them.</p> <p>Whereas, in the UK you get predicted grades from your teachers based on what they think your performance in your final exam will be. Those then go to all the universities, which makes it simple. Different schools work differently for how lenient they are with students requesting to go for higher grades. The way that schools are discouraged from giving everyone higher grades is that that tactic will only work for one year, as the universities <em>supposedly</em> keep records of how accurate the different schools are. You also must apply through your school so you can’t really get around it.</p> <p>Similarly enough, there are parts I do like and don’t about both systems - here are the UK Advantages:</p> <ul> <li>All courses at all universities post minimum grade requirements<sup class="footnote-reference" id="fr-6-1"><a href="#fn-6">7</a></sup>, so there isn’t much faffing around with applying to universities you know you can’t get into. This is in opposition to the US system, where at best I have the middle 50% from MIT and have to guess if I’m good enough.</li> <li>In my humble opinion there’s less possibility for rich people just taking more and more tests to get the grades they want (ignoring the possibility of bribery<sup class="footnote-reference" id="fr-7-1"><a href="#fn-7">8</a></sup>).</li> <li>Here, you’re also being evaluated on more long-term performance - it might just be a problem for me applying to both systems at the same time (and slightly prioritising the UK due to higher acceptance chances at my choices as well as the other advantages that come with going to university domestically<sup class="footnote-reference" id="fr-8-1"><a href="#fn-8">9</a></sup>) I haven’t had as much time to revise for my ACTs but I feel that my Predicteds are more accurate.</li> </ul> <p>And then the US:</p> <ul> <li>The idea of evaluating student performance and potential for application to a pretty big university without any test scores (except for transcript related<sup class="footnote-reference" id="fr-9-1"><a href="#fn-9">10</a></sup>) is pretty nice sounding to me.</li> <li>A-Level & IB Exam Seasons are generally known to be a pretty hellish time of year for all involved - at my school the entire time between maybe mid-late February and July is all spent either revising, taking practice tests or taking actual papers. Its mixed as to whether it’s in school or out of school, but normally A-Level out-of-school revision & taking the exams is about a month and a half. As far as I can see, during this time at US High Schools you don’t do all that much?</li> <li>Whilst I’m not 100% sure about the entrance tests, I can definitely see that they are standardised which I most definitely can’t say about teachers who might be deciding your grades.</li> </ul> <p>Interesting systems across both sides of the pond there, and I’m really not sure which I prefer. Any preference I show for the UK is probably tinted by me just going with the devil I know over the devil I don’t.</p> <h2 id="interviews">Interviews</h2> <p>I’ve spoken <em>extensively</em> about interviews on both sides, so my opinion will be far shorter and simpler.</p> <p>US wins, flat-out. I get that the competitive UK Universities want an idea of aptitude rather than just accomplishment for their courses due to rampant grade inflation<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">11</a></sup>, but I feel that the US one went better because it wasn’t as stressed. In the Cambridge interview I had, I could see that the interviewers had far more candidates than time and needed to get going. Part of that is also influenced by the setting - my US one was in a nice coffee shop (Gails<sup class="footnote-reference" id="fr-11-1"><a href="#fn-11">12</a></sup> vibes at a lower cost) and my UK one was over a zoom<sup class="footnote-reference" id="fr-12-1"><a href="#fn-12">13</a></sup> which stressed me out because of the eternally-present technical issues which even a Computer Science applicant can’t overcome.</p> <h2 id="application-structure">Application Structure</h2> <p>Here, the UK wins although it is a smaller country which probably makes it easier.</p> <p>All applications here are handled through <a href="https://www.ucas.com/">UCAS</a> - you can’t apply without using it. That only means one essay, one place to dump scores, one place to add admin details, one place to look for offers which reduces lots of the overhead from US<sup class="footnote-reference" id="fr-13-1"><a href="#fn-13">14</a></sup>, and one place to pay a relatively small cost (£20 for one university, or £25 for multiple) for applications. This is less cognitive load, and only one email address to have a heart attack about when you see it (<code>track@ucas.ac.uk</code> if you’re interested, and as far as I can tell this is used <em>exclusively</em> for offers which is convenient).</p> <p>As opposed to the US where there are lots of different systems which means inputting my personal details in lots of times which isn’t too bad for 2 universities but I could imagine this getting pretty annoying. The fees are also way higher - to apply to > 13 US Universities (which doesn’t seem too high as far as I can tell), you’re looking at $1000 (going off a cost of $75, which is the MIT cost).</p> <h2 id="receiving-results">Receiving Results</h2> <p>US Universities have specific timestamps for applications decisions (eg. MIT Early Action this year was the 16th of December at 12:16PM Boston time), and I think all the Oxford and Cambridge colleges follow this with their own timings. The rest of the UK Universities (for the most part), just release their results to you whenever they decide. There isn’t really even a proper deadline - if they don’t respond before the first of May then it just counts as a rejection 🤷.</p> <p>There’s advantages and disadvantages either side, but I think the UK wins out. I was pretty stressed about my MIT decision from when I could become stressed about it although it turns out there is a limit to the number of things I can simultaneous stress out about<sup class="footnote-reference" id="fr-14-1"><a href="#fn-14">15</a></sup>! This ended up being after my Cambridge Interview. I had a look at their history and made a little spreadsheet and managed to successfully predict the date<sup class="footnote-reference" id="fr-15-1"><a href="#fn-15">16</a></sup>. That having been said, at least after that date I could theoretically know for final<sup class="footnote-reference" id="fr-16-1"><a href="#fn-16">17</a></sup>? On the other hand, since you have no idea when your UK decisions are coming in, you can theoretically convince your brain to forget and focus on revision, but this does mean that you can’t plan for a nice place to possibly receive a rejection - you just have to politely excuse yourself and hope for the best.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>Since this whole post was just an extended set of closing thoughts for the other university posts, you aren’t going to get much more out of me here. Both systems seem to have their advantages and disadvantages but I’ve been talking about this wrong. I kinda don’t really get any choice in this matter and just have to go with whatever university I want to apply to says. Regardless, in the off chance that any admissions officers read this<sup class="footnote-reference" id="fr-17-1"><a href="#fn-17">18</a></sup> (in which case, thanks for getting to the end!), feel free to take anything or nothing. I think I’ve been speaking about this enough in person, and this was a lovely chance to get to just dump the contents of my brain and get focused on revision for my A-Level Mocks.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>The odds aren’t great though - I think last year it was 146 admitted out of 7892 deferred, which is < 2%. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-1"> <p>Mostly maxing out at around 250 words, except for the Common App longer essay question. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p>This is one small benefit of UCAS imho - you are limited to 5 which somewhat reduces the admin work required to keep up with admissions. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>Why 47? What is a line? idk ask UCAS it just happened to all work out for me. <a href="#fr-3-1">↩</a></p> </li> <li id="fn-4"> <p>How do they score ACT vs SAT? Not a clue, but I’d be interested to know. <a href="#fr-4-1">↩</a></p> </li> <li id="fn-5"> <p>University applications favouring the wealthy? Never! <a href="#fr-5-1">↩</a></p> </li> <li id="fn-6"> <p>As well as information on how those get reduced for Circumstantial Offers. <a href="#fr-6-1">↩</a></p> </li> <li id="fn-7"> <p>If we allow bribery, then both systems get messed up so I’m ignoring that possibility - this is my blog post I can do what I want ^_^. <a href="#fr-7-1">↩</a></p> </li> <li id="fn-8"> <p>And I’ll be honest, there are plenty of advantages ranging from being able to see my friends & family more often, not having to worry about visas as well as the fact that going to university in the US will probably be around 5-6x more expensive. <a href="#fr-8-1">↩</a></p> </li> <li id="fn-9"> <p>Which in my case, is just predicted grades + GCSE scores. <a href="#fr-9-1">↩</a></p> </li> <li id="fn-10"> <p>One of the few areas I disagree with the grade requirements - frankly for the top universities they aren’t realistic. Technically, I’ve applied to universities that are below by multiple grades what I had predicted, but realistically I’ll be lucky to get 3 or 4 offers including my insurance. Again though, I’m applying to highly competitive universities which might mean that my experience is different. <a href="#fr-10-1">↩</a></p> </li> <li id="fn-11"> <p>Relatively posh british bakery/coffee shop chain - the food is expensive but I’ve never had any other cinnamon rolls that come even close. <a href="#fr-11-1">↩</a></p> </li> <li id="fn-12"> <p>I 100% get the reason they do this (make it easier for people who can’t get to Cambridge), but I still would have loved to visit in person. <a href="#fr-12-1">↩</a></p> </li> <li id="fn-13"> <p>Yes, I’m aware of the <a href="https://www.commonapp.org/">Common App</a>, and other such coalitions but they don’t cover <em>every</em> university. <a href="#fr-13-1">↩</a></p> </li> <li id="fn-14"> <p><em>at least consciously stress about, I fear for my heart.</em> <a href="#fr-14-1">↩</a></p> </li> <li id="fn-15"> <p>The pattern for EA seems to be the saturday of the second full week (beginning Monday). <a href="#fr-15-1">↩</a></p> </li> <li id="fn-16"> <p><em>unless you get waitlisted, in which case it’s another three months yay!11!!!11!! (although I did bring this upon myself by applying earlier, and this is also definitely better than a straight rejection)</em> <a href="#fr-16-1">↩</a></p> </li> <li id="fn-17"> <p>This is all going on my MIT <a href="https://mitadmissions.org/apply/firstyear/february-updates-notes-form/">FUN Form</a>, so the chances are non-zero I guess? <a href="#fr-17-1">↩</a></p> </li> </ol> </section> The British Uni Admissions Experience Thu, 07 Dec 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/cambridge-admissions/ https://maguire.tech/ramblings/cambridge-admissions/ <p>I’ve recently finished my Cambridge interview and I’ll now make a vague attempt to document my experience, trials and tribulations. For reference, I applied for <a href="https://www.undergraduate.study.cam.ac.uk/courses/computer-science">Computer Science</a> to <a href="https://www.chu.cam.ac.uk/">Churchill College</a>. This’ll probably end up following a similar structure to <a href="https://maguire.tech/ramblings/mit-interview/">my MIT Post</a>. This is also specifically for the 2023-2024 admissions cycle, as apparently a few things are changing next year for the Personal Statement as well as the Admissions Tests.</p> <h2 id="the-uk-system">The UK System</h2> <p>The UK System is so very different to the US System, so I’ll begin by giving a not-so-brief explanation of how our admissions processes work. I’ll begin by explaining school years, to eventually get to the important years for this discussion.</p> <h3 id="gcses-and-below">GCSEs and Below</h3> <p>In the UK, most people class schools into independent schools and state schools. Independent schools are typically selective and fee-paying, and state schools are paid for by the government and have free enrolment. Then, within state schools there’s the normal ones as well as <a href="https://en.wikipedia.org/wiki/Grammar_school">Grammar Schools</a>, which are more selective but hard to get into due to the high demand. Then, independent schools can be classed in a number of ways - day schools vs boarding schools (all boarding schools are independent as far as I know), public vs private, or even by how expensive the fees are. Quick fun fact - a Public School in the UK is very different to a Public School internationally. Public Schools (as a group noun) refer to a set of schools that were originally open to students regardless of their parents’ jobs. Nowadays, this refers to a set of elite independent schools (like Eton or Harrow) that typically take children very young and continue on until they reach university age. <a href="https://en.wikipedia.org/wiki/Public_school_(United_Kingdom)">See Wikipedia</a>.</p> <p>Most children start in reception aged 4 for a single year. Then mandatory school starts for everyone at a primary school for Years 1 through 6 where students are aged 5 through 11. They then go to secondary school for Years 7 through 11 aged 11 through 16. That having been said, the private school system can be different and they often take in children at a variety of ages - I’ve been at my current school since Year 3 and I’ll be leaving at the end of this year at the end of Year 13.</p> <p>Usually for Year 9 and below, most end-of-year exams don’t contribute to much other than academic streaming for the next year or reports (with the exception of standardised tests at state schools, but I don’t know much about them). Once you get past Year 9, you typically then start directly working on your GCSE (General Certificate of Secondary Education) courses.</p> <p>There are 3 compulsory subjects - Maths, English & Science. Maths is worth 1 GCSE, Science 2 (double award of 3 sciences), and English 2 (separate qualifications for Language and Literature). Typically though, students take more than just 5 GCSEs and will often take more subjects like languages, more complicated Maths or Science courses and different humanities/creatives<sup class="footnote-reference" id="fr-0-1"><a href="#fn-0">1</a></sup>. These are rated 9-1 and theoretically a 9 is supposed to be the equivalent of an A** (with the below grades following like the traditional letters - 8 = A*, 7 = A, 6 = B etc), but grade inflation has impacted that to the point where you’ll hear about independent school students achieving 12 or even more 9s.</p> <h3 id="sixth-form">Sixth Form</h3> <p>After your GCSEs, there’s no requirement from the government for you to take any further education. You can’t just be unemployed at 17 if you’re filthy rich - you have to work a job, or get an apprenticeship but you don’t need to stay in school. However just over half<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">2</a></sup> of students go to a Sixth Form<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">3</a></sup> college (Called either Year 12 & Year 13 or Lower Sixth & Upper Sixth, but always Sixth Form as a collective for ages 16-18).</p> <p>Most students in the Sixth Form study either A-Levels or IB Exams<sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">4</a></sup>. A-Levels are separate examinations by separate boards and typically take around 2 years to complete. Most students take 3 or 4, but in some rare cases at top schools people take 5. There are zero requirements on variety of which exams you have to and don’t have to take, which means that there’s a huge variety so schools will likely offer a subset of the possible exams in common subjects. IB Exams are all set and managed by the International Baccalaureate Organisation, and they set strict requirements on subjects - you have to take at least 1 Science, 1 Mathematical Subject, 1 English Subject, 1 Language & 1 other Humanity. The IB Diploma also has a large emphasis on coursework (internal assessments in every subject, an extended essay, and a <em>Theory of Knowledge</em> exhibition) When you leave Sixth Form, the only things that stay on your record are the grades you achieved - the individual tests you took don’t matter beyond a desire for personal excellence and there’s no concept of a <a href="https://www.timeshighereducation.com/student/advice/what-gpa">GPA</a>. When I’ve been doing US Applications and been learning more about their admissions system, the concept of a GPA terrifies me - every single test and assignment I do going to a number that goes to university? The stress must be unreal.</p> <h2 id="applications-offers">Applications/Offers</h2> <p>Most UK universities have a very similar process for admissions for most subjects. Here I’ll be going over a general overview with a few exceptions that I’ve become familiar with, but for example I won’t be touching Medicine with a 10-foot pole.</p> <p>All of the UK Universities work through one system for undergraduate applications - the Universities and Colleges Admissions Service, commonly known as <a href="https://www.ucas.com/">UCAS</a>. To submit an application through UCAS, you need to register through your school where the following items go onto your application (because they need to submit certain things, which I’ll get onto). Then, you need to submit the following:</p> <ul> <li>Admin Details - these are simple details like where you live, your residency status and any widening access points.</li> <li>Banking Details - the UCAS application costs £20 for one choice, or £25 for multiple. I just selected to have my centre pay, which means that it’ll end up on my school bill.</li> <li>Personal Statement (this is one 4000 character essay about why you want to study that subject. You can take this any way you want, and I’m not going to provide advice because there are far better people to do that.)</li> <li>Your 5 Choices. You get to choose up to 5 courses to go to - you could pick 5 courses at one university, 5 courses at 5 universities or any combination. There aren’t too many restrictions beyond the fact that you’re not allowed to simultaneously apply to Oxford & Cambridge (collectively known as Oxbridge), and you’re not allowed to apply to more than 4 medicine courses.</li> <li>Your reference - typically your teachers write & submit this. Typically there’s a small level of collaboration between the reference writer and the student to make sure you the reference doesn’t overlap with the personal statement, although this is strictly at the start and should not influence the writer.</li> <li>Your Predicteds or Achieved Grades. If you’ve already done your exams, then you just submit the actual grades, and if not then you submit your predicted grades.</li> </ul> <h3 id="choosing-universities">Choosing Universities</h3> <p>There’s a huge variety of different ways to choose a university. First, you might look at the course you want to do (Computer Science for me) and check which universities offer it. There’s a huge number of different league tables which all prioritise different things. All universities are also required to post their minimum grade offers for all their different courses which can help to see if a university is above or below your standards if you compare that to your predicteds. However, sometimes there are other factors to consider (like the prestige of a university - Edinburgh lists A*A*A* for Computer Science, but is easier to get into than Cambridge which lists A*A*A). Most also offer contextual offers, which are for if you’re disadvantaged in some ways compared to your peers - this could be anything from grief affecting your performance to being underprivileged.</p> <p>Then, when you’ve narrowed it down to a shortlist you’ll typically go to an Open Day. These are a few days a year that each university offers, where they’ll have lots of talks on about everything from the Student Union<sup class="footnote-reference" id="fr-4-1"><a href="#fn-4">5</a></sup> to your course to accommodation. Most universities also offer individual tours around the year, but the open days typically offer more things as well as conveniences like more bus services between the local rail station and the university. If you know you’d want to visit a specific university, make sure to book their open day early - I think lots of the nicer slots for the St Andrews one were booked out only a few hours after it opened to booking.</p> <h3 id="predicted-grades">Predicted Grades</h3> <p>Some people from outside the UK might’ve been reading this so far somewhat confused - you just apply with one essay and one reference? How do they know about your academic skills apart from GCSEs?</p> <p>What happens in most schools is that at the end of your Lower Sixth, you take a set of internal exams (as in, not set by any exam board) which your teachers then use (in addition to your performance from the rest of the year, or things like coursework) to provide an estimate of what you’ll get in your final exams. Some schools use AS Levels for this purpose, which are designed to be taken at this point.</p> <p>If you disagree then you can usually ask your teachers to change your grades, but the school has a vested interest in making sure that your grades are accurate. This is because the universities can see which school you applied from, and if schools consistently overpredict then the universities (theoretically) should remember and then discount those school’s opinions. The problem then occurs when somebody actually deserves the high prediction, but might not get the offer they want because of the traditional overestimations from that school.</p> <h3 id="offers">Offers</h3> <p>Then, you submit your UCAS Application - the deadline<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">6</a></sup> is the 31st of January, but my school set a far earlier internal deadline. The idea of that was that you’re stressing out about making a deadline that’s months before the actually important one so if you can’t make the deadline an extension isn’t an issue.</p> <p>Then, the universities must reply to you by the 1st of May. They have 2 options - rejection or offer. If you’re rejected, then you won’t be going to that university. Typically, you can’t ask for feedback. Offers are slightly more complicated - if you’ve applied the year after you did your A-Levels, they’ll give an unconditional offer. If you’ve applied before finishing you’re A-Levels using your predicted grades, then they’ll give a conditional offer which will almost always be at or below your predicted grades and often similar to the minimum grades that they post.</p> <p>Then, you’ve got 2 weeks to decide on your firm and insurance offers. You’ve got to pick 2 universities from the ones that replied to you - if you meet the grade requirements from your firm and want to go to university that year, then you can only go to that university. If you don’t meet those ones, but do meet the requirements for your insurance then you can only go there. If you meet neither, then you can either give up to apply the next year, or you can enter clearing. I don’t know much about clearing, but it seems to be where all students who want to can enter Clearing as a process, and then universities pick the students they want. Typically, the top universities don’t consider any applicants from clearing.</p> <h3 id="a-brief-aside-on-accommodation">A brief aside on Accommodation</h3> <p>Almost all UK universities offer some form of accommodation in the first year - often in Halls of Residence, which can be self-catered, catered, co-ed, sharing, single and prices will change accordingly. Most of the time there’ll be a selection of Halls which have different <em>features</em> like what I mentioned above. THe notable exception is Durham with the colleges system (which I’ll elaborate more on later with the Cambridge stuff). Some universities restrict this to only be guaranteed accommodation if you apply with them as your firm choice.</p> <p>Beyond the first year, you either have to submit an application to the few spaces not taken by first-years<sup class="footnote-reference" id="fr-6-1"><a href="#fn-6">7</a></sup> or take a chance with the rental market which is pretty bad at lots of universities (London is especially bad at this - one of the main reasons I didn’t apply to Imperial<sup class="footnote-reference" id="fr-7-1"><a href="#fn-7">8</a></sup>). The notable exception is Oxbridge, where almost all colleges offer accommodation to all undergraduates on-site although there’s nothing stopping you from renting if you see a lovely place or already have property in Cambridge/Oxford.</p> <h2 id="cambridge-specific">Cambridge-Specific</h2> <p>Here, we get into the incredibly specific. So far, I’ve tried to make sure that this is pretty widely applicable, but here we’re going into detail on my specific application. I’ll go more and more specific as it goes on.</p> <h3 id="the-colleges">The Colleges</h3> <p>Unlike every other university, you don’t just apply to Oxford or Cambridge - you apply to a specific college in your application. As far as I can tell, your supervisors and some of your professors will be from the college you applied to. There’s sports teams and societies which are college specific and for the whole university. One joked about bonus about going to Oxbridge is that you’ve got a higher chance of getting on <a href="https://en.wikipedia.org/wiki/University_Challenge">University Challenge</a> (a quiz show for British university students), as you’re only competing against the other people in your college rather than against everyone in your whole university <sup class="footnote-reference" id="fr-8-1"><a href="#fn-8">9</a></sup>.</p> <h3 id="the-application">The Application</h3> <p>Firstly, Oxbridge have a way earlier admissions deadline (the 22nd of October this year), and gives you decisions way earlier as well (mid-late January) although the acceptance date is the same. There’s the first difficulty.</p> <p>The next difficulty comes from the admissions assessments. Since most people applying will have incredibly high grades and impressive personal statements, they need more ways to differentiate candidates.</p> <h3 id="the-tmua">The TMUA</h3> <p>This takes place in different ways for different courses, but for Computer Science this involves taking the Test of Mathematics for University Admission - the <a href="https://www.admissionstesting.org/for-test-takers/test-of-mathematics-for-university-admission/">TMUA</a>. This is a really difficult maths paper - 2 papers taken back-to-back, which each have 20 multiple choice questions to be done over 75 minutes. It’s all Maths based, but Paper 1 seems to be more about the foundations, and Paper 2 seems to be more about reasoning & logic.</p> <p>My school ran preparation sessions, so I slowly worked my way through all of the old past papers (of which there are only 10, because it hasn’t been around for very long<sup class="footnote-reference" id="fr-9-1"><a href="#fn-9">10</a></sup>), and prepared thoroughly for the exam. I had my half-term holiday start at the end of the week before the test so I got about 4 or 5 days to do nothing but TMUA preparation. That having been said, I was pretty sick for most of that period so I didn’t get all that much productive work done.</p> <p>Then, when the day came around I took the exam at my school and then went for Kokoro with my friends<sup class="footnote-reference" id="fr-10-1"><a href="#fn-10">11</a></sup>, my lungs doing a seal impression the whole time 😔.</p> <h3 id="the-interview">The Interview</h3> <p>From the outside, the cambridge admissions seem to work off a series of successive filters, and I’d just managed to pass the TMUA ones. A few days before I would get my TMUA result back I got an email from my college with my interview time and details.</p> <p>I signed a very thorough non-disclosure (not sure if a solicitor<sup class="footnote-reference" id="fr-11-1"><a href="#fn-11">12</a></sup> would call it that, but it’s probably close enough), so I’ll be pretty shy about any specific details to avoid invalidating my current application.</p> <p>My interview ended up being about 30-40 minutes, and I had it over Zoom - this part gets interesting. So, historically most colleges have done in-person interviews at the Cambridge College, and I’ve had friends who’ve been to up to Cambridge recently for in-person interviews. However, over Covid most colleges switched to Zoom, and quite a few have stayed. It make sense when you think about it - I can pretty easily get to Cambridge from London, but someone living in North Scotland might have more troubles. This way, they can get the most skilled applicants regardless of geography getting in the way.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>Having now gone through some pretty competitive admissions processes in multiple countries (all I’ve got to do now is wait for results 😬), I’m pretty happy to give my thoughts.</p> <p>Doing both at the same time has been pretty stressful - I started off the term worrying about my Personal Statement, then I worried about the TMUA, then it was straight into MIT Essays then my MIT Interview then my Cambridge Interview and that was a challenge. During all that I still had lots of commitments to manage, ranging from being Vice-Captain of my school house to writing these blog posts (although this has been a vibey de-stresser) to a musical production where I took the pretty senior role of <a href="https://maguire.tech/ramblings/technical-theatre-experience/">Stage Manager</a>. That having been said, I wouldn’t change it for the world because of all the knowledge I’ve gained.</p> <p>I think that the US approach is a lot more work (because of the essays, as well as having to do their admissions test last year), but I kinda feel like MIT’ve got a far better idea of the nebulous concept of <em>Jack Maguire™️</em> than Cambridge. That having been said, I think Cambridge have a far better idea of my skills and capabilities - the interview was less casual and was with a professor and a supervisor, which was more stressful on my end but I almost feel more like if I get a rejection there wasn’t much more I could do (apart from being less stressed - that was far better in the MIT Interview, but you can’t really replicate coffee shop vibes in a zoom). I guess I’ll see what happens 😉.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-0"> <p>I took English (Literature + Language), Maths, Further Maths, the 3 Sciences separately, Design & Engineering, Geography, Latin, French & Russian. <a href="#fr-0-1">↩</a></p> </li> <li id="fn-1"> <p>This is a holdover from more old fashioned naming systems still kept in place at some schools, where you progress through different <em>forms</em>, but these vary wildly from school to school. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p>At time of writing, see <a href="https://explore-education-statistics.service.gov.uk/find-statistics/16-18-destination-measures">source</a>. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>There’s also <a href="https://qualifications.pearson.com/en/about-us/qualification-brands/btec.html">BTECs</a>, but I’ve not the foggiest on how they work. <a href="#fr-3-1">↩</a></p> </li> <li id="fn-4"> <p>Most UK Universities have a Students Union. The level of involvement and importance varies, but these typically deal with everything from getting student conditions improved (eg. mental health as a valid excuse for missing an exam) to clubs & societies. Unlike (as far as I’ve seen), lots of US Universities, the senior student union positions cover the entire university and are taken as a sabbatical rather than concurrently with studies. <a href="#fr-4-1">↩</a></p> </li> <li id="fn-5"> <p>Technically, this is the <em>equal consideration</em> deadline. This means that all universities should consider all applications submitted before here equally and by consequence that they’re allowed to discriminate against late applications. <a href="#fr-5-1">↩</a></p> </li> <li id="fn-6"> <p>Unlike the US there isn’t a system of Freshman -> Sophomore -> Junior -> Senior, it’s just first-years to second-years etc. <a href="#fr-6-1">↩</a></p> </li> <li id="fn-7"> <p>Imperial is supposed to be insanely good for Computer Science (and their open day was <strong>impressive</strong>), and they’re in a lovely part of London in South Kensington, but it’s London. I’ve heard awful things about everyone being spread all over London in accommodation over an hour away from the main campus which then screws with lots of social stuff. <a href="#fr-7-1">↩</a></p> </li> <li id="fn-8"> <p>I’m just going to politely pretend that I haven’t heard any stories about colleges which only submit postgraduates, or ones where the teams might have median ages of <strong>45</strong> 😱. <a href="#fr-8-1">↩</a></p> </li> <li id="fn-9"> <p>And looks like it won’t be around for <a href="https://www.admissionstesting.org/news/view/reforms-to-cambridge-assessment-admissions-testing-from-2024/">much longer</a>. <a href="#fr-9-1">↩</a></p> </li> <li id="fn-10"> <p>Those portions are <strong>huge</strong> <a href="#fr-10-1">↩</a></p> </li> <li id="fn-11"> <p>British term for a non-court-based lawyer. Court lawyers are called Barristers - <a href="https://www.youtube.com/watch?v=iId5WDsYxZ4">Saul Goodman</a> would be a Barrister, but the <a href="https://www.youtube.com/watch?v=d3qdju45udE">blood-sucking lawyer from Jurassic Park</a> would probably be a solicitor. <a href="#fr-11-1">↩</a></p> </li> </ol> </section> Transport for London, for Dummies Tue, 21 Nov 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/tfl-for-dummies/ https://maguire.tech/ramblings/tfl-for-dummies/ <p>I’ve had the good fortune of living in an incredible city for most of my life. I’ve also had the fortune to go to lots of other different cities around the world, and most don’t even compare to London for me. The unique combination of food, people, my obvious bias and <strong>obviously</strong> most importantly London’s public transport.</p> <p>I mean, look at the tube map alone!</p> <figure> <img class="no-hover" src="/images/tfl_tube_map.png"/> <figcaption>A work of Art!</figcaption> </figure> <p>If we then compare that to the NYC one, which is serving over double the population of London we can see that one is far better than the other.</p> <figure> <img class="no-hover" src="/images/nyc_subway.png"/> <figcaption>A bit mid.</figcaption> </figure> <p>This will mainly be focused (as per usual), on my experiences. I won’t be talking about the best specific routes, because you should always just use a travel app unless you’re <a href="https://youtu.be/JxD88PflssA">Jay Foreman</a>. I will also take a brief tangent to talk about the UK rail system because there are plenty of train stations in London.</p> <h2 id="payment">Payment</h2> <p>The cheapest way to go for pay-as-you-go journeys is to use an Oyster card. These can be picked up at any tube stations, and you can just dump money onto them. You ‘tap-in’ when you get into a station, take all of your transport on that method<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup> and then ‘tap-out’ when you get off. You can also tap in and out with contactless cards, Apple Pay & Google Pay. Important detail - only one person can travel per card (Apple Pay with a card counts differently to just that card contactless).</p> <p>Please, for the sake of locals use a card. Its far less faff, far more reliable and most importantly it’s fast. You don’t want to know what I’ve heard from less polite Londoners about tourists. People remember British politeness, but also forget our nature of grumbling about problems.</p> <p>For smaller train stations, you might not see actual barriers, and just pads to tap out on, or you might see all of the barriers open. You do still need to tap in and tap out - if they work out that you didn’t tap in or out then they’ll just charge you for whatever they think you did (<a href="https://tfl.gov.uk/fares/how-to-pay-and-where-to-buy-tickets-and-oyster/pay-as-you-go/touching-in-and-out">charges table here</a> and <a href="https://tfl.gov.uk/fares/find-fares/tube-and-rail-fares/single-fare-finder">single fare finder here</a>). Typically, they’ll be open if there isn’t a member of staff present - they need to be able to sort out issues or open the gates for emergencies so if nobody’s there then they have to keep them open.</p> <p>Buses are only tap-on, tubes charge a fee based on which <em>zones</em> you start and finish in and I think trains work the same way<sup class="footnote-reference" id="fr-1-2"><a href="#fn-1">1</a></sup><sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">2</a></sup>. There are also numerous ways to reduce the cost, like getting an Oyster Photocard if you fit certain requirements, and children under 5 travel free. There’s the hopper fare where you can travel on any number of bus journeys within 1 hour once you’ve paid £1.75. There’s also the daily (and weekly) travel caps for the tube. These latter two are applied automatically so you don’t need to worry about them.</p> <h2 id="individual-methods">Individual Methods</h2> <h3 id="tube">Tube</h3> <p>The London tube map looks like confusing at first but that’s mainly due to two requirements - a need for it to be small enough that the stations are legible as well as wanting to vaguely conform to the real world station locations. It looks a mess, but you get used to the bits you use, and you can also just use a travel app for pathfinding.</p> <p>Every tube will stop at every stop on its route (with exceptions like Turnham Green unfortunately) - if the tube you’re on will go past a stop then it will stop there. There are 3 main steps to navigation on the tube: the line, the direction and the terminal stop.</p> <p>The line refers to which colour the tube is on the map - but in London they all have names<sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">3</a></sup> because of their unique history<sup class="footnote-reference" id="fr-4-1"><a href="#fn-4">4</a></sup>. Most stations will serve at maximum two or three lines. The exceptions are the huge Network Rail stations (like Paddington, Euston, Kings Cross and Waterloo) which often serve five or more tube lines as well as trains.</p> <p>Then, each line runs in two directions - either North/South-Bound or West/East-bound. Then, each tube station will have maps right next to the junctions for the different directions. Each tube will also have a map on the inside of just the line that that physical tube serves. The part that some people get confused on is the terminal stop. In each of the directions the tube serves, there will be one or more terminal stops. For example, when going West on the District Line, you can end up at Wimbledon, Richmond or Ealing Broadway. In Central London you don’t need to worry because you’ll be in the core where you can get on any train. You only really need to worry about the terminal stops for getting further outside central London. For example, if you’re going from the Science Museum in South Kensington to the Tennis in Wimbledon on the District Line you’ll need to make sure to catch a Wimbledon train rather than an Ealing Broadway one.</p> <p>In Central London, the tube is <em>almost</em> always the fastest way to get around - faster than a taxi, faster than a bus and faster than walking.</p> <h3 id="bus">Bus</h3> <p>The buses (surprisingly enough) are more complicated but very similar in operation. Each number acts as its own line with a route of stops. You also get the same issue of terminal stops but these are far less of an issue compared to the tube. There are two main differences:</p> <p>Firstly, buses will not stop at stops unless asked. Usually, this isn’t a problem (especially at larger stops), but if nobody presses the button and nobody is waiting then the bus will happily keep on going without letting you off.</p> <p>Next, buses have individual stops for the two directions they will go in which can be quite far apart. If you’re trying to find the stop for Hyde Park Corner to hop onto a 22 to Oxford Circus, make sure to get to the right Hyde Park Corner to make sure you don’t end up in Putney Common.</p> <h3 id="train">Train</h3> <p>Trains in the UK are the transport method with the most variability in fares because all of the train companies are operated separately (with exceptions of conglomerates) and integrate to different degrees with the larger TfL ecosystem. Trains typically have more seats, wider & longer carriages and possibly other amenities like toilets or bike racks.</p> <p>If you’re in London, you can just tap-in and tap-out as normal - be very careful though because there is a border for when you can and can’t tap in/out. There also isn’t the same system with different terminal stops so that’s simple. The complexity comes with the fact that different trains stop at different stops on the same routes. The best bet is to check the departure board before you leave so you know that it stops at the stop you need it to stop at.</p> <p>When buying tickets to go outside of London, most people would book via Trainline. Whilst this will definitely work, for some things they charge fees (like on certain refunds). Most people don’t realise that you can download one of the TOC<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">5</a></sup>’s apps and book tickets for <strong>every other</strong> TOC. Then you don’t need to pay fees for some actions. If you can, I’d also recommend getting some kind of Railcard - there’s ones for students, couples, young people, old people, families and <a href="https://www.railcard.co.uk/">more</a>.</p> <h3 id="river-boats">River Boats</h3> <p>Finally, the river boats. There are only 6 routes, and they stop at every station and every station goes both ways. They also get reliable mobile phone reception, often have cafes and toilets and also give a lovely riverside view of London. The disadvantages are that its insanely expensive (on the scale of public transport at least - on the order of £5/6 per journey), and often slower than the tube or bus.</p> <p>The main clientele is City of London workers who live near the river but don’t want to take a Black Cab (to be explained later). They enjoy it because they can get a fair bit of work done on it, unlike with the tube or bus. People also like using them because they’re a more comfortable and quieter ride compared to the tube/bus and the cafes are always appreciated.</p> <h3 id="private-hire">Private Hire</h3> <p>Private hire is pretty self-explanatory because most cities have it and it isn’t particularly different here.</p> <ul> <li>Uber is the dominant app for quick & relatively cheap cabs and all Uber drivers are all supposed (iirc) to be licensed by TfL.</li> <li>If you have the money, you can also get a Black Cab which is the famous London one - all the drivers are supposed to know Central London off by heart, and they can also use the bus lanes (other bus lane exceptions <a href="https://tfl.gov.uk/modes/driving/red-routes/rules-of-red-routes/bus-lanes">here</a>). Black Cabs also have a very limited number of reasons for refusing fares and <strong>must</strong> have wheelchair ramps.</li> <li>We also have bicycles to hire - <a href="https://tfl.gov.uk/modes/cycling/santander-cycles">Boris bikes</a><sup class="footnote-reference" id="fr-6-1"><a href="#fn-6">6</a></sup> are the official ones, and you can also use the <a href="https://www.li.me/vehicles/electric-bike">Lime Bikes</a>, <a href="https://www.humanforest.co.uk/">Forest Bikes</a> and more that I’ve probably forgotten.</li> <li>There are also rickshaws. Please always be careful with them - some of them aren’t licensed and make sure to always check the prices <strong>before</strong> you start to avoid getting an insanely high fee for a short journey.</li> </ul> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>London’s public transport is insanely good and one of the reasons I so love London - I can leave home, get to the nearest tube stop (via like 4 different buses) and get to anywhere in central London in under half an hour. Unlike in some other cities, the tube really makes the city feel tiny (also aided by the weird development of the city<sup class="footnote-reference" id="fr-7-1"><a href="#fn-7">7</a></sup>) - I can get to the West End, Harrods, Leicester Square, The Ritz, and all of the big tourist attractions with zero effort.</p> <p>Whilst from these footnotes and bracketed sections it might seem that the only concrete rule with TfL is that there will always be exceptions these mostly don’t apply and when they do it’s relatively obvious.</p> <p>I would also like to partially credit TfL with helping me to develop my independence - the ability to be able to go out anywhere anytime without needing to rely on other people has been incredible. TfL most definitely has problems (be it the exceptions, the inaccurate bus times, the long times for fixing things or the endless strikes), but I love it and wouldn’t wish to swap it with any other city’s public transport system.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-1"> <p>There are exceptions though - for example, the Heathrow Express has a ticketed system. Sometimes you can also get on a train via tapping in but you’ll need a ticket at the other end - watch out if you’re going beyond London. <a href="#fr-1-1">↩</a> <a href="#fr-1-2">↩2</a></p> </li> <li id="fn-2"> <p>That having been said, you can sometimes transfer between different methods without going through a gate - these are signposted and you just need to tap the station on the platform. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>For some Londoners, it’s a point of pride to be able to correctly match all of the colours and the names. <a href="#fr-3-1">↩</a></p> </li> <li id="fn-4"> <p>The London Tube is one of the oldest in the world and the lines have their names for a variety of reasons. For example, the Bakerloo Line used to just serve Baker Street & Waterloo, and the Jubilee Line was opened not too far (in time) from the one of the Queen’s Jubilees. <a href="#fr-4-1">↩</a></p> </li> <li id="fn-5"> <p>Train Operating Company (like South Western Rail, Avanti West Coast, Great Western) <a href="#fr-5-1">↩</a></p> </li> <li id="fn-6"> <p>Technically they’re the Santander Cycles, but no self-respecting Londoner would ever call them that. The nickname comes from the Mayor of London who introduced them - Boris Johnson. <a href="#fr-6-1">↩</a></p> </li> <li id="fn-7"> <p>Technically, the City of London is older than England itself. The Normans tried to create a new capital city (in what is now Westminster), but both grew and grew and eventually merged making the distinction pointless. London is technically a <a href="https://en.wikipedia.org/wiki/Conurbation">Conurbation</a> rather than just a normal city. <a href="#fr-7-1">↩</a></p> </li> </ol> </section> The Technical Theatre Experience Mon, 13 Nov 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/technical-theatre-experience/ https://maguire.tech/ramblings/technical-theatre-experience/ <p>Over the last few years, I’ve gone deep into Technical Theatre. This is the kind of blanket term for all the non-actors involved in a theatrical production. I’ve worked on a lot of productions (everything from tiny casts in small spaces to huge musicals in traverse), with a relatively small team and that’s led to me wearing an awful lot of hats. The plan is for me to write about all the roles I’ve done and give some opinions. I’ll then talk about my general experience.</p> <p>I say this multiple times, but it’s good to mention from the start - this is all based off my experiences which are likely not reflective of yours - different shows run differently and people share different responsibilities. There are also probably countless roles I’ve missed - I’m trying to stick to roles I’ve actually had some hand in for the most part here. This is more about my experiences than what you might have done. That having been said, I’m always curious to learn about new things so email me if you’ve got a role I don’t mention here.</p> <p>I’ll also add a warning here - if you actually want to know what to do, the best resource I’ve found is the <a href="https://wiki.cuadc.org/wiki/Cambridge_University_Amateur_Dramatic_Club">ADC Wiki</a>.</p> <h2 id="backstage">Backstage</h2> <p>First, I’ll be talking about all the roles that are on the ground, in the wings and with all the actors.</p> <h3 id="stage-manager-sm">Stage Manager (SM)</h3> <p>The stage manager is the lead role - if something goes wrong, then the buck ultimately stops with you. This is the most senior technical role, and so can sometimes involve extra details around safety in case of serious on-stage problems or guiding audiences to a safe place.</p> <p>Your job amounts to knowing exactly what happens when for the whole show in terms of props, set changes & who’s moving everything around. Then, when something inevitably gets forgotten you need to notice and take steps to a) make sure it isn’t forgotten again and b) try and somehow get it onstage subtly.</p> <p>I’m currently preparing for a Stage Manager job, but I’ve never actually done it so we’ll see what happens.</p> <h3 id="assistant-stage-manager-asm">Assistant Stage Manager (ASM)</h3> <p>The ASMs work for the SM, actually doing all of the set changes and prop pre-setting. Typically you’ll need to memorise transitions because the director will probably have some novel way of moving everything on and off stage. If you forget something, then get ready to be told so over headset from the SM.</p> <p>I’ve been ASM a few times - it’s often categorised by periods of boredom (because you need to stay by the stage to do transitions, but then can’t talk or do much) interspersed by brief periods of high activity. Depending on the show & director, you might only be needed for a few of the bigger transitions, or you’ll be on your feet for the whole night.</p> <h3 id="sound-2-3">Sound 2/3</h3> <p>The main difference between Sound 2 & 3 is how much you’re responsible for - is it just mic-ing up actors or are you also worried about making sure all the packs have enough batteries and are at the right frequencies.</p> <p>That having been said, a lot of the work is just making sure actors have microphones.</p> <p>Depending on the size of the production this can be wildly different in difficulty. In smaller production with lots of talking roles, you might have mic changes with every other scene - once I think I had to do a mic change over about 2 or 3 lines. In larger productions, you might not need to worry much apart from the start and end because you might be able to rent enough for everyone. This is why Sound 3 people can often end up doubling as ASMs.</p> <p>This is one of the more wasteful areas of the theatre - I’ll write more about this later, but the environmental impact of the theatre is not to be understated. Specifically here the consistent director enforces rules around batteries - regardless of how long they’ve been used for they have to be replaced every night. For a 24 mic production, this means 48 AA batteries per night - if you account for rehearsals and say 6 production nights (which is relatively small), you’re then speaking in the quantities of hundreds of batteries. Do that multiple times a year and you’re into the thousands of batteries. And then at that scale, rechargeable batteries become impractical so most batteries just get binned.</p> <h3 id="sound-1">Sound 1</h3> <p>NB: I haven’t ever actually done this because I decided to specialise in lighting - I’ve dabbled and done mic work, but nothing much more so this is extrapolation based off what I’ve seen. I’ll try and keep the opinions to a minimum because I’ve got almost no experience here.</p> <p>Sound 1 is the other major backstage role, although it’s more similar in nature to a box role, as you mainly stay put during the actual production. Before that point, you’ll be running around checking audio cables and making sure all of your equipment works precisely the way you think it does. The reason I’d qualify this as backstage is because you have to do your job from within the auditorium to make sure that you’re hearing the levels accurately.</p> <p>During the actual production, your job is to manage the levels of all the different microphones - band & actors both. Recently, this has just been muting the band’s microphones because they’re so loud and getting the actors as loud as possible. That having been said, that’s probably because for the in-traverse musical<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup>, there are way fewer sound deafening seats than usual.</p> <h2 id="the-box">The Box</h2> <p>I usually find that I prefer box roles more as you can sometimes get away with quiet conversation during quiet parts and sometimes even eating. The one time I’ve ever heard eating/drinking being banned was as an intervention to stop the sound operator drinking so much coffee. It’s also often cooler because you sometimes have AC. There’s also usually a great view.</p> <p>I will add one point to that though - this is mainly from my experiences with one box that is: sound-proofed, air-conditioned for several reasons, has a mini-fridge and large windows. I’ve also seen boxes that are small and stuffy where the fact that the windows opens is a godsend for heat reasons. I’ve done one production in a box that was centimetres away from an audience, and on that occasion roles were reversed because the ASMs and other backstage roles waited outside right next to a stage entrance.</p> <h3 id="lighting-designer">Lighting Designer</h3> <p>Lighting design involves looking at the stage and current lighting plans and deciding whether or not they’re sufficient. If not, then everyone gets involved to bring the lighting bars down and add new lights - either from storage, purchase or rental. This involves a whole lot of tape and safety chains to make sure the cables are as invisible as possible (that is, not drooping down from the bars) and takes a while. Then, they need to be put back in place and focused, a job I’ve never done but looks like it involves spanners and heights which makes me glad I’ve never had to do it - although I would one day like to do it, for curiosity’s sake.</p> <h3 id="lighting-programming">Lighting Programming</h3> <p>Lighting programming mainly involves a mastery of whichever lighting system the theatre uses to control the fixtures - trying follow what the lighting designer says, and tweaking it to fit with the lights installed.</p> <p>Sounds pretty simple, and for smaller shows it definitely can be - the trouble comes with larger shows where me you might jump around in the various sections you’re programming - this can lead to (dun dun duunnununnn!!!) tracking issues! These are the worst bane of lighting programmers and result when cues are incorrectly updated and those changes can persist until that light is used again. You then need to go through every cue between and fix it, which can take a forever and a half.</p> <h3 id="lighting-sound-operator">Lighting/Sound Operator</h3> <p>Lighting & Sound Operator are both very similar and quite simple - you just press the button to advance to the next cue when the Deputy Stage Manager tells you to.</p> <p>The main difference is in the frequency of button-pushing - as a sound operator I once had time to make a cup of tea for the lighting operator, wait for him to drink it and then clean it all up between cues. Lighting changes are often more frequent - I think the longest gap I ever had was during a 7 minute duration cue which signified the transition between day & night.</p> <h3 id="deputy-stage-manager-dsm">Deputy Stage Manager (DSM)</h3> <p>The DSM’s job is to call the lighting changes and communicate any possible on-stage issues to the SM in case they haven’t already noticed. This involves a very well marked up script (often called The Book), but doesn’t involve too much beyond that.</p> <p>This is the most senior box-based role, and on smaller productions you’ll often see the director double up as the DSM to execute the cues exactly to their vision. I’ve seen some quite funny sights of DSMs getting very annoyed at the cast during rehearsals privately to the box crew (who are sworn to secrecy by tradition) once to the point of snapping the nib of their pen.</p> <h2 id="pre-production">Pre-Production</h2> <p>Unfortunately, I have no experience in costumes, set design, prop sourcing or lots of the aspects of the show that start long before production.</p> <p>The only thing I can say is that set building & rigging lights has shown me that <em>Manual Labour</em> can be fun with friends.</p> <h2 id="closing-thoughts">Closing Thoughts</h2> <p>I’ve loved my last few years in technical theatres - including the production I’m working on right now I’ve done 9, as well as multiple events the theatre team is asked to do like running cable for fireworks night. I’m definitely going to continue on when I get to university but I probably need a bit of a healthier attitude towards it - last year I did literally everything I could apart from when it directly clashed with permanent-record acts and I’m a hair tired of it all.</p> <p>Then, on the other hand what can you really be without hardship? I’ve learnt proper transferable skills like teamwork, a hair of leadership & prioritisation. I’ve got my calendar and todo list to a bulletproof state, partially because I’m just that guy, but also because if I didn’t then all the balls I’m carefully juggling would have fallen by now.</p> <p>I haven’t even considered the impact on people other than me here - I’ve already mentioned the batteries earlier but there’s a far larger impact than that. I haven’t mentioned the tape (wouldn’t be surprised if we get through it by the 100s of metres per production and then that all goes in the bin). There’s the paint and the lights. I’ve worked on productions where the lights are so many and so bright that we don’t turn on the lights until right before the audience comes in for heat management. Even just from a heat perspective that’s more power than my computer probably consumes all year just in an evening.</p> <p>All of that having been said, I’ve never quite found anything else that matches the stress, the teamwork & the eventual satisfaction when everything goes well in front of a paying audience. I love the thinking on your feet problem solving it a prop is left on stage, or a lighting cue goes one too early because there’s no time for dallying or indecisiveness. Some people get this by sports, but I’ve never quite had that experience so for now I’ll be staying in the theatre. I guess we’ll have to wait and see what the future holds for my theatrical career.</p> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-1"> <p>This is where most of the seats are moved out of the way to have the actors act between 2 pairs of facing audience seats. Makes the show far more immersive at the cost of seating space. <a href="#fr-1-1">↩</a></p> </li> </ol> </section> MIT Interview Sun, 12 Nov 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/ramblings/mit-interview/ https://maguire.tech/ramblings/mit-interview/ <p>Recently, I’ve been going through the joy that is university admissions in 2023 applying simultaneously to US & UK systems. Trying to get in a halfway decent application for them both without sacrificing my studies and the theatre productions I’m involved in has been, to put it lightly, a little bit of a huge pain.</p> <p>Here I’ll be writing about my experience applying to US universities, specifically MIT (and if plans hold, then a caltech post later this year) from the perspective of a Brit.</p> <h2 id="long-term-prep">Long-Term Prep</h2> <p>One thing that both systems seem to have in common is that you need to be an interesting person who can show passion for their subject. From a Computer Science view this means that I’ve spent the last few years making more projects than I can shake a stick at. I’ve tried to make a variety of projects, but the main focus has been on making sure that the project was fun to code over making it actually useful. I found this insanely helpful to make sure I actually ended up finishing these projects to an acceptable degree, rather than just having a halfway finished physics engine that I wasn’t really all that interested in. If you want to see what I’ve been cooking, I’ve got a few writeups on this site, with more coming.</p> <p>I’ve also kept up with computer science and general tech news, like making sure to flick through <a href="https://this-week-in-rust.org/">TWIR</a> every week or listening to WAN show on my runs.</p> <h2 id="admissions-tests">Admissions Tests</h2> <p>This is the main difference between UK & US admissions as far as I can tell. I’ll take a bit here to explain both systems so everyone understands.</p> <p>In the UK, at the end of the second-last year of school you take an internal set of exams and your school gives you a set of predicted grades - you send these to the universities with a personal statement and then universities hand out conditional offers. These are conditional in that the offer will have a set of requirements for grades which will normally be at or below the predicteds you submitted.</p> <p>Whereas in the US (as far as I can tell), you take exams (like SAT or ACT exams) which get sent to the universities with a metric ton of uni-specific essays (apart from if they’re part of a coalition like Common App where there’s one common essay). Then, they send back unconditional offers. There appear to be loads of exceptions and weird bits though - like applications which are binding if they accept you, or some universities which are test-optional or test-blind.</p> <p>So, I’ve got all my predicteds sorted because I go to a UK school, and then I took a few ACTs last year after doing a prep course.</p> <h3 id="observations-from-a-brit">Observations from a Brit</h3> <p>It’s really interesting what the different systems prioritise. A-Level Physics is about a fundamental understanding of the subject with equations and some bits you’ve got to memorise whereas ACT Science is more about reading, understanding and then answering questions about those experiments. A-Level Further Maths is about long questions with problem solving elements, but ACT Maths is simpler and seems more solve this matrix than prove by induction that this matrix does that.</p> <p>Part of that difference might be that the American Universities also ask about which courses you’ve done, and what your GPA is. I can’t answer them though because I don’t have a GPA. I also don’t have any AP classes because that isn’t really a concept in the UK. In some ways I don’t have to stress as much through the year because I don’t need to worry about every individual test affecting anything that goes on my record or to the universities, but equally if I mess up that one big test at the end of the year then I’m properly screwed.</p> <h2 id="recent-stuff">Recent Stuff</h2> <p>So I filled out my applications and wrote my essays, and ended up submitting early action - very slightly better odds and earlier knowledge of success or failure in return for earlier deadlines.</p> <p>I then waited for a few anxious days and got a text & email from my EC (Education Counsellor - if I’ve seen worked out one thing about MIT so far, it’s that I’ll need to make a pretty big spreadsheet for acronyms). We had a chat, and eventually she sent me a calendar invite for about a week later on a Friday afternoon in a coffee shop.</p> <p>This is another interesting difference - most UK universities seem to work off a series of successive filters, like grade requirements, or personal statement and then interviews. They are themselves a filter, but you’ve got to have passed through a fair few beforehand to make it to that stage so people consider it an achievement. Whereas in the US, every attempt is made to give everyone an interview which is just an interesting change.</p> <p>This was when my MIT interview prep properly started (probably a bit late). If I were doing this again, I probably would’ve started reading the MIT blogs earlier because they’ve been insanely helpful, and you can find them over <a href="https://mitadmissions.org/blogs/landing/">here</a>. Over the last week and a half I’ve read more blogs than I can count, on everything from the best spots to cry or study to organising Splash to struggles with burnout. There are YouTube videos, but I haven’t really found many of them as useful. I also can’t lie - the student bloggers over there are definitely part of my inspiration for this.</p> <p>I also started preparing answers to all kinds of questions from ‘Why MIT?’ to ‘where do you see yourself in 5 years’ to ‘How do you spend your free time’, and, especially relevant <a href="https://www.npr.org/2023/06/29/1181138066/affirmative-action-supreme-court-decision">this year</a>, <em>The Community Question</em>. I’ll happily say that some of these questions were asked and some weren’t and that I also had a fair few questions I hadn’t prepared. What to prepare is kind of up to you - it’s not my place to tell you what questions because every interview will be different. I will say that if you’ve got topics that you’re planning on getting to, then make sure to be ready for them to ask follow up questions.</p> <h2 id="the-day-itself">The Day Itself</h2> <p>Since my interview wasn’t until 3PM, I did my usual friday morning classes rather than sleeping in. I do regret this - I wasn’t really able to focus for lots of that morning and I just felt stressed about when I was planning on leaving. One part that was quite funny was walking into one of my lessons knowing well that I could walk out whenever I damn pleased because I’d have a pretty decent excuse. That teacher had done one of my references so he was happy to just let me go without complaint.</p> <p>I sprinted over and made good use of TfL to get to a Pret near my interview location about an hour and a quarter early. I was lucky that it had a decently sized bathroom which I made use of to get changed, before buying a sandwich and some food. I then realised - the Pret was full of people trying to do exactly what I wanted to do (using the Pret as a warm place to work that had food & sockets). Eventually a table opened up and I sprinted over to get my phone & laptop charging for later use.</p> <p>Then, about 30 minutes before the scheduled start I went outside for a 10 minute talk to the interview place which was a way nicer café than the Pret I’d gone to - nice staff, better vibe & much better Hot Chocolate (I’d rate it as best I’ve ever bought).</p> <p>Then, at about 15:00 on the dot, I instantly heard when my interviewer come in - a US West Coast accent stands out in a sea of Brits. I’d lucked out in this case and got an interviewer who was attended the same ‘School’ at MIT which I wanted to attend, which was the School of Engineering. She’d done Course 2 (Mechanical Engineering), and I want to do something in the Course 6 area (Computer Science).</p> <p>I luckily got a chance to speak about pretty much everything I wanted, and I came up with some pretty decent answers to the questions I hadn’t prepared for. From what I’ve heard, my Cambridge (UK) interview will just be lots of Maths questions so I loved the opportunity to speak about things I’m really passionate about. It had definitely helped that I’d done practice questions in a vaguely interview-like format and from my end it seemed like she made an effort to just let me speak about anything & everything I wanted to which was really lovely.</p> <p>As for the questions I asked her - I think I asked her about things like what she did for accommodation and the things she’d done for her extra requirements but the most interesting answer I got was definitely not from the question I expected.</p> <p>That would have been when I asked if the vibe I’d gotten from the blogs was accurate - work hard, play hard. One of the main social activities for her was doing homework with friends which is certainly a contrast to what I’ve heard about other universities in the UK.</p> <p>My interview ended up lasting a hair over 40 minutes, which went by at a speed you wouldn’t believe. I felt it lasted a good amount of time. I did have more to talk about but I wasn’t 100% sure how to fit it all in.</p> <h2 id="all-done">All done</h2> <p>I’ve got a few things I probably would’ve done differently if I could do it over, like mentioning a few of my outside of school hobbies that didn’t make it into my application but all the moping in the world isn’t going to change anything.</p> <p>All I can do is nervously wait for mid-December. I had a peek at the stats last night, and I’ll share my interpretation - there’s about a 20-30% chance I’ll get outright rejected, which sounds incredible until you factor in that there’s still only a 6-7% chance of acceptance. The rest get waitlisted, and just dumped into the regular application pool. Of those waitlisted, it’s about a 2-3% chance of acceptance which is pretty slim, so a waitlisting isn’t all that far from a rejection 😭.</p> <p>Out of all the international applicants, regardless of admissions stage, the chances are just 2% which is pretty small. That having been said, the UK seems to punch above its weight in terms of students in relation to the population and I can see that about 2 get admitted every year. It’s a shame that they don’t do applicants per country and not just current undergrads because then I could actually work out more stats. Then again, it’s probably not healthy to obsess over those stats so we’ll see what happens 🤷.</p> Vent Sat, 11 Nov 2023 00:00:00 +0000 Jack Maguire https://maguire.tech/posts/vent/ https://maguire.tech/posts/vent/ <p>You know how recipes often have a reputation for rambling sections about the origin of those recipes? Well, let’s just say that I finally understand why.</p> <p><a href="https://github.com/BurntNail/vent">Vent</a> is my biggest project ever, at time of writing (over 4k LOC of Rust, not including the SQL or the templates 😲 which is pretty damn huge for me) and has been my baby for the last 8 months. It’s finally proper time that I actually show it off and explain the backend (this definitely isn’t a disguised way for me to re-familiarise myself with the whole thing 😉).</p> <p>You know the saying <em>Vexation breeds innovation</em>? Maybe not? Well, regardless it’s definitely true for this project. I’d just come into possession of a monstrous spreadsheet for managing various events, as well as a OneDrive folder where the photos got dumped. The old spreadsheet was exclusively for managing the details of the events and had no facilities for managing who was attending the event - this was dealt with by a series of newsletters that died out as people got bored and, in the late stages, teams messages which people were told to react with a 👍 to. The photos weren’t organised at all, were a pain to manage and nobody could see them unless you just asked and then they had to be manually dug out.</p> <p>No more! To fix this, we go to a self-hosted 100% custom solution with <code>Rust</code> using <a href="https://lib.rs/crates/axum"><code>axum</code></a> and <code>postgres</code>.</p> <ul> <li><em>Jack, did you need to do this</em>? God no.</li> <li><em>Jack, was anyone asking you to do this?</em> Also, no.</li> <li><em>Weren’t there any prebuilt solutions?</em> Alas, I couldn’t bring myself to look when I saw the potential for such an exciting project.</li> </ul> <p>The plan for the rest of this is to go over the project as it is now, then to go over the creation and some of my interesting hurdles.</p> <h2 id="overview">Overview</h2> <h3 id="technical-details">Technical Details</h3> <p>As of right now, the project basically just acts as an access-managed front-end for a <code>postgres</code> database serving <code>GET</code> requests using <a href="https://shopify.github.io/liquid/">liquid</a>, and handles all interactive elements via <code>POST</code> requests. For CSS, I’m using <a href="https://getbootstrap.com/docs/5.3/getting-started/introduction/">Bootstrap</a>, which feels 2015-esque but is the easiest way for me to get some vaguely-competent looking CSS. I like doing things myself rather than using pre-built solutions, but I’m willing to limit this project to one new main thing - the sentence above.</p> <p>If you’re wondering where I’m going to mention what Javascript framework I’m using, the answer is none. I’m a systems dude who’s trying to learn, and I’ll be honest - this is as much a project to learn the ropes of postgres, deployment, back-ends and the web as it is a project to manage events. I’ve tried lots of the java/type-script-y frameworks, and they frankly don’t really appeal to me. I’ve become pretty damn hooked on the Rust programming language due to the package management, documentation, tooling and unique features like discriminated unions which are missing from loads of other languages so I didn’t want to really use any. I’ve tried <em>(trust me I’ve really actually tried)</em>, but I can’t quite get javascript to fit inside my head - I had the most success with <code>Svelte</code> but it just doesn’t work for me ;(.</p> <h3 id="user-story">User Story</h3> <p>So, the main way this project works is through events and peoples’ links to events. An event has a number of properties (that came from the original spreadsheet - one of the project requirements was to be able to import stuff from that <em>relatively</em> easy), and then a bunch of links. They all have links to the people participating, the people looking managing the event, and the photos that have been added. Participant-Event links also can be marked as <strong>verified</strong> - this is useful for making sure people have actually attended the event. A manager-level user has to do that after the event, and in my experience the photo comes in very useful for this.</p> <p>Finally, there’s a rewards system where after completing a certain number of events, you can be eligible for different rewards. There’s also a system in place for two different entry points each of which have their own requirements for the rewards.</p> <p>The access control works pretty sensibly imho - for example, someone with manager-level permissions can add and remove whomever they please from events to satisfy reality, but a participant-level user can only add and remove themselves. I won’t go into insane detail - just know that there’s 4 levels (Participant, Manager, Admin, Dev), where the admins can add & remove users, and the dev gets stuff that the admin wouldn’t find as interesting like reloading partial templates.</p> <h2 id="the-story">The Story</h2> <p>Here’s the story of my application - a few items have been moved around to better tell the story, but it’s mostly accurate and if you want the 100% accurate version then feel free to read the <a href="https://github.com/BurntNail/vent/commits/main/">Commit History</a>.</p> <h3 id="beginning">Beginning</h3> <p>This project started with the beginners guide for the <code>axum</code> crate (for this was my first axum project which slowly expanded in capability as I learnt more about axum & async rust), where you get started with a basic GET/POST which worked well for me. I then added a <code>liquid</code> based templating system inspired by <a href="https://fasterthanli.me">Amos Wenger</a> (specifically <a href="https://fasterthanli.me/articles/a-new-website-for-2020">this one</a> iirc), who’s writings I cannot recommend enough. I then slowly by surely added the <em>other things</em> like participants and photos. This part of the project was probably the second most fun, where I could move mountains with little effort - partially because I hadn’t done much yet so any change was big, and partially because I was free to look at all kinds of different approaches before I got entrenched in a pretty specific style.</p> <p>I also had a few interesting constraints to work within:</p> <ul> <li>I wanted to be able to run the whole thing on just the one tiny VPS (a Linode Nanode which has 1GB of RAM, 1 vCPUs and 25GB of storage) I rent whilst still keeping some space for other stuff (like this blog!). This was mainly an attempt to reduce both vendor lock-in as well as costs.</li> <li>I wanted this to keep backwards compatibility with the existing spreadsheets - that meant making sure that I had lots of import/export tools.</li> <li>I wanted this to be easily used by lots of more non-technical users - a JSON feed would never be acceptable and I needed to put lots of work in to get something looking halfway decent (a challenge even with the aid of <code>Bootstrap</code>).</li> </ul> <h3 id="authentication">Authentication</h3> <p>I then took it to the person who actually had final say on adoption on this system and he said that I needed to follow all relevant guidelines that he had to, which in this case included making sure that you couldn’t see photos without logging in. My initial thoughts were something along the lines of <em>balls. i don’t really want to do this, because the more user data i store the more impact my screwups have</em>, but I wanted adoption so I just nodded and said yes. Turns out I was still right, just not in the way I thought I was.</p> <p>At this point I was pretty happy with the general <code>axum</code> ecosystem, and I came across two main crates that seem to be used for logging in - <a href="https://lib.rs/crates/axum-auth"><code>axum-auth</code></a> & <a href="https://lib.rs/crates/axum-login"><code>axum-login</code></a>. <code>axum-auth</code> seemed to have more downloads but was just one rust file to check headers and get them. <code>axum-login</code> seemed to have more features and even <strong>drumroll please</strong> - Session Management, although it had fewer downloads. Generally, I like writing things myself (as has already been mentioned, often for the learning opportunity), but if I’ve learnt one thing in my years of Tom Scott/Computerphile viewership - it’s that you leave password management to other people, preferably open source people with code that’s been explored & checked over. That having been said, it still wasn’t the easiest to work with - there were a few annoyances like having to rewrite the <code>PostgresStore</code> for sessions to work with my system, as well as getting passwords to work right. I started with the <code>0.6</code> version, which has recently been updated with more breaking changes than I’ve had hot dinners (future update coming - <a href="https://github.com/BurntNail/vent/issues/187">issue here</a>). I then used <code>bcrypt</code> to store the <em>hashed</em> passwords.</p> <p>One fun solution to a problem is signing up. Since I know my entire userbase in advance, I can just import them all via CSV with blank passwords set. My original solution was to just set the password to whatever the user entered on first login, before I realised that I’d have a sea of password resets as people logged into other peoples’ accounts for shits & giggles. I eventually ended up going with emailed magic links to set the password.</p> <p>Eventually I got all that working, with the access control coming not long after. As I alluded to earlier, there was still a pain point. This was just managing the access control levels across the whole application - I have checks in a whole lot of places to make sure someone’s not trying to be sneaky with direct <code>POST</code> requests to the server and making sure that no PII leaks.</p> <h3 id="the-modern-web">The Modern Web</h3> <p>At this point though it was just something running on my machine, or just via direct HTTP from my registrar (originally <a href="https://hover.com">Hover</a>, but I transferred to <a href="https://www.cloudflare.com/en-gb/learning/dns/what-is-cloudflare-registrar/">Cloudflare</a> for cheaper renewal fees) to my VPS (<a href="https://www.linode.com/">Linode</a> seems to give a decent compromise between cost, configuration & ease of use). I <em>needed</em> HTTPS if I wanted people to send passwords through here - there’s nothing particularly important, and I’ve got the password hygiene not to use the same password everywhere but I can’t say that for all my users. Luckily, one of my very good friends (<a href="https://github.com/HandyHat">HandyHat</a>) had experience here, and walked me through the entire setup of <a href="https://caddyserver.com/">Caddy</a> for Cloudflare using Reverse DNS to get HTTPS certificates.</p> <p>You might’ve noticed that I’ve put all of my eggs into one basket - if Cloudflare goes down then I’m down. I understand the concern, but I also trust that if Cloudflare goes down then the one instance I run of Vent personally will be of little concerns.</p> <p>I also chose to use a VPS here, rather than a Docker Container running on a <a href="https://www.digitalocean.com/products/droplets">Droplet</a> or something - this is for multiple reasons. First is that I’m definitely not just using the VPS for this so I save the more projects I use. I also find it insanely convenient to quickly fix small issues in things like SQL queries in production without needing some complicated deployment process - I just <code>ssh</code> in, edit the relevant file with <a href="https://micro-editor.github.io/"><code>micro</code></a>, then build the project and restart it in a <code>tmux</code> session I keep around.</p> <h3 id="adoption">Adoption</h3> <p>Eventually, I started some user groups on the system and when nothing went wrong (apart from some misspelt usernames which were easily fixed) I went for the full rollout. Whilst some people still don’t mark themselves as planning to attend, it’s now the de-facto system and those people probably check when and where events are on the main page. I’m really happy with how it went.</p> <p>No first plan survives contact with the enemy though, so I had to rapidly issue a few updates like publicly marking who uploads photos, and improving my logging infrastructure which at that point just forwarded whatever domain specific error it got from whatever caused the error via <a href="https://lib.rs/crates/thiserror"><code>thiserror</code></a> & the <a href="https://lib.rs/crates/tracing"><code>tracing</code></a> ecosystem. You don’t want to know how long its taken to tune the logs 😔.</p> <h2 id="fun-technical-things">Fun Technical Things</h2> <p>You didn’t think I’d finish here, did you? Nah - I’ve got a few fun things which I’ll add to as I remember from the making of this project.</p> <h3 id="threads-not-of-the-zuck-variety">Threads (not of the Zuck variety)</h3> <p><code>axum</code> has a <code>State</code> extractor you can use with all of your methods, and whilst from what I’ve seen most people don’t use it as a state manager and just use it as various components I’ve got it all tricked out. For example, I publish an ICS but don’t want the faff of re-making the file for every request (which definitely isn’t what I was doing for a while which caused event duplication issues 😉) so I have one persistent file that gets updated. How do I know when to update it? Every time an event gets updated in such a way that would affect the calendar, I call a function on the State which sends a blank message through a channel to the calendar updater.</p> <p>I’ve also got threads to delete old sessions (hopefully no longer needed in <code>axum-login</code> 0.7?), and send emails to users.</p> <h3 id="templating">Templating</h3> <p>I’ve mentioned that I use liquid templating, but given no more insight than that. Currently, the partials get cached but the actual templates don’t (there’s an <a href="https://github.com/BurntNail/vent/issues/180">issue</a> and it’s next on my list after <code>axum-login</code> 0.7). The main reason for this is that <code>liquid</code> templates are entirely synchronous and the compiler is created in an environment with no way to access <code>async</code> APIs other than caching them beforehand 🤷, so if ever anyone looks at my code and gets confused then that’s why.</p> <p>If you look over my templates, I’ve also got a couple of invariants (like people always being ordered by group in the view all page when logged in) that allow me to commit some <em>horrendous</em> crimes that make me giggle every time I look at them, and sigh every time I have to think about doing something with that part.</p> <p>Having now got experience with liquid, I can say that it probably wasn’t the right choice - it can be annoyingly limited to work with (something like <a href="https://lib.rs/crates/handlebars"><code>handlebars</code></a> might’ve been better in that respect), but I’m also happy to accept that it’s good enough and that maybe everything doesn’t always need to be perfect.</p> <h2 id="gotchas-for-future-me">Gotchas for future me</h2> <p>These are just a few pointers for people ever embarking on similar projects:</p> <ul> <li>You need to setup a <a href="https://developers.cloudflare.com/support/page-rules/understanding-and-configuring-cloudflare-page-rules-page-rules-tutorial/">page rule</a> to turn off SSL for <code>DOMAIN/.well_known/acme_challenge/*</code> because that prevents Caddy from setting up the certificate.</li> <li>Linode blocks email-related ports (presumably for spam/DDOS reasons?) and doesn’t ever say clearly (just in some <a href="https://www.linode.com/blog/linode/a-new-policy-to-help-fight-spam/">random blog post</a>) so you need to contact support to get them unblocked.</li> <li>If you’re tee-ing JSON logs into a file and then serving that file, you might need to parse that file and reverse all the elements because otherwise all the most recent logs (which are probably the important ones) are right at the bottom.</li> <li><code>axum::debug_handler</code> compiles to nothing in release mode - cover your project in them because all they’ll do is slightly increase compile times and they’ll massively help in errors.</li> <li>Use <a href="https://dbeaver.io/">DBeaver</a> - it’s insanely useful for quickly editing database records and doing other database-adjacent things like complete migrations.</li> <li>People aren’t joking about backups - you wouldn’t believe how easy it is to completely accidentally wipe something important from (or just wipe the whole) production database. It doesn’t matter how many flags you’ve setup in <code>DBeaver</code>.</li> <li><a href="https://www.hostinger.co.uk/">Hostinger</a> can shut off servers with too high activity - make sure to double check that doesn’t happen to you, because I didn’t get emailed about it or realise until I checked wanting to see the events and saw that production was down because <code>Docker</code> was being screwy.</li> </ul>