النشيد السوري

2026-03-17T00:00:00+00:00

كل ما يلي، بما فيها الهوامش، هي كلمات الشيخ علي الطنطاوي رحمه الله </blockquote>

كانت 1</a> نشأتنا الأولى في عهد العثمانيين، وكانت لهم أناشيد يلقونها علينا باللسان التركي، إذ لم نكن نفهم معانيها الضخمة إلا بالترجمة، والترجمة لا تحمل دائما المعنى كله، فلقد كانت تهزنا ألحانها القوية المثيرة التي وضعت لتكون لمشاة الجيش قوة وعونا. وكنا إذا أنشدناها سائرين لا نستطيع أن نقف، وإذا تلوناها واقفين سرنا، وإن قرأناها قاعدين حركنا، من غير قصد منا، أيدينا وأرجلنا—وإنها لتحرك الحجر! ثم جاء عهد فيصل، وكان عهد ازدهار وحياة ونشاط بدا في كل شيء، فنشطت فيه الأناشيد العربية من عقالها. فترجمت أكثر الأناشيد التركية. فكان منها نشيد:

أنا أمي لم تلدني</td> لا للحرب العوان</td> </tr> </table>
بنغمته القوية، ولحنه العاصف—وكان أسْيَرَ أناشيد ذلك الزمان وأشهرها نشيد (أيها المولى العظيم)، الذي اعتبر النشيد الوطني السوري بل العربي، على هلهلة أسلوبه، وضعف معانيه، يليه في الشهرة والذيوع نشيد:

أنت سوريا بلادي</td> أنت عنوان الفخامة</td> </tr>
كل من يأتيك يوماً</td> طامعاً يلقى حمامه</td> </tr> </table>
لفخري بك البارودي، وهو نشيد ضعيف النسج، متهافت البناء، لكن معانيه في الذروة، واشتهرت أناشيد أخرى منها نشيد: (سيروا للمجد طراً سيروا للحرب، واستعيدوا بالمواضي دولة العرب)؛ ونشيد طلاب المدرسة الحربية: (نحن جند الله شبان البلاد)، وهو من خيرها لفظا ومعنى، وقد جمعها (الفلاح العربي) في رسالتيه المعروفتين في تلك الأيام.
ثم لما قضى الله قضاءه فينا في (ميسلون)، ووقعت الواقعة، ودخل العدو ديارنا، منعت هذه الأناشيد كلها، إلا أن تردد همساً، واشتهر يومئذ نشيد الأستاذ أديب التقي، رحمه الله: (في كل صوب حشدت عساكر مدججون)، الذي يصور فيه موقعة ميسلون، فكان ينشد وراء الأبواب، وحيث لا يسمعه الغاصبون، وهو نشيد جيّد، لحنه حزين مؤثر.
وانقطع بعد ذلك سيل الأناشيد الوطنية، حتى قدم علينا من العراق الكشافون في العهد الوطني الأول (سنة ١٩٣٦)، فأخذنا منهم نشيدين اشتهرا فينا وسارا بيننا، حتى كان الطفل الذي لم يتعلم بعد الكلام يدير في حلقه كلمات منهما، وهما (هذا الوطن حق له أن يفتدى بالدما والمهج) و(نحن كشاف العراق)، والشعر فيهما ليس بذاك، واللحن فظيع هو أشبه بصراخ لا دلالة له، منه باللحن الذي يؤثر في الأعصاب ويحرك القلب، ولكنها مع ذلك نشيدان قويان.
ووضعت على أثر ذلك أناشيد جيدة منها (نحن الشباب لنا الغد)، ولكن يعيب لحنه هذه الصيحة المؤنثة في آخر البيت، عند تكرار (نحن الشباب)، فهي صيحة عجوز ثاكل كان لها الأمس، لا صيحة شباب لهم الغد، والنشيد العظيم حقا في نظمه ولحنه، ولفظه ومعناه، نشيد (موطني) لفقيد الشعر إبراهيم طوقان—رحمه الله—ومن أجودها لحنا نشيد الأستاذ حسني كنعان: (أيها الكشاف بادر وارتق أوج العلا)؛ ولحنه نموذج للألحان الحماسية—نسجل هذا للتاريخ!
وصحت النية على وضع نشيد للجمهورية السورية، وكانت مسابقة، ولجنة، وجائزة، ثم عدل عن ذلك واختير النشيد الذي قدمه هذا الأديب الكبير، فلما قرأناه علمنا أنه لوحظ باختياره اسم الشاعر ومنزلته، وأنه لهما لا لبراعة الشعر فرض علينا هذا النشيد، واحتملناه سنين، غير أنه لا يصح أن نتحمله الآن، وقد تم استقلالنا، أو هو قد أشرف على التمام، واستقبلنا عهدا من حياتنا جديدا، ولابد من بيان عيب هذا النشيد لنسبتدل به.

الأصل في النشيد الوطني أن يكون على لسان المتكلم، لأن الأمة هي التي تردده وتنطق به، وهذا النشيد موجه إلى حماة الديار، مطلعه:

حماة الديار عليكم سلام</td> أبت أن تذل النفوس الكرام</td> </tr> </table>
فمن الذي يقول هذا الكلام، ومن المخاطب به؟ إن كان ينطق به الشباب وهم حماة الديار، لم يعقل أن ينادوا أنفسهم، ويسلموا عليها، وليس هذا من (التجريد) الذي كان يألفه شعراء العرب؛ وإن كان يقوله غير الشعب لم يكن مقبولا لأن النشيد يوضع ليقوله الشعب ويترجم به عن آماله ومطامحه.
وهذا السلام أليس أشبه بلهجة أروام الإسكندرية وأرناؤوط الشام، منه بأسلوب الشعراء الأبيناء^{2</a>؟}
ثم يقول بعد هذا:

عرين العروبة بيت حرام</td> وعرش الشموس حمى لا يضام</td> </tr> </table>
فلا يعرف السامع ما عرين العروبة هذا، أهو الجزيرة أم مصر أم الشام أم العراق؟ ولا يعرف المسلم (بيتا حراما) إنما يعرف البيت الحرام، لا ثاني له، فهذا التنكير أولا، وابتذال اسم البيت الحرام في كل مكان ثانيا، كلاهما قبيح. وما هو هذا العرش، والنشيد نشيد جمهورية؟ أفنظمه ليكون النشيد الرسمي لبني أمية، وأي شموس هذه؟ وما هذا الإبهام حتى لا يحسن إلا التصريح والتوضيح؟ يأتي بعد ذلك هذا المقطع العجيب:

ربوع الشآم بروج العلاء</td> تحاكي السماء بعالي السناء</td> </tr>
وأرض زهت بالنجوم الوضاء</td> سماء لعمرك أو كالسماء</td> </tr> </table>
أما (بروج العلاء) هذه فتصح في كل أرض يريد أن يبالغ في مدحها القائل، ولا تدل على ميزة للشام ولا تصفها بصفة فيها ولا تعرّف بها الغريب عنها، ولا تحببها إلى أبنائها، فهي كمراثي الأستاذ علي الجارم التي تصلح لغازي والإسكندر المكدوني وللشيخ المراغي، لأنها تهد الجبال وتبكي السماء، وتقيم القيامة، أو ترسل على الدنيا قنبلة ذرية، لا بحجم البيضة، بل بحجم الفيل، ثم لا تذكر المرثيّ بشيء مما كان عليه. وهذا استطراد نعتذر إلى الشاعر الكبير علي الجارم بك منه، فقد جرته المناسبة.
وما دامت الشام بروج العلاء، وكان ذلك قد تقرر لدى السامع فما معنى كونها تحاكي السماء، وبروج العلاء هي السماء في أفهام الناس كلهم، وهل السماء أسمى سنا من البروج؟ المسألة تحتاج إلى خبير فلكي.
ثم إن الضياء هو السنا بالقصر، أما السناء بالمد فهو الارتفاع، ومن هنا أطلق على المجد مجازا، فصار معنى قوله: (بعالي السناء) برفع الارتفاع، وهو الحشو نفسه وهو إذا قبل في القصيدة لا يقبل في النشيد، لأن النشيد كلمات معدودة وألفاظ محدودة، لا يجوز أن يذهب لفظ واحد منها من غير أن يدل على شيء.
وهو بعد أن جعلها بروج العلاء التي تحاكي السماء، عاد فهبط بها فجعلها (أرضا زهت) ولكن بالشموس الوضاء؟ وما فهمت إلى اليوم ما يرد بهذه الشموس التي يرددها ولا يشبع من ذكرها، إن كان يريد الحقيقة فهي شمس واحدة ما خلق الله سواها؛ ،إن كان يقصد المجاز؛ فليذكر ما يدل عليها ويصرف الفكر إليه، وما كل سامع للنشيد أو تالٍ يستطيع أن يجد له التأويل، هذا إذا كان لهذا الكلام العجيب تخريج أو تأويل.
وأعجب العجب، وأقبح القبح، أن يعود بعد كل ما مر، فيجعلها سماء ثم ينزل بها فيجعلها كالسماء، وهذا ضدّ ما عليه البلغاء في كل عصر، وفي كل أمة، ولا أحسب ذوقا من الدنيا يسيغه، عدِّ من هذا الحشو في كلمة (لعمرك) وعمر من هذا الذي يحلف به؟ ولمن هذا الخطاب؟ والمفروض في النشيد كما بينت أن ينطق به الشعب كله؟!
وما هو مغزى هذا كله، وما دلالته، وأي مجد للشام يذكر، وأي عاطفة تثير؟ لا شيء، إلا هذه المناقشة المزعجة في الشام: هل هي بروج العلاء تشابه السماء برفيع الارتفاع؟ أم هي أرض ولكن زهت بالشموس؟ أم هي سماء (وحياة عمرك ...) أم هي كالسماء؟ هذه هي المشكلة الوطنية الكبرى، ملأ النشيد بذكرها، وهذه هي آمال الوطن ومطامحه، والله المستعان!

وباقي النشيد لا يختلف كثيرا عما ذكرت منه، على حين أن النشيد يجب أن يكون موضوعا على لسان الشعب، وأن يكون قوي العبارة، خاليا من الحشو، واضحا كل الوضوح، صالحا لكل زمان، معبرا عن آمال الشعب وآلامه ومطامحه، مثيرا نخوته وحماسته، مشيرا إلى ماضيه، وجمال أرضه ودياره، إلى غير ذلك مما يوصل إلى الغاية من وضع النشيد، وهي إثارة العزة الوطنية في النفوس، وأن يختار له النغم القوي من غير خشونة، العاطفي بلا ضعف. وحياة النشيد بحلنه وما يهز هذا اللحن من أوتار القلوب، ويحرك على أعصاب السامعين، فإذا كان لنا نشيد يشتمل على هذه كله... وإلا فلا تقولوا: لنا نشيد^{3</a>!!}

ناظم هذا النشيد أديب كبير، وصديق كريم، وهو يعلم أن ليس له عندي إلا الإجلال والتوقير، وأن الشاعر «وإن نبغ» يسبق تارة ويقصر، وإن النقد «وإن قسا» لا يصدر عن حقد ولا يرمي إلى تحقير، وإن مصلحة الوطن بإصلاح نشيده تسوغ مفاجأة الكبير بنقد هذا النشيد. ↩</a> </li>

الأبيناء: جمع بيّن. ↩</a> </li>

ناظم هذا النشيد هو أستاذنا خليل مردم بك رحمه الله، وهو شاعر بليغ، رفيع المقام، فاضل الأخلاق. وابنه الأستاذ عدنان شاعر كبير. وملحن النشيد أستاذنا مصطفى الصواف. ↩</a> </li> </ol> </section>

The Unbearable Frustration of Figuring Out APIs

2026-01-14T00:00:00+00:00

In the ongoing effort for activities that fill the void made by unemployment, I have recently started to learn Chinese. Got into a Chinese language institute and everything. And because not every app supports right-clicking some text and selecting the "Translate" menu option, I found myself launching TextEdit just to do that. So, I figured, maybe I can do a small command line tool where I'd write something like:
translate 你好</code></pre>
And the terminal would happily tell me it means "Hello". Should be easy. A hungry ghost trapped in a jar</a> can probably figure it out in one shot.
First I figured I'd look for some translation API. Almost everything I looked at seemed to want an API token and there is a rate limit (not that I'd hit it), and maybe a credit card number. Then I remembered that macOS has that Translate right click action, surely I can just hook into that.
Note that this article, unlike what I do usually, is written after the fact.

First Steps</h2>
Zig</h3>
I have been using Zig for different projects this year. So I got the Ghost to write the thing for me, and it even gave me the correct flags to pass to the compiler! But then when I tried it, I realized it used the Dictionary service instead of the Translation service. When I asked it for that instead, it was like yeah you cannot call Swift async</code> functions from Zig, you need a Swift shim.
So let's do the thing in Swift instead. I have been meaning to learn Swift anyway.
MyCLI</h3>Thankfully, Swift was already set up on my machine with an LSP and a formatter and the works. I had even followed the basic tutorial</a> and the file is on my machine already, A small edit and I can be on my way. For reference, this is the code at the end of the tutorial: import ArgumentParser import Figlet @main struct FigletTool: ParsableCommand { @Option(help: "Specify the input") public var input: String public func run() throws { Figlet.say(self.input) } }</code></pre> After so long in Rust and Zig lands, the most jarring thing about Swift so far is that import</code> creates glob imports. This is not quite apparent here, but it is there. Also, this is the accompanying Package.swift</code>, comments and all. // swift-tools-version: 5.8 // The swift-tools-version declares the minimum version of Swift required to build this package. import PackageDescription let package = Package( name: "MyCLI", dependencies: [ .package(url: "https://github.com/apple/example-package-figlet", branch: "main"), .package(url: "https://github.com/apple/swift-argument-parser", from: "1.0.0"), ], targets: [ // Targets are the basic building blocks of a package, defining a module or a test suite. // Targets can depend on other targets in this package and products from dependencies. .executableTarget( name: "MyCLI", dependencies: [ .product(name: "Figlet", package: "example-package-figlet"), .product(name: "ArgumentParser", package: "swift-argument-parser"), ], path: "Sources"), ] )</code></pre> This is run with swift run</code>, but to pass arguments, one needs to do the following: swift run MyCLI --input Hello</code></pre> I spent like 15 minutes trying different variations of this # this does NOT work swift run -- --input Hello</code></pre> Anyway, I deleted the Figlet</code> related lines and went from there. False Starts</h2> I asked the Ghost in the Jar for how the API to use it looks like, but every version it gave me was hallucinated in some fashion. So I wrote import Translation</code> at the top of the file, and figured maybe I can figure it out as I go along. Here is the Apple API documentation</a>. A quick look tells you all the functions in here return a some View</code>, and the only function seemingly unrelated to SwiftUI is the init</code> function in TranslationSession</code>, which, annoyingly, needs a known source language. So I do this import ArgumentParser import Translation @main struct Translate: ParsableCommand { @Argument(help: "Specify the input") public var input: String public func run() async throws { print(">\t\(input)") let source = Locale.Language(identifier: "zh_CN") let target = Locale.Language(identifier: "en_US") let session = TranslationSession(installedSource: source, target: target) let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") } }</code></pre> You can see the number of small changes to the original file here. I changed the struct's name (because it shows up in the --help</code> message.) I changed Option</code> to Argument</code> so it does not need a flag. I even turned run</code> into an async</code> function so I can await</code> the session.translate</code> function call. However, I start to hit a couple of snags. First is that TranslationSession.init</code> is restricted to macOS 26. Not wanting to do a bunch of conditional compilation, (after all this tool is for my own use), I will just add the platforms</code> field in Package.swift</code>。 Oh it needs to be before dependencies</code>? Fine, whatever. // swift-tools-version: 5.8 // The swift-tools-version declares the minimum version of Swift required to build this package. import PackageDescription let package = Package( name: "MyCLI", platforms: [ .macOS( /* one problem, I cannot find v26 in the dropdown */) ], dependencies: [ .package(url: "https://github.com/apple/swift-argument-parser", from: "1.0.0") ], targets: [ // Targets are the basic building blocks of a package, defining a module or a test suite. // Targets can depend on other targets in this package and products from dependencies. .executableTarget( name: "MyCLI", dependencies: [ .product(name: "ArgumentParser", package: "swift-argument-parser") ], path: "Sources", ) ] )</code></pre> I cannot find .v26</code> in the LSP's drop down. There is no .Tahoe</code> either. Even if I type it manually, I get an error. I am on Tahoe. I have the latest version of Swift (I checked). why can't I select it from here? You probably already know this, but apparently the first line in the file, that comment, is actually significant. I edited that to say swift-tools-version: 6.2</code>, and now I can write .v26</code> in peace. I hate syntactic comments. Ok, the LSP is happy. The compiler seems happy. Let's get going. Async Woes</h2> Running this with swift run -q MyCLI 你好</code> gives the following message: USAGE: translate <input> ARGUMENTS: <input> Specify the input OPTIONS: -h, --help Show help information.</code></pre> Eh, I clearly passed in something. What gives? I tried changing Argument</code> back to Option</code>, same result. Hm. I realized that the problem maybe is that run</code> is not supposed to be async</code>. Fine, let's remove async</code>, but then how do I call session.translate</code> ? Looking things up, and communing with ghosts, apparently I can wrap the thing in Task</code>. Sure. Task { let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") }</code></pre> And this runs! I rerun the command I do swift run -q MyCLI 你好</code> and I get ... nothing. Just the print message that prints input</code> at the start. I think hah .. I have fallen into the classic trap, I am spawning a Task but I am not waiting for it. How do I wait for it? The ghosts suggested a DispatchSemaphore</code>. This is the full code: import ArgumentParser import Translation @main struct Translate: ParsableCommand { @Argument(help: "Specify the input") public var input: String public func run() throws { print(">\t\(input)") let source = Locale.Language(identifier: "zh_CN") let target = Locale.Language(identifier: "en_US") let session = TranslationSession(installedSource: source, target: target) let semaphore = DispatchSemaphore(value: 0) Task { let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") semaphore.signal() } semaphore.wait() } }</code></pre> Ok .. for some reason, when I tried this right now, it actually works. But when I tried it before, probably with a different arrangement of stuff, it did not. It just froze and gave no feedback. The power of hindsight, I guess. Nonetheless, since it did not work, I figured I was doing something wrong. So I searched for "how to call an async function from a sync function in Swift", and came across What calls the first async function?</a> from Hacking With Swift, and the tl;dr of that article was "not you, just make your main async</code>". Very helpful, Paul. Apparently I needed to make run</code> async. I looked into the docs of swift-argument-parser</code>, I figured surely they have some guidance on how to deal with async</code> functions. Apparently the guidance was just to replace the ParsableCommand</code> protocol with AsyncParsableCommand</code>. Ok, now I felt like an idiot. This is the full code: import ArgumentParser import Translation @main struct Translate: AsyncParsableCommand { @Argument(help: "Specify the input") public var input: String public func run() async throws { print(">\t\(input)") let source = Locale.Language(identifier: "zh_CN") let target = Locale.Language(identifier: "en_US") let session = TranslationSession(installedSource: source, target: target) let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") } }</code></pre> Works like magic now. Except, at the time, it did not. It gave me the following message: > 你好 Error: Unable to Translate</code></pre> Also it leaves the small detail of autodetecting the language, which is here hardcoded as zh_CN</code>.1</a> Language Recognition</h2> The error had zero feedback. Absolutely unhelpful. No idea what's going on. I added a try await session.prepareTranslation()</code> line I saw in some online examples, but .. nothing. I looked a bit more around for tutorials, I could not find anything. All tutorials just regurgitate the same SwiftUI code examples, but I am not using SwiftUI. Ekh. Eventually, I came across the original WWDC video introducing the Translation API</a>, and decided to watch it. And it was actually very helpful, despite it being SwiftUI focused. It does not touch on how to use TranslationSession.init</code>, instead it mentions that a session</code> is given to you to run in a SwiftUI closure. It gives the rationale behind the API, but it also introduces the bit I needed for autodetection, and that is NLLanguageRecognizer</code>. NaturalLanguage</code> is an Apple framework for dealing with natural language (How did you guess?). I pretty much copied the code from the video into its own function (after import NaturalLanguage</code>). func identify_lang(_ sample: String) -> Locale.Language? { let recognizer = NLLanguageRecognizer() recognizer.processString(sample) guard let language = recognizer.dominantLanguage else { return nil } return Locale.Language(identifier: language.rawValue) }</code></pre> And this is my new func run()</code> public func run() async throws { print(">\t\(input)") // unwrapping here but I handle it more gracefully in the code let source = identify_lang(input)! let target = Locale.Language(identifier: "en_US") let session = TranslationSession(installedSource: source, target: target) try await session.prepareTranslation() let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") }</code></pre> The other thing the video pointed out, is that the API user should check language availability. I saw that earlier, but did not particularly care. I know Chinese and English are available. I have been using them! Nonetheless, I copied that code as well just to be thorough, to figure out what the API does. let availability = LanguageAvailability() let is_it = await availability.status(from: source, to: target) switch is_it { case .unsupported: print("> language pairing from \(source.languageCode) to \(target.languageCode) unsupprted") return case .supported: print("> language pairing from \(source.languageCode!) to \(target.languageCode!) not installed") return case .installed: break }</code></pre> This gave me a compile error, as it happens, as apparently the enum is marked the Swift equivelant of #[non_exhaustive]</code>. Going over StackOverflow and the Swift book, told me the answer is to add a @unknown default:</code> case. I could not quickly figure out how to merge it with another case so I did not bother. To my surprise, the pairing returned .supported</code>, and not .installed</code>. In a SwiftUI app, the try await session.prepareTranslation()</code> call creates a pop up to ask the user to download the model. But it is already installed! I am already using it via the right click menu! Nonetheless, that must be the reason for the Unable to Translate</code> error. Installing Models</h2> Communing again with Ghosts, they told me to install the models through System Settings, General, Language, etc. I went where they told me, to this dialogue, where I can install the models locally. You can see that Mandarin is installed now, but the only language installed at the time was English(US). No matter, I installed Mandarin SImplified and Arabic, and added a note to the switch</code> above showing where to install the items. And then I tried again: swift run -q MyCLI 你好</code>. And .. same error. I guessed that it identified the language as Mandarin Traditional for some reason, so I installed that as well, and now it works, finally!! % swift/MyCLI ❭ swift run -q MyCLI 你好 > 你好 > Hello</code></pre> Why was that so hard? Why are the models here separate from the ones in the right click menu? Too many questions. Final Code</h2> This is the full final code, with some proper error names thrown in for good measure. import ArgumentParser import NaturalLanguage import Translation @main struct Translate: AsyncParsableCommand { @Argument(help: "Specify the input") public var input: String public func run() async throws { print(">\t\(input)") let target = Locale.Language(identifier: "en_US") guard let source = identify_lang(input) else { print("> could not identify language") throw CliError.Recognitionfailed } let availability = LanguageAvailability() let is_it = await availability.status(from: source, to: target) switch is_it { case .unsupported: print("> language pairing from \(source.languageCode) to \(target.languageCode) unsupprted") throw CliError.PairingUnsupported case .supported: print("> language pairing from \(source.languageCode!) to \(target.languageCode!) not installed") print("> Go to System Settings > General > Language & Region > Translation Languages and download the models.") throw CliError.PairingNotInstalled case .installed: break @unknown default: print("Unknown status.") } let session = TranslationSession(installedSource: source, target: target) try await session.prepareTranslation() let response = try await session.translate(input) let result = response.targetText print(">\t\(result)") } } func identify_lang(_ sample: String) -> Locale.Language? { let recognizer = NLLanguageRecognizer() recognizer.processString(sample) guard let language = recognizer.dominantLanguage else { return nil } return Locale.Language(identifier: language.rawValue) } enum CliError: Error { case Recognitionfailed case PairingUnsupported case PairingNotInstalled }</code></pre> So I compiled it for release, put it in /usr/local/bin</code>, and called it a day. You can see an example of a failing call if you do swift run -q MyCLI Bonjour</code>, where it prompts to install the dictionary. In Conclusion</h2> To rub some salt on the wound, during the time I was trying to puzzle this out, I realized that Spotlight (with the Cmd+Space key) can already do this exactly how I envisioned it, with around the same number of keystrokes, and it does not need this dance with installing models and whatever.^{2</a>} Also, Swift is frustrating. API of Swift libraries is frustrating. Until later. By the way, there is no documentation anywhere I could find on what the strings accepted here are. The only comment exists is "A Unicode language identifier, like en-US</code>, es-419</code>, or zh-Hant-TW</code>." No list or anything. Am I supposed to guess? ↩</a> </li> Also this supports Chinese input, while Ghostty's quick terminal does not, for some reason. ↩</a> </li> </ol> </section>
A Dumb Introduction to z3 2025-09-15T00:00:00+00:00 Recently I have come across a nice article: Many Hard Leetcode Problems are Easy Constraint Problems</a>, and I figured, I really should learn how to use these things! What else do I really have to do? I have had use for solvers (or as they are commonly called: theorem provers) In a previous article</a>, but then I tried to prove the things with good old algorithms. I looked at z3</code> at the time, but found the whole concept a bit too opaque. Now however, it seemed a bit easier to get into. To be clear, as of writing these words, I have only been looking at z3</code> reading material for two days. I am in no way an expert, and I have not written anything more complex than a solver for the change counter problem (the first example in the article listed above). So I am writing this really knowing nothing about the underlying theory, theorem provers, or whatever the hell "unification" is. There is a good chance you know more about this than I do. There are z3</code> bindings in many popular languages. I will be using z3</code>'s Rust bindings</a>, because I am more comfortable in Rust than, say, Python or JavaScript. The examples I worked with to understand z3</code> however, can be found in two nice documents: First is in Python</a></li> Second in JavaScript</a></li> </ol> What are Solvers?</h2> Solvers are a class of .. tools? libraries? where you input some rules and constraints and have the tool just .. solve it for you. It is not going to be a faster or more optimized solution than a custom made algorithm, but it is much easier to change the rules on the fly. There are many real world uses. They are often used for scheduling and resource allocation problems. Consider the common scenario of a school schedule: Mary cannot work on Tuesdays because she needs to take care of her father; John lives far so he cannot give classes before 10; Class 3-A is full of nerds so their math hours are double; city council regulates no outdoor activity after 12; Susan and Sarah hate each other so you should not have them teach the same class; etc. You can either have two teachers work on it for a week, or just pop it in a solver! The MiniZinc homepage</a> (another popular solver) has a couple of nice examples: a seating chart, rostering, vehicle routing, grid coloring. On that note, you might wonder: why did I go with z3</code> when MiniZinc has a more colorful homepage and is actually referenced by the article linked at the start of this article? The answer is because z3</code> has bindings in Rust. That is pretty much it. A Note on Terminology</h2> Documentation on z3</code> and its API use a lot of jargon,1</a> which makes the whole thing really difficult to wade into without a previous background. I will explain things as I understand when I get to them, but two things really stand out. The first is the word Sort</code>. You see this in the context of arrays and function declarations (we will get to those, I hope). But it has nothing to do with .. well .. sorting. Sort</code> is just the jargon word for types. The second one is constants. They are not what a normal person would call constants: they are actually the knobs the solvers use to solve problems. There are two types of constants: free, which are what one would call variables; and interpreted, which is when you'd type an integer literal and clever type machinations turns it into a constant in the solver. Note that also solvers do not work within the regular type system of the programming language. They have their own types (sorry, sorts), and operations that may or may not map nicely to the language's types and operations. Much of the actual code you are writing is about expressing things in the target solver's language. z3</code> uses a language called "SMT-LIB2" (henceforth called smt2</code>), apparently. And you can actually write your constraints immediately in said language and have the library consume it. Much of what the bindings is take your code and translate internally to this language before feeding it to the solver.2</a> A Simple Equation</h2> Let's start with what might be the simplest, dumbest equation. Solve for x</code>: x + 4 = 7</code></pre> Yes, a child (literally) can solve this. But it is nice. Here is the Rust program to solve it. use z3::{Solver, ast::Int}; fn main() { let solver = Solver::new(); // define the variable. let x = Int::new_const("x"); // define the equation solver.assert((&x + 4).eq(7)); // run the solver _ = solver.check(); let model = solver.get_model().unwrap(); println!("{model:?}"); }</code></pre> This prints out the solution. x</code> equals three. Who would have guessed? x -> 3</code></pre> The Rust bindings have some nice ergonomics here. You can simply do &x + 4</code> and it would do all the bookkeeping behind closed doors to transform the 4</code> (and the 7</code>) into an interpreted constant and have them inserted into the internal model. The reason you have to pass in a string in new_const</code> is that this is the name given to the variable in smt2</code>. It does not have to be "x"</code>, it can be anything. Why do the bindings not autogenerate the name for you? Who knows. If you print the solver (as in println!("{solver:?}");</code>), you will get the following output in the smt2</code>. (declare-fun x () Int) (assert (= (+ x 4) 7))</code></pre> Note that the variable you declared is declared as a function. A free constant is basically a function that takes no input and gives an output (here of type</del> sort Int</code>). The solver finds which version of the function satisfies the assertions. This also explains the arrow in x -> 3</code> earlier. x</code> evaluates to 3. A Jump in Complexity</h2> In school, jumping from solving equations with a single variable to equations with two variables was a real jump on complexity. Everything was doubled! Here is a pair of equations we will try to solve next: x + y = 17 y = 2 * x</code></pre> Here is the program. I am going to print the result of solver.check()</code> first, tho. I just made up those numbers! use z3::{Solver, ast::Int}; fn main() { let solver = Solver::new(); // define the variable. let x = Int::new_const("x"); let y = Int::new_const("y"); // define the equation solver.assert((&x + &y).eq(17)); solver.assert((&x * 2).eq(&y)); println!("{solver:?}"); let c = solver.check(); println!("; {c:?}"); }</code></pre> This prints out the following: (declare-fun y () Int) (declare-fun x () Int) (assert (= (+ x y) 17)) (assert (= (* x 2) y)) ; Unsat</code></pre> Oh it is Unsat</code>. Unsatisfiable. Bummer. This means this cannot be solved as defined. Let's try changing the type to Real</code>.^{3</a> The Real</code> type does not have the same nice ergonomics as Int</code> apparently, so the code will look slightly uglier. This is the new updated code.} use z3::{Solver, ast::Real}; fn main() { let solver = Solver::new(); // define the variable. let x = Real::new_const("x"); let y = Real::new_const("y"); let seventeen = Real::from_rational(17, 1); let two = Real::from_rational(2, 1); // define the equation solver.assert((&x + &y).eq(&seventeen)); solver.assert((&x * &two).eq(&y)); println!("{solver:?}"); let c = solver.check(); println!("; {c:?}"); }</code></pre> Which prints (declare-fun y () Real) (declare-fun x () Real) (assert (= (+ x y) 17.0)) (assert (= (* x 2.0) y)) ; Sat</code></pre> Excellent! Using get_model()</code> and printing the model as before gives us the following answer, presented as a nice rational number. y -> (/ 34.0 3.0) x -> (/ 17.0 3.0)</code></pre> To actually extract the values programmatically, instead of debug printing model</code>, requires some song and dance with the API, but it is simple, really. This is what it would look like. // to avoid panicking on unsatisfiable models if let z3::SatResult::Sat = solver.check() { let model = solver.get_model().unwrap(); // do not ask me what the `true` is for. I don't know. let x = model.eval(&x, true).unwrap().approx_f64(); let y = model.eval(&y, true).unwrap().approx_f64(); println!("x: {x:.3}\ty: {y:.3}"); }</code></pre> Which prints x: 5.667 y: 11.333</code></pre> Nice. Isn't this grand? Multiple Solutions</h2> As I am sure you know from your high school math, some equations have multiple solutions. Here is a simple one. x * x = 4</code></pre> The Rust bindings have a nice method for getting multiple solutions out of a solver, simply called solutions</code>. It works similarly to model.eval()</code> above, and takes the same parameters with the same output. Here is the complete program. (I am going back to Int</code> because I am not cool enough for Real</code> numbers.) use z3::{Solver, ast::Int}; fn main() { let solver = Solver::new(); let x = Int::new_const("x"); solver.assert((&x * &x).eq(4)); println!("{solver:?}"); // This terminates when `check` does not return `Sat` for (idx, s) in solver.solutions(x, true).enumerate() { let s = s.as_i64().unwrap(); println!(";{}:\t{s}", idx + 1); } }</code></pre> Which prints: (declare-fun x () Int) (assert (= (* x x) 4)) ;1: -2 ;2: 2</code></pre> I am not clear really on how to get multiple solutions with the regular check</code> followed by get_model</code> method, but this one is easy enough to use. Also, some problems might have infinitely many solutions, so it is advisable to use take</code> with the solutions</code> iterator. To demonstrate, I will use the circle equation. x * x + y * y = 25</code></pre> Here is the straightforward script followed by the printed out result. Note that the fresh_const</code> API creates a unique name for every invocation. use z3::{Solver, ast::Int}; fn main() { let solver = Solver::new(); let x = Int::fresh_const("x"); let y = Int::fresh_const("x"); let area = &x * &x + &y * &y; solver.assert(area.eq(25)); println!("{solver:?}"); for (idx, (x,y)) in solver.solutions((x,y), true).enumerate() { let x = x.as_i64().unwrap(); let y = y.as_i64().unwrap(); println!("; {}:\t{x:>2},{y:?2} ", idx + 1); } }</code></pre>(declare-fun x!1 () Int) (declare-fun x!0 () Int) (assert (= (+ (* x!0 x!0) (* x!1 x!1)) 25)) ; 1: 0, 5 ; 2: -5, 0 ; 3: -3,-4 ; 4: -4,-3 ; 5: 0,-5 ; 6: 3,-4 ; 7: 4,-3 ; 8: 5, 0 ; 9: 3, 4 ; 10: 4, 3 ; 11: -3, 4 ; 12: -4, 3</code></pre> This goes without saying, but if I used the Real</code> type in this example it would generate infinite solutions. Coin Change Problem</h2> The Coin Change problem is a simple one: given a list of denominations and a total, find the smallest number of coins that add up to said total. Emphasis on smallest. Unlike previous problems, this is an optimization problem. We are looking for a solution that satisfies specific criteria instead of just a solution. Conveniently enough, z3</code> provides an Optimize</code> object which we can use to optimize. Let us set up the parameters of the problem in plain language. The denominations we have are 1, 5, and 10. We need to give 37 money in the least amount of coins. This is simple enough that we can know the solution is three 10 coins, one 5 coin, and two 1 coins. Let's see if we can get the same result. As usual, code followed by output: use z3::{Optimize, ast::Int}; fn main() { let opt = Optimize::new(); let c1 = Int::new_const("c1"); let c5 = Int::new_const("c5"); let c10 = Int::new_const("c10"); let total = (&c1 * 1) + (&c5 * 5) + (&c10 * 10); let count = &c1 + &c5 + &c10; opt.assert(&total.eq(37)); opt.minimize(&count); println!("{opt:?}"); if let z3::SatResult::Sat = opt.check(&[]) { let model = opt.get_model().unwrap(); let c1 = model.eval(&c1, true).unwrap(); let c5 = model.eval(&c5, true).unwrap(); let c10 = model.eval(&c10, true).unwrap(); println!("; c1: {c1:?}, c5: {c5:?}, c10: {c10:?}"); } else { println!("; woe for us"); } }</code></pre>(declare-fun c10 () Int) (declare-fun c5 () Int) (declare-fun c1 () Int) (assert (= (+ (* c1 1) (* c5 5) (* c10 10)) 37)) (minimize (+ c1 c5 c10)) (check-sat) ; c1: 37, c5: 0, c10: 0</code></pre> Oops. This cannot be right. I do not really understand why the answer is so nonsensical here. The problem is that Int</code> really spans the entire natural integers range, so it is accounting for negative amounts of coins. This still does not explain how the optimal solution given is 37 coins. (If you can explain, please let me know.) The solution for this^{4</a> is to constrain the amount of coins to be non-negative. So let's do that. Add these assertions somewhere before check</code>, and Bob's your uncle.} opt.assert(&c1.ge(0)); opt.assert(&c5.ge(0)); opt.assert(&c10.ge(0));</code></pre>(declare-fun c1 () Int) (declare-fun c5 () Int) (declare-fun c10 () Int) (assert (>= c1 0)) (assert (>= c5 0)) (assert (>= c10 0)) (assert (= (+ (* c1 1) (* c5 5) (* c10 10)) 37)) (minimize (+ c1 c5 c10)) (check-sat) ; c1: 2, c5: 1, c10: 3</code></pre> That's more like it. Now let's try with different denominations. Something like 10. 9, and 1. Note that the optimal solution for 37 would be: one 10 coin, three 9 coins, and no 1 coins. The greedy solution would fail to catch that. Here is the output of printing the optimizer and the result. (declare-fun c1 () Int) (declare-fun c9 () Int) (declare-fun c10 () Int) (assert (>= c1 0)) (assert (>= c9 0)) (assert (>= c10 0)) (assert (= (+ (* c1 1) (* c9 9) (* c10 10)) 37)) (minimize (+ c1 c9 c10)) (check-sat) ; c1: 0, c9: 3, c10: 1</code></pre> Success! push</code> and pop</code></h2> Currently, the total 37 is hardcoded. But what if I want the answers for a number of different totals? Thankfully, you do not need to build the optimizer from scratch for every total. Instead, use the magical functions push</code> and pop</code>. The first one essentially creates a bookmark in the stack of assertions. The second removes everything above said bookmark, and the bookmark. It is simple really. Here are the solutions from 30 to 39, because why not. Here is the full main</code>. I will spare you the output. let opt = Optimize::new(); let c1 = Int::new_const("c1"); let c9 = Int::new_const("c9"); let c10 = Int::new_const("c10"); opt.assert(&c1.ge(0)); opt.assert(&c9.ge(0)); opt.assert(&c10.ge(0)); let total = (&c1 * 1) + (&c9 * 9) + (&c10 * 10); let count = &c1 + &c9 + &c10; opt.minimize(&count); println!("; total\tcount\tc1\tc9\tc10"); for t in (30u32..).take(10) { print!("; {t}\t"); opt.push(); opt.assert(&total.eq(t)); if let SatResult::Sat = opt.check(&[]) { let model = opt.get_model().unwrap(); let c1 = model.eval(&c1, true).unwrap().as_u64().unwrap(); let c9 = model.eval(&c9, true).unwrap().as_u64().unwrap(); let c10 = model.eval(&c10, true).unwrap().as_u64().unwrap(); let count = c1 + c9 + c10; println!("{count}\t{c1:?}\t{c9:?}\t{c10:?}"); } else { println!("; woe for us"); } opt.pop(); // no RAII for you }</code></pre> Note that the push</code> and pop</code> API is available for Solver</code> as well. At any rate, back to solving. Sudoku</h2> This is a significant jump in complexity, so bear with me. We are going to solve a Sudoku.5</a> So let's write the constraints first. We can use Rust's arrays or Vec</code> to organize our z3.Int</code>s and check their constraints. First, this is the puzzle we are solving: ....94.3. ...51...7 .89....4. ......2.8 .6.2.1.5. 1.2...... .7....52. 9...65... .4.97....</code></pre> I will forgo the steps to turn that into a [[Option<u8>;9];9]</code>. Instead the code below will get that info from a get_puzzle()</code> function. let solver = Solver::new(); // Note that we're using Rust arrays here. The solver does not really know about them. let grid: [[_; 9]; 9] = array::from_fn(|i| array::from_fn(|j| Int::new_const(format!("x{i}{j}")))); // each cell contains a value 1<=x<=9 grid.iter().flatten().for_each(|i| { solver.assert(i.ge(1) & i.le(9)); }); // each row contains a digit only once for row in &grid { solver.assert(Ast::distinct(row)); } // each column contains a digit only once for idx in 0..9 { let mut col = Vec::with_capacity(9); grid.iter().for_each(|r| col.push(r[idx].clone())); solver.assert(Ast::distinct(&col)) } // each 3x3 contains a digit at most once for x_s in (0..9).step_by(3) { for y_s in (0..9).step_by(3) { let mut square = Vec::with_capacity(9); for x in (x_s..).take(3) { for y in (y_s..).take(3) { // very nested loop square.push(grid[x][y].clone()); } } solver.assert(Ast::distinct(&square)) } } // Finally, assert that each cell equals a provided clue in the given puzzle get_puzzle().iter().flatten().zip(grid.iter().flatten()).for_each(|(clue, variable)| { if let Some(clue) = clue { solver.assert(variable.eq(*clue)); } }); eprintln!("{solver:?}");</code></pre> Printing the solver after each step lets you debug whether you have your constraints correctly. The printout is over 200 lines long, so let's skip that. All we have to do next is to check the value of each cell in grid</code>. if let SatResult::Sat = solver.check() { let model = solver.get_model().unwrap(); for row in grid { for int in row { let result = model.eval(&int, true).unwrap(); print!("{result:?}"); } println!(); } } else { println!("Unsolvable") }</code></pre> And this prints out the result. You can verify for yourself whether this is correct or not. Maybe try other puzzles. Or add more constraints. You can even try the Miracle Sudoku</a>. 715894632 234516897 689723145 493657218 867231954 152489763 376148529 928365471 541972386</code></pre> One thing of note here: which is how dumb the solver is. Note that if you print out the solver, there is no notion of rows and columns and squares. It does not know any Sudoku tricks like X-wings and what have you. All the data is organized on the Rust side of things, and what is given to the solver is "these two variables cannot be the same" over and over and over again. And it just .. tells you what the rest of them are. Another thing that is not obvious at first glance, is that it does not check if there is a unique solution. The puzzle may be badly constructed and have multiple solutions, and it will happily give you one, or two, or how many you ask for. It does, however, check if it is unsolvable! Page Layout</h2> One of the famous examples of using solvers in production is .. layouting. You have a number of elements and you want to arrange them on a page, or a browser window, or whatever. So let's do a rudimentary version of that.6</a> The page we are layouting has an arbitrary size of 190mm width by 270mm tall. We are to put three boxes on the page of varying sizes and rules. I am just spitballing the sizes here: first box is 105mm by 140mm; second is 85 by 135, third is 120 by 110. They should not overlap, and like .. that's it? let page_width = 190; let page_height = 270; let solver = Solver::new(); let box_dims = [(105, 140), (85, 135), (120, 110)]; let box_locs = box_dims.map(|b| { let left = Int::fresh_const("x"); let top = Int::fresh_const("y"); let right = &left + b.0; let bottom = &top + b.1; // assert the boxes fit within the page solver.assert(&left.ge(0)); solver.assert(&top.ge(0)); solver.assert(&right.le(page_width)); solver.assert(&bottom.le(page_height)); // assert each box aligns to lines. line height is 10mm solver.assert(&bottom.rem(10).eq(0)); (left, top, right, bottom) }); // assert boxes do not overlap for i in 0..box_locs.len() { for j in i + 1..box_locs.len() { let fst_box = &box_locs[i]; let snd_box = &box_locs[j]; // tuple structure: (left, top, right, bottom) let cond_1 = fst_box.0.ge(&snd_box.2); // fst left >= snd right let cond_2 = fst_box.1.ge(&snd_box.3); // fst top >= snd bottom let cond_3 = fst_box.2.le(&snd_box.0); // fst right <= snd left let cond_4 = fst_box.3.le(&snd_box.1); // fst bottom <= snd top solver.assert(Bool::or(&[cond_1, cond_2, cond_3, cond_4])); } } // println!("{solver:?}"); if let SatResult::Sat = solver.check() { let model = solver.get_model().unwrap(); for (idx, b) in box_locs.into_iter().enumerate() { print!("box {}\t", idx + 1); let left = model.eval(&b.0, true).unwrap(); let top = model.eval(&b.1, true).unwrap(); let right = model.eval(&b.2, true).unwrap(); let bottom = model.eval(&b.3, true).unwrap(); println!("{left}, {top}, {right}, {bottom}"); } } else { println!("Unsolvable"); }</code></pre> This prints out this neat solution: box 1 85, 0, 190, 140 box 2 0, 5, 85, 140 box 3 0, 140, 120, 250</code></pre> Turned out easier than I thought. Hah. Epilogue</h2> Obviously, all these examples are rather simple. Figuring out how to model the problem in the form of boolean rules and constraints is almost all the challenge. There are some limitations, too: z3</code> cannot solve equations of the sort 2^x == 3</code>; it cannot call external functions to get values (though there are ways to work around that). There is plenty of stuff I have not shown. Array</code>s, which are nothing like programming language arrays and more mappings from one domain to another. BV</code>, bit vectors, which allow bitwise operations on their values like and</code> and or</code> and bit shifting. (Which are the key to solving the Hanging Gardens Problem</a>.) Set</code>s and Seq</code>s and String</code>s and regexes and stuff I have not really looked into. Unfortunately, most resources on the web are a bit heavy on theory, and are not targeted to stupid coders like myself. Until later. Domain specific terminology. ↩</a> </li> Probably not true. For all I know the C API constructs the model directly without the intermediate step. However, I found it much easier to understand them in this manner. ↩</a> </li> Can also use Float</code>. ↩</a> </li> Again, for reasons I do not understand. ↩</a> </li> This example is translated from The z3</code> Guide</a> ↩</a> </li> I got the suggestion for this from Simon Cozens</a>. Admittedly, implementation of this section was aided by a Robot. ↩</a> </li> </ol> </section> Define Zig Enums from Data 2025-08-13T00:00:00+00:00 I am working currently on a small Zig library which I am not quite ready to talk about yet. However, it involves parsing a big table of data and then doing operations based on that data. For the remainder of this article, I shall pretend the library is about wind directions. Note that this article uses Zig 0.14.1</code>. Big changes have happened in the master</code> branch since that was released. Though the parts discussed are probably still the same. </blockquote> Library Structure Outline</h2> There is a data file called WindData.txt</code>. It looks something like this.1</a> 00;Mistral;N 01;Bora;N 02;Sirocco;S 03;Khamaseen;S 04;Chinook;W 05;Zonda;W 06;Föhn;S 07;Santa Ana;E 08;Harmattan;E 09;Leste;E 0A;Pampero;S 0B;Nor'easter;N 0C;Southerly Buster;S 0D;Berg Wind;E 0E;Vendavales;S</code></pre> This text file is parsed by a generate.zig</code> file that organizes winds by their direction. The data is put into a wind_data.zon</code> file, which is added by the build system as an Import</code> so it is simply imported into the code base and .. calculated on. The enum</code> that describes wind directions is duplicated between the main code base and generate.zig</code>. const Direction = enum { N, E, S, W };</code></pre>Problem Statement</h2> What if a new version of the data adds a new wind direction: U</code> (for Upwards)? Now the code fails! It needs a way to create the type out of the data. A rational person would just add the new direction manually, because the algorithms depending on it are going to need to change anyway. The normal solution (in widely used rust crate wind-direction</code>, for example) is good old fashioned code generation using a generate.py</code> script that is run manually every once in a while by the maintainers, which generates a tables.rs</code> file that includes the above enum. But this is Zig. Things are not done with code generation. Things are done with comptime</code> and build systems! generate.zig</code></h2> I will spare you the boring mechanical parts of generate.zig</code> that read a file and iterate over each line and collects the data into some sort of hashmap. The interesting bit I want to talk about is generating type-definition.zon</code>. During file reading, all the direction strings are collected into a set to make sure there are no repeats. This is simple enough code. var set: std.StringArrayHashMapUnmanaged(void) = .empty; // later in the loop, try set.put(gpa, wind_direction_string, {});</code></pre> After the set is collected, it needs to be serialized into zon</code> (which is Zig's own json</code>-like data type). This is as simple as outputting text to a file, but the Zig standard library provides some helpful builtins.2</a> This is easiest way I found to serialize the strings to what Zig calls enum_literal</code>s. (I will get back to that). const out = std.io.getStdOut().writer(); // <-- replace with output file // after finishing up the set. const directions = set.keys(); var sz = std.zon.stringify.serializer(out, .{}); var container = try sz.beginTuple(.{}); for (directions, 0..) |direction, idx| { if (idx != 0) try sz.writer.writeByte(','); // a bit hacky but w/e try sz.ident(direction); } try container.end(); // maybe should print a new line here too?</code></pre> And voila! Here is the happy data.zon</code>. Note that this is written to an output file, not stdout</code> like the code snippet below. .{.N,.E,.S,.W}</code></pre>build.zig</code></h2> This is the least interesting part. It is just plumbing. Here is the code. It would get invoked whenever zig build</code> commands are called, and cached accordingly, hopefully? // calling genrate script const generate_wind_data = b.addExecutable(.{ .name = "wind_dir_data", .root_source_file = b.path("tools/generate.zig"), .target = b.graph.host, }); const generate_step = b.addRunArtifact(generate_wind_data); generate_step.addFileArg(b.path("WindData.txt")); const wind_zon = generate_step.addOutputFileArg("data.zon"); // Main library. const lib_mod = b.createModule(.{ .root_source_file = b.path("src/lib.zig"), .target = target, .optimize = optimize, }); // piping the two lib_mod.addAnonymousImport( "wind_tables", .{ .root_source_file = wind_zon }, );</code></pre> And that's it. Fairly simple code if you're familiar with the Zig build system. The Library</h2> Here is the cool stuff, if I may. Since this for defining a normal, non-generic type, we shall start it thus. All the code in the following snippets will be in there. const WindData = WD: { // All following code is in here. };</code></pre> First, we need to @import</code> the data. That's why it is in a zon</code> file to begin with! But import as what type? In 0.14.1</code>, we still need to define the types zon</code> imports (a restriction I understand is lifted in master</code>). Zig has a special comptime</code> only type called enum_literal</code>. Oddly enough, unlike comptime_int</code> and comptime_float</code>, you cannot just do this: // does not compile const foo: enum_literal = .bar;</code></pre> The type can actually be written out as @TypeOf(.enum_literal)</code>. Sure. A const</code> slice of those would be []const @TypeOf(.enum_literal)</code>. Not that you need the const</code> here for deserialization, as you cannot deserialize into a mutable slice. const data: []const @TypeOf(.enum_literal) = @import("test.zon");</code></pre> Then we need to iterate over the fields and convert them, one by one, to another type. This time it is std.builtin.Type.EnumField</code>. An array of which will become our final enum</code>. We create a BoundedArray</code> at home with just an index and an iterator. This code was pretty much lifted from Mitchell Hashimoto's 'Tagged Union Subsets with Comptime in Zig' article</a>, which is actually the inspiration for this post.3</a> var i: usize = 0; var fields: [data.len]std.builtin.Type.EnumField = undefined; // outer: // uneeded block label. see below for (data) |literal| { const name = @tagName(literal); // // Deduplication code. It is not needed because the file is // // generated from a hash set. // for (fields[0..i]) |f| // if (std.mem.eql(u8, f.name, name)) // continue :outer; fields[i] = .{ .name = name, .value = i, }; i += 1; }</code></pre> Now we simply create the enum</code> with the helpful builtin @Type</code>. break :WD @Type(.{ .@"enum" = .{ .tag_type = usize, .fields = fields[0..i], .decls = &.{}, .is_exhaustive = true, // maybe? } });</code></pre> Fairly simple. One small trick remains. tag_type</code> is not an optional value, so we need to specify it. It is the integer that should hold the results of the enum.. usize</code> is playing it safe, because it is extremely unlikely to hold more than .. whatever the largest usize</code> is. But that's a very big enum for something than potentially fit into one byte. So it is incumbent upon us to create the smallest integer possible. const backing_int: type = @Type(.{ .int = .{ .bits = // what to put here?? .signedness = .unsigned, } });</code></pre> To calculate the smallest number of bits that can hold our fields</code> count, two simple (for computers at least) math operations are needed. First, i</code> (our counter) is raised to the nearest power of two. Then the log2</code> is taken from it. Boom. Two std.math</code> function calls and that is IT. const next_p_of_2 = std.math.ceilPowerOfTwoAssert(usize, i + 1); const bits = std.math.log2_int(usize, next_p_of_2);</code></pre> And this does it. Here is the full code. const WindData = WD: { const data: []const @TypeOf(.enum_literal) = @import("test.zon"); var i: usize = 0; var fields: [data.len]std.builtin.Type.EnumField = undefined; for (data) |literal| { const name = @tagName(literal); fields[i] = .{ .name = name, .value = i, }; i += 1; } const next_p_of_2 = std.math.ceilPowerOfTwoAssert(usize, i + 1); const bits = std.math.log2_int(usize, next_p_of_2); break :WD @Type(.{ .@"enum" = .{ .tag_type = @Type(.{ .int = .{ .bits = bits, .signedness = .unsigned, } }), .fields = fields[0..i], .decls = &.{}, .is_exhaustive = true, } }); };</code></pre> Your enum</code> is ready. Now your data definition is resilient to new types added to the database. You still need to fix all the switch</code> statements, tho. Conclusion</h2> In the actual library I just put out the enum definition into its own separate file that is imported by both generate.zig</code> and the library. But hey, this was a fun exercise. Until later. Generated with Gemini help. ↩</a> </li> Unlike formatting and json</code> serialization, zon</code> serialization does not seem to depend on defining specific magic functions</a> to customize the behaviour. ↩</a> </li> Also, slightly related, is matklad's Partially Matching Zig Enums</a> ↩</a> </li> </ol> </section> LLMs in the Building Industry 2025-07-13T00:00:00+00:00 This is a collection of ideas that may or may not be feasible to implement or to have as the base idea of a start up. It is not a structured document, but more of a stream of thought. IFC</h2> Industry Foundation Classes</a>1</a> is an open source data format for modeling BIM (Building Information Modeling). It is supported by Revit, the market leader. A well known open source implementation is Bonsai BIM</a>. I believe the implementation is inherently text based and can be accessed with programmer oriented tooling such as git</code>. There are graphical tools that allow the user to see the difference between two files visually. An added wall in this revision would show up as green, for example. A deleted wall would be shown in red. This is all existing tooling. For Potential Homeowners</h2> Scenario</h3> Imagine the following: a website where a client can submit their plot details, their budget, and make a write up on what they want in their dream house. For example: They have three kids and their eldest wants to be a ballerina so they need a dance studio space. The father works from home so he needs a home office and a quiet space for virtual meetings. The mother is a hobbyist swimmer so they want a swimming pool. They have no need for a specific guest space, etc. Whatever a potential customer would tell their architect. The website would take the hard data from the plot and the plot's location, and the specifics of the local area building code, and the text description provided by the client (the more detailed the better). Then out of these disparate pieces of the info and combine them together into the ideal house for the subject. Sanity Checks</h3> This would require some sort of sanity checker to verify the results, similar to how LLM-based coding agents work with compilers and language serverso.[lsp] The LLMs hallucinate whatever they want, and the Editor tells them whether the code compiles or not. Sanity checks include, but are not limited to, code compliance, cost overruns, "there is a room without a door", etc. This process can be automated very far. IFC can include data relevant to the other disciplines than Architecture. Most importantly, structural design. This should be an algorithmic task as it is a life and death matter and there is no room for hallucinations</del> creativity. The other disciplines such as plumbing, electric wiring, etc is doable and any mistakes can be easily adjusted (or completely ignored) on site by the installation teams. The IFC files also include dimensions. Combine that with the specifications, or the approved suppliers, (which the client may or may not choose from a drop down menu), and you get a quick and easy to calculate cost. There must be a qualified human reviewer, as the final sanity check. More importantly, to shoulder the legal liability of architectural and structural designs. The price could be fairly cheap: 1,000SAR for one plan. The absolute cheapest office is about 5,000, and these are the ones just pulling a predesigned house out of the drawer. Scale</h3> Working at scale, a similar technology can be used to benefit real estate developers. Instead of the usual cookie cutting measure of a thousand identical houses, variation can be made in each individual unit. Depending on the construction methods used, this might not be more expensive to actually build than cookies. For Engineering Offices</h2> There are many different professions working in a regular Engineering2</a> practice. This is an overview following the different stages of a project. Client Brief</h3> Every project starts with a brief.3</a> The brief includes an intro the project and the different stakeholders and potential desires etc. To respond to the brief, offices are required usually to make both a technical offer and a financial offer. The technical offer is essentially a listing of the Office's capabilities. What is their experience like? What are their core competencies? What big projects they have done before? And what have you. The exact format is usually specified by the client, in the brief, but the details are generally all the same. The same info and files are regurgitated between different projects. The financial offer is more of the same. It is based on however the office wants to to price the subject. The structure of which is also often dictated by the brief. It could be by man hours or by lump sum or depending on the phase of the moon. The cost can depends on the size of the project and the nature of the client (government vor private). This is a lot of clerical and reptitive work that nonetheless must be customized for each client. It involves a large number of staff from different departments: accounting, secretaries, engineering, const estimation (which could fall under engineering). Specifications</h3> Specifications are the most important, and perhaps the most overlooked, part of the design documents. They can either be dictated by the client or, more appropriately, delivered as part of the service. First, a brief explanation of what Specifications are. Take for example masonry. The specification for masonry should specify the following: Potential suppliers,4</a></li> Desired properties: size, material,</li> How to install</li> How to maintain</li> </ul> Sepcifications for large projects can include everything up to the type of screw used to fasten the furniture together. They would be used by the eventual contractor to procure the proper materials. A common practice in offices is to have a local databaee of approved, or potential, suppliers of different building materials. Indeed Engineering offices are often the first line of marketing for any building material supplier. Large offices have their own specifications library. Others make them up as they go along based on the experience of the engineer responsible for them at the time, copy pasting at need from previous projects. Some companies actually sell their specifications, and supplier databses, to potential Architectural offices. One example of that is AIA MasterSpec</a>. A potential use of LLMs here is to genrate these specifications from a local supplier database. Perhaps even contrasting that with the client's brief and their desired level of quality. The Model could be given a template for specifications, or the specirications for a previous project, and directed to generate specificatiosn based on the design drawings and written guidelines and up to date supplier database. Consultancy</h3> Work for Engineering offices is not restricted to design. Indeed the more profitable line of work is construction consultancy, or supervision. Reviewing the contraxctor's submittals for materials, shop drawings,5</a> staff listings, Requests For Clarification (RFCs), and what have you. The reviews could be given to a LLM, as a first pass before a human reviewer looks at them. But I do not think this is a valuable use as the details that would fail a review, especially for technical drawings, are something that is still currently beyond the scope of language models. I will touch on this again when talking about contractors, but a record of all communication is extremely important to maintain throughtout the life cycle of the project. If, say, a feature of the building turned out not to have the desired quality, the consultant can point to the communication where they denied the contractor's shop drawings or chosen materials, but were overruled by the client in the interest of expediency or whatever. Someone, or something, which can read and remember and vaguely understand the conversation, would be invaluable to, well, win arguments. The methods of communication vary by project. Either they are official stamped by the treplicate official letters, or more commonly WhatsApp messages, or anything inbetween. For Construction Companies</h2> Much of the talk about the client briefs and comparing specifications with suppliers database applies here as well. Searching a local database, not necessarily neatly organized with all the relevant info buried in PDFs and websites, to find suppliers whose products match the given specifications. There are also many applications for logistics. How to store the materials and how to transport the materials and how to protect them from the elements are all common challenged. But not unique to construction work. So solutions from other industries (such as manufacturing) could be used. Cost Estimations</h3> This is perhaps a task more appropriate for proper old algorithms than language models. but a common painpoint small contractors have is how to properly estimate the cost of a building, based on drawings and a client brief. The drawings themselves are usually presented as PDFs or as DWG files (sometimes). One would be truly lucky to get the drawings in IFC (or really any BIM) format. PDFs and DWGs include no metadata about the project's elements. The data is provided by things designed for human readers: hatching, dimensions, and labels. There are a few ways this can be approached. Either reading the drawings as a human would: looking up the labels and cross referencing them with the specifications (if any). Measuring the length of different elements, walls and windows and what have you, multiplying that by estimated labor and material costs. All directly from the drawings. The other approach is to create a IFC model based on the drawings, and have the regular BIM tooling to calcluate this data. The prices could be tied to some live database of material prices, or not. Waste Reduction</h3> Note this common particular scenario. Structual steel reinforcement comes in 12m long bars. A common floor height is 4m. However, due to structual requirements that are beyond my understanding, the bars embedded in columns should be one meter longer. This means that one bar can be cut in two 5m long pieces. You end up with a useless wasted 2m piece. There are about 12 bars in every column and the floor of a typical house can have around 25 columns. You do the marh. There are different ways to solve this: a price concious Structural Engineer can reuse the same bar width in different locations allowing every piece to be used for other building elements.6</a> Excellent, and expensive, labor force can find creative uses for the waste. Or the project manager can sell them for scrap to another project that might use them. The problem exists beyond structural steel, however. Masnory units can have up to 15% waste or more, due to very different factors. Getting careful labor that minimizes waste is potentially more expensive than the material wasted. Different building methods, such as bearing walls, are not quite .. approved of in the local market.7</a> A resourceful project manager or designer can find creative uses for the waste. I am actualy unsure how algorithms or AI can help here. I am mostly typing out thoughts. However, in a regular concrete building, a commonly used perishable, but expensive, material is formwork. Most formwork used in Saudi Arabia is wood: either plywood or "white wood". It is expensive to buy. Expensive to rent. And it is necessary to cut to fit the weird concrete shapes architects come up with. And laborers, especially if it is not their wood, are extremely careless in economizing it. A few real life products can be used to solve this problem. Aluminum and Plastic based formwork is infinitely more reusable, but because it cannot be cut, it is not as flexible for different shapes. the ideal solution would be to design, from the start, for minimizing formwork waste. However, as this is the Contractors section, an algorithm to minimize waste on a given design (how to cut the boards in a reusable manner, etc.) would be extremely useful to contractors. Clarifications and Change Orders</h3> Managing clients and consultants is perhaps the Contractor's most difficult task. The scopes keep on expanding and the specifications keep on tightening while prices and schedules are expected to remain the same. The drawings or specificiations are often incomplete and missing vital info, or just plain wrong. Having a record of all communications (often called Document Management) is extremely important for all parties involved. Being able to search and query the often disorganized communication records to find all the different instructions from the client and consultant is evidently important. These clarifications can begin even before signing the contract. When bidding on the project, the contractor is expected, but not really, to completely understand and fully evaluate the contract documents (drawings and specifications), and point out any missing info. A bid is, usually, essentially, an agreement that the contract documents are completely clear. However, analysis and fully reading and grokking the documents is very time consuming, potentally money-losing, work. Automating that discovery phase would be a huge boon to contractors bidding. The Model does not need to be extremely accurate. A few false negatives and false positives would be very tolerated, as long as it is more accuracy than a human reviewer in a reasonable amount of time (a couple of work days, maybe). You would get the contract documents, feed them to the Model, perhaps with prior art, and ask does it have any ommisions or incomplete specifications of unlabelled, unspecified, elements or any contradictions. A human reviewer would go over whatever the Model flags and go on with their day. This verification step cen be useful for Engineering Offices as well. Conclusion</h2> In conclusion, this documents outlines a few ways which Large Language Models and Algorithms can be used to ease the pain of the different people in the construction process. These omit some things like dealing with government regulations and accounting and human resources and what have you: the lements common in almost eveyr business. But it tpouches, instead, what is perhaps unique to the profession. A pun on "Issued For Construction". ↩</a> </li> Including Architectural and Urban Planning practices. ↩</a> </li> Basically what the potential homeowner in the previous example wrote in our website's entry ↩</a> </li> Not allowed in Saudi government contracts, I believe. ↩</a> </li> Exact drawings of what will actually be built. ↩</a> </li> Unfortunately, apprently welding two bars together is not an acceptable option. ↩</a> </li> "What if I need to change the layout?" is a common refrain, even if nobody actually does. ↩</a> </li> </ol> </section> Writing a C Compiler, in Zig 2025-06-17T00:00:00+00:00 This is a series of articles I wrote while writing paella</code></a>, following Nora Sandler's Writing a C Compiler</a>. It was both an exercise to learn Zig and a way to waste time instead of looking for work, as I am currently "between jobs". I did not edit them as I collect them here outside of fixing some broken links. Chapter 1: Intro</a></li> Chapter 2: Unary</a></li> Chapter 3: Binary</a></li> Chapter 4: Logic</a></li> Chapter 5: Variables</a></li> Chapter 6: Conditions</a></li> Chapter 7: Blocks</a></li> Chapter 8: Loops</a></li> Chapter 9: Functions</a></li> Chapter 10: Linking</a></li> </ul> If/when I continue with the book, I shall post the following writeups here. Chapter 10: Linking 2025-06-04T00:00:00+00:00 This is the final chapter of Part 1 of Writing a C Compiler</a>. Something about static variables? Lexer</h2> The Book spends 15 pages talking about linkage and static variabeles and tentative declarations, etc, before it gets to the lexer. Only two new keywrods are needed: static</code> and extern</code>. This is the current keyword static map: pub const keywords = std.StaticStringMap(Tag).initComptime(.{ .{ "int", .type_int }, .{ "return", .keyword_return }, .{ "void", .keyword_void }, .{ "if", .keyword_if }, .{ "else", .keyword_else }, .{ "do", .keyword_do }, .{ "while", .keyword_while }, .{ "for", .keyword_for }, .{ "break", .keyword_break }, .{ "continue", .keyword_continue }, .{ "static", .keyword_static }, .{ "extern", .keyword_extern }, });</code></pre> As a small update to the lexer's API, I added a next_force</code> function that returns an error union rather than an option, as it is easier to deal with at the call site. pub inline fn next_force(self: *Tokenizer) !Token { return self.next() orelse return error.NotEnoughJunk; } // usage var token = try tokens.next_force(); // rather than var token = tokens.next() orlse return error.NotEnoughJunk;</code></pre>AST and Parser</h2> The parser requires deeper changes. First of all, as Prgm</code> changed from a single function to multiple functions last chapter, now it is a list of declarations. Also, static</code> and extern</code> belong, by themselves, to a new semi-node for "storage class specifiers", that is an optional parameter for function and variable declarations. Since a declaration cannot be both static</code> and extern</code>, this makes it a simple enum. pub const Prgm = struct { funcs: std.SegmentedList(Decl, 0), }; pub const StorageClass = enum { static, @"extern" }; pub const FuncDecl = struct { sc: ?StorageClass, // <-- name: []const u8, params: std.SegmentedList(Identifier, 0), block: ?Block, }; pub const VarDecl = struct { sc: ?StorageClass, // <-- name: Identifier, init: ?*Expr, }</code></pre> Changes to parse_prgm</code> are simple enough. There already is a parse_decl</code> so I will just use that one when iterating. Parsing specifiers is interesting and annoying, as I relied before on looking for the int</code> token, now it can be anything! Consider these tqo equally valid variable declarations. static int example1 = 1; int static example2 = 2;</code></pre> There is no set order. Whose genius idea was this? Thankfully I do not have to think of this on my own, as the book privdes a nice pseudocode functon to parse them. When I was doing my Rust implementation, where I was using nom</code> for parsing, I could not actually figure out to make it nicer than what the Book already provided. Here is the function translated from pseudo code to Zig.1</a> fn parse_storage_class( tokens: *lexer.Tokenizer, ) Error!?ast.StorageClass { var type_seen = false; var sc: ?ast.StorageClass = null; while (tokens.next()) |nt| switch (nt.tag) { .type_int => type_seen = true, .keyword_static => sc = if (sc == null) .static else return error.InvalidStorageClass, .keyword_extern => sc = if (sc == null) .@"extern" else return error.InvalidStorageClass, else => { tokens.put_back(nt); break; }, } else return error.NotEnoughJunk; if (!type_seen) return error.InvalidTypeSpecifier; return sc; }</code></pre> Then use this new function in both parse_func_decl</code> and parse_variable_decl</code>. In fact, it is probably easier right now to just simply unify the two into a horrendous giant amalgamation. fn parse_decl( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ty: union(enum) { @"var": ?ast.StorageClass, either }, ) Error!ast.Decl { const sc = if (ty == .@"var") ty.@"var" else try parse_storage_class(tokens); const name = try expect(.identifier, tokens); const new_token = try tokens.next_force(); switch (new_token.tag) { .semicolon => return .{ .V = .{ .sc = sc, .name = .{ .name = name }, .init = null, } }, .equals => { const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.semicolon, tokens); return .{ .V = .{ .sc = sc, .name = .{ .name = name }, .init = expr_ptr, } }; }, .l_paren => { if (ty == .@"var") return error.SyntaxError; var params: std.SegmentedList(ast.Identifier, 0) = .{}; const param_peek = try tokens.next_force(); params: switch (param_peek.tag) { .keyword_void => try expect(.r_paren, tokens), .type_int => { const ident = try expect(.identifier, tokens); try params.append(arena, .{ .name = ident }); const inner_param_peek = try tokens.next_force(); switch (inner_param_peek.tag) { .comma => { try expect(.type_int, tokens); continue :params .type_int; }, .r_paren => {}, else => continue :params .invalid, } }, else => return error.SyntaxError, } const block_peek = try tokens.next_force(); const block = block: { switch (block_peek.tag) { .semicolon => break :block null, .l_brace => { tokens.put_back(block_peek); break :block try parse_block(arena, tokens); }, else => return error.SyntaxError, } }; return .{ .F = .{ .sc = sc, .name = name, .params = params, .block = block, } }; }, else => return error.SyntaxError, } }</code></pre> The ty</code> parameter is because there is one place in the AST that requires a variable declaration specifically, which is the initial part of a for</code> loop. (It is not comptime</code> because I use to sneak in an already parsed sc</code> value, as you will see.) Everywhere else can really take either declaration type. This is the part in for</code> loop parsing that deals with the new parsing functions. const init: ast.Stmt.For.Init = init: { const peeked = try tokens.next_force(); if (peeked.tag == .semicolon) break :init .none else { tokens.put_back(peeked); if (parse_storage_class(tokens)) |sc| { const decl = try parse_decl( arena, tokens, .{ .@"var" = sc }, // <-- snuck in ); const decl_ptr = try utils.create(arena, decl.V); // <-- unwrapped break :init .{ .decl = decl_ptr }; } else |_| { // <-- parse_sc already puts back any other token const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.semicolon, tokens); break :init .{ .expr = expr_ptr }; } } };</code></pre> Running the eye test I quickly ran into a bug. *Sigh. Debugging the Parser</h2> It failed on the first test, so obviously this is a bug in the general algorithm. With all the new changes all at once it is ahrder to pinpoint which one it is. I did not make a separate commit for every refactor, I am afraid. This is the C file it failed on (after removing comments). int putchar (int ch); int print_letters(void) { static int i = 65; putchar(i); { i = i + 1; static int i = 97; putchar(i); i = i + 1; } putchar(10); return 0; } int main(void) { for (int i = 0; i < 26; i = i + 1) print_letters(); }</code></pre> Adding debugging printers to next()</code> and put_back()</code>, as done previously, will let me see where it failed in the file. I could also print the failed token's span, but that would require better architecture and more dedication. I get this stream of consciusness back. I added comments to each line, manually, to be honest, to track which function does which. NEXT type_int 0-3 # parse_prgm REWIND type_int 0-3 # parse_prgm NEXT type_int 0-3 # parse_decl -> parse_storage_class NEXT identifier 4-11 # parse_storage_class REWIND identifier 4-11 # parse_storage_class NEXT identifier 4-11 # parse_decl NEXT l_paren 12-13 # parse_decl NEXT type_int 13-16 # parse_decl # params NEXT identifier 17-19 # parse_decl # params NEXT r_paren 19-20 # parse_decl # params NEXT semicolon 20-21 # parse_decl # block NEXT type_int 22-25 # parse_prgm REWIND type_int 22-25 # parse_prgm NEXT type_int 22-25 # parse_decl -> parse_storage_class NEXT identifier 26-39 # parse_storage_class REWIND identifier 26-39 # parse_storage_class NEXT identifier 26-39 # parse_decl NEXT l_paren 39-40 # parse_decl NEXT keyword_void 40-44 # parse_decl # params NEXT r_paren 44-45 # parse_decl # params NEXT l_brace 46-47 # parse_decl # block REWIND l_brace 46-47 # parse_decl # block NEXT l_brace 46-47 # parse_block NEXT keyword_static 52-58 # parse_block REWIND keyword_static 52-58 # parse_block NEXT keyword_static 52-58 # parse_block_item REWIND keyword_static 52-58 # parse_block_item NEXT keyword_static 52-58 # .. oh REWIND keyword_static 52-58 NEXT keyword_static 52-58 error: SyntaxError</code></pre> Oh yeah here it is. I forgot to update it and parse_block_item</code> still checks for whether something is a declaration or a statement using the int</code> keyword. I could use a similar trick to what I did for the for</code> loop where I can pass the parsed value in instead of peeking and parsing it again. This would require changes to parse_decl</code>s signature, so I will start with that. This is the current signature. fn parse_decl( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ty: union(enum) { @"var": ?ast.StorageClass, either }, ) Error!ast.Decl { const sc = if (ty == .@"var") ty.@"var" else try parse_storage_class(tokens); // etc</code></pre> Instead of smuggling the parsed value through ty</code>, I could change ty</code> to my earlier iteration of an enum</code>, and pass whether it is parsed or not in another variable, like so. fn parse_decl( comptime ty: enum { @"var", either }, arena: std.mem.Allocator, tokens: *lexer.Tokenizer, parsed: union(enum) { yes:?ast.StorageClass, no }, ) Error!ast.Decl { const sc = if (parsed == .yes) parsed.yes else try parse_storage_class(tokens); // etc</code></pre> And changing the call sites, again, but no matter. Here is the new and improved parse_block_item</code>, relying on parse_storage_class</code> putting back any unrecognized tokens. fn parse_block_item( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.BlockItem { if (parse_storage_class(tokens)) |sc| return .decl(try parse_decl(.either, arena, tokens, .{ .yes = sc })) else |_| return .stmt(try parse_stmt(arena, tokens)); }</code></pre> And voila! Running the parser on the same C file above renders the following AST (with some updated printing for variable declarations). PROGRAM FUNCTION putchar ch FUNCTION print_letters static VARIABLE i <- 65 (putchar i); DO (i <- (+ i 1)); static VARIABLE i <- 97 (putchar i); (i <- (+ i 1)); (putchar 10); RETURN 0 FUNCTION main FOR VARIABLE i <- 0; ( (print_letters);</code></pre> The eye test passes. The actual test suite passes. All is done and all is happy with the tiny world of this project. Semantic Analysis</h2> The main difference in here from the previous chapters is that the symbol table needs to survive beyond this phase, as it is used in assembly. So far, in my compiler, the info is not kept in ir_gen</code>. This would probably have to change. Here my idea of doing type checking in the same pass as identifier resolution is put to the test for the first, and probably last, time. One thing to note, is that variables that do have linkage (whether internal or external) do not get their name mangled, because their name is how the linker tracks them. The first thing I will do is resolving file scope variables. Instead of mucking around in resolve_var_decl</code>, it is much smaller code that I will inline in resolve_prgm</code> immediately. pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!void { var variable_map: VariableMap = .empty; defer variable_map.deinit(gpa); var type_map: TypeMap = .empty; defer type_map.deinit(gpa); const bp: Boilerplate = .{ .gpa = gpa, .strings = strings, .variable_map = &variable_map, .type_map = &type_map, }; var iter = prgm.decls.iterator(0); while (iter.next()) |item| switch (item.*) { .F => |*f| try resolve_func_decl(bp, f), // right here, never mind the `.name.name`. // no type checking yet .V => |*v| try variable_map.put(gpa, v.name.name, .{ .name = try strings.get_or_put(gpa, v.name.name), .linkage = .has_linkage, }), }; }</code></pre> Much of the complexity here is deferred in the Book to the type checking stage, so I presume I am heavily editing this part before the section is over. NExt comes resolve_var_decl</code>. Unfortunately, some of the clever shenaningans I did to unify resolution for paramaters and variables are making this a bit more complex, but I persevere. fn resolve_local_var_decl( comptime T: enum { param, @"var" }, bp: Boilerplate, item: switch (T) { .@"var" => *ast.VarDecl, .param => *ast.Identifier, }, ) Error!void { const identifier, const sc: ?ast.StorageClass = switch (T) { .@"var" => .{ &item.name, item.sc }, // <-- getting sc here .param => .{ item, null }, }; if (bp.variable_map.get(identifier.name)) |prev| if (prev.scope == .local) if (!(prev.linkage == .has_linkage and sc == .@"extern")) // <-- return error.DuplicateDecl; if (sc == .@"extern") { // <-- try bp.variable_map.put(bp.gpa, identifier.name, .{ .name = try bp.strings.get_or_put(bp.gpa, identifier.name), .linkage = .has_linkage, }); // no type checking yet return; } // snip --</code></pre> Regarding function declarations, the compiler should throw an error if a local function declaration is static</code>. I am unsure where to put this check. I currently check if a block scope function declaration has a body (which it should not) in resolve_block</code>, so maybe that is where I will stuff that check. fn resolve_block( bp: Boilerplate, current_label: ?utils.StringInterner.Idx, block: *ast.Block, ) Error!void { var iter = block.body.iterator(0); while (iter.next()) |item| switch (item.*) { .S => |*s| try resolve_stmt(bp, current_label, s), .D => |*d| switch (d.*) { .F => |*f| if (f.block) |_| return error.IllegalFuncDefinition else if (f.sc == .static) // <-- here return error.TypeError else try resolve_func_decl(bp, f), .V => |*v| try resolve_local_var_decl(.@"var", bp, v), }, }; }</code></pre> Next up is type checking. Instead of tracking each identifier's type in a type_map</code>, the compiler would track the attributes.^{2</a> So this the previous entry in TypeMap</code>:} const Type = union(enum) { int, func: struct { arity: usize, defined: bool, }, };</code></pre> This becomes the folloiwng. This is not identical to the Book's design as it has its entry as a tuple of type and attributes. I will cross that bridge if I come to it. const Arrtibutes = union(enum) { func: struct { arity: usize, defined: bool, global: bool, }, static: struct { ubut: Init, global: bool, }, local, const Init = union(enum) { tentative, initial: u64, none, }; };</code></pre> Simply updating the existing TypeMap</code> to have the new type as its value is enough for now. Now, back to writing code. Function declarations need to make some checks about their globality: whether they are static</code> or not. Tracking defined functions is improved from last run as well. { // func_decl type checking const gop = try bp.type_map.getOrPut(bp.gpa, nname.real_idx); if (gop.found_existing) { if (gop.value_ptr.* != .func or gop.value_ptr.func.arity != func_decl.params.count()) { return error.TypeError; } else if (gop.value_ptr.func.defined and func_decl.block != null) { return error.DuplicateFunctionDef; } else if (gop.value_ptr.func.global and // <-- new check func_decl.sc == .static) { return error.ConflictingFuncDecls; } gop.value_ptr.func.defined |= func_decl.block != null; // <-- update } else gop.value_ptr.* = .{ .func = .{ .arity = func_decl.params.count(), .defined = func_decl.block != null, .global = func_decl.sc != .static, // <-- new field } }; }</code></pre> File scope variable declarations, which I have stubbed in resolve_prgm</code> above, also need to by type checked. This is significantly longer, and is a bit more complex than I can process at the moment of writing so I copied it from the book without trying to refactor. I am sure I made a mistake anyway. .V => |*v| { // file scope variables const nname = try strings.get_or_put(gpa, v.name.name); try variable_map.put(gpa, v.name.name, .{ .name = nname, .linkage = .has_linkage, }); // TYPE CHECKING var init_value: Arrtibutes.Init = if (v.init) |e| switch (e.*) { .constant => |i| .{ .initial = i }, else => return error.NonConstantInit, } else if (v.sc == .@"extern") .none else .tentative; var global = v.sc != .static; const gop = try type_map.getOrPut(gpa, nname.real_idx); if (gop.found_existing) { if (gop.value_ptr.* != .static) return error.TypeError; if (v.sc == .@"extern") global = gop.value_ptr.static.global else if (gop.value_ptr.static.global != global) return error.ConflictingLinkage; if (gop.value_ptr.static.init == .initial) { if (init_value == .initial) return error.ConflictingDefinitions; init_value = gop.value_ptr.static.init; } else if (init_value != .initial and gop.value_ptr.static.init == .tentative) { init_value = .tentative; } } gop.value_ptr.* = .{ .static = .{ .init = init_value, .global = global, } }; },</code></pre> There is a strong argument to be made that this should live in its own function. Resolving local scope variables is equally annoying. You can probably tell I am bored with this already. fn resolve_local_var_decl( comptime T: enum { param, @"var" }, bp: Boilerplate, item: switch (T) { .@"var" => *ast.VarDecl, .param => *ast.Identifier, }, ) Error!void { const identifier, const sc: ?ast.StorageClass = switch (T) { .@"var" => .{ &item.name, item.sc }, .param => .{ item, null }, }; if (bp.variable_map.get(identifier.name)) |prev| if (prev.scope == .local) if (!(prev.linkage != .none and sc == .@"extern")) return error.DuplicateDecl; if (sc == .@"extern") { const nname = try bp.strings.get_or_put(bp.gpa, identifier.name); try bp.variable_map.put(bp.gpa, identifier.name, .{ .name = nname, .linkage = .has_linkage, }); { // TYPE CHECKING if (item.init != null) { return error.TypeError; } const gop = try bp.type_map.getOrPut(bp.gpa, nname.real_idx); if (gop.found_existing) { if (gop.value_ptr.* == .func) return error.TypeError; } else gop.value_ptr.* = .{ .static = .{ .init = .none, .global = true, } }; } return; } const unique_name = try bp.make_temporary(identifier.name); try bp.variable_map.put(bp.gpa, identifier.name, .{ .name = unique_name }); { // TYPE CHECKING const attr: Arrtibutes = if (sc == .static) ret: { const init_value: Arrtibutes.Init = if (item.init) |e| switch (e.*) { .constant => |i| .{ .initial = i }, else => return error.TypeError, } else .{ .initial = 0 }; break :ret .{ .static = .{ .init = init_value, .global = false, } }; } else .local; const gop = try bp.type_map.getOrPut(bp.gpa, unique_name.real_idx); if (gop.found_existing) { if (gop.value_ptr.* == .func) return error.TypeError; } gop.value_ptr.* = attr; } identifier.* = .{ .idx = unique_name }; if (T == .@"var") if (item.init) |expr| try resolve_expr(bp, expr); }</code></pre> I do not like how this chapters is all these weird intermingling if</code>s. It is very confusing logic and horrible to debug. Running the test suite, I got one failure, which is not catching the error in this C code. int main(void) { int x = 0; /* a variable declared in a for loop header cannot have a storage class. */ for (static int i = 0; i < 10; i = i + 1) { x = x + 1; } return x; }</code></pre> Well, then. I do not think I will add another if</code> statement in that monster function just for for</code> loops. I will go back and reject that at the parsing stage instead. Boom: tests pass. Internal Representation</h2> It makes sense in a weird twisted manner to have the String Interner itself hold the type info that the semantic analsys phase has been collecting. It makes perfect sense if you ask me, and it keeps me from carrying yet another type around. Changing the value of the internal map to ?sema.Attributes</code>, where the strings that do not have related values would simple be null. This, however, requires more work than I am currently willing ti put in, so I will have the semantic analysis function just return the collected type map. pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!TypeMap { // yada yada return type_map; }</code></pre> The IR syntax tree changes in a similar fashion too. Programs are now a list of TopLevel</code>s, which are either a function or a StaticVar</code>. Functions also track whether they are global</code>. pub const Prgm = struct { items: std.ArrayListUnmanaged(TopLevel), }; pub const TopLevel = union(enum) { F: FuncDef, V: StaticVar, }; pub const FuncDef = struct { name: Identifier, global: bool, params: std.ArrayListUnmanaged(Identifier), instrs: std.ArrayListUnmanaged(Instr), }; pub const StaticVar = struct { name: Identifier, global: bool, init: u64, };</code></pre> Now, to generate these new items, the type_map</code> returned from semantic analysis is now an input parameter. This is used to determine which function declarations are global. This is the new prgm_emit_ir</code>. pub fn prgm_emit_ir( alloc: std.mem.Allocator, strings: *utils.StringInterner, type_map: *sema.TypeMap, // might make sense to move this to `utils` prgm: *const ast.Prgm, ) Error!ir.Prgm { var top_level: std.ArrayListUnmanaged(ir.TopLevel) = try .initCapacity( alloc, prgm.decls.len, ); var iter = prgm.decls.constIterator(0); while (iter.next()) |d| if (d.* == .F) if (d.F.block) |_| { var f_ir = try func_def_emit_ir(alloc, strings, &d.F); // assertions galore. so much hidden control flow f_ir.global = type_map.get(f_ir.name.real_idx).?.func.global; try top_level.append(alloc, .{ .F = f_ir }); }; return .{ .items = top_level }; }</code></pre> Secondly, all variable declarations with static</code> or extern</code> specifiers should not be generated. Because they have linkage and therefore belong to the Prgm</code> scope. fn var_decl_emit_ir( bp: Boilerplate, decl: *const ast.VarDecl, ) Error!void { if (decl.sc == null) if (decl.init) |e| { const src = try expr_emit_ir(bp, e); try bp.append(.{ .copy = .init(src, .{ .variable = decl.name.idx }) }); }; }</code></pre> Now, back in prgm_emit_ir</code>, the type_map</code> is traversed for every static variable. Here is hoping the semantic analysis phase tracked these correctly. { // STATIC VARS var iter = type_map.iterator(); while (iter.next()) |entry| if (entry.value_ptr.* == .static) { const name: utils.StringInterner.Idx = .{ // hacks on hacks .real_idx = entry.key_ptr.*, .strings = strings, }; const global = entry.value_ptr.static.global; const init = switch (entry.value_ptr.static.init) { .initial => |i| i, .tentative => 0, .none => continue, }; try top_level.append(alloc, .{ .V = .{ .name = name, .global = global, .init = init, } }); }; }</code></pre> And .. that's fairly it. Doing the eye test on IR generation seems to work fine. Here is one sample. Running the test suite works fine as well. static int my_fun(void); int call_static_my_fun(void) { return my_fun(); } int call_static_my_fun_2(void) { int my_fun(void); return my_fun(); } extern int my_fun(void); static int my_fun(void); int my_fun(void) { static int i = 0; i = i + 1; return i; } //PROGRAM // global FUNCTION call_static_my_fun // fn.1 <- my_fun() // ret fn.1 // ret 0 // global FUNCTION call_static_my_fun_2 // fn.2 <- my_fun() // ret fn.2 // ret 0 // FUNCTION my_fun // add.3 <- i.0 + 1 // i.0 <- add.3 // ret i.0 // ret 0 // VARIABLE i.0 = 0</code></pre> Onwards to codegen and exposing the actual bugs in previous stages. Assembly Generation</h2> Changes to the assembly syntax tree mirror those in the IR's. Also, a new Operand</code> type for rodata</code>. The Operand</code> change is tiny. pub const Operand = union(enum) { imm: u64, reg: Register, pseudo: Identifier, stack: Offset, data: Identifier, // <-- };</code></pre> The changes to Prgm</code> are similar enough to the changes in IR that repeating them here would be a chore. Changes to asm_gen.zig</code> are equally straightforward, for the most part. The new prgm_to_asm</code> almost contains all of it, with functions getting their global</code> in their own function. pub fn prgm_to_asm( alloc: std.mem.Allocator, prgm: ir.Prgm, ) !assembly.Prgm { var items: std.ArrayListUnmanaged(assembly.TopLevel) = try .initCapacity( alloc, prgm.items.items.len, ); for (prgm.items.items) |item| switch (item) { .F => |f| try items.append(alloc, .{ .F = try func_def_to_asm(alloc, f), }), .V => |v| try items.append(alloc, .{ .V = assembly.StaticVar{ .name = v.name, .init = v.init, .global = v.global, } }), }; return .{ .items = items }; }</code></pre>Replacing Pseudos</h3> Previously, pseudoregisters were replaced by stack</code> items. Now, the static ones, as indicated by the type_map</code> (I am never getting rid of this am I), are going to be replaced by data</code> operands. To facilitate this, I am just going to add a type_map</code> field to ir.Prgm</code> and assembly.Prgm</code>. The map passes through all the way to pseudo_to_stack</code> (which might warrant a change of name now), to end up with this. I also changed PseudoMap</code>'s value type to void</code> because I realized I am not using it.^{3</a>} fn pseudo_to_stack( op: assembly.Operand, type_map: *const sema.TypeMap, map: *PseudoMap, ) !assembly.Operand { switch (op) { .reg, .imm, .stack, .data => return op, .pseudo => |name| { const offset: assembly.Operand.Offset = if (map.getIndex(name)) |idx| // already seen @intCast(idx + 1) else if (cond: { const cond = type_map.get(name.real_idx); if (cond == null) break :cond false; break :cond cond.? == .static; }) return .{ .data = name } else ret: { try map.put(name, {}); break :ret @intCast(map.count()); // index of last item + 1 }; return .{ .stack = offset * -4 }; }, } }</code></pre>Fixing Up Instructions</h3> A lot of the old, already implemented rules for moving between different stack positions already apply here for moving between different memory positions, whether they are data</code> or stack</code>. Otherwise, it is the same annoying state machine. To help make things simpler, I created a helper function to tell me if an Operand</code> is a memory location or not. This is the function defined under Operand</code>. pub inline fn is_mem( self: @This(), ) bool { return switch (self) { .stack, .data => true, else => false, }; }</code></pre> And this is the new state machine in all its glory. the only changes is using .is_mem()</code> and changing the states' names from stack</code> to mem</code>. const State = enum { start, mov_mem_mem, cmp_mem_mem, cmp_to_imm, add_mem_mem, sub_mem_mem, mul_to_mem, idiv_const, legal, }; for (func_def.instrs.items) |instr| { state: switch (State.start) { .start => switch (instr) { .mov => |m| if (m.src.is_mem() and m.dst.is_mem()) continue :state .mov_mem_mem else continue :state .legal, .cmp => |m| if (m.src.is_mem() and m.dst.is_mem()) continue :state .cmp_mem_mem else if (m.dst == .imm) continue :state .cmp_to_imm else continue :state .legal, .add => |m| if (m.src.is_mem() and m.dst.is_mem()) continue :state .add_mem_mem else continue :state .legal, .sub => |m| if (m.src.is_mem() and m.dst.is_mem()) continue :state .sub_mem_mem else continue :state .legal, .mul => |m| if (m.dst.is_mem()) continue :state .mul_to_mem else continue :state .legal, .idiv => |o| if (o == .imm) continue :state .idiv_const else continue :state .legal, else => continue :state .legal, }, .mov_mem_mem => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mov.src, .{ .reg = .R10 }) }, .{ .mov = .init(.{ .reg = .R10 }, instr.mov.dst) }, }), .cmp_mem_mem => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.cmp.src, .{ .reg = .R10 }) }, .{ .cmp = .init(.{ .reg = .R10 }, instr.cmp.dst) }, }), .cmp_to_imm => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.cmp.dst, .{ .reg = .R11 }) }, .{ .cmp = .init(instr.cmp.src, .{ .reg = .R11 }) }, }), .add_mem_mem => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.add.src, .{ .reg = .R10 }) }, .{ .add = .init(.{ .reg = .R10 }, instr.add.dst) }, }), .sub_mem_mem => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.sub.src, .{ .reg = .R10 }) }, .{ .sub = .init(.{ .reg = .R10 }, instr.sub.dst) }, }), .mul_to_mem => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mul.dst, .{ .reg = .R11 }) }, .{ .mul = .init(instr.mul.src, .{ .reg = .R11 }) }, .{ .mov = .init(.{ .reg = .R11 }, instr.mul.dst) }, }), .idiv_const => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.idiv, .{ .reg = .R11 }) }, .{ .idiv = .{ .reg = .R11 } }, }), .legal => try out.append(alloc, instr), } }</code></pre> And here, I was done. The eye tests pass, with small hiccups regarding lifetimes and formatting. All is left now is the actual assembly generation. This is fairly mechanical, I think, despite hiding a nasty bug last chapter. Thankfully, all tests pass. I am done with Part 1. Next Thing to Do</h2> I am undecided yet. Either go for Part 3, which is about optimizations, or do Part 1 again in another language, like Swift or .. I dunno .. TypeScript? Maybe I will start working on my own language, or working on a compiler for Hare. Or maybe just go find a job. All in all, this was a fun process. I learned a lot about Zig and reinforced knowledge about writing a C compiler. I recommend buying the Book</a>. It is great. Until later. The Book's function is more complicated that is needed right now but is written this way to accomodate different types in Part 2. As I am not planning to implement Part 2 this run, there is need for the parse_type</code> half. ↩</a> </li> This is actually part of the design of the Book's compiler that gave me much grief in Part 2 during my Rust implementation. I do not quite exactly remember why, right now, but it felt very fiddly and prone to errors. ↩</a> </li> cond</code> is a block expression to make that compariosn. Because for a reason I do not understand type_map.get(name.real_idx) == .static</code> does not work. ↩</a> </li> </ol> </section> Chapter 9: Functions 2025-06-01T00:00:00+00:00 After eight</a> grueling (not really) chapters of Writing a C Compiler</a>, time to implement more assembly instructions. Functions! Linkage! Commas! Lexer, AST, and Parser</h2> Lexer just has a comma now. I thought about adding the comma operator but that didn't seem worth the trouble. The AST has two new additions. Function call expressions and function declarations (which are rebranded and improved function definitions)! Other changes include how the structures themselves are defined. A program is now a list of function declarations, instead of just one. How about that? These are the new AST nodes. I am not sure these compile or not yet, which I will find out when I am done with the parser. The use of SegmentedList</code> is discussed a couple of chapters ago as a more Arena-friendly collection type. pub const Prgm = struct { funcs: std.SegmentedList(FuncDecl, 0), }; pub const Block = struct { body: std.SegmentedList(BlockItem, 0), }; pub const BlockItem = union(enum) { D: Decl, S: Stmt, }; pub const Decl = union(enum) { F: FuncDecl, V: VarDecl, }; pub const FuncDecl = struct { name: []const u8, params: std.SegmentedList(Identifier, 0), block: ?Block, }; pub const VarDecl = struct { name: Identifier, init: ?*Expr, }; pub const Expr = union(enum) { // snip -- func_call: struct { Identifier, std.SegmentedList(Expr, 0) }, }; // This was implemented last chapter fixing the Segmentation Fault! pub const Identifier = union(enum) { name: []const u8, idx: utils.StringInterner.Idx, };</code></pre> It is going to be annoying fixing all the type errors throughout. Nonetheless, the parsing grammar for these new node types are going to change significantly. Here is. This is the new, tentative, parse_prgm</code>. I am not sure this is entirely correct yet. pub fn parse_prgm( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Prgm { var funcs: std.SegmentedList(ast.FuncDecl, 0) = .{}; while (tokens.next()) |next_token| { tokens.put_back(next_token); const func_decl = try parse_func_decl(arena, tokens); try funcs.append(arena, func_decl); } return .{ .funcs = funcs }; }</code></pre> parse_func_decl</code> is the same as the old parse_func_def</code>, but with optional parameters and an optional body. Ok maybe not the same, it is a behemoth. And all this is going to get significantly more complex when adding different types than int</code>. fn parse_func_decl( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.FuncDecl { // same old try expect(.type_int, tokens); const name = try expect(.identifier, tokens); // new stuff !! var params: std.SegmentedList(ast.Identifier, 0) = .{}; { // params try expect(.l_paren, tokens); const next_token = tokens.next() orelse return error.SyntaxError; // labelled switch to loop over mutliple parameters. params: switch (next_token.tag) { // old bahaviour is this: .keyword_void => try expect(.r_paren, tokens), // new optional parameters .type_int => { const ident = try expect(.identifier, tokens); try params.append(arena, .{ .name = ident }); const next_next = tokens.next() orelse return error.SyntaxError; // loops back here. this would be sginifcantly more annoying // to write without labeled switch switch (next_next.tag) { .comma => { try expect(.type_int, tokens); continue :params .type_int; }, .r_paren => {}, // could just retunr here but where is the fun then? else => continue :params .invalid, } }, else => return error.SyntaxError, } } const block = block: { const peeked = tokens.next() orelse return error.SyntaxError; switch (peeked.tag) { .semicolon => break :block null, .l_brace => { tokens.put_back(peeked); break :block try parse_block(arena, tokens); }, else => return error.SyntaxError, } }; return .{ .name = name, .params = params, .block = block }; }</code></pre> There is one big item left, which is BlockItem</code>, which can be a function declaration as well as a variable declaration, and it is not possible to know which is which from the token type_int</code> as done before. The most straightforward way to implement it in the current code base is as follows: look for an type_int</code> token, verify there is an identifier token afterwards, without recording it, then see what the third token is. If it is a semicolon</code> or an equals</code>, it is a variable; if it is a parenthesis, it is a function; otherwise it is illegal. Then the parser is rewinded to the int</code> and the correct function is called. In later chapters, with global variables, it will be apparent that the only place which only accepts one type of declarations is the for</code> loop statement, but I will cross that bridge when I get to it. fn parse_decl( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Decl { const int_token = tokens.next() orelse return error.NotEnoughJunk; if (int_token.tag != .type_int) return error.SyntaxError; _ = try expect(.identifier, tokens); const new_token = tokens.next() orelse return error.NotEnoughJunk; tokens.put_back(int_token); switch (new_token.tag) { .semicolon, .equals => return .{ .V = try parse_var_decl(arena, tokens) }, .l_paren => return .{ .F = try parse_func_decl(arena, tokens) }, else => return error.SyntaxError, } }</code></pre> Parsing function calls is a new challenge. Previously, any identifier is immediately assumed to be a variable. But now, if the identifier is followed by parenthesis, it could be a function call with an arbitrary number of parameters. The new identifier</code> case exhibits a serious case of rightward drift. .identifier => { const next_token = tokens.next() orelse return error.NotEnoughJunk; const name = tokens.buffer[current.loc.start..current.loc.end]; switch (next_token.tag) { .l_paren => return .{ .func_call = .{ .{ .name = name }, try parse_args(arena, tokens), } }, else => { tokens.put_back(next_token); return .{ .@"var" = .{ .name = name } }; }, } }, // elsewhere: fn parse_args( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!std.SegmentedList(ast.Expr, 0) { // assumes l_paren already consumed var ret: std.SegmentedList(ast.Expr, 0) = .{}; const current = tokens.next() orelse return error.NotEnoughJunk; args: switch (current.tag) { .r_paren => return ret, .comma => { const expr = try parse_expr(arena, tokens, 0); try ret.append(arena, expr); const n_token = tokens.next() orelse return error.NotEnoughJunk; continue :args n_token.tag; }, else => { // only actually relevant for the first argument. tokens.put_back(current); continue :args .comma; }, } }</code></pre> Semantic Analysis</h2> This is a bit more involved than usual this chapter. In addition to variable resolution, the compiler needs to do type checking! Do all declarations of functions (which can repeat), have the same number of parameters? Starting with the easier stuff, the identifier resolution pass should handle the new function syntax. Start with function calls. Here is the added part in resolve_expr</code>. .func_call => |*f| { if (bp.variable_map.get(f.@"0".name)) |entry| { f.@"0" = .{ .idx = entry.name }; var iter = f.@"1".iterator(0); while (iter.next()) |item| try resolve_expr(bp, item); } else return error.Undeclaredfunction; },</code></pre> Resolving function declarations had me move the creation of the main variable map back into resolve_prgm</code> instead, and create a new inner map for every function. Also, I need to update the Entry</code> type of the variable map. This is the new Entry</code>. const Entry = struct { name: utils.StringInterner.Idx, scope: enum { local, parent } = .local, linkage: enum { none, external } = .none, // <-- new };</code></pre> Then this is resolve_func_decl</code>, and the inner parameters resolution, which is really just a cheap copy of resolve_var_decl</code> itself rebranded resolve_decl</code>. fn resolve_func_decl( bp: Boilerplate, func_decl: *ast.FuncDecl, ) Error!void { if (bp.variable_map.get(func_decl.name)) |prev| if (prev.scope == .local and prev.linkage != .external) return error.DuplicateFunctionDecl; try bp.variable_map.put(bp.gpa, func_decl.name, .{ .name = try bp.strings.get_or_put(bp.gpa, func_decl.name), .scope = .local, .linkage = .external, }); var variable_map = try bp.variable_map.clone(bp.gpa); defer variable_map.deinit(bp.gpa); var iter = variable_map.valueIterator(); while (iter.next()) |value| value.* = .{ .name = value.name, .scope = .parent, .linkage = value.linkage, }; const inner_bp = bp.into_ineer(&variable_map); var iter = func_decl.params.iterator(0); while (iter.next()) |param| { // a cheap imitation of `resolve_var_decl` // should pribably be in its own function. if (inner_bp.variable_map.get(param.name)) |entry| if (entry.scope == .local) return error.DuplicateVariableDecl; const unique_name = try inner_bp.make_temporary(param.name); try inner_bp.variable_map.put(inner_bp.gpa, param.name, .{ .name = unique_name }); param.* = .{ .idx = unique_name }; } if (func_decl.block) |*block| try resolve_block(inner_bp, null, block); }</code></pre> One more thing, there is need to check that block level function declarations have no body. I am going to do that in resolve_block</code>. fn resolve_block( bp: Boilerplate, current_label: ?utils.StringInterner.Idx, block: *ast.Block, ) Error!void { var iter = block.body.iterator(0); while (iter.next()) |item| switch (item.*) { .S => |*s| try resolve_stmt(bp, current_label, s), .D => |*d| switch (d.*) { .F => |*f| if (f.block) |_| return error.IllegalFuncDefinition else try resolve_func_decl(bp, f), .V => |*v| try resolve_var_decl(bp, v), }, }; }</code></pre> Before delving into type checking, the book suggests to run the test suite, but expect a number of failures. Thankfully, I can check the individual folders separately using the eye test, without running the test suite in the official manner, I can tell that the compiler is passing all the right files it should at this stage, even the ones in invalid_types</code>. Type checking</h2> The Book has the type checking done in its own pass. At first, I tried just stuffing the type checking logic right into the same identifier resolution pass. But the actual problem was that I needed a separate data structure anyway, since function declarations have to match even when they are in different scopes. So I am still doing it in the same run, but with a separate data structure. The type checking consists mostly of the following: not using the same identifier for both int</code>s and functions, making sure functions always have the same number of parameters in all declarations, and make sure a function is not defined (with a body) twice. A new data structure would be needed to stuff this info, global throughout the whole file.^{1</a> A hashmap taking the unique identifiers as keys and their types as values. The type is either an integer or a function with defined arity (number of parameters). I also need to track whether a function has been defined or not.} const TypeMap = std.AutoHashMapUnmanaged( u32, Type, ); const Type = union(enum) { int, func: struct { arity: usize, defined: bool, }, };</code></pre> Then adding a pointer to it to Boilerplate</code>, and adjusting resolve_prgm</code> as follows. pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!void { var variable_map: VariableMap = .empty; defer variable_map.deinit(gpa); var type_map: TypeMap = .empty; defer type_map.deinit(gpa); const bp: Boilerplate = .{ .gpa = gpa, .strings = strings, .variable_map = &variable_map, .type_map = &type_map, }; var iter = prgm.funcs.iterator(0); while (iter.next()) |item| try resolve_func_decl(bp, item); }</code></pre> What follows next is lots of annoying boilerplate. I must make sure every time I add something to a variable_map</code>, I am adding its new name (if any) to type_map</code>. Hairier than usual logic, but should try to straighten it out before stuffing it in a Boilerplate</code> method. For example, in resolve_finc_decl</code>, adding a name to variable_map</code> and type_map</code> is done as follows: { const nname = try bp.strings.get_or_put(bp.gpa, func_decl.name); try bp.variable_map.put(bp.gpa, func_decl.name, .{ .name = nname, .scope = .local, .linkage = .external, }); const gop = try bp.type_map.getOrPut(bp.gpa, nname.real_idx); if (gop.found_existing) { if (gop.value_ptr.* != .func or gop.value_ptr.func.arity != func_decl.params.count()) { return error.TypeError; } else if (gop.value_ptr.func.defined and func_decl.block != null) { return error.DuplicateFunctionDef; } } else gop.value_ptr.* = .{ .func = .{ .arity = func_decl.params.count(), .defined = func_decl.block != null, } }; }</code></pre> A similar thing to do in resolve_var_decl</code>, and inside the small tidbit in resolve_func_decl</code> that reeolves parameters. This is how it looks like. { const gop = try bp.type_map.getOrPut(bp.gpa, unique_name.real_idx); if (gop.found_existing) { if (gop.value_ptr.* != .int) { return error.TypeError; } } else gop.value_ptr.* = .int; }</code></pre> There does not seem enough shared logic right now to try and DRY these. Maybe later. What is left is checking these things in function calls and variable declarations. They are both very funny looking. .@"var" => |name| { if (bp.variable_map.get(name.name)) |un| { if (bp.type_map.get(un.name.real_idx).? == .int) // unwrap optional expr.* = .{ .@"var" = .{ .idx = un.name } } else return error.TypeError; } else return error.UndeclaredVariable; }, .func_call => |*f| { if (bp.variable_map.get(f.@"0".name)) |entry| { const t = bp.type_map.get(entry.name.real_idx).?; if (t == .func and t.func.arity == f.@"1".count()) { f.@"0" = .{ .idx = entry.name }; var iter = f.@"1".iterator(0); while (iter.next()) |item| try resolve_expr(bp, item); } else return error.TypeError; } else return error.UndeclaredFunction; },</code></pre> I think this should be it. I checked for functions being defined twice/ I checked that functions should be functions and types should be types. I checked all functions with the same name should have the same arity. zig build</code> returns no errors. What is left? The proof of the pudding is in the test suite. Time to run the test suite. (which now applies --latest-only</code> by default as not to take too long.) % paella ❱ zig build submit -- --chapter 9 --stage validate ---------------------------------------------------------------------- Ran 61 tests in 157.089s OK</code></pre> Phew. Mind you this does not mean the logic is correct. It just means it is failing the ones it should fail and passing the ones it should pass. The full test suite passes as well, which is cool. I took the time to refactor out the common logic between function parameters and variable declarations. Zig's comptime</code> allows some tricks that Rust would torture me for,2</a> forcing me to use a trait or some weird amalgamation of const generics. It turned out most of the logic is for identifiers, except for initializers in variable declarations. The new function follows. fn resolve_var_decl( comptime T: enum { param, @"var" }, // anoynmous types yay bp: Boilerplate, item: switch (T) { // can simply be `if` but this is more readable i think .@"var" => *ast.VarDecl, .param => *ast.Identifier, }, ) Error!void { const identifier = switch (T) { // pulling out the common logic .@"var" => &item.name, .param => item, }; if (bp.variable_map.get(identifier.name)) |entry| if (entry.scope == .local) return error.DuplicateDecl; const unique_name = try bp.make_temporary(identifier.name); try bp.variable_map.put(bp.gpa, identifier.name, .{ .name = unique_name }); { // TYPE CHECKING const gop = try bp.type_map.getOrPut(bp.gpa, unique_name.real_idx); if (gop.found_existing) { if (gop.value_ptr.* != .int) return error.TypeError; } else gop.value_ptr.* = .int; } identifier.* = .{ .idx = unique_name }; if (T == .@"var") // logic unique to declarations if (item.init) |expr| try resolve_expr(bp, expr); } // called like this, inside `resolve_func_decl` while (params.next()) |param| try resolve_var_decl(.param, inner_bp, param);</code></pre> I ran the test suite again (and the eye tests) after this and everything seems to work out. Internal Represntation</h2> After a long drought, finally time to update the IR syntax tree and generation. Which means new assembly stuff later. Couldn't the Book stick to adding more control flow constructs for ever? Maybe function calls can be implemented by just jumping around. Is that a thing? The syntax tree updatss mirror the AST's. const Identifier = utils.StringInterner.Idx; // type alias pub const Prgm = struct { funcs: std.ArrayListUnmanaged(FuncDef), // <-- }; pub const FuncDef = struct { name: Identifier, params: std.ArrayListUnmanaged(Identifier), // <-- instrs: std.ArrayListUnmanaged(Instr), }; pub const Instr = union(enum) { func_call: struct { name: Identifier, args: std.ArrayListUnmanaged(Value), dst: Value, }, // rest of the owl };</code></pre> And the rest is pretty much the same, sans a few updates to the pretty printers and deinitializers. Note that there are no function declarations, as they are discarded in this stage after being done with the type checking. prgm_emit_ir</code> is simply a small update from the one-function version to the many-functions version, skipping over declarations without bodies pub fn prgm_emit_ir( alloc: std.mem.Allocator, strings: *utils.StringInterner, prgm: *const ast.Prgm, ) Error!ir.Prgm { var funcs: std.ArrayListUnmanaged(ir.FuncDef) = try .initCapacity( alloc, prgm.funcs.len, ); var iter = prgm.funcs.constIterator(0); while (iter.next()) |f| if (f.block) |_| { // skip empty declarations const fir = try func_def_emit_ir(alloc, strings, f); try funcs.append(alloc, fir); }; return .{ .funcs = funcs }; }</code></pre> func_def_emit_ir</code> is also almost exactly the same, except that the identifiers are moved over. fn func_def_emit_ir( alloc: std.mem.Allocator, strings: *utils.StringInterner, func_def: *const ast.FuncDecl, ) Error!ir.FuncDef { const name = try strings.get_or_put(alloc, func_def.name); var params: std.ArrayListUnmanaged(utils.StringInterner.Idx) = try .initCapacity( alloc, func_def.params.count(), ); var iter = func_def.params.constIterator(0); while (iter.next()) |param| try params.append(alloc, param.idx); var instrs: std.ArrayListUnmanaged(ir.Instr) = .empty; const bp: Boilerplate = .{ .alloc = alloc, .strings = strings, .instrs = &instrs, }; try block_emit_ir(bp, &func_def.block.?); try instrs.append(alloc, .{ .ret = .{ .constant = 0 } }); return .{ .name = name, .params = params, .instrs = instrs }; }</code></pre> Function call expressions are new, even though they follow the same pattern as all other expressions. .func_call => |f| { var args: std.ArrayListUnmanaged(ir.Value) = try .initCapacity( bp.alloc, f.@"1".count(), ); const dst = .{ .variable = try bp.make_temporary("fn") }; var iter = f.@"1".constIterator(0); while (iter.next()) |e| { const v = try expr_emit_ir(bp, e); try args.append(bp.alloc, v); } try bp.append(.{ .func_call = .{ .name = f.@"0".idx, .args = args, .dst = dst, } }); return dst; },</code></pre> The last thing left to do here is to make sure that functon declaration as block items are skipped as well. fn block_emit_ir( bp: Boilerplate, block: *const ast.Block, ) Error!void { var iter = block.body.constIterator(0); while (iter.next()) |item| switch (item.*) { .S => |*s| try stmt_emit_ir(bp, s), .D => |d| if (d == .V) try var_decl_emit_ir(bp, &d.V), // <- }; }</code></pre>Small Detour Back to Type Checking</h3> I am honestly not quite sure about some decisions up to now. For example, if function declarations are discarded, why are their parameters given unique identities at all? Maybe I should go back and simply check the count (and ideally the types, but there are no types in here). This would change the end of resolve_func_decl</code> to the following, instead of the if statement only encasing the last line. Since the count is checked earlier if (func_decl.block) |*block| { var params = func_decl.params.iterator(0); while (params.next()) |param| try resolve_var_decl(.param, inner_bp, param); try resolve_block(inner_bp, null, block); }</code></pre> This passes all validations tests, except one. /* Duplicate parameter names are illegal in function declarations as well as definitions */ int foo(int a, int a); int main(void) { return foo(1, 2); } int foo(int a, int b) { return a + b; }</code></pre> Ah well. Amusingly, at least according to the LLM I asked, int foo(int, int)</code> is a legal declaration. Anyway, this was a failed detour. Assembly Generation</h2> This is the first serious brush in the Book with the System V ABI, the most common binary interface for Unixes. As there are currently no types more complex than int</code>, all the compiler needs to worry about is stuffing parameters in the right registers. (And for function bodies, retreiving them from the right registers). The IR has not concerned itself with this, relying on the surrounding passes to make sense out of it. In my Rust implementation, I used clever iterator combinators to write the logic for this section. I am kind of curious to see how it will turn out in Zig. The first step, as usual, is to update the assembly syntax tree. A program needs to be redefined as a list of functions, and there are new instructions and new registers. Some necessary refactorings should be timed now as well. This is assembly.Prgm</code>. pub const Prgm = struct { funcs: std.ArrayListUnmanaged(FuncDef), pub fn fixup( self: *@This(), alloc: std.mem.Allocator, ) !void { for (self.funcs.items) |*func| { // functions called here changed from taking *Prgm to *FuncDef // ideally this should live under FuncDef, but this is fine for now const depth = try pass_pseudo.replace_pseudos(alloc, func); try func.instrs.insert(alloc, 0, .{ .allocate_stack = @abs(depth), }); try pass_fixup.fixup_instrs(alloc, func); } }</code></pre> The new instructions are simple additions. However, I am using two type aliases here: Depth</code> for an unsigned version of stack depth, and Identifier</code> for utils.StringInterner.Idx</code>. allocate_stack: Depth, // old dealloc_stack: Depth, push: Operand, call: Identifier,</code></pre> And aside from the new Registers, which I will not bore you with, this is pretty much it.^{3</a> Implementing the codegen is the next step. parameters in function definitions need to be taken from their correct register and stack positions; and function calls need to stuff them there. prgm_to_asm</code>, the main entry point, is straightforward.} pub fn prgm_to_asm( alloc: std.mem.Allocator, prgm: ir.Prgm, ) !assembly.Prgm { var funcs: std.ArrayListUnmanaged(assembly.FuncDef) = try .initCapacity( alloc, prgm.funcs.items.len, ); for (prgm.funcs.items) |func| try funcs.append( alloc, try func_def_to_asm(alloc, func), ); return .{ .funcs = funcs }; }</code></pre> For functions, it is slightly different. The first six parameters should be passed in these registers in this specific order: DI</code>, SI</code>, DX</code>, CX</code>, R8</code>, and R9</code>. Remaining arguments are pushed into the stack in reverse order. So normally, I will just assign a constant for them registers. const REGISTERS: [6]assembly.Operand.Register = .{ .DI, .SI, .DX, .CX, .R8, .R9 };</code></pre> In func_def_to_asm</code> itself, a zipped for</code> loop is perhaps the most straightforward way of doing this. The first section is simple enough. And since the remaining paramters are pushed into the stack at function call in reverse order, they are in the correct order here. Starting from a 16 stack</code> offset, they are moved one by one. Luckily, this same loop can be used! for (func_def.params.items, 0..) |param, idx| if (idx < REGISTERS.len) try instrs.append(alloc, .{ .mov = .init( .{ .reg = REGISTERS[idx] }, .{ .pseudo = param }, ) }) else { const offset = (idx - REGISTERS.len + 2) * 8; // 16, 24, etc try instrs.append(alloc, .{ .mov = .init( .{ .stack = @intCast(offset) }, .{ .pseudo = param }, ) }); };</code></pre> Implementing function calls is a lot more involved. First, the stack needs to be aligned properly at 16, as per the System V ABI. Copying the passed in expressions to the input registers is easy, but reversing the rest of them requires some shenanigans. First is to have a stack allocated array with the maximum possible size because it is unknown at the outset what the size of the returned slice could be. // the return is of unknown size. // maximum possible size is parameter count * 2 + 4 var ret: std.ArrayListUnmanaged(assembly.Instr) = try .initCapacity(alloc, c.args.items.len * 2 + 4);</code></pre> Then calculate the amount of arguments in the stack and the needed padding to 16. Zig has a saturating subtraction operator -|</code> that is perfect for this. const depth = c.args.items.len -| REGISTERS.len; const padding: assembly.Instr.Depth = if (depth % 2 == 0) 8 else 0; if (padding > 0) try ret.append(alloc, .{ .allocate_stack = padding }); // 1</code></pre> The numbering in the comments is me keeping track of instructions that do not depend on parameters count. After that comes putting the values in their respective registers, and the stack.4</a> for (c.args.items, 0..) |arg, idx| { if (idx >= REGISTERS.len) break; try ret.append(alloc, .{ .mov = .init( .{ .reg = REGISTERS[idx] }, value_to_asm(arg), ) }); } for (0..depth) |idx| { const v_ir = c.args.items[c.args.items.len - 1 - idx]; const v_asm = value_to_asm(v_ir); switch (v_asm) { .imm, .reg => try ret.append(alloc, .{ .push = v_asm }), else => try ret.appendSlice(alloc, &.{ .{ .mov = .init(v_asm, .{ .reg = .AX }) }, .{ .push = .{ .reg = .AX } }, }), } }</code></pre> I think this should work. 0..depth</code> will be .. nothing, and the loop will not run. If it is larger, it starts from the last item (at index len - , 1</code>), then subtracts the current index of the loop. And finally, dealing with the return value, and returning the slice. // emit call instruction try ret.append(alloc, .{ .call = c.name }); // 2 const bytes_to_remove = 8 * depth + padding; if (bytes_to_remove != 0) try ret.append(alloc, .{ .dealloc_stack = bytes_to_remove }); // 3 try ret.append(alloc, .{ .mov = .init( .{ .reg = .AX }, value_to_asm(c.dst), ) }); // 4 return ret.items;</code></pre> Doing the eye test for codegen, I quickly noticed something stupid. I am moving the register's value to an immutable. Oops. int add(int x, int y); int main(void) { return add(1, 2); } // PROGRAM // FUNCTION main // allocate 4 // allocate 8 // mov DI -> imm 1 // mov SI -> imm 2 // call .Ladd // deallocate 8 // mov AX -> stack -4 // mov stack -4 -> AX // ret // mov imm 0 -> AX // ret</code></pre> Fixing that by switching two lines, the eye test passes fine. The test suite does not hit any errors (and, as a reminder, does not test beyond success a nd failure). After that comes the annoying fixup pass adjustments. The depth returned by replace_pseudos</code> is adjusted to save the depth of each function in a field in the function definition. And also, when allocating, rounding that up to the nearest multiple of 16. There is a neat trick to round up a number, n</code>, to the nearest multiple of 16. Basically (n + 15) & ~15</code>. Unfortunately, Zig makes this a pain to write because the integer type inference is not clever enough to understand ~15</code>. There in the standard library is a helpful function to do that math: std.mem.alignForward</code>. Kindly pointed to me by the Zig Discord. So I added this bit at the end of replace_pseudos</code>. func_def.depth = @intCast(pseudo_map.count() * 4); const aligned = std.mem.alignForward(assembly.Instr.Depth, func_def.depth, 16); try func_def.instrs.insert(alloc, 0, .{ .allocate_stack = aligned });</code></pre> So, time to actually emit the assembly. Code Emission</h2> The changes here are pretty small. The new instructions are one thing, but there are new registers and new register sizes for push</code>, which takes 8-width registers. The logic for printing registers got pretty large so I put it in its own function. fn emit_register( reg: Operand.Register, width: usize, writer: anytype, ) !void { p: switch (width) { 1 => switch (reg) { .AX => try writer.print("%al", .{}), .DX => try writer.print("%dl", .{}), .CX => try writer.print("%cl", .{}), .DI => try writer.print("%dil", .{}), .SI => try writer.print("%sil", .{}), .R8 => try writer.print("%r8b", .{}), .R9 => try writer.print("%r9b", .{}), .R10 => try writer.print("%r10b", .{}), .R11 => try writer.print("%r11b", .{}), }, 4 => switch (reg) { .AX => try writer.print("%eax", .{}), .DX => try writer.print("%edx", .{}), .CX => try writer.print("%ecx", .{}), .DI => try writer.print("%edi", .{}), .SI => try writer.print("%esi", .{}), .R8 => try writer.print("%r8d", .{}), .R9 => try writer.print("%r9d", .{}), .R10 => try writer.print("%r10d", .{}), .R11 => try writer.print("%r11d", .{}), }, 8 => switch (reg) { .AX => try writer.print("%rax", .{}), .DX => try writer.print("%rdx", .{}), .CX => try writer.print("%rcx", .{}), .DI => try writer.print("%rdi", .{}), .SI => try writer.print("%rsi", .{}), .R8 => try writer.print("%r8", .{}), .R9 => try writer.print("%r9", .{}), .R10 => try writer.print("%r10", .{}), .R11 => try writer.print("%r11", .{}), }, else => continue :p 4, // default case } }</code></pre> And then in the format function just do the following. There is a bit of redundancy regarding the 4 as a default case. You are not paranoid if they are out to get you! .reg => |r| try emit_register(r, options.width orelse 4, writer),</code></pre> Running the test suite, I get into 16 failures. Oh come on. Most of them seem to be UnrecognizedFlag</code>, which is the error I have in the compiler driver for flags that are unrecognized. That is what I get for not reading the text and jumping straight to the tables. There is a tidbit there about recognizing a -c</code> flag, which is the gcc</code> flag for creating object files instead of executables. This is a small change to the argument parser. That is the new one: pub fn parse_args() !Args { var args = std.process.args(); _ = args.skip(); var path: ?[:0]const u8 = null; var mode: Mode = .compile; var c_flag = false; // new while (args.next()) |arg| { if (arg[0] == '-') { if (arg[1] == 'c') // new c_flag = true else mode = std.meta.stringToEnum(Mode, arg[2..]) orelse return error.UnrecognizedFlag; } else if (path == null) path = arg else return error.PathDuplicated; } return .{ .path = path orelse return error.PathNotFound, .mode = mode, .c_flag = c_flag, // new }; }</code></pre> And mugh below that, when calling the assembler, the -c</code> flag is passed if c_flag</code> is true. And changing the extension to .o</code>. So it becomes the following (changing the extension itself is done elsewhere in a bit of rather annoying code). This is probably the nicest way of doing it. { // assembler var child = std.process.Child.init( if (args.c_flag) &.{ "gcc", "-c", asm_out, "-o", obj } else &.{ "gcc", asm_out, "-o", exe }, gpa, ); const term = try child.spawnAndWait(); if (!std.meta.eql(term, .{ .Exited = 0 })) return error.AssemblerFail; try std.fs.cwd().deleteFile(asm_out); // cleanup }</code></pre> Now running the test suite for real. Debugging</h2> On the plud side, all the programs ar ecompiling and running. On the negative side, I got a lot of bad return codes. There are logic mistakes. Also 3 errors relating to unicode and timeouts? I do not get it. The test suite's output is generally jumbled up and I find it hard to process. Having said that, the easiest way to attack the problem is to look at the individual files and compiles them myself. UnicodeDecodeError</h3> This is one of the weird errors, Nora Sandler's comments and all. The program should call the system's putchar</code>, which prints a character to standard out. #ifdef SUPPRESS_WARNINGS #pragma GCC diagnostic ignored "-Wunused-parameter" #endif int putchar(int c); /* Make sure we can correctly manage calling conventions from the callee side * (by accessing parameters, including parameters on the stack) and the caller side * (by calling a standard library function) in the same function */ int foo(int a, int b, int c, int d, int e, int f, int g, int h) { putchar(h); return a + g; } int main(void) { return foo(1, 2, 3, 4, 5, 6, 7, 65); }</code></pre> This is the output for each stage. PARSING FUNCTION putchar c FUNCTION foo a b c d e f g h (putchar h); RETURN (+ a g) FUNCTION main RETURN (foo 1 2 3 4 5 6 7 65) ================================ VALIDATION FUNCTION putchar c.0 FUNCTION foo a.1 b.2 c.3 d.4 e.5 f.6 g.7 h.8 (putchar h.8); RETURN (+ a.1 g.7) FUNCTION main RETURN (foo 1 2 3 4 5 6 7 65) ================================ IR FUNCTION foo fn.9 <- putchar(h.8) add.10 <- a.1 + g.7 ret add.10 ret 0 FUNCTION main fn.11 <- foo(1, 2, 3, 4, 5, 6, 7, 65) ret fn.11 ret 0 ================================ CODEGEN FUNCTION foo allocate 48 mov DI -> stack -4 mov SI -> stack -8 mov DX -> stack -12 mov CX -> stack -16 mov R8 -> stack -20 mov R9 -> stack -24 mov stack 16 -> R10 mov R10 -> stack -28 mov stack 24 -> R10 mov R10 -> stack -32 allocate 8 mov stack -32 -> DI call .Lputchar deallocate 8 mov AX -> stack -36 mov stack -4 -> R10 mov R10 -> stack -40 mov stack -28 -> R10 add R10 -> stack -40 mov stack -40 -> AX ret mov imm 0 -> AX ret FUNCTION main allocate 16 allocate 8 mov imm 1 -> DI mov imm 2 -> SI mov imm 3 -> DX mov imm 4 -> CX mov imm 5 -> R8 mov imm 6 -> R9 push imm 65 push imm 7 call .Lfoo deallocate 24 mov AX -> stack -4 mov stack -4 -> AX ret mov imm 0 -> AX ret</code></pre> Hold on, before going into assembly, I spot a weird thing here. Why is the call to foo</code> is preceded by .L</code>. This is probably a formatting error which does not matter. But more importantly, the numbers for deallocations seem weird. Why is it deallocating 24? Let's review the logic in instr_to_asm</code> in .func_call</code>. const bytes_to_remove = 8 * depth + padding; if (bytes_to_remove != 0) try ret.append(alloc, .{ .dealloc_stack = bytes_to_remove }); // 3</code></pre> Eh, this makes sense. Counting the arguments passed on the stack there, there are 2. So why is there padding? const padding: assembly.Instr.Depth = if (depth % 2 == 0) 8 else 0; // <--- here if (padding != 0) try ret.append(alloc, .{ .allocate_stack = padding }); // 1</code></pre> Oh I am assigning the padding in reverse. If the number of arguments is even, there should be no padding. Silly me. This does not actually deal with the error I got for this file, which is this: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte</code></pre> This might be the valued printed by putchar</code> failing to read. However, only way to know is to run the test suit again. % paella ❱ zig build submit --release=safe -- --chapter 9 # many moths later ---------------------------------------------------------------------- Ran 61 tests in 60.822s FAILED (failures=12)</code></pre> Hey at least no weird UTF-8 errors. That's 7 less failures than last time (which was 16 failures and 3 errors). AssertionError</h3> Note: Please do not rad this section. It is embarrassing. </blockquote> This is a more readable error than the last one. AssertionError: Incorrect behavior in chapter_9/valid/stack_arguments/test_for_memory_leaks * Bad return code: expected 1 and got 0</code></pre> I can work with this. This is the C file in question. These are a lot of arguments. /* Make sure stack arguments are deallocated correctly after returning from a function call; also test passing variables as stack arguments */ #ifdef SUPPRESS_WARNINGS #pragma GCC diagnostic ignored "-Wunused-parameter" #endif int lots_of_args(int a, int b, int c, int d, int e, int f, int g, int h, int i, int j, int k, int l, int m, int n, int o) { return l + o; } int main(void) { int ret = 0; for (int i = 0; i < 10000000; i = i + 1) { ret = lots_of_args(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, ret, 13, 14, 15); } return ret == 150000000; }</code></pre> I will skip the whole parsing to IR thing and simply give you here the last codegen, with added comments. It is huge. PROGRAM FUNCTION lots_of_args allocate 64 mov DI -> stack -4 ; a mov SI -> stack -8 ; b mov DX -> stack -12 ; c mov CX -> stack -16 ; d mov R8 -> stack -20 ; e mov R9 -> stack -24 ; f mov stack 16 -> R10 mov R10 -> stack -28 ; g mov stack 24 -> R10 mov R10 -> stack -32 ; h mov stack 32 -> R10 mov R10 -> stack -36 ; i mov stack 40 -> R10 mov R10 -> stack -40 ; j mov stack 48 -> R10 mov R10 -> stack -44 ; k mov stack 56 -> R10 mov R10 -> stack -48 ; l mov stack 64 -> R10 mov R10 -> stack -52 ; m mov stack 72 -> R10 mov R10 -> stack -56 ; n mov stack 80 -> R10 mov R10 -> stack -60 ; o mov stack -48 -> R10 mov R10 -> stack -64 ; l mov stack -60 -> R10 ; o add R10 -> stack -64 ; l + o mov stack -64 -> AX ret mov imm 0 -> AX ret FUNCTION main allocate 32 mov imm 0 -> stack -4 ; ret mov imm 0 -> stack -8 ; i => .Lst_for.16 cmp imm 10000000 -> stack -8 mov imm 0 -> stack -12 setl stack -12 cmp imm 0 -> stack -12 jmpe .Lbr_for.16 allocate 8 ; odd number of arguments ; register arguments mov imm 1 -> DI ; a mov imm 2 -> SI ; b mov imm 3 -> DX ; c mov imm 4 -> CX ; d mov imm 5 -> R8 ; e mov imm 6 -> R9 ; f ; stack arguments push imm 15 ; o push imm 14 ; n push imm 13 ; m mov stack -4 -> AX ; ret into l push AX ; l push imm 11 ; k push imm 10 ; j push imm 9 ; i push imm 8 ; h push imm 7 ; g call lots_of_args deallocate 80 ; why is this 80? 9 pushes + padding mov AX -> stack -16 ; l + o here mov stack -16 -> R10 mov R10 -> stack -4 ; assign to ret => .Lcn_for.16 mov stack -8 -> R10 mov R10 -> stack -20 add imm 1 -> stack -20 mov stack -20 -> R10 mov R10 -> stack -8 jmp .Lst_for.16 => .Lbr_for.16 cmp imm 150000000 -> stack -4 mov imm 0 -> stack -24 sete stack -24 mov stack -24 -> AX ret mov imm 0 -> AX ret</code></pre> I am actually at a loss. All this seems to make sense. The generated asembly is more of the same. Let us see another failure: AssertionError: Incorrect behavior in chapter_9/valid/arguments_in_registers/fibonacci * Bad return code: expected 8 and got -11</code></pre> How would you even get -11? int fib(int n) { if (n == 0 || n == 1) { return n; } else { return fib(n - 1) + fib(n - 2); } } int main(void) { int n = 6; return fib(n); }</code></pre> This is the assembly this time. .globl _fib _fib: pushq %rbp ;; function prologue movq %rsp, %rbp subq $48, %rsp ;; allocate 48 movl %edi, -4(%rsp) ;; n cmpl $0, -4(%rsp) ;; does n equal 0? movl $0, -8(%rsp) sete -8(%rsp) cmpl $0, -8(%rsp) ;; is n == 0 true ? jne .Ltrue_or.2 cmpl $1, -4(%rsp) ;; does n equal 1? movl $0, -12(%rsp) sete -12(%rsp) cmpl $0, -12(%rsp) ;; is n == 1 true? jne .Ltrue_or.2 movl $0, -16(%rsp) jmp .Lend_or.3 .Ltrue_or.2: movl $1, -16(%rsp) .Lend_or.3: cmpl $0, -16(%rsp) ;; is the previous expresison false? je .Lelse.7 movl -4(%rsp), %eax ;; if so just return n movq %rbp, %rsp popq %rbp ret jmp .Lend.8 .Lelse.7: movl -4(%rsp), %r10d ;; if not .. movl %r10d, -20(%rsp) ;; move n here subl $1, -20(%rsp) ;; n - 1 movl -20(%rsp), %edi ;; move n-1 to edi call _fib ;; call fib movl %eax, -24(%rsp) ;; stash result of fib(n-1) movl -4(%rsp), %r10d movl %r10d, -28(%rsp) ;; move n here subl $2, -28(%rsp) ;; n - 2 movl -28(%rsp), %edi ;; move n-1 to edi call _fib ;; call fib movl %eax, -32(%rsp) ;; stash result of fib(n-2) movl -24(%rsp), %r10d ;; stashed result of fib(n-1) movl %r10d, -36(%rsp) movl -32(%rsp), %r10d ;; stashed result of fib(n-2) addl %r10d, -36(%rsp) ;; add two results together movl -36(%rsp), %eax ;; move result to proper place movq %rbp, %rsp popq %rbp ret ;; the end .Lend.8: movl $0, %eax ;; this bit here is the useless return 0 movq %rbp, %rsp ;; added at the end of every function popq %rbp ret .globl _main _main: pushq %rbp movq %rsp, %rbp subq $16, %rsp movl $6, -4(%rsp) movl -4(%rsp), %edi call _fib movl %eax, -8(%rsp) movl -8(%rsp), %eax movq %rbp, %rsp popq %rbp ret movl $0, %eax movq %rbp, %rsp popq %rbp ret</code></pre> The logic seems fine. I am not debugging the logic for if</code>s and for</code>s as those test cases pass fine already. This sample here has no stack arguments yet it is still failing. This definitely means there is a memory corruption somewhere, but where? I am sure the smart ones of you spotted it. I am not that smart.^{5</a>} I figured maybe the mistake is from using the subshell. So I tried doing the test suite the manual way I did it previously. But nope, same result. The good news is that my subshell spell works perfectly. On a whim, which is most likely wrong, I thought of changing the sign of the stack offset for function bodies. (So they start from -16 downwards rather than 16 upwards). And that would not affect the failure in the fibonacci</code> example anyway. Surprisingly, that actually affected absolutely nothing: same number of failing cases. I am starting to think this is a parking_lot</code> bug. I moved to compare the codegen stage with my previous rust program, and it is identical. I moved on to comparing the assembly generated. Even using diff</code> with colors to compare the two assemblies. Then there I saw it. The logic is fine, but there is a typo: something I wrote many chapters ago. Behold: stack offsets: -8(%rsp)</code>. It should be from %rbp</code>. Now all the tests pass. Wonder how any of them did in the first place. Lessons Learned</h2> Spell checking is important.</li> diff --color</code> is a useful tool.</li> Unrelated to paella</code> itself, but I made some improvements to the build script, and learned more abot jj</code> and git</code>. I think I will add git</code> tags to commits that finish each chapter, which would make browsing them easier.</li> This week I instealled llm</code></a>, and had it set up to talk to the free version of Github Copilot. Really convenient and useful, and I used to spell check the articles. I will try some local models next but I do not expect they will do much with my Macbook Air M2.</li> Can I have typed integer literals or slightly greedier type inference in Zig please? It is absurd that ~15</code> just does not work.</li> </ol> One chapter left for Part 1. Unsure what to do next. Type errors between different files is very pointedly a not-my-problem. It is a linker error, while this is a compiler. ↩</a> </li> When will Rust have enums as proper const generics? Please, Rust. ↩</a> </li> To be honest, and for all chapters, I am skipping most of the reading and jumping through to the tables at the end of the section to implement the functions. I assume that is what the tables are for. ↩</a> </li> The Book's pseudocode gives different instructions to add if the IR value is converted to an immediate value, register, or a pseudo value. The weird part here is that, well until chapter 15 (where I stopped last time), the IR value is never converted to a register at this stage. This could be something in preparation for the optimizations in Part 3. ↩</a> </li> Gemini LLM told me this is fine. Though to be honest, Github Copilot pointed it out, but I missed it among the five useless things it also mentioned. ↩</a> </li> </ol> </section> Chapter 8: Loops 2025-05-27T00:00:00+00:00 Seven</a> done of Writing a C Compiler</a>, a few more to go. Now is the time for loops. Lexer, AST and Parser</h2> The lexer is just a few new keywords. Been there, done that. Changes to the AST are more interesting. No less than five new statements, one for every new keyword. Here is the new Stmt</code> without further ado: pub const Stmt = union(enum) { @"return": *Expr, expr: *Expr, @"if": struct { cond: *Expr, then: *Stmt, @"else": ?*Stmt }, compound: Block, @"break": ?[:0]const u8, @"continue": ?[:0]const u8, @"while": While, do_while: While, @"for": For, null: void, const While = struct { cond: *Expr, body: *Stmt, label: ?[:0]const u8, }; pub const For = struct { label: ?[:0]const u8, init: Init, cond: ?*Expr, post: ?*Expr, body: *Stmt, pub const Init = union(enum) { decl: *Decl, expr: *Expr, none }, }; }</code></pre> A few notes. Each of these statements have an optional label argument. This label is used later to associate break</code>s and continue</code>s with their associated loops during semantic analysis. You cannot actually assign labels to loops in C as far as I know. Also, usually for a one off subtype like For</code> here, I put it inline. But this one is big and hairy and has a subtype of its own that it warrants the special treatment. Init</code> itself is named because I am naming it later in the parser. To map the fields to the actually for</code> loop in code, it is basically this: for (init, cond, post) body</code></pre> All the three first fields are optional. The reason that none</code> is a state of init</code> instead of using an optional is to shave a few bytes from the size of init</code> (and therefore the whole For</code> type and then the whole Stmt</code> type). Parsing these is a bit more interesting. First, the straightforward break</code> and continue</code>. .keyword_break => { try expect(.semicolon, tokens); return .{ .@"break" = null }; }, .keyword_continue => { try expect(.semicolon, tokens); return .{ .@"continue" = null }; },</code></pre> while</code> and do while</code> are slightly more interesting. They're similar to if</code>. .keyword_while => { try expect(.l_paren, tokens); const cond = try parse_expr(arena, tokens, 0); const cond_ptr = try utils.create(arena, cond); try expect(.r_paren, tokens); const body = try parse_stmt(arena, tokens); const body_ptr = try utils.create(arena, body); return .{ .@"while" = .{ .cond = cond_ptr, .body = body_ptr, .label = null, } }; }, .keyword_do => { const body = try parse_stmt(arena, tokens); const body_ptr = try utils.create(arena, body); try expect(.keyword_while, tokens); try expect(.l_paren, tokens); const cond = try parse_expr(arena, tokens, 0); const cond_ptr = try utils.create(arena, cond); try expect(.r_paren, tokens); try expect(.semicolon, tokens); return .{ .do_while = .{ .cond = cond_ptr, .body = body_ptr, .label = null, } }; },</code></pre> Now time for the final boss. for</code>. There is a funny complication here that I hit on my previous Rust implementation. The parsing code for variable declarations already expects a semicolon, but the one for expressions does not. So if I determine the init</code> statement is a variable declaration, I need not check for a semicolon, whereas I do if it turns out to be empty or an expression. There is another problem that's perhaps unique to the current implementation, which is that, if you remember, when parsing statements divorced from parsing variable declarations, parse_stmt</code> moved into its own home, while parsing declarations still clung to the old home of parse_block_item</code>. Now that variable declarations have a new partner, they need to move to their own home. The divorce is finally complete. fn parse_block_item( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.BlockItem { const current = tokens.next() orelse return error.NotEnoughJunk; tokens.put_back(current); switch (current.tag) { .type_int => return .decl(try parse_var_decl(arena, tokens)), else => return .stmt(try parse_stmt(arena, tokens)), } } fn parse_var_decl( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Decl { try expect(.type_int, tokens); const name = try expect(.identifier, tokens); const new_token = tokens.next() orelse return error.NotEnoughJunk; const init: ?*ast.Expr = switch (new_token.tag) { .equals => ret: { const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.semicolon, tokens); break :ret expr_ptr; }, .semicolon => null, else => return error.SyntaxError, }; return .{ .name = name, .init = init }; }</code></pre> Now that I am properly equipped, time to take on for</code>. After the left parenthesis, take it one piece at a time. First is init</code>: // I needed the type here so I had to name it and make it pub in the AST const init: ast.Stmt.For.Init = init: { const new_token = tokens.next() orelse return error.NotEnoughJunk; switch (new_token.tag) { .semicolon => break :init .none, .type_int => { tokens.put_back(new_token); const decl = try parse_var_decl(arena, tokens); const decl_ptr = try utils.create(arena, decl); break :init .{ .decl = decl_ptr }; }, else => { tokens.put_back(new_token); const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.semicolon, tokens); break :init .{ .expr = expr_ptr }; }, } };</code></pre> The other two are tamer beasts. A small difference between cond</code> and post</code> is the post</code> does not require a semicolon to terminate it, so it is compared to the right parenthesis instead. const cond: ?*ast.Expr = cond: { const new_token = tokens.next() orelse return error.NotEnoughJunk; switch (new_token.tag) { .semicolon => break :cond null, else => { tokens.put_back(new_token); const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.semicolon, tokens); break :cond expr_ptr; }, } }; const post: ?*ast.Expr = post: { const new_token = tokens.next() orelse return error.NotEnoughJunk; switch (new_token.tag) { .r_paren => break :post null, else => { tokens.put_back(new_token); const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(arena, expr); try expect(.r_paren, tokens); break :post expr_ptr; }, } };</code></pre> Then finally, the coup de grace: const body = try parse_stmt(arena, tokens); const body_ptr = try utils.create(arena, body); return .{ .@"for" = .{ .init = init, .cond = cond, .post = post, .body = body_ptr, .label = null, } };</code></pre> This about covers it I think. One tiny complication remains: in implementing the pretty printer last chapter, I used the keyword DO</code> for blocks. So instead of repeating myself, I am going to use a different keyword for do while</code> loops: UNTIL</code>. These look fine. Time to move on. int main(void) { int sum = 0; for (int i = 0; i < 10;) { i = i + 1; if (i % 2) continue; sum = sum + i; } return sum; } // PROGRAM // FUNCTION main // int sum <- 0 // FOR int i <- 0; ( // (i <- (+ i 1)); // IF (% i 2) // CONTINUE // (sum <- (+ sum i)); // RETURN sum</code></pre>int main(void) { int a = 1; do { a = a * 2; } while(a < 11); return a; } // PROGRAM // FUNCTION main // int a <- 1 // UNTIL (< a 11) // (a <- (* a 2)); // RETURN a</code></pre>Semantic Analysis</h2> In addition to variable resolution, this chapter will also do loop labelling, attaching each break</code> and continue</code> to their parent loops. While C does not have labelled loops, goto</code> aside, this is still needed to generate the proper labels for IR and assembly jump</code> instructions. Resolving variables in while</code> and do while</code> is fairly simple, just like an if</code> statement. break</code> and continue</code> have nothing to resolve. for</code> loops, are, once again, more complicated. A new scope is introduced, similarly to compound statements, on top of the scope introduced by the inner compound statement. Instead of duplicating the boilerplate, I thought to combine for</code> statements and compound</code> statements into one branch as follows: .@"for", .compound => { var variable_map = try bp.variable_map.clone(bp.gpa); defer variable_map.deinit(bp.gpa); var iter = variable_map.valueIterator(); while (iter.next()) |value| value.* = .{ .name = value.name, .scope = .parent }; const inner_bp: Boilerplate = .{ .gpa = bp.gpa, .strings = bp.strings, .variable_map = &variable_map, }; switch (stmt.*) { .compound => |*b| try resolve_block(inner_bp, b), .@"for" => |f| { switch (f.init) { .decl => |d| try resolve_decl(inner_bp, d), .expr => |e| try resolve_expr(inner_bp, e), .none => {}, } if (f.cond) |c| try resolve_expr(inner_bp, c); if (f.post) |p| try resolve_expr(inner_bp, p); try resolve_stmt(inner_bp, f.body); }, else => unreachable, } },</code></pre> Regarding loop labelling, the Book does this in a separate pass from variable resolution. I do not particularly feel like creating yet another file (as I have each compiler pass in its own file), so I will try to do it within resolve_stmt</code>. After all, all the related structures are statements. Since it is being called recursively, an additional parameter is needed. I will pass it separately to avoid updating the Boilerplate</code> struct for every time resolve_stmt</code> is called, which would, ironically, increase the boilerplate. This is the new resolve_stmt</code> in all its glory. fn resolve_stmt( bp: Boilerplate, current_label: ?[:0]const u8, stmt: *ast.Stmt, ) Error!void { switch (stmt.*) { .null => {}, .@"break", .@"continue" => |*l| l.* = current_label orelse return error.BreakOutsideLoop, .@"return", .expr => |expr| try resolve_expr(bp, expr), .@"if" => |i| { try resolve_expr(bp, i.cond); try resolve_stmt(bp, current_label, i.then); if (i.@"else") |e| try resolve_stmt(bp, current_label, e); }, .@"while", .do_while => |*w| { const label = try bp.make_temporary("while"); w.label = label; // forgot this on the first pass! try resolve_expr(bp, w.cond); try resolve_stmt(bp, label, w.body); }, .@"for", .compound => { var variable_map = try bp.variable_map.clone(bp.gpa); defer variable_map.deinit(bp.gpa); var iter = variable_map.valueIterator(); while (iter.next()) |value| value.* = .{ .name = value.name, .scope = .parent }; const inner_bp: Boilerplate = .{ .gpa = bp.gpa, .strings = bp.strings, .variable_map = &variable_map, }; switch (stmt.*) { .compound => |*b| try resolve_block(inner_bp, current_label, b), .@"for" => |*f| { const label = try bp.make_temporary("for"); f.label = label; switch (f.init) { .decl => |d| try resolve_decl(inner_bp, d), .expr => |e| try resolve_expr(inner_bp, e), .none => {}, } if (f.cond) |c| try resolve_expr(inner_bp, c); if (f.post) |p| try resolve_expr(inner_bp, p); try resolve_stmt(inner_bp, label, f.body); }, else => unreachable, } }, } }</code></pre> With tiny related adjustments to resolve_block</code>'s body and signature and resolve_func_def</code>'s body, the code is good to go. The Eye Test I set up a couple of chapters earlier allowed me to catch that I was not actually assigning the labels to the blocks, a mistake I would not have caught otherwise until much later. IR Generation</h2> All the new structures in this chapter can be implemented by the same IR instructions done in previous ones. Jumps and Labels. Who needs structured programming anyway? Now, labels are needed for break</code> and continue</code> statements's translation to jumps. To make sure the labels are consistent, they should be generated out of the loop labels constructed last section. So first I am adding another helper function to my Boilerplate</code> struct, that creates new strings out of old ones. fn augment_label( self: @This(), comptime prefix: []const u8, label: [:0]const u8, ) Error![:0]const u8 { const cat = try std.fmt.allocPrint(self.alloc, prefix ++ "_{s}", .{label}); defer self.alloc.free(cat); const name = try self.strings.get_or_put(self.alloc, cat); return name.string; }</code></pre> do while</code> loops are perhaps the easiest. .do_while => |w| { const st = try bp.augment_label("st", w.label.?); const br = try bp.augment_label("br", w.label.?); const cn = try bp.augment_label("cn", w.label.?); try bp.append(.{ .label = st }); try stmt_emit_ir(bp, w.body); try bp.append(.{ .label = cn }); const v = try expr_emit_ir(bp, w.cond); bp.append(.{ .jump_nz = .init(v, st) }); try bp.append(.{ .label = br }); },</code></pre> Everything else is, to be honest, just copied off the Book. .@"while" => |w| { const br = try bp.augment_label("br", w.label.?); const cn = try bp.augment_label("cn", w.label.?); try bp.append(.{ .label = cn }); const v = try expr_emit_ir(bp, w.cond); try bp.append(.{ .jump_z = .init(v, br) }); try stmt_emit_ir(bp, w.body); try bp.append(.{ .jump = cn }); try bp.append(.{ .label = br }); }, .@"for" => |f| { const st = try bp.augment_label("st", f.label.?); const br = try bp.augment_label("br", f.label.?); const cn = try bp.augment_label("cn", f.label.?); switch (f.init) { .decl => |d| try decl_emit_ir(bp, d), .expr => |e| _ = try expr_emit_ir(bp, e), .none => {}, } try bp.append(.{ .label = st }); if (f.cond) |c| { const v = try expr_emit_ir(bp, c); try bp.append(.{ .jump_z = .init(v, br) }); } try stmt_emit_ir(bp, f.body); try bp.append(.{ .label = cn }); if (f.post) |p| _ = try expr_emit_ir(bp, p); try bp.append(.{ .jump = st }); try bp.append(.{ .label = br }); }, .@"break" => |l| try bp.append(.{ .jump = try bp.augment_label("br", l.?), }), .@"continue" => |l| try bp.append(.{ .jump = try bp.augment_label("cn", l.?), }),</code></pre> One fun detail here, the last two calls to augment_label</code> will not allocate the string again. The string interner should just use the same string allocated previously. I'd perhaps test that to make sure, but eh. Segmentation Faults and How to Fix Them</h2> Running the eye test on Chapter 8's valid code, I quickly hit a segmentation fault when printing the IR instructions for this code. int main(void) { int acc = 0; int x = 100; while (x) { int y = 10; x = x - y; while (y) { acc = acc + 1; y = y - 1; } } return acc == 100 && x == 0; }</code></pre> I am sure the smart ones of you figured out the problem back in chapter 3. When I figured out what caused it, I got amused it did not happen sooner. The problem, my dear reader, is that my beloved string interner exceeded its capacity and reallocated its inner ArrayList</code>, invalidating all the pointers I had around the codebase to its slices. Rust would've never! Changing this is really trivial. All I need to do really is replace almost every occurrence of [:0]const u8</code> in the code (which is usually the interned string), with StringInterner.ID</code>. The actual problem, which is why I chose to put the string slices around in the first place, is that I cannot access the String Interner from the pretty printers. Allow me to demonstrate. This is ir.FuncDef</code> in its entirety. pub const FuncDef = struct { name: [:0]const u8, instrs: std.ArrayListUnmanaged(Instr), fn deinit(self: *@This(), alloc: std.mem.Allocator) void { self.instrs.deinit(alloc); } pub fn format( self: @This(), comptime _: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { const w = options.width orelse 0; try writer.writeByteNTimes('\t', w); try writer.print("FUNCTION {s}\n", .{self.name}); // <- printing name here for (self.instrs.items) |instr| try writer.print("{:[1]}\n", .{ instr, w + 1, }); } };</code></pre> I could store the numerical ID that is assigned to each string. But where else am I even using the strings, then? I am also using the same format</code> mechanism to output the final assembly code. The most straightforward, and annoying, solution is to add a pointer to the string interner for almost every single type in my program. The other solution is to have a global StringInterner</code>, and I have avoided using globals in the program so far. A completely insane solution would be to store the pointer to the StringInterner</code> (not a pointer to its bytes</code> array,) in the Idx</code> object I'd use instead of [:0]const u8</code>. So first let's do the easy part. get_or_put</code> now returns a small struct with Idx</code> and a string slice, and I can make it return only the Idx</code>, and let the Zig compiler tell me where I need to fix the types. pub fn get_or_put( self: *StringInterner, gpa: std.mem.Allocator, string: []const u8, ) std.mem.Allocator.Error!Idx { // <-- change here try self.bytes.ensureUnusedCapacity(gpa, string.len + 1); try self.map.ensureUnusedCapacityContext(gpa, 1, .{ .bytes = &self.bytes }); const adapter: std.hash_map.StringIndexAdapter = .{ .bytes = &self.bytes }; const gop = self.map.getOrPutAssumeCapacityAdapted(string, adapter); if (gop.found_existing) return gop.key_ptr.*; // <-- and here const new_id: Idx = @intCast(self.bytes.items.len); self.bytes.appendSliceAssumeCapacity(string); self.bytes.appendAssumeCapacity(0); gop.key_ptr.* = new_id; return new_id; // <-- and here }</code></pre> Simple enough. zig build</code> tells me I am using this type in 4 places. All of them look like name.string</code>. So I will just make them return name</code> (which is really Idx</code>), and fix the return types of their surrounding functions. One example is the one mentioned earlier, augment_label</code>. fn augment_label( self: @This(), comptime prefix: []const u8, label: utils.StringInterner.Idx, // <-- change here ) Error!utils.StringInterner.Idx { // <-- and here const st_label = self.strings.get_string(label).?; // <-- addition const cat = try std.fmt.allocPrint(self.alloc, prefix ++ "_{s}", .{st_label}); defer self.alloc.free(cat); const name = try self.strings.get_or_put(self.alloc, cat); return name; // <-- and here. }</code></pre> So many errors!! Now we go over the types. A search for [:0]const u8</code> is going to return a shitton of results, but persevere I must. The previously mentioned ir.FuncDef</code> for example, becomes the following. Pretend the formatting problem is solved for now. pub const FuncDef = struct { name: utils.StringInterner.Idx, instrs: std.ArrayListUnmanaged(Instr), fn deinit(self: *@This(), alloc: std.mem.Allocator) void { self.instrs.deinit(alloc); } }</code></pre> Then the functions must ne changed. If I am not mistaken, all the functions dealing with the string interner actually live in Boilerplate</code> structs. So time to check those. Here is the new ir_gen.Boilerplate.make_temporary</code>. fn make_temporary( self: @This(), comptime prefix: []const u8, ) Error!utils.StringInterner.Idx { // <- only here return try self.strings.make_temporary(self.alloc, prefix); }</code></pre> There is a lot of leg work here that I will spare you the boring details of. A few details that came up showing how deep the assumption is within the codebase. One of them is the different StringHashMap</code>s around the code. They do not manage the strings, I do. And now they are all invalidated! What if I just solve this with a much bigger starting capacity for the string interner?[^branch] The other big one is that in semantic analysis, I replaced the pointers to the old names (which are allocated from the source) with pointers within the string interner. It means, if I were to keep to the same structure of files, I need to replace the name</code> fields in all AST types dealing with variables with an Identifier</code> type, defined thus. pub const Identifier = union(enum) { name: []const u8, idx: utils.StringInterner.Idx, };</code></pre> And now ast.Decl</code> is like this. pub const Decl = struct { name: Identifier, init: ?*Expr, };</code></pre> After changing all the types, and fixing all the compilation errors, I wanted to make sure this thing works before trying to solve the formatting problem. So I took the C file mentioned above and sent it through all the stages up to codegen. First goes parsing: PROGRAM FUNCTION { 109, 97, 105, 110 } int ast.Identifier{ .name = { 97, 99, 99 } } <- 0 int ast.Identifier{ .name = { 120 } } <- 100 WHILE ast.Identifier{ .name = { 120 } } int ast.Identifier{ .name = { 121 } } <- 10 (ast.Identifier{ .name = { 120 } } <- (- ast.Identifier{ .name = { 120 } } ast.Identifier{ .name = { 121 } })); WHILE ast.Identifier{ .name = { 121 } } (ast.Identifier{ .name = { 97, 99, 99 } } <- (+ ast.Identifier{ .name = { 97, 99, 99 } } 1)); (ast.Identifier{ .name = { 121 } } <- (- ast.Identifier{ .name = { 121 } } 1)); RETURN (&& (== ast.Identifier{ .name = { 97, 99, 99 } } 100) (== ast.Identifier{ .name = { 120 } } 0))</code></pre> Second is semantic analysis. Note how most of the numbers (outside of ast.identifier.etc</code>) are actually variable names, or rather indices to variable names. PROGRAM FUNCTION { 109, 97, 105, 110 } int ast.Identifier{ .idx = 0 } <- 0 int ast.Identifier{ .idx = 6 } <- 100 10: WHILE ast.Identifier{ .idx = 6 } int ast.Identifier{ .idx = 18 } <- 10 (ast.Identifier{ .idx = 6 } <- (- ast.Identifier{ .idx = 6 } ast.Identifier{ .idx = 18 })); 22: WHILE ast.Identifier{ .idx = 18 } (ast.Identifier{ .idx = 0 } <- (+ ast.Identifier{ .idx = 0 } 1)); (ast.Identifier{ .idx = 18 } <- (- ast.Identifier{ .idx = 18 } 1)); RETURN (&& (== ast.Identifier{ .idx = 0 } 100) (== ast.Identifier{ .idx = 6 } 0))</code></pre> Following it up is the IR. 0 <- 0</code> is a funny one. Assigning a constant 0</code> to a variable whose name is at index 0</code>. main</code> amusingly, is actually at index 30. PROGRAM FUNCTION 30 0 <- 0 6 <- 100 => 46 jz 6 => 35 18 <- 10 57 <- 6 - 18 6 <- 57 => 74 jz 18 => 63 85 <- 0 + 1 0 <- 85 91 <- 18 - 1 18 <- 91 jump => 74 => 63 jump => 46 => 35 130 <- 0 == 100 jz 130 => 97 137 <- 6 == 0 jz 137 => 97 119 <- 1 jump => 109 => 97 119 <- 0 => 109 ret 119 ret 0</code></pre> And finally, codegen. PROGRAM FUNCTION 30 allocate 36 mov imm 0 -> stack -4 mov imm 100 -> stack -8 => .L46 cmp imm 0 -> stack -8 jmpe .L35 mov imm 10 -> stack -12 mov stack -8 -> R10 mov R10 -> stack -16 mov stack -12 -> R10 sub R10 -> stack -16 mov stack -16 -> R10 mov R10 -> stack -8 => .L74 cmp imm 0 -> stack -12 jmpe .L63 mov stack -4 -> R10 mov R10 -> stack -20 add imm 1 -> stack -20 mov stack -20 -> R10 mov R10 -> stack -4 mov stack -12 -> R10 mov R10 -> stack -24 sub imm 1 -> stack -24 mov stack -24 -> R10 mov R10 -> stack -12 jmp .L74 => .L63 jmp .L46 => .L35 cmp imm 100 -> stack -4 mov imm 0 -> stack -28 sete stack -28 cmp imm 0 -> stack -28 jmpe .L97 cmp imm 0 -> stack -8 mov imm 0 -> stack -32 sete stack -32 cmp imm 0 -> stack -32 jmpe .L97 mov imm 1 -> stack -36 jmp .L109 => .L97 mov imm 0 -> stack -36 => .L109 mov stack -36 -> AX ret mov imm 0 -> AX ret</code></pre> It occurs to me looking at this last one is that only strings that survive to the final stage are labels and function names. (Only main</code> for now). Even the labels can actually just be a .L35</code>, or whatever number, which is a perfectly valid label identifier. So, now to get a working compiler all I need to actually fix in this stage is printing the function's name. To test that theory, I added a *utils.StringInterner</code> field to both assembly.Prgm</code> and assembly.FuncDef</code> to get the name of the function out. It is a fairly small change in both assembly.zig</code> and asm_gen.zig</code>. You get this final assembly. .globl _main _main: pushq %rbp movq %rsp, %rbp subq $36, %rsp movl $0, -4(%rsp) movl $100, -8(%rsp) .L46: cmpl $0, -8(%rsp) je .L35 movl $10, -12(%rsp) movl -8(%rsp), %r10d movl %r10d, -16(%rsp) movl -12(%rsp), %r10d subl %r10d, -16(%rsp) movl -16(%rsp), %r10d movl %r10d, -8(%rsp) .L74: cmpl $0, -12(%rsp) je .L63 movl -4(%rsp), %r10d movl %r10d, -20(%rsp) addl $1, -20(%rsp) movl -20(%rsp), %r10d movl %r10d, -4(%rsp) movl -12(%rsp), %r10d movl %r10d, -24(%rsp) subl $1, -24(%rsp) movl -24(%rsp), %r10d movl %r10d, -12(%rsp) jmp .L74 .L63: jmp .L46 .L35: cmpl $100, -4(%rsp) movl $0, -28(%rsp) sete -28(%rsp) cmpl $0, -28(%rsp) je .L97 cmpl $0, -8(%rsp) movl $0, -32(%rsp) sete -32(%rsp) cmpl $0, -32(%rsp) je .L97 movl $1, -36(%rsp) jmp .L109 .L97: movl $0, -36(%rsp) .L109: movl -36(%rsp), %eax movq %rbp, %rsp popq %rbp ret movl $0, %eax movq %rbp, %rsp popq %rbp ret</code></pre> I ran the program at its current stage through the test suite, and it passed all tests with flying colors. No segmentation faults! Before actually tackling the printing problem, it would perhaps be better to change the utils.StringInterner.Idx</code> type to be a wrapper around u32</code> rather than a straight up u32</code>. It makes types a little bit safer (no adding two indices together). The change in the declaration is as simple as the following.^{1</a>} pub const Idx = enum(u32) { _, pub fn format( self: @This(), comptime _: []const u8, _: std.fmt.FormatOptions, writer: anytype, ) !void { try writer.print("[{d}]", .{@intFromEnum(self)}); } };</code></pre> And inside StringInterner</code>'s functions, the two builtin @intFromEnum</code> and, you will never guess, @enumFromInt</code>. Putting these in a few strategic places hides the u32</code> implementation, (including the formatting as .Lutils.StringInterner.Idx(28)</code> is not actually a valid label identifier), and the tests keep on passing. Looking at jj diff --stat</code>, all the changes are summarized in the following. Not bad. c_files/nested_loop | 2 ++ src/asm_gen.zig | 5 ++++- src/asm_passes/pseudos.zig | 5 ++++- src/assembly.zig | 30 ++++++++++++++++-------------- src/ast.zig | 34 ++++++++++++++++++++-------------- src/ir.zig | 24 ++++++++++++------------ src/ir_gen.zig | 17 +++++++++-------- src/main.zig | 2 +- src/parser.zig | 8 ++++---- src/sema.zig | 28 ++++++++++++++++------------ src/utils.zig | 44 ++++++++++++++++++++++---------------------- 11 files changed, 110 insertions(+), 89 deletions(-)</code></pre>Fixing the Formatting Problem</h2> As mentioned before, there are four options, on different levels of unhingedness. Add a *utils.StringInterner</code> field to every type that may contain a string (as done already in this example for assembly.FuncDef</code>) and every function that generates them (a lot of them do already via the Boilerplate</code> structs). Then when formatting, one just refers to the inner field.</li> Add a global *utils.StringInterner</code>, and remove it from the Boilerplate</code> structs. I do not like globals too much, but this is the least disruptive to the code.</li> Change the Idx</code> type to hold its own pointer to the string interner. I have no idea how this would even begin to look like and it is probably insane.</del> It works perfectly and it is definitely insane.</li> Just .. increase the starting buffer for interner to a large enough size that it never needs to reallocate. This is a terrible idea for a real program but this is not actually a real program.</li> Forgo pretty printing the middle stages entirely. The current status runs and passes fine as is.</li> </ol> As insane as it sounds, I went with 3. The resulting type is no larger than a slice, which I already had. It keeps the relevant data where it is (in the Idx</code>). I still think it is a totally insane idea. But the tests pass, the printer prints, things work. And this wraps chapter 8. Lessons Learned</h2> Loops are interesting</li> Do not add a for</code> loop to my language.</li> To love Rust more. Rust would never have let me segfault!</li> Er .. refactoring in Zig, I guess? The last section as a whole is a Lessons Learned, to be honest.</li> </ol> Odin's distinct</code> types are so much better for this. ↩</a> </li> </ol> </section> Chapter 7: Blocks 2025-05-25T00:00:00+00:00 Chapter 6 of Writing a C Compiler</a> was a short one. So will Chapter 7. Do not even bother to buckle up. Syntax Tree and Parsing</h2> See? Not even new tokens for lexer. This chapter is all about implementing compound statements. To abstract the similarities between a function body and compound statements, a new Block</code> AST node shall be created. Along with a tiny change in FuncDef</code>, you get this beauty. pub const FuncDef = struct { name: []const u8, body: Block, }; pub const Block = struct { body: std.SegmentedList(BlockItem, 0), };</code></pre> And adding a new statement type in Stmt</code>. pub const Stmt = union(enum) { @"return": *Expr, expr: *Expr, @"if": struct { cond: *Expr, then: *Stmt, @"else": ?*Stmt }, compound: Block, // <-- this one null: void, };</code></pre> I think it does not need a pointer. I will wait to see if the Zig compiler yells at me. Updating the parser requires a new function: parse_block</code>. It is simple enough and the implementation is copied over from parse_func_def</code>. fn parse_block( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Block { try expect(.l_brace, tokens); var body: std.SegmentedList(ast.BlockItem, 0) = .{}; while (tokens.next()) |next_token| { if (next_token.tag == .r_brace) break; tokens.put_back(next_token); const item = try parse_block_item(arena, tokens); try body.append(arena, item); } else return error.NotEnoughJunk; return .{ .body = body }; }</code></pre> And parse_func_def</code> is adjusted accordingly: fn parse_func_def( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.FuncDef { try expect(.type_int, tokens); const name = try expect(.identifier, tokens); try expect(.l_paren, tokens); try expect(.keyword_void, tokens); try expect(.r_paren, tokens); const block = try parse_block(arena, tokens); return .{ .name = name, .block = block }; }</code></pre> And that is it. Pretty Printing Tricks</h2> I encountered an interesting problem in the AST pretty printer. Because I am using an indentation based scheme, rather than braces, it makes it slightly difficult to know where blocks begin and end if I just, increase the indentation. So, for a regular block, I add the keyword DO</code>. int main(void) { int a = 2; int b; { a = -4; int a = 7; b = a + 1; } return b == 8 && a == -4; }</code></pre>PROGRAM FUNCTION main int a <- 2; int b; DO a <- (- 4); int a <- 7; b <- (+ a 1); RETURN (&& (== b 8) (== a (- 4)))</code></pre> But when the block is in the if</code> statement, this would look something like this, which is .. eh, unseemly. IF a DO int b <- 2; RETURN b</code></pre> So instead I get rid of DO</code> and just have indent immediately after IF</code>. The thing is, the logic for compound statement printing does not know, normally, whether it is preceded by IF</code> (and later FOR</code> and WHILE</code>), or if it is standing by its own. Doing the custom formatting when printing IF</code> would lead to a lot of duplicated code for every new control flow statement. So time to abuse another feature of the formatter! One of the options in srd.fmt.FormatOptions</code> is alignment</code>, which is normally used to align the printed items right, left, or center, within given padding. In the fmt</code> string, <</code> indicates left, ^</code> indicates center, and ></code> indicates right, which is the default value. In other words, when printing the statement following IF</code>, all I need to do there is this, which sets the alignment</code> option in the called function to .center</code>. try writer.print("{:^[1]}", .{ cs.then, w + 1 }); // ^ right here</code></pre> And now I can do the check right within .compound</code>: .compound => |b| { var iter = b.body.constIterator(0); var cw: usize = undefined; if (options.alignment != .center) { // <-- here try writer.writeAll("DO"); cw = w + 1; } else { if (iter.next()) |item| try writer.print("{}", .{item}); cw = w; } while (iter.next()) |item| { try writer.print("\n{:[1]}", .{ item, cw, }); } },</code></pre> The other issue, which you might have noticed, is that the new line characters which separate the internal block items are printed before each statement. The reason for that is for the block not to end in a newline character, as the new line is usually taken care of by the parent block. This simply avoids an additional blank line after every block. This prints the AST nicely, even for ridiculous C code. int main(void) { int ten = 10; {} int twenty = 10 * 2; {{}} return ten + twenty; }</code></pre>PROGRAM FUNCTION main int ten <- 10; DO int twenty <- (* 10 2); DO DO RETURN (+ ten twenty)</code></pre> I could just as well omit the empty blocks right here and then.^{1</a> Maybe I will do that if I ever write a real compiler.} Semantic Analysis</h2> I wish this were as easy as filling up the blanks, but that is actually where the meat of this chapter is. Because C allows shadowing in inner scopes. The following is perfectly valid C code. int main(void) { int x = 1; // <-- x0 { int x = 2; // <-- x1 if (x > 5) { // <-- x1 x = 3; // <-- x1 int x = 4; // <-- x2 } return x ; // <-- x1 } return x; // <-- x0 }</code></pre> What this means in practice is that each new scope (read: new block), requires its own variable_map</code>. Previously, the variable_map</code> was defined at program scope in resolve_prgm</code>: pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!void { var variable_map: std.StringHashMapUnmanaged([:0]const u8) = .empty; defer variable_map.deinit(gpa); const bp: Boilerplate = .{ .gpa = gpa, .strings = strings, .variable_map = &variable_map, }; try resolve_func_def(bp, prgm.func_def); }</code></pre> The variable_map</code> is discarded by the end of this function because, by the time the function is done, all the user-supplied variable names have been given unique names. The scopes are identified and done at this point. It does not serve to create a new empty variable_map</code> for each new block, because blocks can use variables from parent blocks.[^global] So for every new block, or compound statement, a copy of the variable map is passed instead of the original, and existing keys are replaced by the bindings in the inner scope. However, to avoid double declarations, a marker is needed to note whether this variable belongs to the current scope (so it cannot be declared again), or a parent one, (so it can). This is the part where double declarations are checked: fn resolve_decl( bp: Boilerplate, decl: *ast.Decl, ) Error!void { if (bp.variable_map.contains(decl.name)) return error.DuplicateVariableDecl; // snip --</code></pre> The data for whether this is shadowing a parent variable (legal) or repeating a local declaration (illegal) is not tracked yet. The simplest way to use a Entry</code> type that has both the new name, and a little boolean toggle whether it is local. The data oriented way is perhaps to create a new map for each new scope, but this seems complicated when the scopes get too deep and you need to keep a stack of maps, etc. I will stick with a boolean, or at least an enum. const Entry = struct { new_name: [:0]const u8, scope: enum { local, parent } = .local, };</code></pre> I really love anonymous types. Then change the variable map to a std.StringHashMapUnmanaged(Entry)</code>. This currently needs a couple of call sites to update, but all very straightforward. The default value of .local</code> makes initialization simpler, just a .{ .name = unique_name }</code> when inserting into the map. Now a small change to the duplicate declaration checking can be done. I need to check if it exists and if it is a .local</code> scope. A cute zig syntactic trick makes this nicely terse. Who needs if let</code> chains. if (bp.variable_map.get(decl.name)) |entry| if (entry.scope == .local) return error.DuplicateVariableDecl;</code></pre> All that is left is creating a new, duplicate, variable_map</code> when resolving a compound statement. Before I do that, I will do some refactorings moving the old variable_map</code> into FuncDef</code> scope instead, and moving the old resolve_func_def</code> logic into a new resolve_block</code> that can be reused. This also simplifies later chapters when each function has its own scope as well. pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!void { try resolve_func_def(gpa, strings, prgm.func_def); } fn resolve_func_def( gpa: std.mem.Allocator, strings: *utils.StringInterner, func_def: *ast.FuncDef, ) Error!void { var variable_map: std.StringHashMapUnmanaged(Entry) = .empty; defer variable_map.deinit(gpa); const bp: Boilerplate = .{ .gpa = gpa, .strings = strings, .variable_map = &variable_map, }; try resolve_block(bp, &func_def.block); } fn resolve_block( bp: Boilerplate, block: *ast.Block, ) Error!void { var iter = block.body.iterator(0); while (iter.next()) |item| switch (item.*) { .S => |*s| try resolve_stmt(bp, s), .D => |*d| try resolve_decl(bp, d), }; }</code></pre> Now back to compound statements. First the variable_map</code> is copied into a new one, with all the .local</code>s changes into .parent</code>s. Zig helpfully has a clone</code> method. then iterate over all the values and update the scope</code> field. // inside resolve_stmt .compound => |*b| { var variable_map = try bp.variable_map.clone(bp.gpa); defer variable_map.deinit(bp.gpa); var iter = variable_map.valueIterator(); while (iter.next()) |value| value.* = .{ .name = value.name, .scope = .parent }; const inner_bp: Boilerplate = .{ .gpa = bp.gpa, .strings = bp.strings, .variable_map = &variable_map, }; try resolve_block(inner_bp, b); },</code></pre> This compiles fine. The eye test passes on all C files in chapter 7's tests in the test suite. Time to actually run the test suite, and thankfully all tests pass. Now what? Almost nothing. No changes to IR (except handling the new statement) or any following stage necessary. Running the full tests, up to compilation, works fine. Time to move on to chapter 8. Lessons Learned</h2> The most fun I had here was experimenting with the AST pretty printer. Cracking the nut of the duplicate new line problem. Or pull a Python and output ...</code> or pass</code>. ↩</a> </li> </ol> </section> Chapter 6: Conditions 2025-05-24T00:00:00+00:00 This is chapter 6 of implementing Writing a C Compiler</a> in Zig. So, without further ado, time to get on with if</code> statements and conditional expressions. Lexer</h2> New keywords! if</code> and else</code>, and two new tokens ?</code> and :</code>. Apparently the word query</code> is an acceptable single word name for a question mark so I am going with that in the lexer. The new keywords are simple, enough, but it requires updating the keyword map. pub const keywords = std.StaticStringMap(Tag).initComptime(.{ .{ "int", .type_int }, .{ "return", .keyword_return }, .{ "void", .keyword_void }, // new lines : .{ "if", .keyword_if }, .{ "else", .keyword_else }, });</code></pre> The two new tokens are trivial to add and require no new states in the lexer's state machine. So onto AST and parsing. AST</h2> One new statement and one new expression today. Compound statements survive another day without being implemented. pub const Stmt = union(enum) { // snip -- @"if": struct { cond: *Expr, then: *Stmt, @"else": ?*Stmt }, };</code></pre>pub const Expr = union(enum) { // snip -- ternary: struct { *Expr, *Expr, *Expr }, };</code></pre> And that is pretty much it for the AST. No revision of previous major design decisions. Parser</h2> So how do you parse an expression like c ? a : b</code>?. The rule for it is that it is right associative (not unlike the assignment operator =</code>), but the expression in the middle is treated as it is part of the expression itself or between two parenthesis. In other words, think of ? a :</code> as an binary operator between c</code> and b</code> all by itself, only for parsing purposes. So first update the binop_precedence</code> function pub fn binop_precedence(self: @This()) ?struct { u8, u8 } { return switch (self) { .query => .{ 3, 0 }, // the rest }; }</code></pre> Then update parse_expr</code> to deal with it. This is slightly trickier than usual. The middle expression should be parsed before the right hand side is. Therefore this silly dance, before parsing rhs</code>. const then_ptr: ?*ast.Expr = if (current.tag == .query) t: { const then = try parse_expr(arena, tokens, 0); try expect(.colon, tokens); break :t try utils.create(ast.Expr, arena, then); } else null;</code></pre> Then, in the big switch board, this tiny line is added. .greater_equals => .{ .binop_ge = bin_op }, .query => .{ .ternary = .{ lhs_ptr, then_ptr.?, rhs_ptr } }, // <-</code></pre> And voila. Ternary expressions are parsed now. This creates a very tiny bit of waste creating the boiler-plate reducing bin_op</code> (which is shared between all the other expressions). But all is fair in the name of easier to write code. For parsing the if</code> statement, I hit upon a previous design decision. In adding declarations last chapter, I decided to forgo a separate parse_stmt</code> in exchange for a simpler, at the time, single parse_block_item</code>, and discriminating then between declarations and statements. Now, since the then</code> argument of the if</code> statement can only be a statement and not a declaration, this requires a divorce, and a separate parse_stmt</code>. Declarations are not ready to leave the house yet and therefore will remain in parse_block_item</code>. This is the trimmed parse_block_item</code>, followed by parse_stmt</code> building its own house. fn parse_block_item( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.BlockItem { const current = tokens.next() orelse return error.NotEnoughJunk; switch (current.tag) { .type_int => { const name = try expect(.identifier, tokens); const new_token = tokens.next() orelse return error.NotEnoughJunk; const init: ?*ast.Expr = switch (new_token.tag) { .equals => ret: { const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, expr); try expect(.semicolon, tokens); break :ret expr_ptr; }, .semicolon => null, else => return error.SyntaxError, }; return .decl(.{ .name = name, .init = init }); }, else => { tokens.put_back(current); const stmt = try parse_stmt(arena, tokens); return .stmt(stmt); }, } } fn parse_stmt( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Stmt { const current = tokens.next() orelse return error.NotEnoughJunk; switch (current.tag) { .semicolon => return .null, .keyword_return => { const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, expr); try expect(.semicolon, tokens); return .{ .@"return" = expr_ptr }; }, // if statement goes here else => { tokens.put_back(current); const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, expr); try expect(.semicolon, tokens); return .{ .expr = expr_ptr }; }, } }</code></pre> The if</code> statement parsing is slightly more involved than usual. The condition is always an expression that is parenthesized, and the then</code> statement always exists. The else</code> statement's existence depends on the existence of the keyword else</code>, so I have to peek. But because of the way I implemented peeking, by "putting back" the consumed token, it makes for some weird looking code, making more than usual use of block expressions. .keyword_if => { try expect(.l_paren, tokens); const cond = try parse_expr(arena, tokens, 0); const cond_ptr = try utils.create(ast.Expr, arena, cond); try expect(.r_paren, tokens); const then = try parse_stmt(arena, tokens); const then_ptr = try utils.create(ast.Stmt, arena, then); const peek = tokens.next() orelse return error.NotEnoughJunk; // weird looking code starts here. const else_ptr: ?*ast.Stmt = if (peek.tag == .keyword_else) s: { const e = try parse_stmt(arena, tokens); break :s try utils.create(ast.Stmt, arena, e); } else n: { tokens.put_back(peek); break :n null; }; return .{ .@"if" = .{ .cond = cond_ptr, .then = then_ptr, .@"else" = else_ptr, } }; },</code></pre> Now, before eating the pudding, the semantic analysis phase requires some tiny updates. There are no changes other than handling the new statements and expressions, and adding thoise is 11 lines total. // resolve_stmt .@"if" => |i| { try resolve_expr(bp, i.cond); try resolve_stmt(bp, i.then); if (i.@"else") |e| try resolve_stmt(bp, e); }, // resolve_expr .ternary => |t| { try resolve_expr(bp, t.@"0"); try resolve_expr(bp, t.@"1"); try resolve_expr(bp, t.@"2"); },</code></pre> And that is it, really. Back to proving the pudding. Testing it on this lovely and non-confusing C file: int main(void) { int a = 0; if (!a) if (3 / 4) a = 3; else a = 8 / 2; return a; }</code></pre> Gives me this result, which judging by the formatting is probably parsed correctly. PROGRAM FUNCTION main int a <- 0; IF (! a) IF (/ 3 4) a <- 3; ELSE a <- (/ 8 2); RETURN a</code></pre> All parsing and semantic analysis are succeeding and failing where they should. That's good news. The Eye Test</h2> Since I implemented all this nice pretty parsing, I would like to inspect on all the valid test files of the chapter. At least, visually confirm that there is nothing out of the ordinary. Same as I have been doing for C files in each chapter, but instead of copying a couple of interesting ones and running my app on them, I can do a batch comparison between each C file and the visual result of parsing it or validation or whatever. The first thought was to do it with a shell script. But shell is voodoo to me.^{1</a> build.zig</code> is right there, so maybe I can use it to that effect.} The new command would need two arguments: the stage, which is then passed directly to the executable, and the target folder. I cannot just pass the target chapter because not all chapters have the same structure. What followed was one of the most aggravating and annoying parts of this series. The first hurdle was navigating Zig's standard library's file system API. The docs are not very clear and the different APIs are organized in weird places, and how does any of that tie back in the build system? There is a walk</code> function.2</a> Excellent, but it needs a std.fs.Dir</code> object. All I have is a std.Build.LazyPath</code>. How do I make one into the other? Turns out LazyPath</code> has a getPath3</code> method (getPath</code> and getPath2</code> are deprecated) that gives you a std.Build.Cache.Path</code> object. And that in its turn has an openDir</code> method that gives you a std.fs.Dir</code> that then you can walk</code>. Try figuring that out only from the docs. The document traversal afterwards was simple enough. Basic Zig code. while (try walker.next()) |entry| if (entry.kind == .file and std.mem.endsWith(u8, entry.basename, ".c")) // not to do .s files { // insert entry handling here };</code></pre> The second hurdle, and if you actually know what I know now this will all seem very stupid, is a bit more involved. At first I passed the arguments as I did for the other previous commands. So like this zig build eye -- path/to/test/folder --parse</code></pre> And I desugared the arguments accordingly. const args: []const []const u8 = b.args orelse &.{ "./c_files/", "--lex" }; // least harmful default;</code></pre> So this way, if I don't pass anything, these would pass and the tree would be walked properly. Now zig build eye</code> worked fine. The problem is, if I run another command than eye</code>, (say .. run</code>), the directory walk would run anyway and then the build script would fail because the first argument is not a valid directory or whatever. I did know that, btw. Looking for help again someone suggested that the solution is to create a custom step, and even provided helpful code to do so. Here was a nice Gist I found on Google</a>. This seems perfect. So I do this: const std = @import("std"); pub fn build(b: *std.Build) !void { // stuff unchanged { // `zig build eye` command const eye_step = b.step("eye", "eye test all the files in a given directory"); var closure = b.allocator.create(Closure) catch unreachable; closure.* = .{ .exe = exe, .step = std.Build.Step.init(.{ .id = .custom, .name = "inner_eye", .makeFn = make_eye_step, .owner = b, }) }; eye_step.dependOn(&closure.step); } } const Closure = struct { exe: *std.Build.Step.Compile, step: std.Build.Step, }; fn make_eye_step(step: *std.Build.Step, _: std.Build.Step.MakeOptions) !void { const b = step.owner; const closure: *const Closure = @fieldParentPtr("step", step); const exe = closure.exe; const args: []const []const u8 = b.args orelse &.{ "./c_files/", "--lex" }; // least harmful default; // directory api boilerplate const lazy = b.path(args[0]); const path = lazy.getPath3(b, null); const dir = try path.openDir("", .{ .access_sub_paths = false }); var walker = try dir.walk(b.allocator); defer walker.deinit(); var prev_run_cmd: ?*std.Build.Step.Run = null; while (try walker.next()) |entry| { if (entry.kind == .file and std.mem.endsWith(u8, entry.basename, ".c")) { const file = entry.path; const bat = b.addSystemCommand(&.{ "bat", file }); bat.setCwd(lazy); // I am proud of myself for finding this bat.stdio = .inherit; // each invokation of `bat` depends on the previous run step so they're sequential if (prev_run_cmd) |c| bat.step.dependOn(&c.step); const run_cmd = b.addRunArtifact(exe); run_cmd.setCwd(lazy); run_cmd.addArg(file); run_cmd.addArgs(args[1..]); run_cmd.stdio = .inherit; // exactly what the default for `zig build run` does. run_cmd.step.dependOn(b.getInstallStep()); // keeping the squence run_cmd.step.dependOn(&bat.step); prev_run_cmd = run_cmd; } } // making sure the OG step depends on the tail of the chain. step.dependOn(&prev_run_cmd.?.step); }</code></pre> The way I see it, I am doing all the right things. All the steps depend on the proper steps, and best of all, the other zig build</code> commands like run</code> and test</code> and what have you, all work fine. Except this does not work. And I could not figure out. Running this would dutifully go through the loop properly, as I was able to verify by inserting a few prints</code>, but the commands do not run. And asking for help on the Discord only led me into StackOverflow-esque answers that I am not interested in repeating. Asking for help on Ziggit</a> actually gave me enough info to understand the real problem. This reply in particular</a> answered the dilemma for me. Do you know where the two phases of the build system are mentioned in the Official Build System Documentation and Intro</a>? Because I cannot find it. It is mentioned in passing in the documentation of some methods like getPath3</code> and run</code>, but with no proper explanation. Anyway, it clicked, and I was able to use the user provided options to insert the needed details. { // `zig build eye` command const eye_step = b.step("eye", "Eye test all the files in a given directory"); if (b.option(std.Build.LazyPath, "folder", "Path to eye")) |lazy| { // same logic as before, more or less try walk_tree(b, exe, eye_step, lazy); } else { // *this* step, and only this one would fail // if the `folder` option is not set, because it // depends on `fail` only if the path does not exist const fail = b.addFail("folder needed for eye"); eye_step.dependOn(&fail.step); } }</code></pre> b.option</code> creates a new argument for the user of the build system. This way, if the argument is present, the directory tree is walked and the proper dependencies are added to the eye</code> command. If it is not, then the eye</code> command will fail. Other commands are unchanged and unaffected. The only difference from my first envisioning it is the way it is called, which I am not happy with, but eh. Pick your battles. zig build eye -Dfolder="./c_files/" -- --parse</code></pre> And it works exactly like I wanted. It calls bat</code> on every file in a given directory followed by paella</code> at the given stage, allowing for a quick eye test between the two. I'd paste the whole output here but I will settle for one file. File: multiple_if.c 1 int main(void) { 2 int a = 0; 3 int b = 0; 4 5 if (a) 6 a = 2; 7 else 8 a = 3; 9 10 if (b) 11 b = 4; 12 else 13 b = 5; 14 15 return a + b; 16 } PROGRAM FUNCTION main int a <- 0; int b <- 0; IF a a <- 2; ELSE a <- 3; IF b b <- 4; ELSE b <- 5; RETURN (+ a b) ================================</code></pre>Automatic Formatting</h2> While mucking around in the documentation I discovered a nice addFmt</code> method that is there to check for proper formatting in CI, but can just format the code for you. Adding these two lines pretty much anywhere in the file formatted everything every time I zig build</code>. const fmt_step = b.addFmt(.{ .paths = &.{"./"} }); exe.step.dependOn(&fmt_step.step);</code></pre> Internal Representation</h2> Now is the time to get on with new IR generation. Similarly to lat chapter, no new instructions need be appended. So after the IR generation stage, it is done. Short chapter all around. Generating the IR for if</code> statements is straightforward, with a slight complication around the optional else</code> statement. .@"if" => |c| { const cond = try expr_emit_ir(bp, c.cond); const else_label = try bp.make_temporary("else"); try bp.append(.{ .jump_z = .{ .cond = cond, .target = else_label } }); try stmt_emit_ir(bp, c.then); if (c.@"else") |@"else"| { const end_label = try bp.make_temporary("end"); try bp.append(.{ .jump = end_label }); try bp.append(.{ .label = else_label }); try stmt_emit_ir(bp, @"else"); try bp.append(.{ .label = end_label }); } else try bp.append(.{ .label = else_label }); },</code></pre> For conditional expressions, it is much of the same, except expr_emit_ir</code> is called instead and its value is returned, and there is no optional else</code> clause. .ternary => |t| { const else_label = try bp.make_temporary("else"); const end_label = try bp.make_temporary("end"); const dst_name = try bp.make_temporary("ter"); const dst: ir.Value = .{ .variable = dst_name }; const cond = try expr_emit_ir(bp, t.@"0"); try bp.append(.{ .jump_z = .{ .cond = cond, .target = else_label } }); const then = try expr_emit_ir(bp, t.@"1"); try bp.append(.{ .copy = .init(then, dst) }); try bp.append(.{ .jump = end_label }); try bp.append(.{ .label = else_label }); const else_ = try expr_emit_ir(bp, t.@"2"); try bp.append(.{ .copy = .init(else_, dst) }); try bp.append(.{ .label = end_label }); return dst; },</code></pre> And that's it. I do not even have to run the intermediate codepaths as no changes to instructions or assembly generation this chapter. There is a small extra credit of implementing goto</code>, but I am skipping extra credit this time. Lessons Learned</h2> Not to repeat ranting at Zig's documentation, but I definitely now understand Zig's build system at a deeper level than before. If all else fails I can get some LLM to generate it for me. It would work fine. ↩</a> </li> Turns out it is recursive and I actually wanted iterate()</code>. But that is on me, it is clearly written. ↩</a> </li> </ol> </section> Chapter 5: Variables 2025-05-21T00:00:00+00:00 Reflecting on the past chapters so far</a>, I am starting to think that this project is perhaps the perfect vehicle to explore new languages. Right off the bat, one deals with file systems, cross compiling, data structures, memory management, pretty formatting, and what have you. But as the project goes on, especially if you are familiar with the problem (read: have done it before in a previous language), most of the tasks become purely mechanical. Refactoring, updating requirements, satisfying compiler errors. It shows you how the language handles churn, and whether it is pleasant or not. It also shows how helpful the community around the language when you inevitably have to debug your language knowledge and hit weird snags. So, yeah. If you want to learn a new programming language, get yourself Writing a C Compiler</a>, do it once (at least halfway) with your favorite language. Then you can do it with any language as a test run. Way less math than a ray tracer. Anyway, now is Chapter 5. Chapter 5 is about, let me look, local variables. Oh finally some new statements. Lexer and Parser</h2> One new token, =</code>. Adding this is trivial so I am sparing you the details. There is one thing to update that, while belongs to the parser, is placed in lexer.zig</code>. The precedence chart. Operator Associativity</h3> Previously, all operators were left associative. This one, however, is right associative. One could make a special case for it, but another upcoming operators1</a> are right associative as well. So care must be taken. The binop_precedence</code> function is implemented as a method in Token.Tag</code>, and this is how it looks after adding =</code>. It returns a tuple now instead of just a u8</code>. The reason left</code> is a u8</code> as well is because the two numbers are added later and changing from bool</code> to u8</code> in Zig is a torture. Even if (left) 1 else 0</code> does not work as it actually returns a comptime int</code> .... pub fn binop_precedence(self: @This()) ?struct { u8, u8 } { return switch (self) { .asterisk, // * .f_slash, // / .percent, // % => .{ 50, 1 }, .hyphen, // - .plus, // + => .{ 45, 1 }, .lesser_than, .lesser_equals, .greater_than, .greater_equals, => .{ 35, 1 }, .double_equals, .bang_equals, => .{ 30, 1 }, .double_ambersand => .{ 10, 1 }, .double_pipe => .{ 5, 1 }, .equals => .{ 1, 0 }, else => null, }; }</code></pre> It would be perhaps more idiomatic to a create a proper struct and give the fields proper name and have a proper enum for left vs right associativity but, eh. Too much boilerplate for a one use function. In parse_expr</code>, the rules are slightly amended. Instead of this while</code> loop: while (next_token.tag.binop_precedence()) |prec| { if (prec < min_prec) break; const rhs = try parse_expr(alloc, tokens, prec + 1); // etc</code></pre> It now becomes this: while (next_token.tag.binop_precedence()) |r| { const prec, const left = r; // destructuring yay if (prec < min_prec) break; const rhs = try parse_expr(alloc, tokens, prec + left); // etc</code></pre> And that .. is pretty much it. Just map the new operator to the new expression that I have not written yet. New Statements and Expressions!</h3> Two new expressions: pub const Expr = union(enum) { constant: u64, @"var": [:0]const u8, assignment: BinOp, // etc</code></pre> And two new statements, finally. This is the entire Stmt</code> type, with its formatter, too. pub const Stmt = union(enum) { @"return": *Expr, expr: *Expr, null: void, pub fn format( self: @This(), comptime _: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { const w = options.width orelse 0; try writer.writeByteNTimes('\t', w); switch (self) { .@"return" => |expr| try writer.print("RETURN {}", .{expr}), .expr => |expr| try writer.print("{};", .{expr}), .null => try writer.print(";", .{}), } } };</code></pre> There is a new AST node, too! Decl</code> is a variable declaration. (Maybe it should be VarDecl</code> considering what is to come?) It is a simple one. pub const Decl = struct { // no type because there is only one type so far: int name: []const u8, expr: ?*Expr, };</code></pre> Here is a fun note. In the AST proper, all the strings are backed by the source code. So they are not null-terminated ([]const u8</code>). However, in the following stages, the strings are backed by StringInterner</code>, so they are null-terminated ([:0]const u8</code>). I could use normal slices or null-terminated pointers ([*:0]const u8</code>) in the second case just as well, but extra safety does not hurt a toy compiler like this one. Also, since function bodies can include either declarations or statements, there is a need for a simple BlockItem</code> union. Here it is with its formatter that just delegates to the sub type. I will probably never have to touch this again. pub const BlockItem = union(enum) { D: Decl, S: Stmt, pub fn format( self: @This(), comptime fmt: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { switch (self) { inline else => |i| try i.format(fmt, options, writer), } } };</code></pre> And then update FuncDef</code> to have an collection of BlockItem</code>s, rather than a pointer to one Stmt</code>.Since these pointers are arena backed, it makes little sense for it to be an ArrayList</code>, because this collection reallocates a lot. A better idea is either a linked list, or a segmented list which is a halfway between the two. The main downside is that I cannot just take a slice from either of these two collections. I will cross that bridge when I come to it. The Zig's SinglyLinkedList</code> type has an api change on the master</a>. So to avoid potential future churn in a future where I upgrade Zig's version, I am going to go with SegmentedList</code>. Here I also revise an earlier design decision. Previously, every parsing structure returns a pointer to its result. But the new FuncDef</code> makes no use of pointers, as they are added to the SegmentedList</code>. So I rewrote all the functions to remove the allocation on the return value, and place instead on where it is actually needed at assignment. Plenty of refactoring. Because of extensive changes throughout almost every function in parser.zig</code>, here is the entire file in its current state. I have not tried compiling it yet, so beware of typos. const std = @import("std"); const ast = @import("ast.zig"); const lexer = @import("lexer.zig"); const utils = @import("utils.zig"); pub fn parse_prgm( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Prgm { const func_def = try parse_func_def(arena, tokens); const func_ptr = try utils.create(ast.FuncDef, arena, func_def); // now that we are done, check the tokenizer is empty. if (tokens.next()) |_| return error.ExtraJunk; return .{ .func_def = func_ptr }; } fn parse_func_def( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.FuncDef { try expect(.type_int, tokens); const name = try expect(.identifier, tokens); try expect(.l_paren, tokens); try expect(.keyword_void, tokens); try expect(.r_paren, tokens); try expect(.l_brace, tokens); var body: std.SegmentedList(ast.BlockItem, 0) = .{}; while (true) { const next_token = tokens.next() orelse return error.SyntaxError; if (next_token.tag == .r_brace) break; tokens.put_back(next_token); const item = try parse_block_item(arena, tokens); try body.append(arena, item); } return .{ .name = name, .body = body }; } fn parse_block_item( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.BlockItem { const next_token = tokens.next() orelse return error.SyntaxError; switch (next_token.tag) { .type_int => { const name = try expect(.identifier, tokens); const new_token = tokens.next() orelse return error.SyntaxError; const expr: ?*ast.Expr = switch (new_token.tag) { .equals => ret: { const res = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, res); try expect(.semicolon, tokens); break :ret expr_ptr; }, .semicolon => null, else => return error.SyntaxError, }; return .{ .D = .{ .name = name, .expr = expr } }; }, // statements .semicolon => return .{ .S = .null }, .keyword_return => { const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, expr); try expect(.semicolon, tokens); return .{ .S = .{ .@"return" = expr_ptr } }; }, else => { tokens.put_back(next_token); const expr = try parse_expr(arena, tokens, 0); const expr_ptr = try utils.create(ast.Expr, arena, expr); try expect(.semicolon, tokens); return .{ .S = .{ .expr = expr_ptr } }; }, } } fn parse_expr( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, min_prec: u8, ) Error!ast.Expr { var lhs = try parse_factor(arena, tokens); const lhs_ptr = try utils.create(ast.Expr, arena, lhs); var next_token = tokens.next() orelse return error.SyntaxError; while (next_token.tag.binop_precedence()) |r| { const prec, const left = r; if (prec < min_prec) break; const rhs = try parse_expr(arena, tokens, prec + left); const rhs_ptr = try utils.create(ast.Expr, arena, rhs); const bin_op: ast.Expr.BinOp = .{ lhs_ptr, rhs_ptr }; lhs = switch (next_token.tag) { .plus => .{ .binop_add = bin_op }, .hyphen => .{ .binop_sub = bin_op }, .asterisk => .{ .binop_mul = bin_op }, .f_slash => .{ .binop_div = bin_op }, .percent => .{ .binop_rem = bin_op }, .equals => .{ .assignment = bin_op }, .double_ambersand => .{ .binop_and = bin_op }, .double_pipe => .{ .binop_or = bin_op }, .double_equals => .{ .binop_eql = bin_op }, .bang_equals => .{ .binop_neq = bin_op }, .lesser_than => .{ .binop_lt = bin_op }, .greater_than => .{ .binop_gt = bin_op }, .lesser_equals => .{ .binop_le = bin_op }, .greater_equals => .{ .binop_ge = bin_op }, else => unreachable, }; next_token = tokens.next() orelse return error.SyntaxError; } tokens.put_back(next_token); return lhs; } fn parse_factor( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, ) Error!ast.Expr { const current = tokens.next() orelse return error.SyntaxError; switch (current.tag) { .identifier => return .{ .@"var" = tokens.buffer[current.loc.start..current.loc.end], }, .number_literal => { const lit = tokens.buffer[current.loc.start..current.loc.end]; const res = std.fmt.parseInt(u64, lit, 10) catch return error.InvalidInt; return .{ .constant = res }; }, .hyphen => { const inner_exp = try parse_factor(arena, tokens); return .{ .unop_neg = try utils.create(ast.Expr, arena, inner_exp) }; }, .tilde => { const inner_exp = try parse_factor(arena, tokens); return .{ .unop_not = try utils.create(ast.Expr, arena, inner_exp) }; }, .bang => { const inner_exp = try parse_factor(arena, tokens); return .{ .unop_lnot = try utils.create(ast.Expr, arena, inner_exp) }; }, .l_paren => { const inner_exp = try parse_expr(arena, tokens, 0); try expect(.r_paren, tokens); return inner_exp; }, else => return error.SyntaxError, } } inline fn expect( comptime expected: lexer.Token.Tag, tokens: *lexer.Tokenizer, ) Error!ExpectResult(expected) { if (tokens.next()) |actual| { if (actual.tag != expected) return error.SyntaxError; switch (expected) { .identifier => return tokens.buffer[actual.loc.start..actual.loc.end], else => {}, } } else return error.SyntaxError; } fn ExpectResult(comptime expected: lexer.Token.Tag) type { switch (expected) { .identifier => return []const u8, else => return void, } } const Error = std.mem.Allocator.Error || error{ SyntaxError, InvalidInt, ExtraJunk, }; test "precedence" { const t = std.testing; var a_a = std.heap.ArenaAllocator.init(t.allocator); defer a_a.deinit(); const a = a_a.allocator(); { const src = "3 * 4 + 5;"; var tokens = lexer.Tokenizer.init(src); const result = try parse_expr(a, &tokens, 0); try t.expect(result.* == .binop_add); try t.expectFmt("(+ (* 3 4) 5)", "{}", .{result}); } { const src = "4 + 3 && -17 * 4 < 5;"; var tokens = lexer.Tokenizer.init(src); const result = try parse_expr(a, &tokens, 0); try t.expect(result.* == .binop_and); try t.expectFmt("(&& (+ 4 3) (< (* (- 17) 4) 5))", "{}", .{result}); } }</code></pre> Well, that was a trip. The ir_gen</code> functions are broken too so they were replaced by a stub for now. Here is the C file I am using as my science mouse this chapter: int main(void) { int a = 2147483646; int b = 0; int c = a / 6 + !b; return c * 2 == a - 1431655762; }</code></pre> And this is the parser output. It oviously does not cover every path the code can take but that's what a test suite is for. Which they all pass, so hooray. PROGRAM FUNCTION main int a <- 2147483646; int b <- 0; int c <- (+ a (! b)); RETURN (== c (- a 1431655762))</code></pre> It is important to keep in mind that the test suite at this stage does not check for correct logic, but for whether the compilers fails the programs it should. Correct logic is checked either by custom unit tests (which I discussed last chapter) and testing the final compiler output. Wait hold on. Where did the a/6</code> go? Unit Testing Considered Useful</h3> Checking the unit tests I did last chapter, they are failing too. What gives? The code is all up there. Can you spot the error? Poking around, I realized the error happened when I changed all functions to return the nodes immediately rather than pointers to them. It is this pair of lines in the beginning of parse_expr</code>: var lhs = try parse_factor(arena, tokens); const lhs_ptr = try utils.create(ast.Expr, arena, lhs);</code></pre> And then I keep the first and only lhs_ptr</code> around assigning to the left side of all following expressions. Bah. Running the fix on the C code snippet above gets me this result now. PROGRAM FUNCTION main int a <- 2147483646; int b <- 0; int c <- (+ (/ a 6) (! b)); RETURN (== (* c 2) (- a 1431655762))</code></pre> This is the new and improved parse_expr</code>. fn parse_expr( arena: std.mem.Allocator, tokens: *lexer.Tokenizer, min_prec: u8, ) Error!ast.Expr { var lhs = try parse_factor(arena, tokens); var next_token = tokens.next() orelse return error.SyntaxError; defer tokens.put_back(next_token); while (next_token.tag.binop_precedence()) |r| { const prec, const left = r; if (prec < min_prec) break; const lhs_ptr = try utils.create(ast.Expr, arena, lhs); // <-- fix const rhs = try parse_expr(arena, tokens, prec + left); const rhs_ptr = try utils.create(ast.Expr, arena, rhs); const bin_op: ast.Expr.BinOp = .{ lhs_ptr, rhs_ptr }; lhs = switch (next_token.tag) { // same as before }; next_token = tokens.next() orelse return error.SyntaxError; } return lhs; }</code></pre> Semantic Analysis</h2> Oh that's a new one. This is where things like 2 = a * 3</code> are rejected. Other compilers seem to call this pass sema</code>, so I am adding a sema.zig</code>. This file is concerned with transforming and checking the AST: double declarations, no declarations, shadowing, scoping, what have you. The pseudocode functions in the Book have a variable_map</code> input. it maps each name, apparently, to a unique name (that I assume will be interned later?). I am unsure how this would look eventually, but for now I will be naive and use a std.StringHashMap([]const u8)</code>. Here is the Book's resolve_decl</code> pseudocode into Zig-ish pseudocode: fn resolve_decl( decl: *ast.Decl, variable_map: std.StringHashMap([]const u8), ) Error!void { if (variable_map.contains(decl.name)) return error.DuplicateVariableDecl; const unique_name = make_temporary(); // todo try variable_map.put(decl.name, unique_name); const init: ?*Expr = if (decl.init) |*expr| try resolve_expr(expr, variable_map) // todo else null; // where is the expr allocated? return .{ .name = unique_name, .init = init }; }</code></pre> So many todos and questions. This requires filling up piece by piece. New Names for Old Variables</h3> First of all, the new unique_name</code>. This is going to be used, as is, in the next compiler passes. So it makes sense for it to be on the StringInterner. Which means, the StringInterner and its backing allocator should be passed into the function. I am going to save myself some trouble and just go ahead and create a Boilerplate</code> struct like I did in ir_gen</code> previously. // first pass const Boilerplate = struct { gpa: std.mem.Allocator, strings: *utils.StringInterner, variable_map: *std.StringHashMapUnmanaged([:0]const u8), fn make_temporary( self: @This(), comptime prefix: []const u8, ) Error![:0]const u8 { // todo } };</code></pre> The next problem is that, to make sure the variable is unique in future compiler passes, it should use the counter. The counter is currently self contained within ir_gen</code>'s make_temporary</code>, but now it has to be elevated into its own thing. So, I am changing it to make it a method of StringInterner</code>, and have both new and old make_temporary</code> functions use that one. This is the new method on StringInterner</code>: pub fn make_temporary( self: *StringInterner, gpa: std.mem.Allocator, prefix: []const u8, ) ![:0]const u8 { // zig static variables const static = struct { var counter: usize = 0; }; var buf: [16]u8 = undefined; const name_buf = try std.fmt.bufPrint( &buf, "{s}.{}", .{ (if (prefix.len == 0) "tmp" else prefix), static.counter }, ); const name = try self.get_or_put(gpa, name_buf); static.counter += 1; return name.string; }</code></pre> One change from the previous function: prefix</code> is not comptime</code> known any more as it can be a user-specified variable. I am not quite sure, to be honest, if this code compiles, but I will find out by the end of this section. This is the new make_temporary</code> in both ir_gen</code> and the new sema</code>. Note that self</code> (and Error</code> for that matter) are different types for both functions, but it has the same interface. fn make_temporary( self: @This(), prefix: []const u8, ) Error![:0]const u8 { return try self.strings.make_temporary(self.alloc, prefix); }</code></pre>Resolutions</h3> This is the new resolve_decl</code> after the nip and tuck and adjusting to change the Decl</code> in place: fn resolve_decl( bp: Boilerplate, decl: *ast.Decl, ) Error!void { if (bp.variable_map.contains(decl.name)) return error.DuplicateVariableDecl; const unique_name = try bp.make_temporary(decl.name); try bp.variable_map.put(bp.gpa, decl.name, unique_name); if (decl.init) |expr| try resolve_expr(bp, expr); }</code></pre> Only resolve_expr</code> remains a stub. So naturally the next step is the relatively simple resolve_stmt</code>. fn resolve_stmt( bp: Boilerplate, stmt: *ast.Stmt, ) Error!void { switch (stmt.*) { .@"return", .expr => |expr| try resolve_expr(bp, expr), .null => {}, } }</code></pre> And now this is resolve_expr</code>. To be frank I am not quite sure I still have the right incantations regarding *</code> and .*</code> and &</code>. I trust the compiler will correct me when it comes to it.^{2</a>} fn resolve_expr( bp: Boilerplate, expr: *ast.Expr, ) Error!void { switch (expr.*) { .constant => {}, .assignment => |*b| { if (b.@"0".* != .@"var") return error.InvalidLValue; try resolve_expr(bp, b.@"0"); try resolve_expr(bp, b.@"1"); }, .@"var" => |name| expr.* = if (bp.variable_map.get(name)) |un| .{ .@"var" = un } else return error.UndeclaredVariable, .unop_neg, .unop_not, .unop_lnot, => |u| try resolve_expr(bp, u), .binop_add, .binop_sub, .binop_mul, .binop_div, .binop_rem, .binop_and, .binop_or, .binop_eql, .binop_neq, .binop_ge, .binop_gt, .binop_le, .binop_lt, => |b| { try resolve_expr(bp, b.@"0"); try resolve_expr(bp, b.@"1"); }, } }</code></pre> This leaves resolve_prgm</code> and resolve_func_def</code> which are barely interesting, if not for setting up the Boilerplate</code>. pub fn resolve_prgm( gpa: std.mem.Allocator, strings: *utils.StringInterner, prgm: *ast.Prgm, ) Error!void { var variable_map: std.StringHashMapUnmanaged([:0]const u8) = .empty; const bp: Boilerplate = .{ .gpa = gpa, .strings = strings, .variable_map = &variable_map, }; try resolve_func_def(bp, prgm.func_def); } fn resolve_func_def( bp: Boilerplate, func_def: *ast.FuncDef, ) Error!void { var iter = func_def.body.iterator(0); while (iter.next()) |item| switch (item.*) { .S => |*s| try resolve_stmt(bp, s), .D => |*d| try resolve_decl(bp, d), }; }</code></pre> Now, the proof of the pudding is in the eating. Let's wire it in the compiler driver (with an all new validate</code> option), and see the results. NoSpaceLeft</h3> Running the test suite came up with odd and confusing errors and long info dumps. Time to investigate. First of all, run the new code against the sample C program from before. and it fails with a .. memory leak? Turning off the memory leak detection (which is simple enough with Zig's DebugAllocator</code>) It passes fine, but the variable names seem unchanged. Hmm. Trying the test suite again with the turned off leak checking (just to see if it passes), however, is also failing with NoSpaceLeft</code>. One file seems to fail, and it is following: int main(void) { int first_variable = 1; int second_variable = 2; return first_variable + second_variable; }</code></pre> Looks simple enough. Running it through, the zig compilers gives me a very helpful, and very very long, stack tarce. It fails in make_temporary</code>. Oh the buffer does not have enough space. I made a 16 byte buffer to avoid a heap allocation. Silly me. Let's change it to a 64 byte buffer.3</a> // was var buf: [16]u8 = undefined; const name_buf = try std.fmt.bufPrint( &buf, "{s}.{}", .{ (if (prefix.len == 0) "tmp" else prefix), static.counter }, ); // becomes var buf: [64]u8 = undefined; const name_buf = try std.fmt.bufPrint( &buf, "{s}.{}", .{ (if (prefix.len == 0) "tmp" else prefix), static.counter }, );</code></pre> Well, this fixes the issue at least. This is the output of --validate</code> on that file now: PROGRAM FUNCTION main int first_variable <- 1; int second_variable <- 2; RETURN (+ first_variable.0 second_variable.1)</code></pre> As you can see, the names are updated alright, but apparently not in Decl</code>'s name</code> field. That's a silly oversight. Let's fix that now. // end of resolve_decl decl.name = unique_name; // <-- new line }</code></pre> Well this takes care of that. Now let me figure out the memory leak. Leak</h3> Let's run the same program again and look through the call trace the zig compiler very helpfully provides. Oh I forgot to deinit variable_map</code>. Silly me. This would happen in resolve_prgm</code> because I have no use for it afterwards right now. Yay no memory leaks. That ws easy. IR Generation</h2> No changes to the IR syntax tree today. Only generating IR for the new AST nodes. This also means there are no assembly changes either. Almost there to a stable IR backend.4</a> The most interesting thing in here is that variable declarations are simply discarded. They have done their job in semantic analysis, and now they are shown the door. But let's go backwards. The new expressions emitting IR are just @"var"</code> and assignment</code>. var</code> is cast to a null terminated pointer but that is it. It is already backed by the interner from the semantic analysis phase. assignment</code> puts the items in a weird bizarro order, where dst</code> is the first item. .@"var" => |v| return .{ .variable = @ptrCast(v) }, .assignment => |b| { // bizarro order const dst = try expr_emit_ir(bp, b.@"0"); // guaranteed to be a var const src = try expr_emit_ir(bp, b.@"1"); try bp.append(.{ .copy = .init(src, dst) }); return dst; },</code></pre> The new statements just pass through: fn stmt_emit_ir( bp: Boilerplate, stmt: *const ast.Stmt, ) Error!void { switch (stmt.*) { .null => {}, .@"return" => |e| try bp.append(.{ .ret = try expr_emit_ir(bp, e), }), .expr => |e| _ = try expr_emit_ir(bp, e), } }</code></pre> The interesting new part is func_def_emit_ir</code>. Since body</code> is now a list of stuff rather than just one statement, I need to iterate over it and generate IR from each item. But there is no decl_emit_ir</code> yet, so I will do that first. If there is no initialization expression, do nothing. If there is, evaluate the expression and copy it to the variable's name. fn decl_emit_ir( bp: Boilerplate, decl: *const ast.Decl, ) Error!void { if (decl.init) |e| { const src = try expr_emit_ir(bp, e); try bp.append(.{ .copy = .init(src, .{ .variable = @ptrCast(decl.name) }) }); } }</code></pre> Now func_def_emit_ir</code> has to construct the IR instructions in a body. So a loop over the block items to process them one at a time is the simplest way to do it. The Book recommends adding a simple return 0;</code> instruction to the end as an edgecase for functions written without a return value. This is the C standard for main</code>, but as omitting a return value is undefined behaviour for other functions, I can just do whatever, so adding a return 0</code> is entirely within the spec. It would not get processed if the code author did not forget their return</code>. fn func_def_emit_ir( alloc: std.mem.Allocator, strings: *utils.StringInterner, func_def: *const ast.FuncDef, ) Error!ir.FuncDef { const name = try strings.get_or_put(alloc, func_def.name); var instrs: std.ArrayListUnmanaged(ir.Instr) = .empty; const bp: Boilerplate = .{ .alloc = alloc, .strings = strings, .instrs = &instrs, }; var iter = func_def.body.constIterator(0); while (iter.next()) |item| switch (item.*) { .S => |s| try stmt_emit_ir(bp, s), .D => |d| try decl_emit_ir(bp, d), }; try instrs.append(alloc, .{ .ret = .{ .constant = 0 } }); return .{ .name = name.string, .instrs = instrs }; }</code></pre> And that's it. There is no new assembly generation because there are no new IR instructions. So time to test the whole shebang, which it all passes. So Hurray. Lessons Learned</del> Rant About Zig</h2> How much I hate Zig's unergonomic optionals, for one, and how unergonomic integer literals are, for two. For all the good Zig has, some things are just way, way harder than they have any right to be. Take for example parse_expr</code>. while (current.tag.binop_precedence()) |r| { const prec, const left = r; if (prec < min_prec) break; const rhs = try parse_expr(arena, tokens, prec + left); // <- look here // snip --</code></pre> The reason I made binop_precedence</code> return a u8</code> for left-associativity, is how absurdly difficult it is and ugly it is to work with otherwise. The natural way of modeling this is to use either a bool</code> or an enum. But if left</code> was a bool, this very normal looking, perfectly readable code would not compile: const rhs = try parse_expr(arena, tokens, prec + if (left) 1 else 0);</code></pre> The reason for that is integer literals (1</code> and 0</code> in this snippet) are of type comptime_int</code>. The type inference is not smart enough to get there. Which is like, fine, whatever. That would not be an issue if it was not for the fact you literally cannot create typed literals. 1u8</code>, like Rust or Odin would do, is not the Zig way. It is not explicit enough. The way to do it is @as(u8, 1)</code>. No joke. const rhs = try parse_expr(arena, tokens, prec + if (left) @as(u8, 1) else 0);</code></pre> At least the type inference is smart enough to infer the 0</code>, thank god. Wait, you say, you can cast. So why not cast the bool</code> to a u8</code>? Haha you fool. Casting a bool</code> to an integer makes no sense! There is a special builtin for that, called @intFromBool</code>. Why not use that, you ask? Because this built in returns a u1</code>. Which is really a u8</code> in memory but the type system will not automatically cast it to the wider integer because ... fuck if I know.^{5</a> So you end up with this beauty: @as(u8, @intFromBool(left))</code>.} Absolutely inane design decisions that all they do is annoy and frustrate the user. So I just returned a u8</code> from binop_precedence</code>. Incorrect, but less fuss. I will leave the optionals rant for later. Meanwhile, enjoy this contraption that should have been in the core language. (Yes it does work.) pub fn compare_optional( opt: anytype, value: @typeInfo( @typeInfo(@TypeOf(opt)).optional.child, ).@"union".tag_type.?, ) bool { return ((opt orelse return false) == value); }</code></pre> I speak, of course, of the beloved ternary operator. ↩</a> </li> I will have you know I went back and corrected things as the compiler bid me, so now it looks like I got it right the first time. ↩</a> </li> I looked it up, and the maximum length for internal C identifirs is 63 bytes, and anything afterwards is discarded. This is tighter, as it does not account for the added stuff, but whatever. ↩</a> </li> Not even close. ↩</a> </li> To add insult to injury, ?u1</code> is not niche optimized and has a size of two bytes, despite it being only two bits of info. ↩</a> </li> </ol> </section> Chapter 4: Logic 2025-05-19T00:00:00+00:00 Three chapters</a> out of who knows from Writing a C Compiler</a>. Time onto Chapter 4. Chapter 4 is about, let me check, "logical and relational operators". Great. More operators. Lexer</h2> You would expect the lexer by now to be a boring done deal, and you would be right. However, this is slightly more interesting because I get to add more states to the state machine! The tokens for today are the following: !</code>, &&</code>, ||</code>, ==</code>, !=</code>, <</code>, ></code>, <=</code>, and >=</code>. As I am not supporting bitshift operations, I will consider those tokens (specifically &</code> and |</code>) a failure state. The tests will not have them, either way. Adding the token tags is straightforward. The State</code> enum however, should have a new state for every partial application. This is the new State</code>: const State = enum { start, identifier, int, hyphen, // new bang, ambersand, pipe, equals, lesser_than, greater_than, };</code></pre> The character in which the tokenizer enters these states is, I think, self explanatory. This, for example, is pipe</code>, and lesser_than</code>. .start => switch (self.buffer[self.index]) { // snip -- '|' => continue :state .pipe, '<' => continue :state .lesser_than, // snip -- ], .pipe => { self.index += 1; switch (self.buffer[self.index]) { '|' => { self.index += 1; result.tag = .double_pipe; }, else => result.tag = .invalid, } }, .lesser_than => { self.index += 1; switch (self.buffer[self.index]) { '=' => { self.index += 1; result.tag = .lesser_equals; }, else => result.tag = .lesser_then, } },</code></pre>Parser</h2> The parser is a relatively simpler affair. More binary operations with different precedence levels. None of them are right associative, so nothing but updating the AST and the various switch</code> statements. Nothing much to write about. Unit Tests</h2> To spice up this section, I have decided to take on writing unit tests for the parser. Zig has first class support for unit tests. Make a block titled test</code> and the compiler sees it. The only problem is hooking it in the build system. Back in chapter 2, I think, I rejigged the zig build test</code> command to run the Book's test suite. As in, it just compiles and installs the binary as normal, then runs a shell command. Zig's built in testing support requires a different kind of wiring. Thankfully, the default output with zig init</code> contains all the necessary pieces, I can reuse those. One question remains is whether to tie them to the suit's test suite, so both run with zig build test</code>, or have it a separate command. I believe having it a separate command is best, to cut down on the, already slow, testing time. I shall change the existing test</code> command to submit</code> (so I'd type zig build submit</code>), and have the test</code> command for unit tests. For reference, this is the old step after changing the name. (The block is organizational). { // `zig build submit` command const test_step = b.step("submit", "Run the Book's test suite"); // subshells. how do they work. const inner_command = try std.mem.join(b.allocator, " ", &.{ "../writing-a-c-compiler-tests/test_compiler", b.pathJoin(&.{ b.exe_dir, "paella" }), try std.mem.join( b.allocator, " ", b.args orelse &.{""}, ), }); // does this work like i think it does? const test_command = b.addSystemCommand( &.{ "arch", "-x86_64", "zsh", "-c", inner_command }, ); test_command.step.dependOn(b.getInstallStep()); test_step.dependOn(&test_command.step); }</code></pre> Copying from the default build.zig</code> and trimming unnecessary things, I get this: { // `zig build test` command const test_step = b.step("test", "Run unit tests"); const exe_unit_tests = b.addTest(.{ .root_module = exe_mod, }); const run_exe_unit_tests = b.addRunArtifact(exe_unit_tests); test_step.dependOn(&run_exe_unit_tests.step); }</code></pre> And .. that should be it? To test whether the test command run, I shall write a test</code> to test whether 2 + 2 equals 4 in main.zig</code>. test "test the test" { try std.testing.expect(2 + 2 == 4); }</code></pre> Ok zig build test</code> comes and goes and says nothing. Let's see if it can detect a failure. I shall expect 2 + 2 does not equal 4. test "test the test" { try std.testing.expect(2 + 2 != 4); }</code></pre> And it fails. So our tests work! Putting it parser.zig</code> however, does not seem to work. Tests pass no matter what I expect. What gives? After asking around in the Zig discord, this is a known thing. Zig's compiler is lazy, and does not look at functions not referenced from main.zig</code>. But because the test executable does not run fn main</code>, it does not see any modules that are referenced within main.zig</code> but not referenced in main.zig</code>'s tests. It is dumb as fuck, if you ask me. And honestly I cannot tell if this is a deliberate design decision or just a known bug. The solution is, well, to write a test</code> block in main.zig</code> that references the files in which I'd like to run tests.^{1</a> The most straightforward way to do this is this:} test { std.testing.refAllDeclsRecursive(@This()); }</code></pre> And now the tests in parse.zig</code> pass and fail as expected. This is apparently a hack and not recommended. The recommended way is even more stupid looking, which only imports parser.zig</code>, and I'd have to do it for every file and every sub import. test { _ = parser; }</code></pre> Having gone that far, let's actually write our unit test in parse.zig</code>. test "precedence 1" { const t = std.testing; const src = "3 * 4 + 5;"; var tokens = lexer.Tokenizer.init(src); var a_a = std.heap.ArenaAllocator.init(t.allocator); defer a_a.deinit(); const a = a_a.allocator(); const result = try parse_expr(a, &tokens, 0); try t.expect(result.* == .binop_add); try t.expectFmt("(+ (* 3 4) 5)", "{}", .{result}); }</code></pre> My first drafts of the test failed because I was leaking memory, hence the adding an arena ceremony. It is a bit more ceremony than I expected, but it is fine. Who needs unit testing anyway? End to end testing is where it is at. Internal Representation</h2> The IR this chapter is a lot more interesting than usual. Since &&</code> and ||</code> are short circuting operators, it means now is the time have jumps and labels and goto</code>s and jump-if-zeros. With conditionals, anything is possible. the new instructions are a lot. Here is the new ir.Instr</code>. You will note that there are no direct instructions for and</code> and or</code>.2</a> pub const Instr = union(enum) { ret: Value, copy: Unary, jump: [:0]const u8, jump_z: JumpIf, jump_nz: JumpIf, label: [:0]const u8, unop_neg: Unary, unop_not: Unary, unop_lnot: Unary, // <-- new binop_add: Binary, binop_sub: Binary, binop_mul: Binary, binop_div: Binary, binop_rem: Binary, binop_eql: Binary, binop_neq: Binary, binop_lt: Binary, binop_le: Binary, binop_gt: Binary, binop_ge: Binary, pub const Unary = struct { src: Value, dst: Value, pub fn init(src: Value, dst: Value) @This() { return .{ .src = src, .dst = dst }; } }; pub const Binary = struct { src1: Value, src2: Value, dst: Value, pub fn init(src1: Value, src2: Value, dst: Value) @This() { return .{ .src1 = src1, .src2 = src2, .dst = dst }; } }; pub const JumpIf = struct { cond: Value, target: [:0]const u8, pub fn init(cond: Value, target: [:0]const u8) @This() { return .{ .cond = cond, .target = target }; } }; };</code></pre> The jump</code> instruction is straight up goto</code>. 3</a> The two conditional jumps simply evaluate their cond</code> operand, and act accordingly. One of them is enough, but it is apparently simpler to do it this way. These conditional jumps are the backbone of &&</code> and ||</code>, and the ternary operator come chapter 6. The operations other than &&</code> and ||</code> have a straightforward impl similar to the previous ones. It is &&</code> and ||</code> that depend on the new instructions. This is the implementation of &&</code>. .binop_and => |b| { const false_label = try bp.make_temporary("false_and"); const end_label = try bp.make_temporary("end_and"); const result = try bp.make_temporary("dst_and"); const src1 = try expr_emit_ir(bp, b.@"0"); try bp.append(.{ .jump_z = .init(src1, false_label) }); const src2 = try expr_emit_ir(bp, b.@"1"); try bp.append(.{ .jump_z = .init(src2, false_label) }); const dst : ir.Value = .{ .variable = result }; try bp.append(.{ .copy = .init(.{ .constant = 1 },dst ) }); try bp.append(.{ .jump = end_label }); try bp.append(.{ .label = false_label }); try bp.append(.{ .copy = .init(.{ .constant = 0 }, dst) }); try bp.append(.{ .label = end_label }); return dst; },</code></pre> Let me walk this one bit by bit, just to make sure I understand it. &&</code> short circuits if the first operand is false, or zero. evaluate src1</code>.</li> if src1</code> is zero, the result is zero, and we jump to where result</code> is set to zero.</li> if not, then we evaluate src2</code>.</li> if src2</code> is zero, the result is zero, and we jump to where result</code> is set to zero.</li> if not, then the result is one, and we jump to where the result</code> is set to one.</li> </ol> For ||</code> this would be the following: evaluate src1</code>.</li> if src1</code> is one, the result is one, and we jump to where result</code> is set to one.</li> if not, then we evaluate src2</code>.</li> if src2</code> is one, the result is one, and we jump to where result</code> is set to one.</li> if not, then the result is zero, and we jump to where the result</code> is set to zero.</li> </ol> Which means this should be our ||</code> IR generation. .binop_or => |b| { const true_label = try bp.make_temporary("true_or"); const end_label = try bp.make_temporary("end_or"); const result = try bp.make_temporary("dst_or"); const src1 = try expr_emit_ir(bp, b.@"0"); try bp.append(.{ .jump_nz = .init(src1, true_label) }); const src2 = try expr_emit_ir(bp, b.@"1"); try bp.append(.{ .jump_nz = .init(src2, true_label) }); const dst : ir.Value = .{ .variable = result }; try bp.append(.{ .copy = .init(.{ .constant = 0 }, dst) }); try bp.append(.{ .jump = end_label }); try bp.append(.{ .label = true_label }); try bp.append(.{ .copy = .init(.{ .constant = 1 }, dst) }); try bp.append(.{ .label = end_label }); return dst; },</code></pre> This should work. This is the C file I am experimenting with this evening: int main(void) { return 0 || 0 && (1 / 0); }</code></pre> And this is the generated IR: PROGRAM FUNCTION main jnz 0 => true_or.0 jz 0 => false_and.3 div.6 <- 1 / 0 jz div.6 => false_and.3 dst_and.5 <- 1 jump => end_and.4 => false_and.3 dst_and.5 <- 0 => end_and.4 jnz dst_and.5 => true_or.0 dst_or.2 <- 0 jump => end_or.1 => true_or.0 dst_or.2 <- 1 => end_or.1 ret dst_or.2</code></pre> Eh. This looks wrong at first glance, but working through it, it seems to be .. fine? The division by 0 is immediately jumped over to greener pastures and the function returns 0. The tests pass, which means I have corrected all the typos and exhausted all the switches and there are no panics. Verification of the logic will come later. Assembly Generation</h2> The Book goes into a lengthy explanation of the "flags" mechanism in c86 assembly, in addition with the first brush with Undefined Behaviour. No point in repeating that here, go read the book. I just want to write code. The new Assembly AST includes new assembly instructions, as well as cond_code</code>s: the various flags that will be set and unset depending on given comparisons. These are kinda free floating right now. Maybe I will stuff them within the Instr</code> namespace later. const CondCode = enum { e, ne, g, ge, l, le };</code></pre> That's it. The rest comes when adding doubles and floats and nasty other stuff. And here are the new instructions: pub const Instr = union(enum) { // snip -- cmp: Mov, jmp: [:0]const u8, jmp_cc: struct { CondCode, [:0]const u8 }, set_cc: struct { CondCode, Operand }, label: [:0]const u8, // snip -- };</code></pre> cmp</code> just uses the Mov</code> struct, which is really an Operand</code> tuple. The other tuples are not reused enough for me to give them a name (yet?). Then comes the conversion between the IR and the Assembly. fn instr_to_asm( alloc: std.mem.Allocator, instr: ir.Instr, ) ![]assembly.Instr { switch (instr) { // snip== .unop_lnot => |u| { const src = value_to_asm(u.src); const dst = value_to_asm(u.dst); return try alloc.dupe(assembly.Instr, &.{ .{ .cmp = .init(.{ .imm = 0 }, src) }, .{ .mov = .init(.{ .imm = 0 }, dst) }, .{ .set_cc = .{ .e, dst } }, }); }, // snip -- .binop_eql, .binop_neq, .binop_lt, .binop_le, .binop_gt, .binop_ge => |b| { const src1 = value_to_asm(b.src1); const src2 = value_to_asm(b.src2); const dst = value_to_asm(b.dst); const cc: assembly.Instr.CondCode = switch (instr) { .binop_eql => .e, .binop_neq => .ne, .binop_lt => .l, .binop_le => .le, .binop_gt => .g, .binop_ge => .ge, else => unreachable, }; return try alloc.dupe(assembly.Instr, &.{ .{ .cmp = .init(src2, src1) }, .{ .mov = .init(.{ .imm = 0 }, dst) }, .{ .set_cc = .{ cc, dst } }, }); }, .label => |s| return try alloc.dupe(assembly.Instr, &.{ .{ .label = s }, }), .jump => |s| return try alloc.dupe(assembly.Instr, &.{ .{ .jmp = s }, }), .jump_z, .jump_nz => |j| return try alloc.dupe(assembly.Instr, &.{ .{ .cmp = .init(.{ .imm = 0 }, value_to_asm(j.cond)) }, .{ .jmp_cc = .{ if (instr == .jump_z) .e else .ne, j.target } }, }), .copy => |u| return try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init( value_to_asm(u.src), value_to_asm(u.dst), ) }, }), } }</code></pre> The rest of replacing pseudo registers is fairly mechanical and boring and I hate it. Here is the current fixup_instrs</code>, just to see what it has grown into in three short chapters. pub fn fixup_instrs( alloc: std.mem.Allocator, prgm: *assembly.Prgm, ) !void { var out: std.ArrayListUnmanaged(assembly.Instr) = try .initCapacity( alloc, prgm.func_def.instrs.capacity, ); defer { std.mem.swap( std.ArrayListUnmanaged(assembly.Instr), &out, &prgm.func_def.instrs, ); out.deinit(alloc); } const State = enum { start, mov_stack_stack, cmp_stack_stack, cmp_to_imm, add_stack_stack, sub_stack_stack, mul_to_stack, idiv_const, legal, }; for (prgm.func_def.instrs.items) |instr| { state: switch (State.start) { .start => switch (instr) { .mov => |m| if (m.src == .stack and m.dst == .stack) continue :state .mov_stack_stack else continue :state .legal, .cmp => |m| if (m.src == .stack and m.dst == .stack) continue :state .cmp_stack_stack else if (m.dst == .imm) continue :state .cmp_to_imm else continue :state .legal, .add => |m| if (m.src == .stack and m.dst == .stack) continue :state .add_stack_stack else continue :state .legal, .sub => |m| if (m.src == .stack and m.dst == .stack) continue :state .sub_stack_stack else continue :state .legal, .mul => |m| if (m.dst == .stack) continue :state .mul_to_stack else continue :state .legal, .idiv => |o| if (o == .imm) continue :state .idiv_const else continue :state .legal, else => continue :state .legal, }, .mov_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mov.src, .{ .reg = .R10 }) }, .{ .mov = .init(.{ .reg = .R10 }, instr.mov.dst) }, }), .cmp_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.cmp.src, .{ .reg = .R10 }) }, .{ .cmp = .init(.{ .reg = .R10 }, instr.cmp.dst) }, }), .cmp_to_imm => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.cmp.dst, .{ .reg = .R11 }) }, .{ .cmp = .init(instr.cmp.src, .{ .reg = .R11 }) }, }), .add_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.add.src, .{ .reg = .R10 }) }, .{ .add = .init(.{ .reg = .R10 }, instr.add.dst) }, }), .sub_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.sub.src, .{ .reg = .R10 }) }, .{ .sub = .init(.{ .reg = .R10 }, instr.sub.dst) }, }), .mul_to_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mul.dst, .{ .reg = .R11 }) }, .{ .mul = .init(instr.mul.src, .{ .reg = .R11 }) }, .{ .mov = .init(.{ .reg = .R11 }, instr.mul.dst) }, }), .idiv_const => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.idiv, .{ .reg = .R11 }) }, .{ .idiv = .{ .reg = .R11 } }, }), .legal => try out.append(alloc, instr), } } }</code></pre> I kind of like how it looks, in that there is a clear listing of each illegal state, and a direct transformation from a legal form to the other, and the discovery code is split up from the replacement code. It might even be easier to review? I do not know. All tests pass. Even the earlier dividng by zero one. If you are curious, here is the generated final assembly. .globl _main _main: pushq %rbp movq %rsp, %rbp subq $12, %rsp movl $0, %r11d cmpl $0, %r11d jne .Ltrue_or.0 movl $0, %r11d cmpl $0, %r11d je .Lfalse_and.3 movl $1, %eax cdq movl $0, %r11d idivl %r11d movl %eax, -4(%rsp) cmpl $0, -4(%rsp) je .Lfalse_and.3 movl $1, -8(%rsp) jmp .Lend_and.4 .Lfalse_and.3: movl $0, -8(%rsp) .Lend_and.4: cmpl $0, -8(%rsp) jne .Ltrue_or.0 movl $0, -12(%rsp) jmp .Lend_or.1 .Ltrue_or.0: movl $1, -12(%rsp) .Lend_or.1: movl -12(%rsp), %eax movq %rbp, %rsp popq %rbp ret</code></pre> There is one complication. Even tho the tests passed, while reviewing through the book I realized I am supposed to emit different names for the registers when they are in the set_cc</code> instruction. Since the tests passed I can just .. ignore this? But let's do it. I am going to abuse the width</code> property (same one I used to get indentation) to pass in the register's width. For reference, this the current emission/printing code for set_cc</code>: .set_cc => |s| try writer.print("\tset{s:<7}{gen}", .{ @tagName(s.@"0"), s.@"1" }),</code></pre> Fairly straightforward and avoids special formatting logic for CondCode</code>: just use the tag name! The :_<7</code> syntax is how you're supposed to use the width</code> property, by passing optional padding. For the Operand, all the change needed here is to change {gen}</code> to {gen:1}</code>, passing a width of 1</code> into the downstream formatter. In the Operand</code> formatter, this is the Register printing code: .reg => |r| switch (r) { .AX => try writer.print("%eax", .{}), .DX => try writer.print("%edx", .{}), .R10 => try writer.print("%r10d", .{}), .R11 => try writer.print("%r11d", .{}), },</code></pre> And this is when a conditional for width == 1</code> is added. I am treating the default value as before. .reg => |r| if (options.width == 1) switch (r) { .AX => try writer.print("%a1", .{}), .DX => try writer.print("%d1", .{}), .R10 => try writer.print("%r10b", .{}), .R11 => try writer.print("%r11b", .{}), } else switch (r) { .AX => try writer.print("%eax", .{}), .DX => try writer.print("%edx", .{}), .R10 => try writer.print("%r10d", .{}), .R11 => try writer.print("%r11d", .{}), },</code></pre> All tests continue to pass. So that is nice I guess. Lessons Learned</h2> Not much really. This is a fairly mechanical chapter. The new info I had already learned when going through the Book the first time. Maybe I learned not to target x86 assembly directly for my own language. Why not target wasm</code>? It does not have goto</code>. Yet another point for debugging your language knowledge. And do not tell me this is not actually part of the language. ↩</a> </li> If I were developing a compiler for a language other than C, say Odin</a>, this is probably where I'd diverge from this Book on the IR. So early. But it seems annoying to me to encode the goto</code> logic straight in my implementation. I guess it depends on the backend. ↩</a> </li> This is necessary because C, and x86 assembly for that matter, has goto</code>, which is implemented later in the book, and it fits the C semantics of switch</code>. So implementing goto</code> is the least friction option. wasm</code>, however, does not have goto</code>, and uses structured code instead with break</code> out of blocks. I am not actually sure how C's goto</code> compiles to wasm</code>. Investigation is required. ↩</a> </li> </ol> </section> Chapter 3: Binary 2025-05-17T00:00:00+00:00 Two chapters</a> of Writing a C Compiler</a> done, many more to go, and it is time to implement binary expressions. But before I get into that, let me rant a bit about Zig. The War Against Tabs</h2> Zig's grammar does not tolerate the tab character anywhere in the file. This is unfortunate, as I like tabs. With source code, it is whatever, as zig fmt</code> takes care of it, but can I please type my tabs in multiline string literals please? I like to indent the output, whether it is the parser's debugging output or the final assembly output, with tabs. It just makes sense. It is simpler conceptually to type one tab twice for the second level of indentation than to type a space 4 or 8 or 16 times. Put a tab, and let the user choose the preferred tab width. But Zig's grammar will have none of it. There can be no tab characters in the file (albeit it is tolerated a bit at indentation before zig fmt</code> gets rid of it all.) This puts me in a dilemma. Consider this piece of code for emitting the ret</code> instruction. .ret => try writer.writeAll( "\tmovq %rbp, %rsp\n" ++ "\tpopq %rbp\n" ++ "\tret" ),</code></pre> Reasonable, right? But zig fmt</code> takes a machete to it, and they no longer nicely align. .ret => try writer.writeAll( "\tmovq %rbp, %rsp\n" ++ "\tpopq %rbp\n" ++ "\tret", ),</code></pre> The next reasonable solution is Zig's rather clever and very nice multiline string literals. It is literally my favorite piece of syntax in Zig. So much more ergonomic and grokkable and understandable than Swift's """</code> or Rust's .. nothing. So naturally, I'd type it like this: .ret => try writer.writeAll( \\ movq %rbp, %rsp \\ popq %rbp \\ ret ), // ^^ tab character here.</code></pre> But this does not compile. Just does not. No tab characters allowed. One could argue that this decision reduces confusion over a variable width character like tab and hides the true intentions of whatever. I do not care. I want my tabs, goddammit. I would be ok with escaping tabs '\t'</code>, but escapes do not work in multiline string literals. I could use std.fmt.comptimePrint</code>, but that adds too much obfuscation and makes using the literals as the fmt</code> arguments in print</code> statements a lot more complicated. It is just annoying all around. Anyway, with some magic of comptime</code>, and trial and error of Zig's comptime rules, I came up with this macro</del> function: inline fn indent( comptime text: []const u8, ) []const u8 { comptime { var iter = std.mem.splitScalar(u8, text, '\n'); var res: []const u8 = ""; while (iter.next()) |line| res = if (line.len > 0 and line[0] != '_') res ++ "\t" ++ line ++ "\n" else res ++ line ++ "\n"; return res[0 .. res.len - 1]; } }</code></pre> inline fn</code> and the comptime</code> block conspire together to make sure the body of this function always ever runs at comptime</code> and never at run time, with no ceremony at the calling site. The code itself is fairly straightforward: it separates the input by lines; adds a new line to all lines; and indents (with tabs!) any line that is not empty and does not start with _</code>, for label and function names. This is specific to my use case, but it is fine. It is used like this. .ret => try writer.writeAll(indent( \\movq %rbp, %rsp \\popq %rbp \\ret )),</code></pre> Almost invisible, What's better is that, since it returns a comptime</code> known string, it can be used as the fmt</code> argument in print functions, like the following snippet for the function's prelude (which also showcases the _</code> and the empty line rules). try writer.print(indent( \\.globl _{0s} \\_{0s}: \\pushq %rbp \\movq %rsp, %rbp \\ ), .{self.name});</code></pre> This prints, for name "main"</code>, as intended, as follows: .globl _main _main: pushq %rbp movq %rsp, %rbp</code></pre> So much pain would have been avoided if I could just type the damn byte. Now, back to business. Lexer</h2> Updating the lexer requires adding four new tokens, +</code>, *</code>, /</code>, and %</code>. Where is subtraction, you say? We already lex it, dummy. A failure mode for ++</code>, like done in Chapter 2, is not necessary, because 1 ++ 2</code> would be rejected by the parser later anyway. And there is no ambiguity about prefix operators, as the +</code> unary prefix operator is not implemented either. This makes things so considerably simple that I will not bother writing the update down. On to parsing, which is way more interesting. Parser</h2> As I am not using a parser combinator library this time, I have to actually get down and implement the legendary precedence claimbing/pratt parsing algorithm. So, sleeves rolled. First to update the Expr</code> type, splatting operations as before. BinOp</code> is just a helper tuple alias. One note: due to this helper type, the formatting for these is a bit more interesting than usual. Instead of creating an anonymous struct literal as second argument for print</code> (as usual), I can just pass the BinOp</code> payload, and it works. pub const Expr = union(enum) { constant: u64, unop_negate: *Expr, unop_complement: *Expr, binop_add: BinOp, binop_sub: BinOp, binop_mul: BinOp, binop_div: BinOp, binop_rem: BinOp, pub const BinOp = struct { *Expr, *Expr }; pub fn format( self: @This(), comptime _: []const u8, _: std.fmt.FormatOptions, writer: anytype, ) !void { switch (self) { .constant => |c| try writer.print("{d}", .{c}), // RPN .unop_negate => |e| try writer.print("{} --", .{e}), .unop_complement => |e| try writer.print("{} ~", .{e}), .binop_add => |b| try writer.print("{} {} +", b), // <-- NOT `.{b}` .binop_sub => |b| try writer.print("{} {} -", b), .binop_mul => |b| try writer.print("{} {} *", b), .binop_div => |b| try writer.print("{} {} /", b), .binop_rem => |b| try writer.print("{} {} %", b), } } };</code></pre> This is an article about me writing Zig, so I will not bore with yet another explanation of precedence climbing. Plenty of those online, and they are described in literally every compiler book.^{1</a> I am trying to write code here! Most of the work will be in the function parse_expr</code>. Here is the current version.} fn parse_expr( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Expr { const current = tokens.next() orelse return error.SyntaxError; switch (current.tag) { // literal .number_literal => { const lit = tokens.buffer[current.loc.start..current.loc.end]; const res = try std.fmt.parseInt(u64, lit, 10); return try create(ast.Expr, alloc, .{ .constant = res }); }, // unary operations .hyphen => { const inner_exp = try parse_expr(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_negate = inner_exp }); }, .tilde => { const inner_exp = try parse_expr(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_complement = inner_exp }); }, // groups .l_paren => { const inner_exp = try parse_expr(alloc, tokens); try expect(.r_paren, tokens); return inner_exp; }, else => return error.ExpectExpr, } }</code></pre> Well the first step is to rename this to parse_factor</code> (as the book calls it, or atom</code> is fine.). This is possible because unary operators have, almost, the highest precedence in C</a>, and can almost be assumed to be part of the literal. In fact the only operators with higher precedence are postfix, so the grammar works out naturally. Updating the recursive calls it in order to, except that whatever is between the parenthesis is a full expression, and not just a factor, so it becomes this. fn parse_factor( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Expr { const current = tokens.next() orelse return error.SyntaxError; switch (current.tag) { .number_literal => { // snip -- }, .hyphen => { const inner_exp = try parse_factor(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_negate = inner_exp }); }, .tilde => { const inner_exp = try parse_factor(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_complement = inner_exp }); }, .l_paren => { const inner_exp = try parse_expr(alloc, tokens); try expect(.r_paren, tokens); return inner_exp; }, else => return error.ExpectFactor, } }</code></pre> The new parse_expr</code> is where the magic happens. This is a first draft with no precedence for addition and subtraction: fn parse_expr( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Expr { var lhs = try parse_factor(alloc, tokens); var next_token = tokens.next() orelse return error.SyntaxError; while (next_token.tag == .plus or next_token.tag == .hyphen) { const rhs = try parse_factor(alloc, tokens); const new_lhs: ast.Expr = switch (next_token.tag) { .plus => .{ .binop_add = .{ lhs, rhs } }, .hyphen => .{ .binop_sub = .{ lhs, rhs } }, else => unreachable, }; lhs = utils.create(ast.Expr, alloc, new_lhs); next_token = tokens.next() orelse return error.SyntaxError; // unwrapping is probably correct. } return lhs; }</code></pre> I was about to go on, but I realized there is a bug in this implementation: It always consumes one more token than needed. The book uses a helper function called peek</code> that does not consume the token stream. This is fine as long as it is parsed in the same function, but it is not fine when it is used as a signal to stop parsing, and the next parser is expected to see it. Some sort of cache is required.2</a> Either in the tokenizer itself or external to it. The tokenizer has two fields: buffer</code>, which is the source code; and index</code> which is a cursor over the source code. The simplest way to do a cache is to add a third field: peeked</code> which would hold an optional token. But there is a funnier way: a put_back</code> method. See, every token has its span in the source code attached. So all I need to do, in theory, is just reset the index to that span's start. This is more work for the tokenizer as it has to run the tokenizing state machine over it again, but it is funny enough and simple enough and does-not-change-how-the-rest-of-the-code-works enough that I am going to try it. It probably doesn't even need to be a method, so inline</code>. pub inline fn put_back(self: *Tokenizer, token: Token) void { self.index = token.loc.start; }</code></pre> Done. I will put that in parse_expr</code>. It probably works. // snip -- tokens.put_back(next_token); return lhs; }</code></pre>Precedence</h3> The previous function parses only left-associative expressions with a single level of precedence. To parse proper PEMDAS, the precedence climbing algorithm should only parse expressions that are higher than a given, starting precedence, that is passed in as a parameter. Also, a precedence table is needed, which could be just baked into the binary. This requires a small helper method on the Tag</code> type to determine the precedence of each tag (and whether it is a binary operator to begin with). However, the changes to parse_expr</code> are minimal, aside from adding a new parameter. fn parse_expr( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, min_prec: u8, // <-- new parameter ) !*ast.Expr { var lhs = try parse_factor(alloc, tokens); var next_token = tokens.next() orelse return error.SyntaxError; while (next_token.tag.binop_precedence()) |prec| { // <- helper function if (prec < min_prec) break; const rhs = try parse_expr(alloc, tokens, prec + 1); const bin_op: ast.Expr.BinOp = .{ lhs, rhs }; const new_lhs: ast.Expr = switch (next_token.tag) { .plus => .{ .binop_add = bin_op}, .hyphen => .{ .binop_sub = bin_op}, .asterisk => .{ .binop_mul = bin_op }, .f_slash => .{ .binop_div = bin_op }, .percent => .{ .binop_rem = bin_op }, else => unreachable, }; lhs = utils.create(ast.Expr, alloc, new_lhs); next_token = tokens.next() orelse return error.SyntaxError; } tokens.put_back(next_token); return lhs; }</code></pre> And since everything is left associative, that's it. It is done. Aside from Zig being unable to infer the error union type, which means annotating an actual error type. This is the C file I am using as a scratch board this chapter: int main(void) { return (3 + 4 * 5 + -4); }</code></pre> And this is parsing output. I think I have come up with a better C syntax, don't you think? RPN is great. PROGRAM FUNCTION main RETURN 3 4 5 * + 4 -- +</code></pre> Playing around a bit, let's see how it looks as S-expressions. Much fun can be had. PROGRAM FUNCTION main RETURN (+ (+ 3 (* 4 5)) (- 4))</code></pre> Anyway, all chapter 3 parsing tests pass. So the cute put_back</code> function works! Having Fun with Closures</h2> Zig does not have closures or anonymous functions. If you want to build something of the sort, well, don't. The language will fight you. But if you insist, it is a bit of fun. Consider these couple of lines from parse_expr</code>: while (next_token.tag.binop_precedence()) |prec| { if (prec < min_prec) break; // snip --</code></pre> If you have ever done functional programming, or just used Rust, you can immediately tell this is an Option::filter</code>! (Maybe the Haskellers have another name for it.) So I figured I will try doing something similar here in Zig. I did not commit this code, as it replaces these two lines by two function definitions and one struct definition. It is a fun attempt and it worked and passed the tests as well. Might even compile to the same code, too. So we need to define a filter over optionals. It basically checks if the payload satisfies a predicate. The most straightforward way to do it that's generic enough is this: fn filter(T: type, opt: ?T, closure: anytype) ?T { const inner = opt orelse return null; if (closure.filter(inner)) return opt else return null; }</code></pre> closure</code> here is an anytype</code> that has a filter</code> method (which takes a T</code> and returns bool</code>). I do not like anytype</code> but I am not committed enough to this bit to find a better abstraction. Now inside our function, or file, we define a Closure</code> struct (or any other name, w/e) like this: const Closure = struct { min_prec: u8, fn filter(self: @This(), item: u8) bool { return item >= self.min_prec; } };</code></pre> Then replace our while</code> loop invocation with this beauty: // here is the part that makes it a closure vv while (filter(u8, next_token.tag.binop_precedence(), Closure{ .min_prec = min_prec })) |prec| { // snip --</code></pre> And it works! This is almost how closures work in languages that have them. Now back to boring imperative code. Intermediate Representation</h2> The IR pass here is fairly straightforward, and is a direct 1 to 1 translation from the AST. The code is short and very similar to the previousc chapter's implementation of unary instructions that it is entirely uninteresting to write about again. Seriously, much of the same. The only difference is that the container type, Binary</code>, has two src</code> fields. Here it is next to chapter 2's champion Unary</code>: pub const Unary = struct { src: Value, dst: Value, pub fn init(src: Value, dst: Value) @This() { return .{ .src = src, .dst = dst }; } }; pub const Binary = struct { src1: Value, src2: Value, dst: Value, pub fn init(src1: Value, src2: Value, dst: Value) @This() { return .{ .src1 = src1, .src2 = src2, .dst = dst }; } };</code></pre> Another thing I did to reduce the boilerplate in ir_gen.zig</code>, which was to create a Boilerplate</code> type, that holds all the pointers used in every function. This is the full type definition as of now. const Boilerplate = struct { alloc: std.mem.Allocator, strings: *utils.StringInterner, instrs: *std.ArrayListUnmanaged(ir.Instr), // helper for everywhere fn append( self: @This(), instr: ir.Instr, ) Error!void { try self.instrs.append(self.alloc, instr); } // helpers for `expr_emit_ir` fn unary( self: @This(), e: *ast.Expr, comptime prefix: []const u8, ) Error!ir.Instr.Unary { const src = try expr_emit_ir(self, e); const dst_name = try make_temporary(self.alloc, self.strings, prefix); const dst: ir.Value = .{ .variable = dst_name }; return .init(src, dst); } fn binary( self: @This(), b: ast.Expr.BinOp, comptime prefix: []const u8, ) Error!ir.Instr.Binary { const src1 = try expr_emit_ir(self, b.@"0"); const src2 = try expr_emit_ir(self, b.@"1"); const dst_name = try make_temporary(self.alloc, self.strings, prefix); const dst: ir.Value = .{ .variable = dst_name }; return .init(src1, src2, dst); } };</code></pre> And here is an example of it being used in the tiny stmt_emit_ir</code>, which saved on so much parameter space! fn stmt_emit_ir( bp: Boilerplate, stmt: *ast.Stmt, ) Error!void { switch (stmt.*) { .@"return" => |e| try bp.append(.{ .ret = try expr_emit_ir(bp, e), }), } }</code></pre> As a bonus, here is the IR generated for the sample C program above. PROGRAM FUNCTION main mul.0 <- 4 + 5 add.1 <- 3 + mul.0 neg.2 <- - 4 add.3 <- add.1 + neg.2 ret add.3</code></pre> Assembly</h2> Addition, subtraction, and multiplication are straightforward implementations, as they have their corresponding assembly instructions. Integer division .. is not. The first task, as usual, is to update the assembly syntax tree to include all the new instructions, and two new registers. pub const Instr = union(enum) { mov: Mov, ret: void, // unary operations neg: Operand, not: Operand, // binary operations // NEW add: Mov, sub: Mov, mul: Mov, idiv: Operand, cdq: void, // sign extension, don't ask. allocate_stack: u64, const Mov = struct { // just a pair of operands src: Operand, dst: Operand, pub fn init(src: Operand, dst: Operand) @This() { return .{ .src = src, .dst = dst }; } }; }; pub const Operand = union(enum) { imm: u64, reg: Register, pseudo: [:0]const u8, stack: i64, pub const Register = enum { AX, DX, R10, R11 }; };</code></pre> As it turned out, I did the basic work correctly last chapter. So the rest of this code turned out to be mechanical as well. These are the new cases in instr_to_asm</code>. Since these commands have similar structure, they are implemented the same, with another internal switch</code> for where they differ. It makes separating them easier later on. .binop_add, .binop_sub, .binop_mul => |b| { const src1 = value_to_asm(b.src1); const src2 = value_to_asm(b.src2); const dst = value_to_asm(b.dst); return try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init(src1, dst) }, switch (instr) { .binop_add => .{ .add = .init(src2, dst) }, .binop_sub => .{ .sub = .init(src2, dst) }, .binop_mul => .{ .mul = .init(src2, dst) }, else => unreachable, }, }); }, .binop_div, .binop_rem => |b| { const src1 = value_to_asm(b.src1); const src2 = value_to_asm(b.src2); const dst = value_to_asm(b.dst); const dst_reg: assembly.Operand.Register = if (instr == .binop_div) .AX else .DX; return try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init(src1, .{ .reg = .AX }) }, .cdq, .{ .idiv = src2 }, .{ .mov = .init(.{ .reg = dst_reg }, dst) }, }); },</code></pre> Replacing pseudo registers turned out to be straightforward too. I just had to add the .add, .sub. .mul</code> to the .mov</code> case (because they similarly take two operands); and .idiv</code> to the unaries. This is the full function now. pub fn replace_pseudos( alloc: std.mem.Allocator, prgm: *assembly.Prgm, ) !assembly.Instr.Depth { var pseudo_map: PseudoMap = .init(alloc); defer pseudo_map.deinit(); for (prgm.func_def.instrs.items) |*instr| { switch (instr.*) { .mov, .add, .sub,.mul => |*m| m.* = .init( try pseudo_to_stack(m.src, &pseudo_map), try pseudo_to_stack(m.dst, &pseudo_map), ), .neg, .not , .idiv=> |*v| v.* = try pseudo_to_stack(v.*, &pseudo_map), .ret, .cdq, .allocate_stack => {}, } } return @intCast(pseudo_map.count() * 4); }</code></pre> Fixing illegal instructions is, perhaps, the most annoying part of the book. idiv</code> cannot take a constand and the binaries cannot between two stac locations or whatever. It is a terribly unintresting part of the process and a hell to debug. Since I modeled it as a state machine, it might prove to be a bit more interesting than my Rust implementation. Here are the new fixups: idiv</code> cannot tak a constant. Change it to mov</code> to r10d</code> then idiv r10d</code></li> add</code> and sub</code> cannot be between two stack places. Change to mov</code> then `add.</li> imul</code> cannot have its destination be a stack place. Change to mov</code> to r11d</code>, then imul</code>, then mov</code> back from r11d</code>.</li> </ol> Do you see how mind numbing this is? Thankfully the state machine idea made copying and pasting previous code slightly trivial. I can perhaps even refactor it later, or not. const State = enum { start, mov_stack_stack, legal, // new states add_stack_stack, sub_stack_stack, mul_to_stack, idiv_const, }; for (prgm.func_def.instrs.items) |instr| { state: switch (State.start) { .start => switch (instr) { .mov => |m| if (m.src == .stack and m.dst == .stack) continue :state .mov_stack_stack else continue :state .legal, // new code starts here .add => |m| if (m.src == .stack and m.dst == .stack) continue :state .add_stack_stack else continue :state .legal, .sub => |m| if (m.src == .stack and m.dst == .stack) continue :state .sub_stack_stack else continue :state .legal, .mul => |m| if (m.dst == .stack) continue :state .mul_to_stack else continue :state .legal, .idiv => |o| if (o == .imm) continue :state .idiv_const else continue :state .legal, else => continue :state .legal, }, .mov_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mov.src, .{ .reg = .R10 }) }, .{ .mov = .init(.{ .reg = .R10 }, instr.mov.dst) }, }), // new code starts Here .add_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.add.src, .{ .reg = .R10 }) }, .{ .add = .init(.{ .reg = .R10 }, instr.add.dst) }, }), .sub_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.sub.src, .{ .reg = .R10 }) }, .{ .sub = .init(.{ .reg = .R10 }, instr.sub.dst) }, }), .mul_to_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mul.dst, .{ .reg = .R11 }) }, .{ .mul = .init(instr.mul.src, .{ .reg = .R11 }) }, .{ .mov = .init(.{ .reg = .R11 }, instr.mul.dst) }, }), .idiv_const => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.idiv, .{ .reg = .R11 }) }, .{ .idiv = .{ .reg = .R11 } }, }), .legal => try out.append(alloc, instr), } } }</code></pre> Lots of copying and pasting. Maybe when I refactor this I can make the State</code> enum carry payloads. The quicker I am out of assembly instructions fixup, however, the better. Assemblers should just be smarter. All codegen</code> tests. Pass, which here simply means it is failing when it should and is not hitting any panics in the passed codepaths. Always a plus. Filling up code emission would prove the pudding. Thanks to the already implemented boilerplate last chapter, this is also a nice and straightforward task. And, silly typos aside, everything works smoothly and all tests pass. Here is the sample C program in "codegen</code>" form, and final assembly form.^{3</a>} PROGRAM FUNCTION main allocate 16 mov imm 4 -> stack -4 mov stack -4 -> R11 mul imm 5 -> R11 mov R11 -> stack -4 mov imm 3 -> stack -8 mov stack -4 -> R10 add R10 -> stack -8 mov imm 4 -> stack -12 neg stack -12 mov stack -8 -> R10 mov R10 -> stack -16 mov stack -12 -> R10 add R10 -> stack -16 mov stack -16 -> AX ret</code></pre> .globl _main _main: pushq %rbp movq %rsp, %rbp subq $16, %rsp movl $4, -4(%rsp) movl -4(%rsp), %r11d imull $5, %r11d movl %r11d, -4(%rsp) movl $3, -8(%rsp) movl -4(%rsp), %r10d addl %r10d, -8(%rsp) movl $4, -12(%rsp) negl -12(%rsp) movl -8(%rsp), %r10d movl %r10d, -16(%rsp) movl -12(%rsp), %r10d addl %r10d, -16(%rsp) movl -16(%rsp), %eax movq %rbp, %rsp popq %rbp ret</code></pre> Extra Credit</h2> The Book has an extra credit section for doing bitwise operations. I am skipping extra credit this run. They are useful and interesting but also more churn in the following chapters. This is the last time you will hear me talk of it. Lessons Learned</h2> Implemented a macro</del> comptime</code> function.</li> Closures at home.</li> Positive reinforcement of previous design decisions.</li> </ol> Yet to see a compiler book explaining the shunting yard algorithm. ↩</a> </li> The Zig tokenizer does not have a peek</code> function. But the compiler just collects the entire token stream in an array before passing it to the parser. ↩</a> </li> Now is not the time for constant folding and optimizations. These come in Part 3 of the Book. ↩</a> </li> </ol> </section> Chapter 2: Unary 2025-05-14T00:00:00+00:00 So, with the first chapter</a> of Writing a C Compiler</a> behind us, it is time to start on the second chapter: Unary Operators. But first, a word from build.zig</code> land. Running the tests with zig build test</code></h2> The tests for this Book live in their own repository</a>, which I have dutifully cloned and run tests in. The process I followed to run the tests is fairly manual: navigate to the test directory.</li> run the command arch -x86_64 zsh</code> (which I have cleverly aliased to x86</code>). This runs a x86 version of zsh</code>.</li> run the tests.</li> </ol> When I was doing my Rust implementation</a>, I kept a terminal accessible with a global hotkey and I kept the inner shell running at all times. With build.zig</code>, I looked for a way to do that automatically. The Build</code> object, which is central to build.zig</code> has a nice addSystemCommand</code> method, which I could use to run my stuff in. Now remained the problem of how to run the tests in the x86 shell. My first thought was to have the command as arch -x86_64 test_compiler ..etc</code> (test_compiler</code> being the test script provided). While it worked, as in the tests for chapter 1 all ran fine, upon further reading I realized that the arch</code> actually chooses which binary to run from a macOS Universal Binary. It has no effect on python scripts, and the simple tests on chapter 1 worked fine either way. So I asked around, and some kind soul pointed me to the right thing to search for: Subshells. See, this might seem obvious if you've lived in Unix land all your life, but shells are applications too. You can pass command line arguments to them!! So, bash -c "echo foo"</code>, runs bash</code>, runs the command echo foo</code> in bash</code>, then exits. Perfect. Putting these two together, this was the added bit to build.zig</code>: const test_step = b.step("test", "Run tests"); // subshells. how do they work. const inner_command = try std.mem.join(b.allocator, " ", &.{ "../writing-a-c-compiler-tests/test_compiler", b.pathJoin(&.{ b.exe_dir, "paella" }), try std.mem.join( b.allocator, " ", b.args orelse &.{""}, ), }); // does this work like i think it does? const test_command = b.addSystemCommand( &.{ "arch", "-x86_64", "zsh", "-c", inner_command }, ); test_command.step.dependOn(b.getInstallStep()); test_step.dependOn(&test_command.step);</code></pre> I am unreasonably happy with this. Using regular Zig standard library tools like std.mem.join</code> allowed to join together the commands I am passing to the zsh</code> subshell, and voila!! Now the tests (say for chapter 1) can be ran "simply" with zig build test -- --chapter 1</code>. Great success! Lexer</h2> There are three tokens to add in this chapter: ~</code>, -</code>, and --</code>. The decrement operator is only being tokenized here to reject illegal C syntax like return --2</code>, but otherwise will not be implemented. So our Tag</code> enum grows1</a> to have these three tokens: tilde, hyphen</code>. As I am not planning to implement the book's extra credit this time around, I will just have the double hyphen lex into the invalid</code> token. This would reject lexing valid code like x--</code>, but I do not think this is going to be present into the test cases. We will see. The State</code> enum is a bit more interesting. It cannot accept two consecutive hyphens, so seeing a hyphen would put it into the hyphen</code> state, where if it sees another hyphen, it puts out Tag.invalid</code>, or Tag.hyphen</code> otherwise. So, if the state is at .start</code>, and the lexer encounters a '-'</code> character, it just switches gear to the .hyphen</code> state. Labelled switch</code> in action. Then in there, triage happens. The code is honestly copied, again, from the Zig compiler, with adjustments. state: switch (State.start) { .start => switch (self.buffer[self.index]) { '~' => { result.tag = .tilde; self.index += 1; }, '-' => continue :state .hyphen, // etc .hyphen => { self.index += 1; switch (self.buffer[self.index]) { '-' => result.tag = .invalid, else => result.tag = .hyphen, } }, // etc }</code></pre> This should be it. Time to put the new testing command into action. Ran 43 tests in 5.432s OK</code></pre> Both of them work. Time for AST and parsing. Parsing</h2> Changes to the AST are fairly minimal. There are two unary operations: negation, and complement. Instead of adding a separate UnaryOp</code> enum as the Book suggests, I am going to embed them right into Expr</code> as separate tags, updating the pretty printer while I am at it.2</a> pub const Expr = union(enum) { constant: u64, unop_negate: *Expr, unop_complement: *Expr, pub fn format( self: @This(), comptime _: []const u8, _: std.fmt.FormatOptions, writer: anytype, ) !void { switch (self) { .constant => |c| try writer.print("{d}", .{c}), .unop_negate => |e| try writer.print(" -{}", .{e}), .unop_complement => |e| try writer.print(" ~{}", .{e}), } } };</code></pre> Updating the parsing functions is a bit more involved, however. Now is the start of PEMDAS, and time to start our recursive descemt into madness.3</a> This is pretty much the same function provided in the book, except that the operation is chosen immediately. fn parse_expr( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Expr { const current = tokens.next() orelse return error.ExpectExpr; switch (current.tag) { .number_literal => { const lit = tokens.buffer[current.loc.start..current.loc.end]; const res = try std.fmt.parseInt(u64, lit, 10); // vvv this helper function just allocates and sets return try create(ast.Expr, alloc, .{ .constant = res }); }, .hyphen => { const inner_exp = try parse_expr(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_negate = inner_exp }); }, .tilde => { const inner_exp = try parse_expr(alloc, tokens); return try create(ast.Expr, alloc, .{ .unop_complement = inner_exp }); }, .l_paren => { const inner_exp = try parse_expr(alloc, tokens); try expect(.r_paren, tokens); return inner_exp; }, else => return error.ExpectExpr, } }</code></pre> Now the parsing is done, but to test a small annoyance needs to be fixed. In asm_gen.zig</code>, there is a switch over expressions, and it needs to handle the new modes, even tho it is not being called right now. This tedium and the need to repeat it for every step is partially why I stopped the Rust implementation. Adding a else => @panic("unimplemented"),</code> to that switch</code> smoothes things over for now. Ran 43 tests in 6.363s OK</code></pre> Yuppee. Last but not least, even though it is not needed for the first chapters, as the only identifier is main</code>, I figured I'd try some comptime</code> magic to improve my expectation experience. I changed expect</code> a little bit so it returns a string when I am asking for an identifier, but void</code> otherwise.4</a> fn expect( comptime expected: lexer.Token.Tag, tokens: *lexer.Tokenizer, ) !if (expected == .identifier) []const u8 else void { if (tokens.next()) |actual| { if (actual.tag != expected) return error.SyntaxError; if (expected == .identifier) return tokens.buffer[actual.loc.start..actual.loc.end]; } else return error.SyntaxError; }</code></pre> Intermediate Representation</h2> The Book uses its own version of Intermediate Representation, called TACKY</code>. It is a variation of a popular IR strategy called three-address code, hence the name. TACKY</code> uses its own AST, and it would slot between the program's AST and the assembly's AST. This is the first pass. So many strings. const std = @import("std"); pub const Prgm = struct { func_def: *FuncDef, }; pub const FuncDef = struct { name: []const u8, // for similar reasons to its use in assembly AST instrs: std.ArrayListUnmanaged(Inst), }; pub const Inst = union(enum) { ret: Value, // decided to splat the operator here, like discussed in chapter 1. unop_negate: Unary, unop_complement: Unary, // to avoid excessive typing pub const Unary = struct { src: Value, dst: Value, pub fn init(src: Value, dst: Value) @This() { return .{ .src = src, .dst = dst }; } }; }; pub const Value = union(enum) { constant: u64, variable: []const u8, };</code></pre> The goal of this IR is to disentangle nested expressions by separating them into what is, at least in IR form, separate operations. This means that for an expression like 1 + 2 * 3</code>5</a>, it is transformed into the following representation: tmp0 = 2 * 3 tmp1 = 1 + tmp0 return tmp1</code></pre> The Book provided expr_emit_ir</code>, with my splatted data structures, looks like the following: fn expr_emit_ir( alloc: std.mem.Allocator, expr: *ast.Expr, instrs: *std.ArrayListUnmanaged(ir.Instr), ) !ir.Value { switch (expr.*) { .constant => |c| return .{ .constant = c }, .unop_negate => |e| { const src = try expr_emit_ir(alloc, expr, instrs); const dst_name = try make_temporary(alloc, "neg"); const dst: ir.Value = .{ .variable = dst_name }; try instrs.append(alloc, .{ .unop_negate = .init(src, dst) }); return dst; }, .unop_complement => |e| { const src = try expr_emit_ir(alloc, expr, instrs); const dst_name = try make_temporary(alloc, "cml"); const dst: ir.Value = .{ .variable = dst_name }; try instrs.append(alloc, .{ .unop_complement = .init(src, dst) }); return dst; }, } }</code></pre> A bit repetitive, isn't it? So I put together the repeated logic into its own function, unary_helper</code>: fn unary_helper( alloc: std.mem.Allocator, expr: *ast.Expr, instrs: *std.ArrayListUnmanaged(ir.Instr), comptime prefix: []const u8, ) !struct { ir.Value, ir.Value } { const src = try expr_emit_ir(alloc, expr, instrs); const dst_name = try make_temporary(alloc, prefix); const dst: ir.Value = .{ .variable = dst_name }; return .{ src, dst }; }</code></pre> Nothing fancy. make_temporary</code> over here is just a function with an ever increasing static variable for creating always different variable names. However, I hit here the weirdest zig compile error I have seen, yet. error: unable to resolve inferred error set</code>. What gives? Turns out when two recursive functions call each other, both with an inferred error type, Zig throws in the towel. Swift would've never given up and compiled, eventually; Apparently I had to specify the error type myself. const helper_error = error{ MakeTemporary, // return error value of `make_temporary` UnaryHelper, };</code></pre> ... and changed unary_helper</code> to this: fn unary_helper( alloc: std.mem.Allocator, expr: *ast.Expr, instrs: *std.ArrayListUnmanaged(ir.Instr), comptime prefix: []const u8, ) helper_error!struct { ir.Value, ir.Value } { const src = expr_emit_ir(alloc, expr, instrs) catch return error.UnaryHelper; const dst_name = try make_temporary(alloc, prefix); const dst: ir.Value = .{ .variable = dst_name }; return .{ src, dst }; }</code></pre> And now it compiles. The rest of ir_gen.zig</code>, is rahter a straightforward one to one between the two trees, and rather boring to implement. Instead, I will show you make_temporary</code>. Zig has a weird way to define local static variables. fn make_temporary( alloc: std.mem.Allocator, comptime prefix: []const u8, ) helper_error![]const u8 { // zig static variables const coun = struct { var ter: usize = 0; }; const name = std.fmt.allocPrint( alloc, if (prefix.len == 0) "temp" else prefix ++ ".{}", .{coun.ter}, ) catch return error.MakeTemporary; coun.ter += 1; return name; }</code></pre> First of all, non-Atomic is fine. All this is single threaded anyway. Second of all, any variable directly declared in a struct</code> scope is static. So if you need a local static variable you define a struct</code> with a .. variable. This is exactly the same semantics as declaring a global variable (or at file scope) except the scope here is a lot smaller.6</a> Mind bending a bit, but makes sense. To save on pretty printing this time, I figured to try std.json.stringify</code> to get the IR output. This is the C file I am working on. So small and non-threatening. int main(void) { return ~-3; }</code></pre> And this is the IR output: { "func_def": { "name": "main", "instrs": { "items": [ { "unop_negate": { "src": { "constant": 3 }, "dst": { "variable": "neg.0" } } }, { "unop_complement": { "src": { "variable": "neg.0" }, "dst": { "variable": "cml.1" } } }, { "ret": { "variable": "cml.1" } } ], "capacity": 5 } } }</code></pre> Very longwinded, with some useless info. Why do I need to know the ArrayListUnmanaged</code> capacity? Time to work on pretty printing and allocation bookkeeping. Not very interesting, to be honest, so I will skip writing about it. Except for string interning. that is interesting. String Interning</h2> String interning is having a single storage for strings that are used throught the program. Keeping the source around is not enough, because of the new strings for the IR's temporary variables. Giving ownership of the strings to ir.Value</code> is fine, except it is really easy to double free the strings when cleaning up. In fact, I did that when I was trying to not implement the string interner, and Zig's DebugAllocator</code> dutifully caught it. The basic idea behind the interner is to have all strings owned by one repository that you can free at once at the end. One can use an arena, but the interner's structure is more efficient because it also avoids duplication of strings. It can be backed by a hashmap, so it can also attach information to each string. All in all it is a useful memory and resource management technique. The friendly people at the Zig discord, especially InKryption</a> not only helped me out understand the data idea, bu actually provided me with a complete implementation. It makes use of Zig's, hoenstly weird, sentinel-terminated arrays: a generalization of C's null-terminated strings. The idea is as follows: a regular slice (say the []comst u8</code> we use and love) stores its length within it. It knows what length it is at all times. So if you read from it or iterate on it, it stops dutifully at the length it knows. A null-terminated string, however, would be [:0]const u8</code>. When you read from it or iterate on it, it keeps going until it hits the 0</code>. (Obviously you can use any sentinel value but the 0</code> is very convenient for strings.) So the String Interner keeps as its backing data a regular ArrayListUnmanaged(u8)</code> that I have been using before. Say it is called bytes</code>.) But for any new string in the application, it appends it to the ArrayList then adds 0</code> byte at the end. And I keep its starting index around. When I want to use it, I take slice like so bytes[starting_idx..:0]</code> and voila .. I have a slice of my string. The remaining pieces of the puzzle is two items in the Zig standard library: StringIndexContext</code> and StringIndexAdapter</code>. They're specifically designed for this specific use case. Who knew? A Context</code> in Zig's hash maps is a simple enough idea. It just defines the eql</code> and hash</code> functions said hash map will use, with an optional backing data structe. An Adapter</code> is used to override these two functions for speciifc operations. The StringIndex</code> duo have slightly different implementation of the eql</code> function, and if you're curious you can check them out in the standard library's code.7</a> Without further ado, here is the full StringInterner</code> provided thankfully by InKryption. I honestly had to read multiple times to wrap my head around what's going on, so I commented some parts of it. const std = @import("std"); const StringInterner = struct { // state bytes: std.ArrayListUnmanaged(u8), map: std.HashMapUnmanaged( Id, void, std.hash_map.StringIndexContext, std.hash_map.default_max_load_percentage, ), // management pub const init: StringInterner = .{ .bytes = .empty, .map = .empty, }; pub fn deinit(self: *StringInterner, allocator: std.mem.Allocator) void { self.bytes.deinit(allocator); self.map.deinit(allocator); } // type alias. store this in stuff. pub const Idx = u32; // checks if a string is in or not. returns ID pub fn getIdx( self: *const StringInterner, string: []const u8, ) ?Idx { return self.map.getKeyAdapted(string, self.hmAdapter()); } // get string from id pub fn getString( self: *const StringInterner, idx: Idx, ) ?[:0]const u8 { if (!self.map.containsContext(idx, self.hmCtx())) return null; // cast is necessary for type inference reasons. const slice_sentinel: [:0]const u8 = @ptrCast(self.bytes.items[idx..]); return std.mem.sliceTo(slice_sentinel, 0); } // the insert function. returns aither an existing idx or a new idx. pub fn getOrPut( self: *StringInterner, allocator: std.mem.Allocator, string: []const u8, ) std.mem.Allocator.Error!Idx { // reserves capacity in both the backing array and the map try self.bytes.ensureUnusedCapacity(allocator, string.len + 1); try self.map.ensureUnusedCapacityContext(allocator, 1, self.hmCtx()); // getOrPut returns a reference to the key and the value. If it existed, they're // valid pointers. If not, their contents are undefined waiting for you to fill them. const gop = self.map.getOrPutAssumeCapacityAdapted(string, self.hmAdapter()); gop.value_ptr.* = {}; // this is void, but it can be anything!! note void is zero-width if (gop.found_existing) return gop.key_ptr.*; // unlikelt to hit that. the source files are tiny. if (self.bytes.items.len > std.math.maxInt(Idx)) return error.OutOfMemory; const new_idx: Idx = @intCast(self.bytes.items.len); // indertion happens here self.bytes.appendSliceAssumeCapacity(string); // don't forget to append the null byte !! self.bytes.appendAssumeCapacity(0); // update the map through the pointer. gop.key_ptr.* = new_idx; return new_idx; } // helper functions to avoid a self referential struct fn hmCtx(self: *const StringInterner) std.hash_map.StringIndexContext { return .{ .bytes = &self.bytes }; } fn hmAdapter(self: *const StringInterner) std.hash_map.StringIndexAdapter { return .{ .bytes = &self.bytes }; } };</code></pre> I will currently use this structure as is, only changing the names to be snake_case</code> as I cannot read camelCase</code>. Later on I can use it to attach data to the strings. Fairly useful. My first go at this was to replace every string with StringInterner.Idx</code>. Just make sure to pass a pointer to the interner when constructing these structures. pub const Prgm = struct { func_def: *FuncDef, }; pub const FuncDef = struct { name: utils.StringInterner.Idx, instrs: std.ArrayListUnmanaged(Instr), }; // and so on.</code></pre> While this works and compiles fine, it turned out to be problematic when pretty printing. Idx</code> is just a u32</code> and stores no reference to the interner. And the format</code> functions do not have an allocator or a user-space reference to use. So instead, I am changing the type to [:0]const u8</code>, then storing slices from the interner rather than indices. Then adjust our ir_gen</code> functions to pass a pointer to the interner through all of them. That or creating a makeshift Ctx</code> type. Time will tell. For an example, now, this the unary_helper</code> mentioned earlier and the new make_temporary</code>, with the new structure threaded in. fn unary_helper( alloc: std.mem.Allocator, interner: *utils.StringInterner, expr: *ast.Expr, instrs: *std.ArrayListUnmanaged(ir.Instr), comptime prefix: []const u8, ) helper_error!struct { ir.Value, ir.Value } { const src = expr_emit_ir(alloc, interner, expr, instrs) catch return error.UnaryHelper; const dst_name = try make_temporary(alloc, interner, prefix); const dst: ir.Value = .{ .variable = dst_name }; return .{ src, dst }; } fn make_temporary( alloc: std.mem.Allocator, interner: *utils.StringInterner, comptime prefix: []const u8, ) helper_error![:0]const u8 { const static = struct { var counter: usize = 0; }; const name = std.fmt.allocPrint( alloc, if (prefix.len == 0) "temp" else prefix ++ ".{}", .{static.counter}, ) catch return error.MakeTemporary; defer alloc.free(name); const name_idx = interner.get_or_put(alloc, name) catch return error.MakeTemporary; const name_res = interner.get_string(name_idx).?; // <- unwrap optional static.counter += 1; return name_res; }</code></pre> So this all compiles and runs and passes tests just fine. All is left is implementing the pretty printing myself, as the json</code> module seemed to break for reasons I do not understand (related to the interner).8</a> The technique to do so was discussed in the Chapter 1 article, so no point in repeating it here. It is time to move on to the next section. Codegen</h2> The first task here is almost invariably to update the assembly AST to include more instructions. Again deviating from the book, I will splat the unary operator into the instructions. This will totally bite me later. The updated parts of the new AST look as follows: pub const Inst = union(enum) { mov: Mov, ret: void, // unary operations neg: Operand, not: Operand, allocate_stack: i64, // same type as `.stack` below. const Mov = struct { src: Operand, dst: Operand, pub fn init(src: Operand, dst: Operand) @This() { return .{ .src = src, .dst = dst }; } }; }; pub const Operand = union(enum) { imm: u64, reg: Register, stack: i64, pseudo: [:0]const u8, // these get replaced pub const Register = enum { AX, R10 }; };</code></pre> Oh there is a string there. Our string interner will come in handy. Another note: I am not perfectly happy with the way the book represents register. Because later in the book, when long</code> and pointers are implemented, different names have to be chosen for each register, and I had to encode that info in the Register</code> type itself, which led to weird amount of code duplications. I have not looked at how the reference OCaml implementation</a> does it. But I will cross that bridge when I come to it. So far all registers are 32 bits wide. Now asm_gen.zig</code> is reworked to take the IR AST as input instead. This is almost a complete rewrite except for the first, trivial, Prgm</code> transformation. Complicates the matter a bit is that IR instructions and Assembly instructions do not map 1 to 1, but each IR instruction can be one or more Assembly instructions. So I have to return a slice. Manual memory management strikes again. It is easier to start from the bottom up. This is IR values to Assembly operands. No allocator needed! Just copy the string's reference as is. fn value_to_asm( value: ir.Value, ) assembly.Operand { switch (value) { .constant => |c| return .{ .imm = c }, .variable => |v| return .{ .pseudo = v }, } }</code></pre> And this is more involved Instr</code> porter: fn instr_to_asm( alloc: std.mem.Allocator, instr: ir.Instr, ) ![]assembly.Instr { switch (instr) { .ret => |v| { const src = value_to_asm(v); const ret = try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init(src, .{ .reg = .AX }) }, .ret, }); return ret; }, .unop_complement => |u| { const src = value_to_asm(u.src); const dst = value_to_asm(u.dst); const ret = try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init(src, dst) }, .{ .not = dst }, }); return ret; }, .unop_negate => |u| { const src = value_to_asm(u.src); const dst = value_to_asm(u.dst); const ret = try alloc.dupe(assembly.Instr, &.{ .{ .mov = .init(src, dst) }, .{ .neg = dst }, }); return ret; }, } }</code></pre> Not very DRY, and a lot is going on. Certainly a lot of (what is going to be very temporary) allocations. Returning pointers to stack space is a rookie mistake. (Thanks Rust!) The classic solution to a lot of temporary allocations in similar programs is to use an arena allocator, and then free it all at once when the function is done. This would be set up from func_def_to_asm</code>. So here is the first attempt at it: fn func_def_to_asm( alloc: std.mem.Allocator, func_def: ir.FuncDef, ) !assembly.FuncDef { var arena_allocator = std.heap.ArenaAllocator.init(alloc); defer arena_allocator.deinit(); const arena = arena_allocator.allocator(); var instrs = std.ArrayListUnmanaged(assembly.Instr).empty; for (func_def.instrs.items) |instr| { // note the different allocators for each function. const ret = try instr_to_asm(arena, instr); try instrs.appendSlice(alloc, ret); } return .{ .name = func_def.name, .instrs = instrs, }; }</code></pre> And finally Prgm</code> is trivial. pub fn prgm_to_asm( alloc: std.mem.Allocator, prgm: ir.Prgm, ) !assembly.Prgm { const func_def = try utils.create( assembly.FuncDef, alloc, try func_def_to_asm(alloc, prgm.func_def.*), ); return .{ .func_def = func_def }; }</code></pre> And the proof of the pudding is in the eating. But not before filling in all the @panic("todo")</code>s and cleaning up the allocated memory. The code so far works and produces the expected result on the previously mentioned tiny C program, but codegen is not over yet. Fixing Up Instructions</h2> There are two final tasks before actually generating the assembly. First is to replace all them .pseudo</code> operands with .stack</code> operands. The Book used the pseudo</code> operands as a placeholder, as the compiler does not have the stack info at hand yet. It needs to count the amount of intermediate, temporary variables, created during IR generation. The second is to rewrite illegal assembly instructions, such as moving from a stack to a stack, into legal ones. These are essentially additional compiler passes, and I am loath to do the same thing I did in my Rust implementation and stuff everything in one file. So a new directory is born, titled asm_passes</code>, with two new Zig files in toe. If you are keeping track, this is my current folder structure. ./src ├── asm_gen.zig ├── asm_passes │ ├── fixup.zig │ └── pseudos.zig ├── assembly.zig ├── ast.zig ├── ir_gen.zig ├── ir.zig ├── lexer.zig ├── main.zig ├── parser.zig └── utils.zig</code></pre> Most of them are self descriptive. utils</code> is where the string interner definition lives and, currently, one create</code> function, that allocates a value on the heap. Pseudoregisters</h3> This simply goes over the instrs</code> array stored in FuncDef</code> to change up the arrays. The interesting challenge here is maintaining a map from identifiers to stack offsets. Storing the strings as map keys is horribly wasteful, so the string interner's Idx</code> type is used instead, so it passes in. I probably made this a lot harder for myself. I tried to implement my own hash map adapter type, then had a struggle with Zig's syntax for references. Eventually, Good prevailed and the code was compiled. The building block is this "simple" function, with a minimal amount of parameters. getOrPut</code> is a standard library function that almost does what it says on the tin, but gives you pointers to place the key and value yourself as you please. I opted to use a managed (not Unmanaged</code>) hash map here to spare myself an allocator parameter. I could shave off strings</code> too if I manage to figure out how Context</code> works.9</a> fn pseudo_to_stack( op: assembly.Operand, strings: *utils.StringInterner, map: *std.AutoHashMap( utils.StringInterner.Idx, assembly.Operand.StackDepth, ), ) !assembly.Operand { const static = struct { var counter: assembly.Operand.StackDepth = 0; }; switch (op) { .reg, .imm, .stack => return op, .pseudo => |name| { const idx = strings.get_idx(name).?; const gop = try map.getOrPut(idx); if (gop.found_existing) { return .{ .stack = gop.value_ptr.* }; } else { static.counter -= 4; gop.key_ptr.* = idx; gop.value_ptr.* = static.counter; return .{ .stack = gop.value_ptr.* }; } }, } }</code></pre> The public facing function that runs on assembly.Prgm</code> comes next. The &</code> and *</code> and .*</code> sprinkled over the loop's variables took a lot of trial and error and Zig discord help to get right. But the shared payload between .neg</code> and .not</code> does not to be duplicated. pub fn replace_pseudos( alloc: std.mem.Allocator, strings: *utils.StringInterner, prgm: *assembly.Prgm, ) !void { var pseudo_map: std.AutoHashMap( utils.StringInterner.Idx, assembly.Operand.StackDepth, ) = .init(alloc); defer pseudo_map.deinit(); for (prgm.func_def.instrs.items) |*instr| { // ^ no address operator as it is already a slice switch (instr.*) { .mov => |*m| { // ^ this was the tricky one to find const src = try pseudo_to_stack(m.src, strings, &pseudo_map); const dst = try pseudo_to_stack(m.dst, strings, &pseudo_map); m.* = .init(src, dst); }, .neg, .not => |*v| v.* = try pseudo_to_stack(v.*, strings, &pseudo_map), .ret, .allocate_stack => {}, } } }</code></pre> It works. Thankfully. This little bit made me tear my hair out.10</a> As a small comparison, this is the C file in question, followed by the pretty printed codegen before and after replacing pseudoregisters. // to save you a scroll int main(void) { return ~-3; }</code></pre>#before PROGRAM FUNCTION main mov imm 3 -> pseudo neg.0 neg pseudo neg.0 mov pseudo neg.0 -> pseudo cml.1 not pseudo cml.1 mov pseudo cml.1 -> AX ret</code></pre># afer PROGRAM FUNCTION main mov imm 3 -> stack -4 neg stack -4 mov stack -4 -> stack -8 not stack -8 mov stack -8 -> AX ret</code></pre> This is a horribly inefficient compiler, but it clearly works as intended. Optimization comes later in Part 3 of the book. Fixing Instructions</h3> The first fixup is to insert the .allocate_stack</code> instruction with the depth of the stack. But this is not being returned anywhere! So a slight rejiggering of pseudo_to_stack</code> to use a passed in counter instead of a local static one, and having replace_pseudos</code> return the stack depth, would solve the problem. Simple, if annoying, code change. Then adding the first instruction this way becomes fairly simple. Bad performance for this one step but I am not paying too much attention to that. const depth = try pass_pseudo.replace_pseudos(alloc, strings, prgm_asm); try prgm_asm.func_def.instrs.insert( alloc, 0, .{ .allocate_stack = @intCast(@abs(depth)) }, // ^ this retunrs an unsigned int. ); // next step gets called here</code></pre> The other, more involved step is fixing up illegal instructions. If you know anything about x86 assembly, you would know that you cannot make a mov</code> instruction between two stack places. There are reasons for that that I do not particularly care about. Why cannot the Book compile to a saner assembly language like WASM? I am not even on a x86 machine! Thankfully, this is the only thing to deal with in this chapter. Here is my first pass at the code. pub fn fixup_instrs( alloc: std.mem.Allocator, prgm: *assembly.Prgm, ) !void { var out: std.ArrayListUnmanaged(assembly.Instr) = try .initCapacity( alloc, prgm.func_def.instrs.capacity, ); defer { std.mem.swap( std.ArrayListUnmanaged(assembly.Instr), &out, &prgm.func_def.instrs, ); out.deinit(alloc); } for (prgm.func_def.instrs.items) |instr| { switch (instr) { .mov => |m| switch (m.src) { .stack => switch (m.dst) { .stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(m.src, .{ .reg = .R10 }) }, .{ .mov = .init(.{ .reg = .R10 }, m.dst) }, }), else => try out.append(alloc, instr), }, else => try out.append(alloc, instr), }, else => try out.append(alloc, instr), } } }</code></pre> So much rightwards drift. It would perhaps be more readable to express it as a state machine, like the tokenizer works. I will make sure this works as intended first then try to streamline it, as this is untenable. Time to run the codegen tests. Ran 43 tests in 5.153s OK</code></pre>Code Emission</h2> Before we get into the state machine, it is better to be done with code emission. It is a fairly simple and mechanical exercise as it just requires updating the printing code, but it could expose pretty much any bugs in the program. and ... Ran 43 tests in 5.147s FAILED (errors=19)</code></pre> Oh come on. This turns out to be a combination of some commented out code when I was updating the compiler driver to include IR, and dumb spelling mistakes I did in this fairly simple and mechanical task, such as writing &</code> instead of %</code>. Simple mistakes, really. All tests pass now: Ran 43 tests in 14.731s OK</code></pre> State Machine</h2> Following from the tokenizer's code, I defined a simple State</code> enum: const State = enum { start, mov_stack_stack, legal, };</code></pre> And the main switch</code> inside the loop turns into a labelled switch</code> over State</code>, in which each state has its own logic, but any legal instruction just funnels to .legal</code>. To be honest, writing this was easier than I expected it to be. It even worked on the first iteration, after I fixed the variable names. This is the second. Jury's up whether the state machine is more readable than the nested switch</code>es, but I think I will stick with it for the time being. It passes all tests. for (prgm.func_def.instrs.items) |instr| { state: switch (State.start) { .start => switch (instr) { .mov => |m| if (m.src == .stack and m.dst == .stack) continue :state .mov_stack_stack else continue :state .legal, else => continue :state .legal, }, .mov_stack_stack => try out.appendSlice(alloc, &.{ .{ .mov = .init(instr.mov.src, .{ .reg = .R10 }) }, .{ .mov = .init(.{ .reg = .R10 }, instr.mov.dst) }, }), .legal => try out.append(alloc, instr), } }</code></pre> Just for comparison's sake, this is the equivalent code in my Rust implementation</a>. for instr in std::mem::take(&mut self.instrs) { match instr { Instr::Mov { src: src @ Operand::Stack(_), dst: dst @ Operand::Stack(_), } => out.extend([ Instr::Mov { src, dst: Operand::Reg(Register::R10), }, Instr::Mov { src: Operand::Reg(Register::R10), dst, }, ]), other => out.push(other), } }</code></pre> Pattern matching is the best. And this concludes Chapter 2. Lessons Learned</h2> Building a cool state machine from zero.</li> More manual memory management magic tricks. (Couldn't push the alliteration further.)</li> Subshells, and a more automated testing process.</li> First taste of comptime</code>. ULTIMATE POWER.</li> </ol> Maybe next time I will compile the tests to be always in ReleaseSafe</code> instead of Debug</code> mode. Would certainly make things go more smoothly. Need to figure that one out. I refactored the code a bit after chapter 1 and added proper support for the identifier</code> token. It is a trivial change so no point in detailing it further, except that the new expect</code> is cooler than before. ↩</a> </li> There is an argument to be made for replacing my custom pretty printer with standard library's json</code> serializer. The result is more verbose and has more vertical space and redundant info, but on the plus side, it requires zero maintenance. ↩</a> </li> Sounds good for a book title or something, doesn't it? ↩</a> </li> Inspired by matklad</a>. ↩</a> </li> Binary operations are in the next chapter. ↩</a> </li> Helps to keep in mind that every file in Zig is a struct, too. ↩</a> </li> A useful read, which was educational for me on Contexts and Adaptors (in current Zig version, at least), are the Zig issues #21917</a> and #23872</a>, especially mulgg</a>'s comments. On the zig discord, mlugg gave me this useful summary: "they're quite similar. a context is essentially an adapter but for the 'real' key type; a context tells you how to hash/eql the actual comptime K: T</code>, whereas an adapter allows you to insert different key types as long as you can implement compatible hash/eql on them". ↩</a> </li> The crash disappeared when I removed the needless reference to the interner from the ir.Prgm</code> struct, but I hit a new problem. The null terminated strings were printing as "main\u0000"</code>. I am not sure if this is a standard library bug or a bug in my code. Definitely a case of debugging my language knowledge. ↩</a> </li> I ended up replacing the hashmap here with std.StringHashMap</code> which does exactly what I want and allowed me to avoid passing the string interner. My mistake was assuming, for some reason, that StringHashMap</code> stores the strings inernally, but it just stores the slices already backed by the interner. ↩</a> </li> I was debugging my language knowledge, it seems. ↩</a> </li> </ol> </section> Chapter 1: Intro 2025-05-10T00:00:00+00:00 In a constant drive to spend, or waste, time while I am looking for work, I got and worked through Nora Sandler's Writing a C Compiler</a> in Rust</a>. It is an excellent introduction to things like intermediate representation and x86_64</code> assembly. About halfway through the book, however, I lost focus. Continuing to have nothing particularly productive to do, I figured that I could try following the book again in a different language: either Swift or Zig. I do want to learn Swift eventually as writing Apps is probably more useful than writing operating systems, but the tooling leaves something to be desired. So Zig it is, for the time being. Zig</a> is a very opinionated "C replacement". It has a lot of weird and arbitrary design decisions: no typed integer literals (@as(u8, 1)</code> instead of 1u8</code>) for example. But it also has a lot of nice ideas and it is being posited as the hottest thing since sliced bread, so why not Zig. Swift can happen later. Zig, and the Zig Version Manager</a> were already on my machine from previous flirtations with the language. 0.14.0</code> is the latest zig "release", so that is what I am using. I decided to call this project paella</code>. I like paella. So without further ado, let's go. build.zig</code></h2> Zig is not being marketed, as it were, as a language. It is being marketed as a build system that just happens to have a language attached. zig init</code> in the terminal creates a template project with full explanations of the basic building blocks. Editing it to what I need, this is where I landed: // build.zig const std = @import("std"); pub fn build(b: *std.Build) void { // entry point const exe_mod = b.createModule(.{ .root_source_file = b.path("src/main.zig"), // this is the target for the Book and my machine. .target = b.resolveTargetQuery(.{ .cpu_arch = .x86_64, .os_tag = .macos, }), // choose optimization based on --release .optimize = switch (b.release_mode) { .off => .Debug, else => .ReleaseFast, } }); const exe = b.addExecutable(.{ .name = "paella", .root_module = exe_mod, }); b.installArtifact(exe); // the rest is boilerplate for `zig build run` and `zig build test` // ... }</code></pre> I did not include all the boilerplate that would allow me to use zig build run</code> instead of zig run main src/main.zig</code>. Not much better, just nicer. Also a build system currently would let me add dependencies easier later. build.zig.zon</code> is whatever the initial command spit up, tbh. But it is where dependencies are to be listed later on. The test suite and the book expect everything to use the x86_64</code> CPU architecture, which is a little inconvenient in macOS due to it being on aarm64</code>. This is howver well explained in the book and the environment set up for it is already done when doing the Book previously in Rust. The Compiler Driver</h2> The next step is what is called in the book the Compiler Driver, or rather the framework which surrounds the compiler bits the Book is actually about. The Driver does several things: Takes a file path and an optional phase flag as arguments.</li> Calls the system's compiler, gcc</code> to preprocess the C source file at said path, (so no macros or comments or compiler flags or whatever),</li> Calls the to-be-implemented compiler to transform the preprocessed file on disk into assembly, or stop at the requested compilation phase.</li> And finally, calls gcc</code> again to assemble the assembly file into machine code.</li> </ol> The reason I am actually spelling it out, even though it is spelt out in the Book, is that it got me really confused at first. I thought the Driver was a separate thing that we will get to later. But no, it was pretty much step zero. Argument Parsing</h3> The simple argument parsing needed doesnt warrant getting a whole library in. The program will be called by the test suite and is not really meant for public consumption. So the bare minimum it is. (The returned errors are for my benefit.) const Args = struct { path: [:0]const u8, // a string in zigland mode: Mode, }; pub const Mode = enum { lex, parse, codegen, compile, assembly }; fn parse_args() !Args { var args = std.process.args(); _ = args.skip(); const path = args.next() orelse return error.PathNotFound; const mode: Mode = if (args.next()) |arg| std.meta.stringToEnum(Mode, arg[2..]) orelse return error.UnrecognizedFlag else .compile; return .{ .path = path, .mode = mode }; }</code></pre> This code snippet makes use of some of the higher level Zig features, including two ways to unwrap optional values. const optional_int: ?i32 = 5; // ?i32 is an optional i32 // first one const If_is_an_expression: i32 = if (optional_int) |int| // `int` is in scope here and we can do things int else 7; // second one // unwraps the result or returns a value. const definite_int: i32 = optional_int orelse 7; // the two are exactly the same.</code></pre>Calling Commands</h3> To run the preprocessor before compiling and the assembler after compiling, one needs to make use of std.process.Child</code>. The quickest and least painful way to do so is Child.run()</code>. However, this by default allocates, and returns, stdout</code> and stderr</code>. If the intent is to forward them to the parent process (which it is), the easiest way is this: var child = std.process.Child.init( &.{ "echo", "something" }, allocator, // zig is all about passing allocators ); // change options of `child` here: for example // child.stdout_behavior = .Ignore; // term is the returned result of said process. const term = try child.spawnAndWait();</code></pre>Manipulating File Extensions</h3> However, before that, let's get started at the much more complex problem of changing a file's extension.^{1</a> Zig's standard library provides a few ways to get the components, as it were, from a file path. They live in the std.fs.path</code> and std.mem</code> namespaces. After a bunch of trials and errors, I came up with the folliwng.} const path = "foo/bar.txt"; const stem = try std.fs.path.join( alloc, &.{ std.fs.path.dirname(path) orelse "", std.fs.path.stem(path), }, ); defer alloc.free(stem); const new_path = try std.mem.join( alloc, ".", &.{ stem, "TXT", }, ); defer alloc.free(new_path); std.debug.assert(std.mem.eql(u8, new_path, "foo/bar.TXT"));</code></pre>Finally!(?)</h3> Putting two and two together, this is the first draft of the compiler driver. I am not even sure if it compiles or if I made any dumb mistakes, because the hole in the middle is rather large and it needs filling first. const std = @import("std"); pub fn main() !void { var debug_allocator = std.heap.DebugAllocator(.{}).init; defer _ = debug_allocator.deinit(); // leak detection in Debug const gpa = debug_allocator.allocator(); const args = try parse_args(); try run(gpa, args); } pub fn run( alloc: std.mem.Allocator, args: Args, ) !void { const input_path = args.path; const pp_out, const asm_out, const exe = try get_output_paths(alloc, input_path); defer { // yay for manual memory management alloc.free(pp_out); alloc.free(asm_out); alloc.free(exe); } { // preprocessor var child = std.process.Child.init( &.{ "gcc", "-E", "-P", input_path, "-o", pp_out }, alloc, ); const term = try child.spawnAndWait(); if (!std.meta.eql(term, .{ .Exited = 0 })) return error.PreprocessorFail; } { // compiler _ = args.mode; // mode controls compilation here // todo // take from path `pp_out` output to path `asm_out` } { // assembler var child = std.process.Child.init( &.{ "gcc", asm_out, "-o", exe }, alloc, ); defer std.fs.cwd().deleteFile(asm_out) catch {}; // cleanup const term = try child.spawnAndWait(); if (!std.meta.eql(term, .{ .Exited = 0 })) return error.AssemblerFail; } } pub const Args = struct { path: [:0]const u8, mode: Mode, }; pub const Mode = enum { lex, parse, codegen, compile, // default assembly, // unused by test script - useful for debugging }; pub fn parse_args() !Args { var args = std.process.args(); _ = args.skip(); const path = args.next() orelse return error.PathNotFound; const mode: Mode = if (args.next()) |arg| std.meta.stringToEnum(Mode, arg[2..]) orelse return error.UnrecognizedFlag else .compile; return .{ .path = path, .mode = mode }; } fn get_output_paths( alloc: std.mem.Allocator, input_path: []const u8, ) !struct { []const u8, []const u8, []const u8, } { const exe = try std.fs.path.join( alloc, &.{ std.fs.path.dirname(input_path) orelse "", std.fs.path.stem(input_path), }, ); errdefer alloc.free(exe); const pp = try std.mem.join( alloc, ".", &.{ exe, "i" }, ); errdefer alloc.free(pp); // keywords and arbitrary strings become identifiers in zig like so const @"asm" = try std.mem.join( alloc, ".", &.{ exe, "s" }, ); return .{ pp, @"asm", exe }; }</code></pre> Lexer</h2> The Book takes things very gradually. So in the first chapter it just focuses on build the bare minimum of the compiler. The C program to be compiled in that chapter is this: int main (void) { return 0; }</code></pre> No, seriously, that's it. And the preprocessor takes care of pesky things like comments. The lexer is absurdly simple, for now. In Rust, I did all the lexing ahd parsing with nom</code>, the lovely parser combinator library. However, here things should be done the old-fashioned, imperative way. That's why the 0.14.0</code> Zig release introduced a new feature: labelled switch</code>!2</a> const State = enum { a, b, c, d }; loop: switch (State.a) { .a => { std.debug.print("reached .a\n", .{}); // fallthrough at home continue :loop .b; }, .b => { std.debug.print("reached .b\n", .{}); if (today == .monday) continue :loop .c else continue :loop .d; }, .c => std.debug.print("reached .c\n", .{}), .d => std.debug.print("reached .d\n", .{}), }</code></pre> The biggest, clearest example of using it is in the Zig compiler itself</a>. It is a cool design so I decided to just copy the design as is.3</a>. There are a few moving parts, so I added a bunch of comments to explain. // lexer.zig const std = @import("std"); // each Token carries a tag and a span (called `loc` here.) pub const Token = struct { tag: Tag, loc: Loc, // location within the source file pub const Loc = struct { start: usize, end: usize, }; // keywords identified here with their corresponding tags. pub const keywords = std.StaticStringMap(Tag).initComptime(.{ .{ "return", .keyword_return }, .{ "int", .type_int }, .{ "void", .keyword_void }, .{ "main", .keyword_main }, }); // helper function for the above pub fn getKeyword(bytes: []const u8) ?Tag { return keywords.get(bytes); } // the tag collection pub const Tag = enum { // punctuation l_paren, r_paren, l_brace, r_brace, semicolon, number_literal, // keywords type_int, keyword_void, keyword_return, keyword_main, // identifiers happen later // helpers identifier, // useful for state for now invalid, // this and the one below it are unused for now, but maybe useful later? pub fn lexeme(tag: Tag) ?[]const u8 { return switch (tag) { .invalid, .number_literal, .identifier, => null, .l_paren => "(", .r_paren => ")", .l_brace => "{", .r_brace => "}", .semicolon => ";", .type_int => "int", .keyword_void => "void", .keyword_return => "return", .keyword_main => "main", }; } pub fn symbol(tag: Tag) []const u8 { return tag.lexeme() orelse switch (tag) { .invalid => "invalid token", .identifier => "an identifier", .number_literal => "a number literal", else => unreachable, }; } }; }; // the lexer state machine pub const Tokenizer = struct { // pointer to source buffer: [:0]const u8, // where in source we are index: usize = 0, pub fn init(buffer: [:0]const u8) Tokenizer { return .{ .buffer = buffer }; } // states. this will grow in complexity with more tokens. const State = enum { start, identifier, int, }; // the loop itself pub fn next(self: *Tokenizer) ?Token { // the eventually returned value. var result: Token = .{ .tag = undefined, .loc = .{ .start = self.index, .end = undefined, }, }; state: switch (State.start) { // the starting state for every new token. .start => switch (self.buffer[self.index]) { 0 => { // nullbyte if (self.index == self.buffer.len) { return null; // eof } else { result.tag = .invalid; } }, ' ', '\n', '\t', '\r' => { // whitespace self.index += 1; // advance cursor result.loc.start = self.index; continue :state .start; // and restart }, 'a'...'z', 'A'...'Z', '_' => { result.tag = .identifier; continue :state .identifier; // move to identifier state }, '0'...'9' => { result.tag = .number_literal; self.index += 1; continue :state .int; // move to integer state }, // all of the following is self-explanatory really '(' => { result.tag = .l_paren; self.index += 1; }, ')' => { result.tag = .r_paren; self.index += 1; }, ';' => { result.tag = .semicolon; self.index += 1; }, '{' => { result.tag = .l_brace; self.index += 1; }, '}' => { result.tag = .r_brace; self.index += 1; }, else => result.tag = .invalid, }, .identifier => { self.index += 1; switch (self.buffer[self.index]) { // keep going until ... 'a'...'z', 'A'...'Z', '_', '0'...'9' => continue :state .identifier, // .. something other than those happens else => { const ident = self.buffer[result.loc.start..self.index]; // check if it s a keyword if (Token.getKeyword(ident)) |tag| { result.tag = tag; } }, } }, .int => switch (self.buffer[self.index]) { '0'...'9' => { self.index += 1; continue :state .int; }, // integers not allowed to have letters after them, for now. 'a'...'z', 'A'...'Z' => result.tag = .invalid, else => {}, }, } // tidy up. result.loc.end = self.index; return result; } };</code></pre> It is a bit cut down from the original behemoth, but this will grow in complexity as I advance through the book. The corresponding file in my Rust implementation eventually became full of macros and slowed rust-analyzer to a crawl. The stub in main.zig</code> is filled as follows: // main.zig { // compiler // the use of the more complex function here is to specify the sentinel null terminator ... const src = try std.fs.cwd().readFileAllocOptions( alloc, pp_out, std.math.maxInt(usize), null, @alignOf(u8), 0, // .. this null terminator.here. The rest are defaults. ); try std.fs.cwd().deleteFile(pp_out); // cleanup defer alloc.free(src); var tokenizer = lexer.Tokenizer.init(src); while (tokenizer.next()) |token| { // just to see the results for now. std.debug.print("{?}: {s}\n", .{ token.tag, src[token.loc.start..token.loc.end], }); switch (token.tag) { .invalid => return error.LexFail, else => {}, } } if (args.mode == .lex) return; // to be continued }</code></pre> Unlike the original implementation, there are a few differences: it does not attempt to recover on new lines, but breaks as soon as one invalid token happens. This is actually a terrible idea but it is passable in a toy compiler. Also there is no eof</code> token, choosing to return null</code> on source file end. Not very data oriented of me, as it increases the size by a byte (probably), but I will live. Bugs shall be found when I tie it to the book's test suite. Setting Up Tests</h2> This is actually so dumb it is not worth its own section. Just clone the book's tests repo and create a symlink to the compiler in the base directory. For future reference, this is what I typed to create said symlink: # while in the tests folder ln -s ../paella/zig-out/bin/paella</code></pre> Then I just call the tests as follows from the same said folfer. ./test_compiler ./paella --chaoter 1 --stage lex</code></pre> All tests should pass now. Ran 24 tests in 2.008s FAILED (failures=19)</code></pre> Oh come on. All of these errors seem to be hitting the error.UnrecognizedFlag</code> specified in the argument parser. Debugging the situation, with friendly std.debug.print</code>, showed me that not all calls to the function have the same order of arguments, which is definitely a surprise. The argument parsing code clearly needs to be revised. pub fn parse_args() !Args { var args = std.process.args(); _ = args.skip(); var path: ?[:0]const u8 = null; var mode: Mode = .compile; while (args.next()) |arg| { if (arg[0] == '-') { mode = std.meta.stringToEnum(Mode, arg[2..]) orelse return error.UnrecognizedFlag; } else if (path == null) { path = arg; } else { return error.PathDuplicated; } } return .{ .path = path orelse return error.PathNotFound, .mode = mode, }; }</code></pre> Let's try again. ---------------------------------------------------------------------- Ran 24 tests in 2.420s OK</code></pre> Yay! AST</h2> Dealing with abstract syntax trees (ASTs) is pretty much the reason the book tells the reader to use a functional language with sum types^{4</a> and pattern matching. In fact, the reference implementation is in relatively easy to read OCaml.} To explore how to represent the AST in Zig, I mostly just looked for C resources. This very nice article</a> explains how to represent a basic AST in C, and as a bonus compares it to an equivalent OCaml (as it happens) implementation. I also had a read on Mitchell Hashimoto's series on the Zig compiler</a>. The series is very informative, but to be honest there is no real need for this project to be super data oriented design driven with maximum cache friendliness and whatever may be. A, relatively, naive implementation of an AST is perhaps much more useful for learning purposes. That said, I can adapt some of the interesting ideas mentioned, like "splatting" some enum</code>s. For example, assume an expression can be either a binary operation or a unary operation. It would be represented naively as follows: const Expr = union(enum) { number: u64, binary: struct { BinOp, *Expr, *Expr }, unary: struct { UnOp, *Expr }, }; const BinOp = enum { add, sub, and_, or_ }; const UnOp = enum { negate, not };</code></pre> However, splatting the Expr</code> enum would make the whole type slightly smaller. As an added benefit, many parts of the code would have different behaviour depending on what type of binary or unary operator it is, leading to switch</code> statements within switch</code> statements. The flatter design would look liek this: const Expr = union(enum) { number: u64, bin_add: struct { *Expr, *Expr }, bin_sub: struct { *Expr, *Expr }, bin_and: struct { *Expr, *Expr }, bin_or: struct { *Expr, *Expr }, un_negate: *Expr, un_not: *Expr, };</code></pre> It looks slightly more verbose, but it is exactly the same amount of data. The types are being summed in our sum types! In the actual Zig compiler, the pointers are replaced by indices into the other Expr</code> nodes where they are stored in a giant array: Handles being the better pointers and all that. As I do not particualy need that kind of optimization, I will just throw every node into an Arena</code> and free it all at once once I am done with the AST (in a later stage from parsing.) The needs for this chapter are a lot simpler, however. The entirety of our current AST can be summarized as follows. const Prgm = struct { func_def: *FuncDef, }; const FuncDef = struct { name: []const u8, body: *Stmt, }; const Stmt = union(enum) { @"return": *Expr, }; const Expr = union(enum) { constant: u64, };</code></pre>Parser</h2> Writing a parser is a bit more involved, and it is something I have never done before. In my previous implementation I used a parser combinator library</a>. It was probably the easiest and smoothest part. There are parser combinator libraries for Zig. mecha</code></a> is the first result on Google.5</a> I think at first I will try my handrolling the parser. Should that prove to be arduous, I will maybe switch to mecha</code>. The pseudocode given in the book translates, in Zig, to the next listing. I am passing the allocator as all items should be allocated in an Arena, as mentioned before. I am also passing the tokenizer as a reference. I do not expect to have a lot of backtracking, as it is C I am compiling, after all, so the tokens are allocated nowhere and the tokenizer just produces them as it goes along. fn parse_stmt( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Stmt { try expect(.keyword_return, tokens); const expr = try parse_expr(alloc, tokens); // stub try expect(.semicolon, tokens); const ret = try alloc.create(ast.Stmt); ret.* = .{ .@"return" = expr }; return ret; } fn expect( expected: lexer.Token.Tag, tokens: *lexer.Tokenizer, ) !void { if (tokens.next()) |actual| { if (actual.tag != expected) return error.SyntaxError; } else return error.SyntaxError; }</code></pre> Looks simple enough. Let's try the rest. parse_expr</code> has currently only one thing: an integer. It should have some structure, however. It does not use expect</code> as expect</code> discards its token. fn parse_expr( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Expr { const current = tokens.next() orelse return error.ExpectExpr; switch (current.tag) { .number_literal => { const lit = tokens.buffer[current.loc.start..current.loc.end]; const res = try std.fmt.parseInt(u64, lit, 10); const ret = try alloc.create(ast.Expr); ret.* = .{ .constant = res }; return ret; }, else => return error.ExpectExpr, } }</code></pre> The other two, parse_func_def</code> and parse_prgm</code> are as straightforward as they get. I am putting off dealing with identifiers for now, because I do not want to put them in the arena, but use some sort of global hashmap instead. Maybe for the next chapter. fn parse_prgm( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Prgm { const func_def = try parse_func_def(alloc, tokens); const ret = try alloc.create(ast.Prgm); ret.* = .{ .func_def = func_def }; return ret; } fn parse_func_def( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.FuncDef { try expect(.type_int, tokens); // this should be an identifier but i am taking a rain check try expect(.keyword_main, tokens); try expect(.l_paren, tokens); try expect(.keyword_void, tokens); try expect(.r_paren, tokens); try expect(.l_brace, tokens); const body = try parse_stmt(alloc, tokens); try expect(.r_brace, tokens); const ret = try alloc.create(ast.FuncDef); ret.* = .{ .name = "main", .body = body }; return ret; }</code></pre> Great. Now to call them from main.zig</code> and build and run the tests. AssertionError: Didn't catch error in chapter_1/invalid_parse/extra_junk.c ---------------------------------------------------------------------- Ran 24 tests in 4.164s FAILED (failures=1)</code></pre> Pfft. This is extra_junk.c</code> int main(void) { return 2; } // A single identifier outside of a declaration isn't a valid top-level construct foo</code></pre> Oh that's why. Adding a small check to parse_prgm</code> should solve the issue. pub fn parse_prgm( alloc: std.mem.Allocator, tokens: *lexer.Tokenizer, ) !*ast.Prgm { const func_def = try parse_func_def(alloc, tokens); // funny piece of syntax. if (tokens.next()) |_| return error.ExtraJunk; const ret = try alloc.create(ast.Prgm); ret.* = .{ .func_def = func_def }; return ret; }</code></pre> This passes all tests. To be honest I do not if I am doing things correctly yet. Less so in writing a compiler part but more so in the not leaking any memory part. The DebugAllocator</code> does tell if there are leakages, but the test runner hides the output from me unless there is a test failure. It does not matter in the writing a compiler side of things, but I guess it does matter on the learning properly side of things. Whatever. What's the next task? Pretty Printing the AST</h2> Before I start on the actual next task, now would be a pretty good time to start implementing pretty printing. It would help massively down the line with debugging. It is possible in Zig to create custom formatters for types, which I plan to use. However it relies on weird generic pseudo-interface shenanigans, very unlike what is in the Rust system. Zig's interface like behaviour is structural: it checks if there is a function a specific name and type in there, and if not, does something else. First let's show the result without pretty printing, fir the following C program. int main (void) { return 0; }</code></pre> When parsed into an AST, it prints by default like that: ast.Prgm{ .func_def = ast.FuncDef{ .name = { 109, 97, 105, 110 }, .body = ast.Stmt{ .return = ast.Expr{ ... } } } }</code></pre> Not particularly useful or helpful. The way to create a custom formatter is to add this function to the type's definition: pub fn format( self: @This(), comptime fmt: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { // do stuff }</code></pre> The writer</code> field seems to have the functions print</code> and writeAll</code>, based on various examples, but I do not actually know from this signature what is vailable.^{6</a> This resolves at compile time via C++ template-like SFINAE behaviour (as far as I understand it anyway).} FormatOptions</code> is a lot more interesting. this is the definition from the Zig standard library: pub const FormatOptions = struct { precision: ?usize = null, width: ?usize = null, alignment: Alignment = default_alignment, fill: u21 = default_fill_char, };</code></pre> Like in did in my Rust implementation, I can use the optional width</code> variable to pass the indentation level to each child element. If this does not make sense now, it might in a minute. The formatting apparatus in Zig is also explained in the standard library documentation, which is explained in neat examples in zig.guide</a>. Here is the format written out for your convenience: /// `{[argument][specifier]:[fill][alignment][width].[precision]}`</code></pre> There was one missing piece though. How do I pass a variable to the width</code> parameter? All the examples seem to use a literal. This took some tinkering to figure out, but figure it out I did. The signature of std.debug.print</code> accepts in its second argument an anytype</code>, which is analyzed at comptime</code> and matched with the given formatting string (which must be comptime</code> known), and it fails to compile if they do not. I always passed on a tuple and relied on its general ordering, like so. std.debug.print("{?}\n", .{prgm});</code></pre> But the args</code> parameter does not have to be a tuple. The struct elements can have names!! This explained very briefly in the docs (found in std.fmt.zig</code> in the standard library) like so: /// - *argument* is either the numeric index or the field name of the argument that should be inserted /// - when using a field name, you are required to enclose the field name (an identifier) in square /// brackets, e.g. {[score]...} as opposed to the numeric index form which can be written e.g. {2...}</code></pre> I did not know that. But that laos meant I can pass the width</code> as a variable as well if I pass it in as a named field and surround it in square brackets. // magic happens here vvvvvvv try writer.print("{[def]:[width]}", .{ .def = self.func_def, .width = w, // passed on from before. }); // this also works. vvv index try writer.print("{:[1]}", .{ self.func_def, w, });</code></pre> This passes the w</code> variabe to the FormatOptions</code> of the called formatter. voila!! What follows is conceptually simple. This is my format</code> function implemented for ast.Prgm</code>: pub fn format( self: @This(), comptime _: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { try writer.print("PROGRAM\n", .{}); try writer.print("{[def]:[w]}", .{ .def = self.func_def, .w = (options.width orelse 0) + 1, }); }</code></pre> And this is the one for FuncDef</code>: pub fn format( self: @This(), comptime _: []const u8, options: std.fmt.FormatOptions, writer: anytype, ) !void { const w = options.width orelse 0; try writer.writeByteNTimes('\t', w); try writer.print("FUNCTION {s}\n", .{self.name}); try writer.print("{[body]:[w]}", .{ .body = self.body, .w = w + 1, }); }</code></pre> And the rest, as chess players say, is just technique. Now running our program on the C snippet gives this glorious output. PROGRAM FUNCTION main RETURN 0</code></pre> This creates an additional maintenance burden the bigger the compiler grows, but it is worth it for the debugging goodness. Now back to work. Assembly Generation</h2> Straight to the point. I like it. No intermediate representations or anything tacky of the sort.^{7</a> So let's get on with it.} The first step in assembly generation is to actually put up an syntax tree for assembly. the second tree is whwat we use to actually generate the assembly code at the end. So, the first thing I will do here is to create an assembly.zig</code> file with the similar structure as ast.zig</code>. This is fairly straightforward. pub const Prgm = struct { func_def: *FuncDef, }; pub const FuncDef = struct { name: []const u8, instrs: std.ArrayListUnmanaged(Inst), }; pub const Inst = union(enum) { mov: struct { src: Operand, dst: Operand }, ret: void, }; pub const Operand = union(enum) { imm: u64, reg: void, };</code></pre> To generate this AST from the source AST, I am going to stick the logic in yet another file I will call asm_gen.zig</code>. So many files. I am not quite sure about storing the body of instructions this way, but it saves from the allocation dance every time I need to edit the contents down the line. Using a SimplyLinkedList</code> might be an option too. For now, this is it. The Unmanaged</code> part is sems intimidating, but it just really means that it does not keep a reference to its allocator, putting the responsibility of making sure the same allocator is passed every time on the caller. My understanding is that this is mean to be the default interface for ArrayList</code> going forard. Back to task. After wrangling a bit with zig's compiler errors and its type checks8</a>, I came up with these intimidating looking functions with all kinds of memory leaks in the sad path (I fear). pub fn prgm_to_asm( alloc: std.mem.Allocator, prgm: ast.Prgm, ) !assembly.Prgm { const func_def = try alloc.create(assembly.FuncDef); errdefer alloc.destroy(func_def); func_def.* = try func_def_to_asm(alloc, prgm.func_def.*); return .{ .func_def = func_def }; }</code></pre>fn func_def_to_asm( alloc: std.mem.Allocator, func_def: ast.FuncDef, ) !assembly.FuncDef { const name = try alloc.dupe(u8, func_def.name); errdefer alloc.free(name); const instrs = try stmt_to_asm(alloc, func_def.body.*); return .{ .name = name, .instrs = instrs, }; }</code></pre> This next one took a lot of wrangling until I figured out the syntax for a slice of unions. fn stmt_to_asm( alloc: std.mem.Allocator, stmt: ast.Stmt, ) !std.ArrayListUnmanaged(assembly.Inst) { switch (stmt) { .@"return" => |value| { var result: std.ArrayListUnmanaged(assembly.Inst) = .empty; try result.appendSlice(alloc, &.{ .{ .mov = .{ .src = try expr_to_asm(alloc, value.*), .dst = .reg, } }, // I could just ype `.ret,` here but that might be confusing .{ .ret = {} }, }); return result; }, } }</code></pre> And last but not least, the absolute bottom of the totem pole. The allocator is passed in here just in case because I think I might need it later. fn expr_to_asm( _: std.mem.Allocator, expr: ast.Expr, ) !assembly.Operand { switch (expr) { .constant => |i| return .{ .imm = i }, } }</code></pre> Testing this out with the tiny C program gives me a rather ugly output. Implementing pretty printing (based on our previous attempt) for our assembly AST gives us this nice output. PROGRAM FUNCTION main mov imm 0, register ret</code></pre> Obviously this can just be repurposed to generate the actual assembly. So that is what is going to happen next. Code Emission</h2> Here is a funny thing. The format</code> function that needs to be implemented for the formatting interface has a field that I have so far discarded. This field is one of the option the user tells the formatter what to do, and it is used all the time for everyone. For example, when you want to print a digit as a digit, you do print("{d}, .{my_digit})</code>. Wheh nyou want an ASCII character, print("{c}, .{my_digit})</code> is your friend. This tiny d</code> or c</code> is the string passed in through the hitherto discarded input. And I can define it to be anything I want!! I actually tried the phrase free syria</code> with a space, and some Arabic text,, and it passed through as expected. I have been printing to the terminal like so: std.debug.print("{}\n", .{prgm});</code></pre> Which gives me the (arguably) nice debugging output seen above. But if I, say, call it like so: std.debug.print("{gen}\n", .{prgm});</code></pre> I can have it emit the assembly code wanted just fine. And since it works with any Writer</code>, I can write it to a file immediately. Pretty cool, and a similar interface to what Rust does, anytype</code> notwithstanding. Without boring you with the details, the formatting prints this really nice assembly (which does not forget the underscore since I am on macOS): .globl _main _main: movl $0, %eax ret</code></pre> And this concludes all tests for this chapter, which pass with flying colors. Lessons Learned</h2> This is, currently, a tiny do-nothing program. However, in the course of implementing it, I learned the followings things: More comfort with Zig's syntax.</li> Learned a bit about Zig's standard library (environment, file system, child processes, collections), and build system.</li> Not shown here, but I managed to figure out how to actually add dependencies. It is a bit more involved than cargo add</code>, but it is nice enough.</li> </ol> I think this program is a decent testing ground for someone wanting to learn a new language, especially if done before in a familiar one. Jury's out on whether to continue with chapter 2 with Zig or try reimplementing chapter 1 in Swift, (or maybe Gleam? How do you even run Gleam programs?) That's all, and thanks for reading. Rust provides a nice, encapsulated, function for this: with_extension</code></a>. But this is Zig: why have a convenient function for a common operation when one can instead do things the complicated and explicit way? Explicitness is extremely important, you know. ↩</a> </li> Which is really just a loop in switch clothing. ↩</a> </li> An interesting tidbit: the Zig lexer/tokenizer only supports the ASCII letters for identifiers. If one wants to use Unicode, it should be escaped like keywords do inside @""</code>. This has the benefit of the Unicode data not being a dependency of the compiler, and it simplifies lexing. ↩</a> </li> like Zig union(enum)</code>s or Rust enum</code>s. The "sum" terminology comes from math, apparently. An enum is one of several things, so the total possible values are a sum of each value. A struct</code>, in pretty much any language, in comparison is a product type, where its number of possible states is the product of its individual elements possible states. ↩</a> </li> The other option is to use a parser generator. I looked and failed to find any that output Zig code. If you know of one, please let me know. ↩</a> </li> Further internet browsing revealed it was std.io.GenericWriter</a> which has a few other potentially useful functions. ↩</a> </li> That is introduced in later chapters, and is called TACKY. ↩</a> </li> Why just why do compilers print the first error first? They do that, then the terminal scrolls up, and I cannot find where the first error begins and the last error of the previous commands ends. Can you all add a pager or print the first error last? ↩</a> </li> </ol> </section> Placemaking Syria 2025-05-03T00:00:00+00:00 Note: This was written as a draft socument. The ideas are haphazard and not properly organized. </blockquote> The cities of Syria are, to put it lightly, in bad shape. Decades of neglect followed by the war and large-scale destruction</a> have destroyed the cities. Everyone in Syria wants to rebuild, but we need to re-build it right. We should build them for humans. But cities should be designed for humans. Humans should be able to walk the streets safely and freely. Chuldren should be able to run and play. Women with strollers and the elderlies with canes and disabled with wheelchairs should be able to get around. Our environment should pleaze our senses. Unfortunately, modern cities are designed first and foremost for cars. Streets are made for cars, and humans walking around are merely a necessity. Car pollute our enivornment with noise and smells and eye-sores. Our children cannot walk safely on streets. All the pavements and asphalt makes it difficult to walk around with a stroller (seriously, try it one day.) But it does not have to be this way. We can use this opportunity to build our cities and streets again to cater for living things. The idea has many names in Academia and Practice. You'll often see it called Placemaking, or Humanization</a>, or the 15-minute City</a>. The theory behind these concepts are entirely uncotroversial among academics and practioners, but where they stumble is often in convincing policy makers (which brings about all the brand names). The Real Problem with Cars</h2> Cars are often brought up as a major contributor to air pollution and global warming. And the solution, very often, is posited as enviironmentally friendly cars, or (in case of Syria) newer less polluting cars. Another usual suggested solution is to mandate carpools</a>, or odd-even rationing</a> or widening roads at the expense of all else</a>.But all that is just kicking the can down the road instead of picking it up. Cars inside cities are terrible as a concept. They are the absolute worst, least efficient way, to move people around a city (second only to a boat, perhaps.) To illustrate the point, I invite you to watch this old car commercial</a>. Note how empty the world looks, with all that space that is taken away by cars could be instead used for pavements and trees and shops and housing and other humans. The ad is mwant, ironically, to advertise cars; but it only shows how ridiculous they are. The irony of this is a self reinforcing cycle. The more cars you have, the more space you need of car infrastructure, the less space there is for actual useful humans things, the further everything is, the more you need cars. Imagine all the additional space you could have in your city if there just was not any cars? Any Examples?</h2> Unfortunately, policy makers are generally very resistant to city humanization policies. There are many rational (and irrational) reasons for that. People do not like change. And people love their cars. Most importantly, infrastructure is expensive, and decades of sunk costs reinforce these positions. The solutions also require maintenance and long term policy commitments. But there are plenty of positive examples of cities that have done a lot to make their streets walkable and human friendly, with various levels of commitment. This showcases that this is not an all or nothing position, (as much as I would like it to be an "all" position,) but someting to aspire to. Chicago, the L, and the Parks</h3> The Chicago "L"</a> (known as the 'L') is one of the most successful examples around the world for a "subway". It is integral to the walkability and ease of transit around the city, allowing residents to move anywhere within the city in walkable distance, aided by the system. [^suburbs] [^suburbs:] No love for the Chicago suburbs, but that is really an all-american ailments that is not remedied at the City level but at the state or "muhafaza" level. The L does not just connect the different sections of Chicago to each other, and to its downtown, the Loop (called so because there is Loop of L lines,) but it also connects them to the various huge parks around Chicago</a>. The parks themselves are an important part of what makes Chicago a livable beautiful city. [The Burnham Park] (after legendary Chicago architect Daniel Burnham1</a>) on the lakefron is a place for residents and tourists alike. Other parks like the Millenium Park and the Jackson Park celebrate famous historical events in the city. Not unlike the Sword of Damascus, really. Riyadh</h3> The City of Riyadh, capital of Saudi Arabia, is as dependent on cars a city can be. It is one of the least dense cities in the world, where it is virtually impossible to exist without a car, or without paying exhoberant taxi fares. Road deadlocks are a regular fact of life. This has been, largely, due to the historical mistake (commonly attributed to Greek urban planner Doxiadis</a>) of expanding the city horizontally over the desert to the north instead of expanding around the various valleys and natural water sources that exist to the south. Riyadh (which is Arabic for "Gardens") is actually green. It has pelnty of natural parks, but the development on the city, and the rapid expansion (aided by cars) has led to the poor state of the city as it is today. However, the Municipality of Riyadh has over the last decade been meaning to change that, and to remake Riyadh into a beautiful walkable cty that is fit for her name. The natural parks around the city (such as Wadi Hanifa and Wadi Namar) have been developed to become regular resident spots. The massvie Riyadh Metro development that took over the city (which was not without its pains) has finally launched last year, has done much to improve the walkability and reach of the various parts of the city and, to paraphrase one news article, "Promore walking culture amoung Saudis". Even the desert, horizontal parts of Riyadh are being developed. The Green Riyadh</a> is planning to plant around 7.5 million trees around mosques and schools and shopping malls. The grand avenues of the city (like Olaya and Tahlia) are frequent hanging out spots. Changes to the building code in Wadi Hanifa and north of the city are emphasisng denser development. Paid street level parking is being introduced all over to discourage car abandonment on streets. Mind that this is not a quick and painless work. There have a lot of pains around the redevelopment (and redevelopment of redevelopment) of Riyadh that caused a lot of busienesses and areas to wither and suffer during it. The Riyadh Metro project was undertaken all at once, in the way only Saudi Arabia could, istead of stages. Similarly with the current ongoing Green Riyadh and Sports Boulevard</a> shutting down many businesses and areas during the work. But the benefites from the already done projects and the projected benefits of the ones about to conclude are worth it. Riyadh is actually becoming quite pleasant to live in. Issues and Challenges</h2> These positions require a lasting policy commitment that recognizes their long term value. People respond to economic incentives, not to long term quality of life ones. If going around the city in a single occupant car is more economically viable (cheaper) than riding the metro, then people will do that. The improvement of public transport and city walkability should coincide with making cars harder and more annoying to use: restricted parking, road tolls (both of which actually help with public service maintenance) are two tools that can be used effectively to do that. The need to start planning this is right now. Public transport and urban design take an incredibly long time to design and execute and maintain. Public and policy support should be cultivated and used to support such projects, and view public transport and proper, walkable streets as a necessity, not a luxury. in Syria, the silver lining is that the bandaid is already ripped off: The infrastrudtcure is destroyed. There is no sunk cost any more, but we need to start planning now. Aleppo has plenty of public parks and wide streets and beautiful features. Did you know Aleppo has a river in it that flows through the city center? Connections around the city with an elevated sytem like Chiago's would help reduce transport and petrol costs for residents. It would be easier to the Citadel from Nile Street or Hamdania without an expensive cab. Excessively wide streets (like Faisal Road, which is one-direction anyway) and and excessively narrow ones (like the roundabout around the Citadel) could beasily transformed into human-first spaces. However, what actually prompted me to write this piece is this image of Damascus: Imagine this beautiful grand space without cars or asphalt. How buatiful would that shot have been? Additional Reading and Viewing</h2> (Arabic) Sharik Podcast: Meeting with Abdul Rahman al-Sayel: City Humanization and Sustainabolty</a>. Al-Sayel's Twitter account</a> is a worth a follow.</li> (English) What is Placemaking, Project for Publix Spaces</a></li> If you don't mind the vulgarity: the subreddit r/fuckcars</a> has a nice write up with additional resouces.</li> </ul> Protagonist of The Devil in the White City</a>. ↩</a> </li> </ol> </section> Parsing Hearthstone Deck Codes 2024-11-14T00:00:00+00:00 In the last few days I have been mucking around with Mimiron</code></a>, my Hearthstone API library and Discord bot. One of the main functions of the bot, and the one it is used for the most, is parsing Hearhstone deck codes and presenting them in a pretty fashion. I had a working parser for a while, but I wanted to experiment with it so I decided to rewrite it in nom</a>. Here are both parsers. Deck Code Format</h2> The deck code fromat is explained by HearthSim</a>, the company behind HSReplay</a> As far as I know, there is no official specification of deck codes encoding, so HearthSim's description is what everyone relies on. The page is actually outdated, but their example implementations (in C#</a>, Python</a>, and TypeScript</a>) explain it clearly enough. The deck code is essentially a base64</code> encoded series of integers.</li> These integers are encoded as unsigned varint</code>s, variable width integers</a>1</a>.</li> The first two integers are always 0</code> and 1</code>. They happen to be a byte each, simplifying matters.</li> The third integer is the Format. 1</code> is Wild, 2</code> is Standard, 3</code> is Classic, and 4</code> is Twist.</li> </ul> After Format</code>, every block of integers starts with a count, then a listing of items, usually Card IDs (often called DBF IDs). Card IDs are unique numerical IDs that uniquely2</a> identify each card in the game. The blocks are as follows: Hero. Conveniently, this block only ever has one element.</li> Single-copy cards.</li> Double-copy cards.</li> N-copy cards. Each item in this block is a Card ID followed a by a count. This is only used for weird one-off formats or Arena codes.</li> A block of blocks? Unclear.</li> </ol> The block of blocks is not clearly documented. Sussing it out from existing parsers, it only has one item, which is the Sideboard block. The Sideboard block starts with a count, as usual, but each item is a pair of Card IDs. The first is the card itself, and the second is the Card ID of the Card it is under. There are, as of this time, only two Sideboard cards in the game. After that is whatever. It apparently does not matter. It seems to usually be two 0</code>s? hitherto undefined blocks? WHo knows. Example</h3> Take this deck code for example: AAECAdrJBgbHpAaopQbUpQaX4Qat4Qaq6gYM5p4GpKcGqKcG66kGw74GzsAG0MAG0dAGmOEGmOIG5OoGjfgGAAED9bMGx6QG97MGx6QG6N4Gx6QGAAA=</code></pre> This is the list of numbers it represents, with comments. 0 1 2 # Format 1 107738 # Hero ID 6 # Single Copy IDs 102983 # <-- Note this ID is the same as the Sidebaord Card later. 103080 103124 110743 110765 111914 12 # Double Copy IDs 102246 103332 103336 103659 106307 106574 106576 108625 110744 110872 111972 113677 0 # N-Copy IDs. This is only used in weird formats. 1 # no idea really 3 # Sideboard IDs 104949 102983 104951 # first part of pair: Card IN Sideboard 102983 # The Sideboard Card 110440 102983 0 # who knows? Future proofing maybe? 0</code></pre> It is a very simple format really. Desired Output and Shared Elements</h2> This is the Object which I need to fill up. struct Data { format: Format, hero: usize, cards: Vec<usize>, sideboard_cards: Vec<(usize, usize)>, }</code></pre> And this is the Format</code> type with its conversion from a usize</code>. #[derive(Default)] enum Format { Standard, #[default] Wild, Classic, Twist, } impl TryFrom<usize> for Format { type Error = (); // Petend it is something useful. fn try_from(value: usize) -> Result<Self, Self::Error> { Ok(match value { 1 => Self::Wild, 2 => Self::Standard, 3 => Self::Classic, 4 => Self::Twist, _ => return Err(()), }) } }</code></pre> To be able to implement and test the different parsers without much of the surrounding code interfering with it, I decided to wrap the whole thing into a tiny executable to be able to pass the codes directly on the command line. I'd also need the base64</code> crate. This is the venerable Cargo.toml</code>: [package] name = "deck_parser" version = "0.1.0" edition = "2021" [dependencies] base64 = "0.22.1" pico-args = "0.5.0" # more to xome</code></pre> And this is main.rs</code>, without the types defined above: use base64::{prelude::BASE64_STANDARD, Engine}; use pico_args::Arguments; fn main() { let mut args = Arguments::from_env(); let raw = args.contains("--raw"); let Ok(decoded) = args.free_from_fn(|s| BASE64_STANDARD.decode(s)) else { eprintln!("Not base64 encoded"); std::process::exit(1); }; if raw { print_nums(&decoded); } else { if print_data(&decoded).is_err() { eprintln!("Invalid Deck Code"); std::process::exit(1); } } } // Print the List of Numbers fn print_nums(decoded: &[u8]) { todo!() } // Print the Raw Data obejct. fn print_data(decoded: &[u8]) -> Result<(), Box<dyn Error>> { todo!() }</code></pre> The --raw</code> flag is mostly to test the varint</code> reader, and check it always gets the same output. Straightforward Parser</h2> This is the first parser that I implemented. I used the integer_encoding</code> crate</a> because I did not want to think about how varint</code> encoding works, and it presents a nice API. It required wrapping the buffer in a std::io::Cursor</code>, but that was it. It reads right from there. Consdiering the varint</code> parsing is relegated to a crate, print_nums</code> is fairly straightforward. use integer_encoding::VarIntReader; // way at the top of the file. use std::io::Cursor; fn print_nums(decoded: &[u8]) { let mut buffer = Cursor::new(decoded); while let Ok(n) = buffer.read_varint::<usize>() { println!("{n}"); } }</code></pre> And one can quickly test that it works with the following command. Should you run it with the deck code earlier, you will find it lists the same numbers. cargo run -- --raw <insert-deck-code-here></code></pre> Excellent. This means that integer_encoding</code> does its job splendidly. The second one is a bit more involved, but also fairly straightforward. read_varint</code> advances the Cursor</code>, so all that is needed is to assign the values in the correct order. There is a small footgun hiding in the API, however, which is that unsigned varint</code>s have a different encoding for signed varint</code>s. Considering literals are, by default, i32</code>, Rust's type inference works against you here. fn print_data(decoded: &[u8]) -> Result<(), Box<dyn Error>> { let mut buffer = Cursor::new(&decoded); // Format is the third number. buffer.set_position(2); let format = buffer .read_varint::<usize>()? .try_into() .unwrap_or_default(); // Hero ID is the fifth number. buffer.set_position(4); let hero = buffer.read_varint()?; let mut cards = Vec::new(); // Single copy cards let count = buffer.read_varint()?; for _ in 0_usize..count { // <-- Type Inference footgun!! let id = buffer.read_varint()?; cards.push(id); } // Double copy cards let count = buffer.read_varint()?; for _ in 0_usize..count { let id = buffer.read_varint()?; cards.push(id); cards.push(id); // twice } // N-copy cards let count = buffer.read_varint()?; for _ in 0_usize..count { let id = buffer.read_varint()?; let n = buffer.read_varint()?; for _ in 0_usize..n { cards.push(id); } } // Sideboard cards. Are they always available? let mut sideboard_cards = Vec::new(); if buffer.read_varint::<usize>().is_ok_and(|i| i == 1) { let count = buffer.read_varint()?; for _ in 0_usize..count { let id = buffer.read_varint()?; let sb_id = buffer.read_varint()?; sideboard_cards.push((id, sb_id)); } } let result = Data { format, hero, cards, sideboard_cards, }; println!("{:#?}", result); // Don't forget to dervie Debug Ok(()) }</code></pre> Not a very pretty printing. But it works. Good base implementation. Unsigned Variable Integers</h2> When I started trying to rewrite the parser in nom</code>, I looked for an extension nom</code> crate that can read varint</code>s. I found two! Reading through their source, however, I realized that it is really simple, and decided to bring it in the code base myself. Unsigned varint</code>s' encoding is, as it turned out, fairly simple. Honestly it is actually easier to just show the code than explain how the encoding works. fn parse_varint(input: &[u8]) -> Option<usize> { // a usize can only be as big as 9 bytes in varint encoding const MAX_BYTES: usize = 9; let mut num = 0; for (idx, &byte) in input.iter().take(MAX_BYTES).enumerate() { // 7 bits from each byte are used let byte = byte as usize & 0x7F; let offset = idx * 7; num |= byte << offset; // The last byte's most significant bit is 0. if byte & 0x80 == 0 { return Some(num); } } None // Insufficient data or overflow }</code></pre> Signed varint</code>s are a whole other beast, and they do not concern this article outside of the type inference footgun from the previous section. nom</code> Parser</h2> This is what actually prompted writing this article.3</a> I already used nom</code> elsewhere in Mimiron</code>, and I wanted to exercise more with it. The nom</code> way is all about small, composable functions. The clearest small composable function to do is parse_varint</code>. I am not going to use the imperative version shown above, I am writing one using nom</code>'s parser combinators and the standard library's Iterator</code> combinators. // no nom codebase is complete without a giant use statement on top // this includes combinators for all the functions below use nom::{ bytes::complete::{ take, // takes a number of bytes unconditionally take_till, // takes until a condition is met and stops before }, combinator::{ cond, // calls a parser only if a condition is met map, // maps the result of a parser map_opt, // maps the result of a parser to an Option opt, // optional parser, for a value. returns Option verify, // verifies the result satisfies a condition }, multi::{ length_count, // gets a number from first parser, // then applies second parser that many times many0, // repeats a parser until it fails. returns Vec }, sequence::{ pair, // call first parser, then second parser. preceded, // discards result of first parser tuple, // same as pair, but up to 21 separate parsers }, IResult, // nom's Result type. // Returns remaining input and successful results // or an error with the input inside the Error. }; fn parse_varint(input: &[u8]) -> IResult<&[u8], usize> { let is_last = |b| b & 0x80 == 0; let is_in_bounds = |p: &[u8]| p.len() < 9; let try_varint = |(p, lb): (&[u8], &[u8])| { // this is the same imperative code up above with some error checking p.iter() .chain(lb) .enumerate() .try_fold(0, |acc, (idx, byte)| { ((*byte as usize) & 0x7F) .checked_shl(idx as u32 * 7) .map(|n| acc | n) }) }; map_opt( pair(verify(take_till(is_last), is_in_bounds), take(1u8)), try_varint, )(input) }</code></pre> Same, but slightly more unreadable (I am going to have so much fun with this): fn parse_varint(input: &[u8]) -> IResult<&[u8], usize> { map_opt( pair( verify(take_till(|b| b & 0x80 == 0), |p: &[u8]| p.len() < 9), take(1u8), ), |(p, lb): (&[u8], &[u8])| { p.iter() .chain(lb) .enumerate() .try_fold(0, |acc, (idx, byte)| { ((*byte as usize) & 0x7F) .checked_shl(idx as u32 * 7) .map(|n| acc | n) }) }, )(input) }</code></pre> Using this small function to generate the list of numbers, going straight for the unreadable verion: pub fn print_nums(decoded: &[u8]) { _ = many0(map(parse_varint, |n| println!("{n}")))(decoded); }</code></pre> The second function is a bit more involved, but it generally follows the structure of the previous section's solution. I think it is easy enough to follow. pub fn print_data(decoded: &[u8]) -> Result<(), Box<dyn Error + '_>> { // starting from 2 as Format is the third number. let (rem, format) = map( parse_varint, |f| f.try_into().unwrap_or_default() )(&decoded[2..])?; // skip 1 byte for Hero ID. let (rem, _) = parse_varint(rem)?; let (rem, hero) = parse_varint(rem)?; let (rem, cards) = map( tuple(( // single cards length_count(parse_varint, parse_varint), // double cards length_count(parse_varint, map(parse_varint, |id| [id; 2])), // n-count cards length_count( parse_varint, map( pair(parse_varint, parse_varint), (id, n)| [id].repeat(n) ), ), )), |(v1, v2, vn)| { v1.into_iter() .chain(v2.into_iter().flatten()) .chain(vn.into_iter().flatten()) .collect() }, )(rem)?; let (rem, c) = opt(parse_varint)(rem)?; let (_rem, sideboard_cards) = map( cond( c.is_some_and(|c| c == 1), length_count(parse_varint, pair(parse_varint, parse_varint)), ), Option::unwrap_or_default, )(rem)?; let result = super::Data { format, hero, cards, sideboard_cards, }; println!("{result:#?}"); Ok(()) }</code></pre> But this can be made worse: the whole thing can be collapsed into one function. rustfmt</code> is pulling a lot of weight here, to be honest.4</a> pub fn print_data(decoded: &[u8]) -> Result<(), Box<dyn Error + '_>> { let result = map( tuple(( map(parse_varint, |f| f.try_into().unwrap_or_default()), preceded(parse_varint, parse_varint), map( tuple(( length_count(parse_varint, parse_varint), length_count( parse_varint, map(parse_varint, |id| [id; 2]) ), length_count( parse_varint, map( pair(parse_varint, parse_varint), |(id, n)| [id].repeat(n) ), ), )), |(v1, v2, vn)| { v1.into_iter() .chain(v2.into_iter().flatten()) .chain(vn.into_iter().flatten()) .collect() }, ), preceded( verify(parse_varint, |i| *i == 1), length_count(parse_varint, pair(parse_varint, parse_varint)), ), )), |(format, hero, cards, sideboard_cards)| Data { format, hero, cards, sideboard_cards, }, )(&decoded[2..])? .1; println!("{:#?}", result); Ok(()) }</code></pre> One tiny thing before the end. Note the +'_</code> bound on the return type. Since nom</code>'s errors contain the input, which is a reference, converting the errors can sometimes cause Rust's type inference to declare that your functions expect a 'static</code> lifetime. For example, here, when the +'_</code> bound is removed, or when converting nom</code>'s errors to anyhow</code>'s errors using the ?</code> operator.</a>. It is not a footgun, per se, as the compiler yells at you. But the error is frankly inscrutable unless you know what you are looking for. Final Thoughts</h2> Rewriting in nom was a bit easier than I thought it would be. And even the compact "one-liner" is more readable than I thought. The hardest part is probably how to translate the problem into nom</code>'s API. (Which I struggled with when I considered using nom</code> for Advent of Code</a>). Who knows? Maybe this year. If you would like to do this with another Rust parser library (chumsky</code> or winnow</code> or combine</code> or pest</code>), please do share, and I will happily link it here. This Wikipedia page gives a terrible and obtuse explanation. The implementation turned out to very simple. ↩</a> </li> IDs can be deleted from the game, leading to outdated deck codes out there, but that's out of scope of this article. In Mimiron</code>, I work around that by looking them up in the third party database everyone else uses: HearthSim's</a>, and "canonizing" the IDs. My bot is somewhat unique for relying first on Blizzard's official API</a>. ↩</a> </li> If you are somehow this far into this post and do not know what nom</code> is, it is a Rust parser combinator library, probably the first and most famous. Combinators are small higher order functions (read: functions that can take and/or output other functions: function functions, if you will), that can be composed together. Parser Combinator Libraries are, in a way, their own Domain Specific Language. nom</code>'s documentation has a nice intro</a>. ↩</a> </li> 2024-11-26 update: I found out that this code has a bug: it rejects valid decks without a sideboard. How would one fix that? ↩</a> </li> </ol> </section> The Hanging Gardens Problem 2024-10-17T00:00:00+00:00 This is an interesting puzzle inspired by Christian Freeling's tile set The China Cube</a>. As it really has nothing to do with China, I call it the Hanging Gardens Problem. Imagine each cube as a section of the Garden, and connections to other cubes are paths and stairways.1</a> Here is a nice drawing showing the full set of 64 Cubes, from Freeling's site. Problem Formulation</h2> There are 64 cubes.</li> Each cube is unique.</li> Each cube has a specific number of Red and Blue faces.</li> Cubes do not rotate or change orientation in anyway.</li> Direction of the faces is specific: North, East, South, West, Top, Bottom.</li> </ul> It follows from these that there is One cube with all Red faces (the Red cube, bottom Cube in the earlier drawing.)</li> One cube with all Blue faces (the Blue cube, top Cube in the earlier drawing)</li> Six cubes with one Red face.</li> Six cubes with one Blue face.</li> ... etc.</li> </ul> All possible permutations of Red and Blue faces are represented (hence 64 cubes). Here is the puzzle: Assemble the cubes so the Blue faces touch each other, and all the Red faces are exposed.2</a> In other words, a Blue face must always face, touching, another Blue face. A Red face must not do that. However, a Red face might wave to another Red face from a distance (at least one cube far.) There is most likely is a solution. The solution does not have to be one big shape. It actually cannot, because of the Red Cube. tl;dr I gave up on the problem due to time and memory constraints. However, while I was at it, I found solutions to two, equally interesting, sub problems, and nice graphics. </blockquote> Wave Function Collapse</h2> I asked around in a number of Discord servers on how to approach solving the problem programmatically, outside of 3-D printing the set and physically assembling it by hand. The Gorilla Sun</a> Creative Coding server suggested that using the Wave Function Collapse</a> algorithm might be the best fit for the problem, and suggested a very excellent Youtube video from the Coding Train</a> implementing the algorithm in Javascript. It is a very good video that I suggest watching. As explained at the start of the video, this is simply the Sudoku solving algorithm, generalized to creating random tile sets. Apparently, the Quantum Mechanics inspired name is simply for flavor and searchability. So, how does one solve a Sudoku? First, identify the connection rules and the tile set. (Which, in Sudoku, need no explaining.)</li> Second, Observation: mark each empty cell in the grid with the candidate tiles</del> numbers that go in them.</li> Third, the Collapse in Wave Function Collapse: identify the cells with the fewest number of candidates. 1. Any cell with 1 candidate "collapses" into that candidate, and the number is inked in. 2. If there are not any cells with one candidate, pick one cell with the fewest number of candidates and pick a candidate at random, "collapsing" it.3</a></li> Fourth, Propagation: with the new state of things, update the candidates in each call. Here is where Sudoku people use X-wings and Swordfishes and what have you.</li> Repeat the Collapse, until the grid is full.</li> If there ever, after Propagation, a cell with zero candidates, this means an illegal state was reached. Either Backtrack to the last random choice, (and, well, choose another branch), or simply start the whole thing over.</li> </ol> This approach can work just fine for the Hanging Gardens. The Tile set and the connection rules are clearly defined. It is only a matter of putting it into code and .. running it. The problem most apparent to me, however, is that the search space is potentially huge. So, I first need to try this approach with a smaller problem set. Square Gardens</h2> Phrasing the problem in terms of Squares, and 2D connections, rather than Cubes, shrinks the problem space by an order of magnitude. Plus, there is a known solution given by Freeling</a>! Tiles and Air</h3> First, I need to formulate the tiles. There are two kinds of Tiles. The first kind is the actually Squares. This is a finite set. The second kind, is an "Air" tile. This empty tile is infinitely available, and is automatically placed at the Propagation phase connected to every Red face. A Red face must connect to an Air tile, and a Blue face can never connect to one. An Air tile can connect from any side to any Air tile with no restriction. I am still having fun with Odin</a>, so I will try this solution in it first. Here are the starting types: Side :: enum { North, East, South, West } Tile :: distinct bit_set[Side] Air :: distinct struct {} Cell :: union { Air, Tile }</code></pre> I am not too happy about this because while the data can neatly fit into one byte, Odin puts this in two. This looks cleaner and easier to understand than weird hacks, so I will stick to it for the time being. Odin's union</code> is nice here, semantically, as it actually has three states: nil</code>, Air</code> and Tile</code>, exactly as much as needed. The Squares are only 16 (count them in the image above). It is possible to safely ignore the Red (unconnected) Square, because it can simply be put anywhere in space and does not need to connect to anything. Generating a list of the Squares is as simple as counting to 15, or 0xF</code>. Running the below snippet to confirm: main :: proc () { list : [dynamic]Cell for i in u8(0x1) ..= 0xF { fmt.println(transmute(Tile)i) append(&list, transmute(Tile)i) // <-- Odin casting magic here } fmt.println(len(list)) }</code></pre> Prints the following: Tile{North} Tile{East} Tile{North, East} Tile{South} Tile{North, South} Tile{East, South} Tile{North, East, South} Tile{West} Tile{North, West} Tile{East, West} Tile{North, East, West} Tile{South, West} Tile{North, South, West} Tile{East, South, West} Tile{North, East, South, West} 15</code></pre> Which is exactly what is needed. Hooray!! Grid and Candidates</h3> An infinite grid is infeasible due to being in a finite universe and using computers with finite resources. So what is a good size for a finite grid? There are 16 Tiles, so a 16 by 16 grid is a clear upper bound, and clearly too big, due to the impossibility of all squares being in one line. I will go with a 15 by 15 grid, which is also safe as it is too big, but it gives me a nice center cell to fix the Blue square in, and branch everywhere from there. To track the candidates in each cell, the simplest option, perhaps, is to create a set of tiles. Since each tile corresponds neatly to a number (as per the transmute</code> above,) simply tracking the numbers should be enough. As mentioned also, the Red square can be placed anywhere in space, so it does not need to be tracked. Giving its number, 0, to track the Air Tile should be enough, as it can later replace almost any Air tile in the final grid. Candidates :: distinct bit_set[0..=15; u16] Grid :: struct { cells: [15][15]Cell, // central cell is [7][7] candidates: [15][15]Candidates }</code></pre> Having candidates</code> in a separate field like that is convenient. Any cell with no candidates (a nil</code> bit set), would signal an illegal position reached. Any cell with only one candidate is considered collapsed, and the collapse would then be reflected in the cells</code> field, which represents the actual contents. All this, of course, is subject to be iterated out later. This is the first draft of how to initialize the grid. Note that in Odin ~</code> is the bitwise NOT</code> operator. So ~Candidates{15}</code> is the full set except for 15. And ~Tile{}</code> is the full set. grid_init :: proc() -> (ret: Grid) { // Blue Square ret.cells[7][7] = ~Tile{} // Fill the Candidates Grid ret.candidates = ~Candidates{15} // broadcast magic ret.candidates[7][7] = {15} return ret }</code></pre>Propagation</h3> An astute Sudoku solver would quickly be able to see that this initial list of candidates is, in fact, useless and incorrect. Yes it removes the placed Tile from all candidate positions, but it does not take into account the placed Tile's connections.4</a> For this to work the Candidate grid must be updated to match the new reality. It follows that for each placed Cell</code> in cells</code>, it must update the surrounding candidates. So a function that gives out the candidates per face is needed. This can almost just be hardcoded, since each number maps neatly to a specific square. It is possible to hardcode it for each tile individually, but where is the fun in that. Candidate_Set :: [Side]Candidates // Enumerated arrays, a nice Odin feature. tile_connections :: proc (c: Cell) -> (ret: Candidate_Set) { // Why not just hardcode it all the way? switch t in c { case Tile: ret[.North] = {4, 5, 6, 7, 12, 13, 14, 15} if .North in t else {0} ret[.East] = {8, 9, 10, 11, 12, 13, 14, 15} if .East in t else {0} ret[.South] = {1, 3, 5, 7, 9, 11, 13, 15} if .South in t else {0} ret[.West] = {2, 3, 6, 7, 10, 11, 14, 15} if .West in t else {0} case Air: ret[.North] = {0, 1, 2, 3, 8, 9, 10, 11} ret[.East] = {0, 1, 2, 3, 4, 5, 6, 7} ret[.South] = {0, 2, 4, 6, 8, 10, 12, 14} ret[.West] = {0, 1, 4, 5, 8, 9, 12, 13} case: ret = ~{} } return }</code></pre> This is nice, mainly because the Candidates</code> type, being a bit set, can be AND</code>ed together to quickly eliminate candidates from Cells in the relevant direction. This can be run for each new state in a loop (or maybe just twice) to propagate candidates of collapsed Air</code> Cells. The grid_propagate</code> procedure will have to do a few things. First, it should scan cells</code> for known placements. If there is a known placement, update candidates</code> accordingly. Then scan candidates</code> for any collapsed cells (where there is only one left candidate), and update those in cells</code> accordingly, then repeat. Assertions could be placed if a known placement in cells</code> does not match a collapsed candidates set in candidates</code>. An error would be returned if, during propagation, a candidates</code> cell was found with 0 candidates, indicating an illegal branch reached. After some trial and error and stack overflows and segmentation faults (due to said stack overflows), I settled on this recursive version It is probably not efficient, as it calculates the same thing multiple times, but it is the most straightforward way i could think of. grid_propagate :: proc (grid: ^Grid) -> bool { // Update candidates based on cells for y in 0..<15 do for x in 0..<15 { cell := grid.cells[x][y] tc := tile_connections(cell) // North if y < 14 do grid.candidates[x][y + 1] &= tc[.North] // East if x < 14 do grid.candidates[x + 1][y] &= tc[.East] // South if y > 0 do grid.candidates[x][y - 1] &= tc[.South] // West if x > 0 do grid.candidates[x - 1][y] &= tc[.West] // if cell is a tile, make sure it is not a candidate anywhere else. if tile, ok := cell.(Tile); ok { // Weird hack cand := Candidates{ int(transmute(u8)tile) } // remove it everywhere with broadcast magic grid.candidates -= cand // then add it in the appropriate place grid.candidates[x][y] = cand } } any_collapsed := false // Update cells based on candidates for y in 0..<15 do for x in 0..<15 { cand := grid.candidates[x][y] if cand == {} do return false // illegal state reached // COLLAPSE // obviously only a new collapse if it is not collapsed already if grid.cells[x][y] == nil && card(cand) == 1 do for id in cand { any_collapsed = true // What did actually collapse grid.cells[x][y] = Air{} if id == 0 else transmute(Tile)i8(id) } } success := true if any_collapsed do success = grid_propagate(grid) // recursion!! return success }</code></pre> Then just update grid_init</code> like this grid_init :: proc () -> (ret: Grid) { // Fill the Candidates Grid ret.candidates = ~Candidates{} // Place Blue square ret.cells[7][7] = ~Tile{} grid_propagate(&ret) return ret }</code></pre> And it works, nicely! Place any of the Squares instead of ~Tile{}</code>, and it works correctly. Even propagates Air</code> cells and their neighbors!! Controlled Demolition</h3> The grid_propagate</code> procedure takes of care of collapsing cells with one candidate left. But as soon as right after the initial position, there are still many cells with unresolved state. That's why Controlled Demolition is needed, to force things to collapse, randomly.5</a> The best candidates for Controlled Demolition are the cells with the least amount of candidates that are not already collapsed. That's to say, the least amount of candidates more than 1. Recording these is probably easy. Choosing one randomly, with even chances all over, is, at first impression, not. It is easy to choose arbitrarily, but not necessarily randomly. If a robust backtracking algorithm exists, then arbitrary is probably better than random. But I do not currently plan to backtrack, but instead simply start over whenever an illegal position is encountered. How should one go about this? With so much randomness, the probably most straightforward course of action is to randomize the indices 0-14 three times. The first two randomized arrays are to loop over candidates</code> in a somewhat random order. The third one is to loop over the candidates in the final found cell, and collapse to the first one it matches over. Sounds absolutely boneheaded, but it should work just fine. grid_controlled_demolition :: proc (grid: ^Grid) -> bool { // all the shuffling rows := []int{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 } rand.shuffle(rows) cols := []int{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 } rand.shuffle(cols) ids := []int{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 } rand.shuffle(ids) // target collapse stored here t_x, t_y: int // tracking the minimum min_card := 15 // loop over the grid "randomly" for y in cols do for x in rows { c := card(grid.candidates[x][y]) (c > 1) or_continue // already collapsed if c < min_card { t_x = x t_y = y min_card = c } if min_card == 2 do break // no point in looking further } // loop over the set flags randomly and collapse to the first match. for id in ids do if id in grid.candidates[t_x][t_y] { // COLLAPSE if id == 0 { grid.cells[t_x][t_y] = Air{} grid.candidates[t_x][t_y] = {0} } else { tile := transmute(Tile)i8(id) cand := Candidates{ int(transmute(u8)tile) } grid.cells[t_x][t_y] = tile grid.candidates -= cand grid.candidates[t_x][t_y] = cand } break } // then propagate the new changes, returning the legality result return grid_propagate(grid) }</code></pre> This should be enough? Calling grid_controlled_demolition</code> a couple of times seems to produce the expected result. main :: proc () { grid := grid_init() // crude grid_controlled_demolition(&grid) grid_controlled_demolition(&grid) grid_controlled_demolition(&grid) grid_controlled_demolition(&grid) grid_controlled_demolition(&grid) for row, ri in grid.candidates do for c, ci in row { fmt.println(ri, ci, c) } }</code></pre> Sample output.</summary> 0 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 0 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 1 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 2 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 3 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 4 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 8 Candidates{0, 1, 4, 8, 12} 5 9 Candidates{2, 3, 7, 10, 11} 5 10 Candidates{2, 3, 7, 10, 11} 5 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 5 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 7 Candidates{0, 2, 4, 8, 10, 12} 6 8 Candidates{0} 6 9 Candidates{9} 6 10 Candidates{13} 6 11 Candidates{4, 7, 12} 6 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 6 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 7 Candidates{1, 3, 7, 11} 7 8 Candidates{6} 7 9 Candidates{0} 7 10 Candidates{0} 7 11 Candidates{0, 1, 2, 3, 8, 10, 11} 7 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 7 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 7 Candidates{1, 3, 7, 11} 8 8 Candidates{15} 8 9 Candidates{5} 8 10 Candidates{4, 7} 8 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 8 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 7 Candidates{1, 3, 7, 11} 9 8 Candidates{14} 9 9 Candidates{0} 9 10 Candidates{0, 1, 2, 3, 8, 10, 11} 9 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 9 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 8 Candidates{8, 10, 11, 12} 10 9 Candidates{0, 1, 2, 3, 4, 7} 10 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 10 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 11 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 12 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 13 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 0 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 1 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 2 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 3 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 4 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 5 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 6 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 7 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 8 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 9 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 10 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 11 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 12 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 13 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12} 14 14 Candidates{0, 1, 2, 3, 4, 7, 8, 10, 11, 12}</code></pre></details> Success and Failure</h3> This seems to work fine. But it is still missing one thing: it needs to know when to stop. If it is a successful solution, (where every cell has exactly one candidate), then stop and print it out. If it is a failed solution, try again. Thankfully, this could be expressed in a couple of lines, with the assumption that the grid has collapsed completely if the first element in the grid, as unlikely as it is to be reached, equals {0}</code>, an Air</code> cell.: main :: proc () { grid := grid_init() outer: for { for grid_controlled_demolition(&grid) { if grid.candidates[0][0] == {0} do break outer } grid = grid_init() } grid_print(&grid) // uninteresting code. it is a grid. }</code></pre> Here are two sample results, which appear to be correct. The second result is interesting as it has a floating element. Obviously the code is undeterministic: each time it runs it gives a different result.6</a> // try 1 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |02| | | | | | | | | | | | | |03|07|12| | | | | | | | | | | | | |11|14| | | | | | | | | | | | | | |09|15|04| | | | | | | | | | | | | | |10| | | | | | | | | | | | | |01|05|13|06| | | | | | | | | | | | | | | |08| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | // try 2 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |01|06| |02| | | | | | | | | | | | | |10| |08| | | | | | | | | | | | |03|14| | | | | | | | | | | | | | |11|15|07|05|04| | | | | | | | | | | |09|13|12| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |</code></pre> 3D Cubes here I come!! Packed Gardens</h2> As the solution for the Square Gardens works, it is obvious that simply changing the data structures and the hardcoded values would solve the 3D solution. I will get to that, but first, I want to try a Packed solution first. The Packed solution has a much simpler to state goal: Pack all 64 cubes into a 4x4x4 cube. Obviously, in this version, the Red faces can (actually have to) face each other. Only Red faces are allowed to face outside. In his page about the set</a>, Freeling does actually provide one Packed solution, with a caveat. The caveat here is that the Red Cube must be contained inside, and not on the edge. It is a difficult requirement, but the provided solution achieves it, at the cost of another floating group at the bottom left of the image above. Initialization</h3> Nonetheless, as it is only a 4x4x4 cube, the search space for the solution is much smaller. There are no Air</code> tiles in here: it is packed. Adding the additional requirement of the Red Cube being wholly contained actually simplifies the solution's initialization. Start the Blue cube at (1,1,1)</code> coordinates, and the Red cube at (2,2,2)</code> coordinates. Any other placement of the two Cubes would actually be a mirror image, or a rotation, of this specific arrangement. Here are the types, which, for the most part, are shared between the Packed Gardens and the Hanging Gardens, and the initialization function as described. Side :: enum u8 { North, East, South, West, Top, Bottom } Tile :: distinct bit_set[Side; u8] Cell :: Maybe(Tile) Candidates :: distinct bit_set[0..<64; u64] Grid :: struct { cells: [4][4][4]Cell, candidates: [4][4][4]Candidates, } Candidate_Set :: [Side]Candidates grid_init :: proc () -> (ret: Grid) { // Fill the candidate Grid ret.candidates = ~Candidates{ 0, 63 } // Place Blue and Red cubes respectively ret.cells[1][1][1] = ~Tile{} ret.cells[2][2][2] = Tile{} // Placeholder for now grid_propagate(&ret) return } </code></pre>Cube Edges</h3> Unlike in the Square Gardens, this is not enough. While the candidates for the edges were safely ignored there, (as the the solution is unlikely to reach them), they must be encoded here, because they are integral. Time to hardcode the implementation! This is how the values are generated: generate_consts :: proc () { for side in (~Tile{}) { fmt.printf("%v_CONNECTED :: Candidates{{", side) for i in 0..<64 { tile := transmute(Tile)u8(i) if side in tile do fmt.print(i, ",") } fmt.println("}") fmt.printf("%v_NOT_CONNECTED :: Candidates{{", side) for i in 0..<64 { tile := transmute(Tile)u8(i) if side not_in tile do fmt.print(i, ",") } fmt.println("}") } }</code></pre> And this is the generated code (with light formatting editing). I have no idea if it is correct. The proof of the pudding is in the eating. NORTH_CONNECTED :: Candidates{ 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63 } EAST_CONNECTED :: Candidates{ 2, 3, 6, 7, 10, 11, 14, 15, 18, 19, 22, 23, 26, 27, 30, 31, 34, 35, 38, 39, 42, 43, 46, 47, 50, 51, 54, 55, 58, 59, 62, 63 } SOUTH_CONNECTED :: Candidates{ 4, 5, 6, 7, 12, 13, 14, 15, 20, 21, 22, 23, 28, 29, 30, 31, 36, 37, 38, 39, 44, 45, 46, 47, 52, 53, 54, 55, 60, 61, 62, 63 } WEST_CONNECTED :: Candidates{ 8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31, 40, 41, 42, 43, 44, 45, 46, 47, 56, 57, 58, 59, 60, 61, 62, 63 } TOP_CONNECTED :: Candidates{16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 } BOTTOM_CONNECTED :: Candidates{32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 } NORTH_NOT_CONNECTED :: Candidates{ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62 } EAST_NOT_CONNECTED :: Candidates{ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17, 20, 21, 24, 25, 28, 29, 32, 33, 36, 37, 40, 41, 44, 45, 48, 49, 52, 53, 56, 57, 60, 61 } SOUTH_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 32, 33, 34, 35, 40, 41, 42, 43, 48, 49, 50, 51, 56, 57, 58, 59 } WEST_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23, 32, 33, 34, 35, 36, 37, 38, 39, 48, 49, 50, 51, 52, 53, 54, 55 } TOP_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 } BOTTOM_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 }</code></pre>tile_connections :: proc (c: Cell) -> (ret: Candidate_Set) { switch t in c { case Tile: ret[.North] = .North in t ? SOUTH_CONNECTED : SOUTH_NOT_CONNECTED ret[.East] = .East in t ? WEST_CONNECTED : WEST_NOT_CONNECTED ret[.South] = .South in t ? NORTH_CONNECTED : NORTH_NOT_CONNECTED ret[.West] = .West in t ? EAST_CONNECTED : EAST_NOT_CONNECTED ret[.Top] = .Top in t ? BOTTOM_CONNECTED : BOTTOM_NOT_CONNECTED ret[.Bottom] = .Bottom in t ? TOP_CONNECTED : TOP_NOT_CONNECTED case: ret = ~{} } return }</code></pre> Then edit grid_init</code> to adjust the candidates in the edge cubes accordingly. However, care must be taken as how these values translate there. For example, the westernmost cubes are the ones that do not have a connection on the .West</code> side. Again, I am not currently sure how correct this is. The pudding is yet to be eaten. grid_init :: proc () -> (ret: Grid) { // Fill the candidate Grid ret.candidates = ~Candidates{ 0, 63 } for z in 0..<4 do for y in 0..<4 do for x in 0..<4 { if x == 0 do ret.candidates[x][y][z] &= WEST_NOT_CONNECTED if x == 3 do ret.candidates[x][y][z] &= EAST_NOT_CONNECTED if y == 0 do ret.candidates[x][y][z] &= SOUTH_NOT_CONNECTED if y == 3 do ret.candidates[x][y][z] &= NORTH_NOT_CONNECTED if z == 0 do ret.candidates[x][y][z] &= BOTTOM_NOT_CONNECTED if z == 3 do ret.candidates[x][y][z] &= TOP_NOT_CONNECTED } // Place Blue and Red cubes respectively ret.cells[1][1][1] = ~Tile{} ret.cells[2][2][2] = Tile{} // still a placeholder grid_propagate(&ret) return }</code></pre>Propagation and Controlled Demolition</h3> Applying minor changes from the Square Gardens' grid_propgagate</code> and grid_controlled_demolition</code>, they end up like this. grid_propagate :: proc (grid: ^Grid) -> bool { // Update candidates based on cells for z in 0..<4 do for y in 0..<4 do for x in 0..<4 { cell := grid.cells[x][y][z] tc := tile_connections(cell) if y < 3 do grid.candidates[x][y + 1][z] &= tc[.North] if x < 3 do grid.candidates[x + 1][y][z] &= tc[.East] if y > 0 do grid.candidates[x][y - 1][z] &= tc[.South] if x > 0 do grid.candidates[x - 1][y][z] &= tc[.West] // Top if z < 3 do grid.candidates[x][y][z + 1] &= tc[.Top] // Bottom if z > 0 do grid.candidates[x][y][z - 1] &= tc[.Bottom] if tile, ok := cell.(Tile); ok { // Weird magic cand := Candidates{ int (transmute(u8)tile) } grid.candidates -= cand grid.candidates[x][y][z] = cand } } any_collapsed := false // Update cells based on candidates for z in 0..<4 do for y in 0..<4 do for x in 0..<4 { cand := grid.candidates[x][y][z] if cand == {} do return false if grid.cells[x][y][z] == nil && card(cand) == 1 do for id in cand { any_collapsed = true grid.cells[x][y][z] = transmute(Tile)u8(id) } } success := true if any_collapsed do success = grid_propagate(grid) // recursion!! return success }</code></pre>grid_controlled_demolition :: proc (grid: ^Grid) -> bool { // all the shuffling rows := []int{ 0, 1, 2, 3 } rand.shuffle(rows) cols := []int{ 0, 1, 2, 3 } rand.shuffle(cols) verts := []int{ 0, 1, 2, 3 } rand.shuffle(verts) ids := []int{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63} rand.shuffle(ids) t_x, t_y, t_z: int min_card := max(int) for z in verts do for y in cols do for x in rows { c := card(grid.candidates[x][y][z]) (c > 1) or_continue if c < min_card { t_x = x t_y = y t_z = z min_card = c } if min_card == 2 do break } for id in ids do if id in grid.candidates[t_x][t_y][t_z] { tile := transmute(Tile)i8(id) cand := Candidates{ id } grid.cells[t_x][t_y][t_z] = tile grid.candidates -= cand grid.candidates[t_x][t_y][t_z] = cand break } return grid_propagate(grid) }</code></pre> And this does it. Figuring out how to ascertain how every cell in the grid has only one candidate took a bit of finessing and experimenting with Odin's core library slice.all_in_proc</code>, but I ended up doing it manually. main :: proc () { grid := grid_init() outer: for { inner: for grid_controlled_demolition(&grid) { for z in 0..<4 do for y in 0..<4 do for x in 0..<4 { if card(grid.candidates[x][y][z]) != 1 do continue inner } break outer } grid = grid_init() } grid_draw(grid) // <-- this one is a bit more interesting. }</code></pre> This works, and thankfully halts. However, the execution time, in the few times I ran it, varied anything between 3 seconds and one minute. This obviously depends on how lucky the random seed is. Using threads would help to get a result faster. Printing the solution naïvely prints a bunch of numbers, with no repeats (as far I could tell), but to verify that the solution is correct it needs to be outputted into a readable format. THe "easy" solution is to put together a dictionary of images and make a collage, much like the image at the top of the section, but that is not very interesting. Solution Output</h3> Simply printing out the resultant grid to console is not a very useful output. With the Square Gardens, it is possible to verify the correctness of the result by simply checking, visually, that each number/square is the only one in the grid that has neighbors in a specific configuration. In other words, look at this grid given from before (with pointless empty spaces removed and the Red Square 00</code> added): | | | | | | | | |00| | |02| | | | |03|07|12| | | | |11|14| | | | | |09|15|04| | | | | |10| | | | |01|05|13|06| | | | | | |08| | | | | | | | |</code></pre> It is easy to verify that, for example, 08</code> is literally the only tile that has only one connection to the top (and whether in code that's North</code> or South</code> is irrelevant to the correctness of the solution.) 14</code>, for example, is the only tile that has only three connections to the top, left, and bottom, and so on for all of them. For the packed solution, it is harder to verify that over text output. One solution would be to create a little image for every cube, (as Freeling's provided solution does) and pack them together. The other solution is to render a 3D image of the solution and visually inspect it that way. Looking over Odin's vendor</code> libraries, and asking around for ideas how to do that, I could not figure out how to draw on an image canvas (the way the imageproc</code></a> lets me do in Rust.) But there are a few bindings to different graphics libraries, and word on the street was that raylib</code></a> is the simplest solution for this sort of thing. I spent a day playing around with raylib</code>'s tutorials and asking for help on the Discord. The API is ... weirder than I am used to, to say the least. But it looks simple enough. To draw a window with a render scene, the code is the following: import rl "vendor:raylib" grid_draw :: proc (grid: Grid) { SCREEN_WIDTH :: 400 SCREEN_HEIGHT :: 200 // raylib must start with this. rl.InitWindow(SCREEN_WIDTH, SCREEN_HEIGHT, "") // and end with this. defer rl.CloseWindow() camera := rl.Camera3D{ position = ({10.0, -5.0, 5.0} + 2.5), target = {2.5, 2.5, 2.5}, up = {0.0, 1.0, 0.0}, fovy = 50, projection = .ORTHOGRAPHIC } for !rl.WindowShouldClose() { rl.UpdateCamera(&camera, .THIRD_PERSON) // This honestly weird pattern is how everything in raylib must work rl.BeginDrawing() defer rl.EndDrawing() rl.ClearBackground(rl.WHITE) rl.BeginMode3D(camera) defer rl.EndMode3D() // draw 3D things here. } }</code></pre> Just to test things out, I will try drawing each cube with a different shade based on how many connections it has. The Blue cube shall be blue and the Red cube shall be red, and the other cubes will be different shades of purple. for z in 0..<4 do for y in 0..<4 do for x in 0..< 4 { tile := grid.cells[x][y][z].(Tile) or_continue grade := u8(card(tile)) b := (max(u8) / 6) * grade r := max(u8) - b rl.DrawCubeV({f32(x), f32(y), f32(z)}, 1.0, {r, 0, b, 255}) }</code></pre> And now: the moment of truth odin run .</code> I have no idea what's going on here. Everything is mushed together and the screen is tiny. Upping the screen's resolution (to 800, 400</code>), multiplying the position by 2.0</code> would be a start, and bringing the camera closer would be a decent start.7</a> Ah .. much better already. This actually rotates (but not zooms) with the mouse movement, thanks to that .THIRD_PERSON</code> setting up there. Time to draw the lines, and better organize the code while I am at it. Here is the full function. (I do not actually expect this to be correct.) grid_draw :: proc(grid: Grid) { SCREEN_WIDTH :: 1200 SCREEN_HEIGHT :: 600 CUBE_DISTANCE :: 3.0 CENTER_POINT :: 2.5 * CUBE_DISTANCE rl.InitWindow(SCREEN_WIDTH, SCREEN_HEIGHT, "") defer rl.CloseWindow() camera := rl.Camera3D { position = ({10.0, -5.0, 5.0} + CENTER_POINT), target = CENTER_POINT, up = {0.0, 1.0, 0.0}, fovy = 30, projection = .ORTHOGRAPHIC, } for !rl.WindowShouldClose() { rl.UpdateCamera(&camera, .THIRD_PERSON) rl.BeginDrawing() defer rl.EndDrawing() rl.ClearBackground(rl.WHITE) rl.BeginMode3D(camera) defer rl.EndMode3D() for z in 0 ..< 4 do for y in 0 ..< 4 do for x in 0 ..< 4 { tile := grid.cells[x][y][z].(Tile) or_continue grade := u8(card(tile)) pos := rl.Vector3{f32(x), f32(y), f32(z)} * CUBE_DISTANCE size := rl.Vector3{ 1.0, 1.0, 1.0 } b := (max(u8) / 6) * grade r := max(u8) - b color := rl.Color{ r, 0, b, 255} rl.DrawCubeV( pos, size, color ) if .North in tile do rl.DrawLine3D(pos, pos + {0, CUBE_DISTANCE / 2, 0}, rl.BLACK) if .South in tile do rl.DrawLine3D(pos, pos + {0, -CUBE_DISTANCE / 2, 0}, rl.BLACK) if .East in tile do rl.DrawLine3D(pos, pos + { CUBE_DISTANCE / 2, 0, 0}, rl.BLACK) if .West in tile do rl.DrawLine3D(pos, pos + {-CUBE_DISTANCE / 2, 0, 0}, rl.BLACK) if .Top in tile do rl.DrawLine3D(pos, pos + {0, 0, CUBE_DISTANCE / 2}, rl.BLACK) if .Bottom in tile do rl.DrawLine3D(pos, pos + {0, 0, -CUBE_DISTANCE / 2}, rl.BLACK) } } } </code></pre> This looks .. ok? half decent? Rotating it around does not show any mistakes. The centering is off, however. And due to the Orthographic projection and camera work I cannot zoom in to see how everything connects in more detail. Faster Iteration of Visuals</h3> Also, frankly, I am getting tired of waiting for the solution to solve before the render show up, so I am changing the code a bit to, maybe, see the cube solve it in real time. This makes the solver much slower, but it makes for a fun animation. SCREEN_WIDTH :: 800 SCREEN_HEIGHT :: 800 CUBE_DISTANCE :: 3.0 CENTER_POINT :: 1.5 * CUBE_DISTANCE main :: proc () { rl.InitWindow(SCREEN_WIDTH, SCREEN_HEIGHT, "") defer rl.CloseWindow() rl.SetTargetFPS(60) camera := rl.Camera3D { position = ({10.0, -5.0, 5.0} + CENTER_POINT), target = CENTER_POINT, up = {0.0, 1.0, 0.0}, fovy = 30, projection = .ORTHOGRAPHIC, } grid := grid_init() solved := false paused := false for !rl.WindowShouldClose() { rl.UpdateCamera(&camera, .THIRD_PERSON) if rl.IsKeyPressed(.SPACE) do paused ~= true { // Draw current state rl.BeginDrawing() defer rl.EndDrawing() rl.ClearBackground(rl.WHITE) rl.BeginMode3D(camera) defer rl.EndMode3D() for z in 0 ..< 4 do for y in 0 ..< 4 do for x in 0 ..< 4 { tile := grid.cells[x][y][z].(Tile) or_continue grade := u8(card(tile)) pos := rl.Vector3{f32(x), f32(y), f32(z)} * CUBE_DISTANCE size := rl.Vector3{ 1.0, 1.0, 1.0 } b := (max(u8) / 6) * grade r := max(u8) - b color := rl.Color{ r, 0, b, 255} rl.DrawCubeV( pos, size, color ) if .North in tile do rl.DrawLine3D(pos, pos + {0, CUBE_DISTANCE / 2, 0}, rl.BLACK) if .South in tile do rl.DrawLine3D(pos, pos + {0, -CUBE_DISTANCE / 2, 0}, rl.BLACK) if .East in tile do rl.DrawLine3D(pos, pos + { CUBE_DISTANCE / 2, 0, 0}, rl.BLACK) if .West in tile do rl.DrawLine3D(pos, pos + {-CUBE_DISTANCE / 2, 0, 0}, rl.BLACK) if .Top in tile do rl.DrawLine3D(pos, pos + {0, 0, CUBE_DISTANCE / 2}, rl.BLACK) if .Bottom in tile do rl.DrawLine3D(pos, pos + {0, 0, -CUBE_DISTANCE / 2}, rl.BLACK) } } if !paused && !solved { if grid_controlled_demolition(&grid) { solved = true for z in 0..<4 do for y in 0..<4 do for x in 0..<4 { if card(grid.candidates[x][y][z]) != 1 { solved = false break } } } else { grid = grid_init() solved = false } } } }</code></pre> </video> Well this is a lot more pleasant to look at, isn't it? One nicety: it pauses when the Space bar is pressed, easier to inspect the connections. It will find a solution in a million years, but until that happens it would be easier to iterate the presentation's colors and distances. Miracolously, howver, no matter when I pause it, the connections between adaject cubes are always correct. There are no cubes that are next to each other with one line hanging midways between them. Every high five is reciprocated. This is a fancy screenshot for an attempted solution. Final Visuals</h3> After a bit of trial and error and experimenting with different colors and distances and camera positions, I ended up with these constants and draw functions. SCREEN_WIDTH :: 800 SCREEN_HEIGHT :: 800 CUBE_DISTANCE_X :: 6.0 CUBE_DISTANCE_Y :: 6.0 CUBE_DISTANCE_Z :: 2.5 CENTER_POINT :: [3]f32 { 1.5 * CUBE_DISTANCE_X, 1.5 * CUBE_DISTANCE_Y, 1.5 * CUBE_DISTANCE_Z } main :: proc () { // boring boilerplate camera := rl.Camera3D { position = ({-7, -7, 15} + CENTER_POINT), target = CENTER_POINT, up = {0.0, 0.0, 1.0}, fovy = 35, projection = .ORTHOGRAPHIC, } // other boring boilerplate rl.ClearBackground(rl.Color{ 240, 240, 240, 255 }) // you know pos := rl.Vector3{ f32(x) * CUBE_DISTANCE_X, f32(y) * CUBE_DISTANCE_Y, f32(z) * CUBE_DISTANCE_Z, } // more of it rl.DrawCubeV(pos, size, color) rl.DrawCubeWiresV(pos, size, rl.BLACK) if .North in tile { end_pos := pos + {0, CUBE_DISTANCE_Y / 2, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.ORANGE) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .South in tile { end_pos := pos + {0, -CUBE_DISTANCE_Y / 2, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.ORANGE) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .East in tile { end_pos := pos + { CUBE_DISTANCE_X / 2, 0, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.DARKGREEN) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .West in tile { end_pos := pos + {-CUBE_DISTANCE_X / 2, 0, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.DARKGREEN) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .Top in tile { end_pos := pos + {0, 0, CUBE_DISTANCE_Z / 2} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.YELLOW) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .Bottom in tile { end_pos := pos + {0, 0, -CUBE_DISTANCE_Z / 2} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.YELLOW) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } // end of proc boilerplate }</code></pre> Producing an Image</h3> Well, now that apparently all the logic and visuals are top notch and as clear as they can be, I do not actually want a visual of the solver solving and taking for ever. I only want a final image. So I will revert main</code> to what it was before, and adjust grid_draw</code> so it only saves an image to disk, rather than opening a window and waiting for me to take a screenshot. The code is mostly the same as before, only without starting a loop. raylib</code> has functions to render a texture and save it as an image without third party libraries, so I just use those. This is the final grid_draw</code> and its resultant image grid_draw :: proc(grid: Grid) { rl.SetConfigFlags({.WINDOW_HIDDEN}) rl.InitWindow(SCREEN_WIDTH, SCREEN_HEIGHT, "") defer rl.CloseWindow() camera := rl.Camera3D { position = ({7, 7, -15} + CENTER_POINT), target = CENTER_POINT, up = {0.0, 0.0, 1.0}, fovy = 35, projection = .ORTHOGRAPHIC, } txtr := rl.LoadRenderTexture(SCREEN_WIDTH, SCREEN_HEIGHT) { // Drawing here rl.BeginTextureMode(txtr) defer rl.EndTextureMode() rl.BeginDrawing() defer rl.EndDrawing() rl.ClearBackground(rl.Color{ 240, 240, 240, 255 }) rl.BeginMode3D(camera) defer rl.EndMode3D() for z in 0 ..< 4 do for y in 0 ..< 4 do for x in 0 ..< 4 { tile := grid.cells[x][y][z].(Tile) or_continue grade := u8(card(tile)) pos := rl.Vector3{ f32(x) * CUBE_DISTANCE_X, f32(y) * CUBE_DISTANCE_Y, f32(z) * CUBE_DISTANCE_Z } size := rl.Vector3{1.0, 1.0, 1.0} b := (max(u8) / 6) * grade r := max(u8) - b color := rl.Color{r, 0, b, 255} rl.DrawCubeV(pos, size, color) rl.DrawCubeWiresV(pos, size, rl.BLACK) if .North in tile { end_pos := pos + {0, CUBE_DISTANCE_Y / 2, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.ORANGE) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .South in tile { end_pos := pos + {0, -CUBE_DISTANCE_Y / 2, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.ORANGE) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .East in tile { end_pos := pos + {CUBE_DISTANCE_X / 2, 0, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.DARKGREEN) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .West in tile { end_pos := pos + {-CUBE_DISTANCE_X / 2, 0, 0} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.DARKGREEN) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .Top in tile { end_pos := pos + {0, 0, CUBE_DISTANCE_Z / 2} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.YELLOW) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } if .Bottom in tile { end_pos := pos + {0, 0, -CUBE_DISTANCE_Z / 2} rl.DrawCylinderEx(pos, end_pos, 0.1, 0.1, 12, rl.YELLOW) rl.DrawCylinderWiresEx(pos, end_pos, 0.1, 0.1, 12, rl.BLACK) } } } img := rl.LoadImageFromTexture(txtr.texture) rl.ExportImage(img, "result.png") }</code></pre> Hanging Gardens</h2> With the Square Gardens and the Packed Gardens done, time to return to the original problem. Much of the different pieces of the solution are already in place, and it is just a question of assembling them together. The main problem left is how to determine the proper size of the grid. It needs to be big enough to potentially accept any variation of the problem, but small enough that it does not cause a stack overflow. When I was experimenting with it, I tried a grid of size 31x31x31 and the Odin compiler warnned^{8</a> me that it is so big it might cause said overflow. Or might just put it on the heap and let the OS take care of cleaning it from memory.} Frankenstein's Monster</h3> This is the full program, with a helpful GRID_SIZE</code> constant that allowed me to iterate and debug a bit with a smaller grid size. package hanging_gardens import "core:fmt" import "core:math/rand" GRID_SIZE :: 31 MID_POINT :: (GRID_SIZE - 1) / 2 Side :: enum u8 { North, East, South, West, Top, Bottom } Tile :: distinct bit_set[Side; u8] Air :: distinct struct {} Cell :: union { Air, Tile } Candidates :: distinct bit_set[0..<64; u64] Grid :: struct { cells: [GRID_SIZE][GRID_SIZE][GRID_SIZE]Cell, candidates: [GRID_SIZE][GRID_SIZE][GRID_SIZE]Candidates, } Candidate_Set :: [Side]Candidates NORTH_CONNECTED :: Candidates { 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63 } EAST_CONNECTED :: Candidates { 2, 3, 6, 7, 10, 11, 14, 15, 18, 19, 22, 23, 26, 27, 30, 31, 34, 35, 38, 39, 42, 43, 46, 47, 50, 51, 54, 55, 58, 59, 62, 63 } SOUTH_CONNECTED :: Candidates { 4, 5, 6, 7, 12, 13, 14, 15, 20, 21, 22, 23, 28, 29, 30, 31, 36, 37, 38, 39, 44, 45, 46, 47, 52, 53, 54, 55, 60, 61, 62, 63 } WEST_CONNECTED :: Candidates { 8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31, 40, 41, 42, 43, 44, 45, 46, 47, 56, 57, 58, 59, 60, 61, 62, 63 } TOP_CONNECTED :: Candidates {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 } BOTTOM_CONNECTED :: Candidates {32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 } NORTH_NOT_CONNECTED :: Candidates{ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62 } EAST_NOT_CONNECTED :: Candidates{ 0, 1, 4, 5, 8, 9, 12, 13, 16, 17, 20, 21, 24, 25, 28, 29, 32, 33, 36, 37, 40, 41, 44, 45, 48, 49, 52, 53, 56, 57, 60, 61 } SOUTH_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 32, 33, 34, 35, 40, 41, 42, 43, 48, 49, 50, 51, 56, 57, 58, 59 } WEST_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23, 32, 33, 34, 35, 36, 37, 38, 39, 48, 49, 50, 51, 52, 53, 54, 55 } TOP_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 } BOTTOM_NOT_CONNECTED :: Candidates{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 } tile_connections :: proc (c: Cell) -> (ret: Candidate_Set) { switch t in c { case Tile: ret[.North] = SOUTH_CONNECTED if .North in t else {0} ret[.East] = WEST_CONNECTED if .East in t else {0} ret[.South] = NORTH_CONNECTED if .South in t else {0} ret[.West] = EAST_CONNECTED if .West in t else {0} ret[.Top] = BOTTOM_CONNECTED if .Top in t else {0} ret[.Bottom] = TOP_CONNECTED if .Bottom in t else {0} case Air: ret[.North] = SOUTH_NOT_CONNECTED ret[.East] = WEST_NOT_CONNECTED ret[.South] = NORTH_NOT_CONNECTED ret[.West] = EAST_NOT_CONNECTED ret[.Top] = BOTTOM_NOT_CONNECTED ret[.Bottom] = TOP_NOT_CONNECTED case: ret = ~{} } return } grid_propagate :: proc(grid: ^Grid) -> bool { // Update candidates based on cells for z in 0..<GRID_SIZE do for y in 0..<GRID_SIZE do for x in 0..<GRID_SIZE { cell := grid.cells[x][y][z] tc := tile_connections(cell) // North if y < GRID_SIZE - 1 do grid.candidates[x][y + 1][z] &= tc[.North] // East if x < GRID_SIZE - 1 do grid.candidates[x + 1][y][z] &= tc[.East] // Top if z < GRID_SIZE - 1 do grid.candidates[x][y][z + 1] &= tc[.Top] // South if y > 0 do grid.candidates[x][y - 1][z] &= tc[.South] // West if x > 0 do grid.candidates[x - 1][y][z] &= tc[.West] // Bottom if z > 0 do grid.candidates[x][y][z - 1] &= tc[.Bottom] // if cell is a tile, make sure it is not a candidate anywhere else. if tile, ok := cell.(Tile); ok { cand := Candidates{ int ( transmute(u8)tile ) } grid.candidates -= cand grid.candidates[x][y][z] = cand } } any_collapsed := false // Update cells based on candidates for z in 0..<GRID_SIZE do for y in 0..<GRID_SIZE do for x in 0..<GRID_SIZE { cand := grid.candidates[x][y][z] if cand == {} do return false // illegal state reached // COLLAPSE // obviously only a new collapse if it isnt collapsed already if grid.cells[x][y][z] == nil && card(cand) == 1 do for id in cand { any_collapsed = true // What did actually collapse grid.cells[x][y][z] = Air{} if id == 0 else transmute(Tile)i8(id) } } success := true if any_collapsed do success = grid_propagate(grid) return success } grid_init :: proc (grid: ^Grid) { // different grid_init from before to reuse memory grid.cells = {} grid.cells[MID_POINT][MID_POINT][MID_POINT] = ~Tile{} grid.candidates = ~Candidates{} grid_propagate(grid) return } grid_controlled_demolition :: proc(grid: ^Grid) -> bool { // all the shuffling rows := [GRID_SIZE]int{} for i in 0..<GRID_SIZE do rows[i] = i cols := rows verts := cols rand.shuffle(rows[:]) rand.shuffle(cols[:]) rand.shuffle(verts[:]) ids := []int{ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63 } rand.shuffle(ids) // target collapse stored here t_x, t_y, t_z: int // tracking the minimum min_card := max(int) for z in verts do for y in cols do for x in rows { c := card(grid.candidates[x][y][z]) (c > 1) or_continue // already collapsed if c < min_card { t_x = x t_y = y t_z = z min_card = c } if min_card == 2 do break // no point in looking further in this case. } for id in ids do if id in grid.candidates[t_x][t_y][t_z] { // COLLAPSE if id == 0 { grid.cells[t_x][t_y][t_z] = Air{} grid.candidates[t_x][t_y][t_z] = {0} } else { tile := transmute(Tile)i8(id) cand := Candidates{ id } grid.cells[t_x][t_y][t_z] = tile grid.candidates -= cand grid.candidates[t_x][t_y][t_z] = cand } break } return grid_propagate(grid) } main :: proc () { grid := new(Grid) grid_init(grid) outer: for i in 1 ..< max(int) { for grid_controlled_demolition(grid) { if grid.candidates[0][0][0] == {0} do break outer } grid_init(grid) if i % 10 == 0 do fmt.eprint("try", i, "\r") } for z in 0..<GRID_SIZE do for y in 0..<GRID_SIZE do for x in 0..<GRID_SIZE { fmt.println(x, y, z, grid.candidates[x][y][z]) } }</code></pre> Running this takes forever, by the way. Even when compiling it optimized for speed with odin build . -o:speed</code>, the first run at the solution took more than 290,000 iterations and more than 18 minutes before I gave up on it, and decided to change allocations a bit. Doubt rised in my heart whether this halts at all, or if it has a nasty bug that is difficult to find due to, well, the long runtime. Threading will probably not make this faster. The longest time I ran it, it went for 25,771,590 iterations and around 27 hours of usertime before I shut it off. Maybe, just maybe, completely random attempts are the wrong approach. Maybe I actually need to implement proper backtracking to ensure that the same tree is not being barked multiple times. I have spent a few days mulling it over, and the simplest way to implement is to have a dynamic array in Grid</code> to save every state where an arbitrary (not random) choice was made, in addition to the index of the choice. And whenever reaching an illegal state, the grid is reset to the previous state, and the index is incremented. Should the index reach a certain value (10 or whatever), then the last state is discarded and the index is incremented on the one before it. Grid Enhancements</h3> So, first things first, here is the new Grid</code> struct, with a few type aliases to lighten the mood. Cell_Grid :: [GRID_SIZE][GRID_SIZE][GRID_SIZE]Cell Candidate_Grid :: [GRID_SIZE][GRID_SIZE][GRID_SIZE]Candidates Demolition_Point :: struct { grid: Cell_Grid, idx: int, } Grid :: struct { cells: Cell_Grid, candidates: Candidate_Grid, demolitions: [dynamic]Demolition_Point }</code></pre> Having the dynamic array there should, in theory, be safe. The Grid is initialized once, And whenever the controlled demolition fails, the index of the last item in demolitions</code> is updated. Or, if it exceeds some threshold (preferably exhausting all the candidates in a cell), the last entry is removed and the second to last entry increments its index instead. Now, for the procedures used, grid_propagate</code> should probably be the same. While grid_init</code> and grid_controlled_demolition</code> should perhaps be overhauled. A potential enhancement, to avoid the algorithm being stuck on the same start every time, is to reuse the shuffling above in initialization. Shuffling the IDs for every new Demolition_Point</code> would be enough, unlike shuffling rows and columns. the reason for that is, if the program exhausted every candidate from a cell, then there is no point in trying other cells from the same state. This is what that would look like. Demolition_Point :: struct { grid: Cell_Grid, idx: int, ids_shuffle: [64]int } // ... grid_init :: proc (grid: ^Grid) { grid.cells[MID_POINT][MID_POINT][MID_POINT] = ~Tile{} grid.candidates = ~Candidates{} grid_propagate(grid) start_point := Demolition_Point { grid = grid.cells, idx = 0, ids_shuffle = IDS, // IDS is numbers from 0 to 63. } rand.shuffle(start_point.ids_shuffle[:]) append(&grid.demolitions, start_point) return }</code></pre>Demolition Revision</h3> Previously, grid_init</code> was used to reset the state after every failed attempt. This time a different procedure is needed, which may be called grid_backtrack</code>, or increment</code>, who knows. This proc would only be called for a failed controlled demolition. Demolition can fail for two reasons: either the candidate chosen forced an illegal state, or all candidates from the lowest cell were exhausted. This call for an enum</code>! grid_controlled_demolition</code> can return this enum and based on whivh grid_backtrack</code> decides what to do. Demolition_Result :: enum { Success, Exhausted_Candidates, Illegal_State }</code></pre> And this would be the new version of grid_controlled_demolition</code>, with less shuffling and small changes: grid_controlled_demolition :: proc(grid: ^Grid) -> Demolition_Result { t_x, t_y, t_z: int min_card := max(int) for z in RANKS do for y in RANKS do for x in RANKS { // RANKS is numbers 0 to 30 c := card(grid.candidates[x][y][z]) (c > 1) or_continue if c < min_card { t_x = x t_y = y t_z = z min_card = c } if min_card == 2 do break } demo_pt := grid.demolitions[len(grid.demolitions) - 1] // should never be empty counter := 0 exhausted := true for id in demo_pt.ids_shuffle do if id in grid.candidates[t_x][t_y][t_z] { if counter < demo_pt.idx { // Skip till idx is reached. counter += 1 continue } // COLLAPSE if id == 0 { grid.cells[t_x][t_y][t_z] = Air{} grid.candidates[t_x][t_y][t_z] = {0} } else { tile := transmute(Tile)i8(id) cand := Candidates{ id } grid.cells[t_x][t_y][t_z] = tile grid.candidates -= cand grid.candidates[t_x][t_y][t_z] = cand } exhausted = false break } // Here is the interesting part. if exhausted do return .Exhausted_Candidates ok := grid_propagate(grid) return .Success if ok else .Illegal_State }</code></pre>The Main Loop</h3> Before figuring out how the supposed grid_backtrack</code> looks like, it might help to construct how the loop looks. With a side of Odin flavoured golf: main :: proc () { grid := new(Grid) grid_init(grid) // inner tag is only for readability outer: for i in 1 ..< max(int) do inner: for { result := grid_controlled_demolition(grid) switch result { case .Success: if grid.candidates[0][0][0] == {0} do break outer demo_pt := Demolition_Point { grid = grid.cells, idx = 0, ids_shuffle = IDS } rand.shuffle(demo_pt.ids_shuffle[:]) append(&grid.demolitions, demo_pt) case .Exhausted_Candidates, .Illegal_State: grid_backtrack(grid, result) if i % 10 == 0 do fmt.eprint("try", i, "\r") // debug land break inner } } grid_draw(grid) // <-- later }</code></pre>The Step Back</h3> For backtracking, what is done depends on the error returned. If it is an illegal state, just increment the index and restart. If it is exhausted candidates, remove the last state, then increment the index of the previous one. This is the initial idea: grid_backtrack :: proc (grid: ^Grid, result: Result) { switch result { case .Success: unreachable() case .Exhausted_Candidates: pop(&grid.demolitions) case .Illegal_State: // nothing } last := slice.last_ptr(grid.demolitions[:]) last.idx += 1 grid.cells = last.grid grid_propagate(grid) }</code></pre> Unfortunately, putting all these together and running it, quickly hits a segfault. I could not figure out why at first, but then peppering a few fmt.eprint</code> around, I realized that the state of grid.candidates</code> wasn't being reset properly between iterations. So any changes made to it in grid_propagate</code> would persist. I fixed that by adding a line just before the call to grid_propagate</code> in grid_backtrack</code>: grid.cells = last.grid grid.candidates = ~Candidates{} // <-- this one grid_propagate(grid)</code></pre> Ok, not segfaulting. Good news. The peppered around fmt.eprint</code>s show behaviour that make sense. Also, similarly to the reset the world behaviour experimented with at first, this is still taking a long time to run, with no idea whether it halts or not. Building it with -o:speed</code> as done before, and running it overnight might give the needed answer. Out of Memory</h3> I woke up to this: I spent a few days trying different things out and seeing what is causing the ballooning in memory. For whatever reason, the program seemed to be allocating a lot more memory than I expected, and the dynamic array of demolitions</code> is ballooning uncontrollably. In theory, there would be 31x31x31 successful demolitions at most. I tested this by using Odin's core library's Small_Array</code>, which is a "dynamic" array with a fixed capacity, giving it 31x31x31 size, and it quickly filled up and panicked as it attempted appending to a full array. This indicates, to me, a logic error somewhere. I tried different variations of the main loop, including inlining grid_backtrack</code>. I removed the ID shuffling per Demolition_Point</code>. I made the Grid</code> smaller. I tried not shuffling IDS</code>. All of them either ran into the same memory issue or just .. kept running.^{9</a> I am at a complete loss of what the issue might be. Clearly this approach to backtracking is not working. I am honestly out of spoons thinking about this.} Giving Up and Lessons Learned</h2> To be honest, it has been almost two months since I started this article, and I am tired of iterating on the problem. Frankenstein's Monster</a> works in theory, given infintie time, and for now this is enough for me. Maybe one day I will go back to figuring how to get to a slution in human time. The part I enjoyed the most was iterating on the graphics with raylib</code>. The library is really easy to use, if not as flexible as I hoped. Maybe for the next project I will do some sort of game, or just use it as generative art. I should watch more Coding Train videos. The original tile set inspired a number of games by Christian Freeling. One of them is Dominions, for which I wrote an implementation in Odin</a>. ↩</a> </li> This is what Freeling calls a "Transcendental solution". ↩</a> </li> This might sound heretical in Sudoku, but in well constructed Sudoku puzzles there is always at least one cell with exactly one candidate in any given puzzle state thought solving. If ever there are not any, either it is not a well constructed puzzle, or the solver is unaware of the Ring of Phistomefel. ↩</a> </li> Nor does it into account the edges of the Grid (where only Red faces may, well, face.) However, the edges of the Grid are unlikely to be reached, so I will ignore that for now. ↩</a> </li> Yes I am totally mixing metaphors right now. Way beyond Quantum Mechanics and well into AEC. Roll with it. ↩</a> </li> More interesting variations can include a search for only symmetrical solutions. This would require additional, more complex constraints in candidate placements. Nothing that cannot be dome but a whole lot more code. ↩</a> </li> Iterating on this would be a lot more pleasant if generating the solution was faster. Maybe I should not have skipped that distributing work across threads step. ↩</a> </li> "Declaration of 'grid' may cause a stack overflow due to its type 'Grid' having a size of 297912 bytes</code>" ↩</a> </li> There was one exception where it terminated in 9 seconds. I realized that 0</code> being the first ID it always tests makes it fill everything with Air</code> tiles. ↩</a> </li> </ol> </section> My Ideal Code Editor 2024-09-24T00:00:00+00:00 I have been doing some thinking about what my ideal code editor should be. I currently use VSCode, both to edit this site and to play around with code. However, VSCode is actually too powerful for my needs. And it lacks one crucial feature (that's pretty much lacking from almost every text/code editor outside of TextEdit and Notepad): proper Arabic language and Right-to-Left (RTL) support.1</a> I tried several editors (honestly none more than 5 minutes) and I always go back to VSCode because of how comfortable it is. I tried Helix</a>, but could not live in the Terminal or modal editing. Tried Zed .. and it honestly just felt weird. Looked at CodeEdit</a>, which is a really interesting project that brings the Xcode aesthetic down to code editor level, but it is still a work in progress, and it still doesn't deal with RTL properly.2</a> This post is more for my own records than anything. Whenever I start on the great My Ideal Code Editor (Code name: MICE, or KTB, or whatever), I have a nice feature list to stick to. Has Not</h2> MICE lacks the following features: An integrated Terminal. My usual terminal works just fine.</li> An integrated Git GUI. I can learn a few Git commands on the terminal.</li> An integrated remote connection. I am a hobbyist who works solo.</li> Built-in support for any language. Focus merely on the text features.</li> Any sort of preview or built-in compiler. With a possible concession to Markdown. (I really like VSCode's Markdown preview pane)</li> </ul> Has</h2> However, MICE has these nice features: High Level Features:</h3> Native UI. I am on a Mac. I would like a native Mac UI.</li> Arabic text support. If I have an Arabic paragraph inside of an English document, I would like said paragraph to be right-aligned.</li> Tabs. Directory pane.</li> A command line tool to open directories, like VSCode's code .</code> command. No idea currently on what else it needs to do.</li> Rely on the the system's spell check in txt</code> and md</code> files.</li> </ul> Coding Features</h3> LSP support: To provide IDE capabilities for any language. Intellisense, Go To Definition, Rename Symbol, Inlay Hints, the works.</li> Tree-Sitter support: Syntax Highlighting (with perhaps LSP fallback), Multi-Cursor stuff, Code folding, Indentation. (Maybe through WASM plugins)</li> Syntax Highlighting can have multiple typefaces in it</a>.</li> Smart editor features like auto-closing parentheses and select-and-type-a-brace-to-surround-block.</li> While no language support is Built-In, the default installation should come with the web's defaults: HTML, CSS, JavaScript, and some useful additions like Sass, Markdown, and some Templating language like Tera</a>, the one used for this site.</li> Easy to install "packages" for additional languages.</li> </ul> What else? I don't know. I might add to this page once I think of anything or start any project. There are Markdown editors that support Arabic decently enough, such as Bear</a> and Obsidian. But those do not handle HTML and Sass and what goes into editing a website, even ones as simple as this. ↩</a> </li> 2024-09-26: I also came across CotEditor</a>. Looks nice and has good CJK support, with vertical text. ↩</a> </li> </ol> </section> The Garden of Odin 2024-09-01T00:00:00+00:00 In an escape from the worries of life, I have been reading recently into the Odin programming language</a>. I wanted to learn something that is closer to the metal than Rust, and Odin seems nice. For a project to do with the language, I figured I would build a library for the abstract board game Dominions, by Christian Freeling</a>. (Sensei's Library link</a>). The game can be described as a Go variant with distinct pieces, as opposed to Go itself where every "piece" is identical. The game is weird. It is much less known than Freeling's other games like Havannah (for which I wrote an implementation for in Rust</a>) and Grand Chess (which I designed and built a physical set for). Even Freeling himself does not put much stock into it, and is more interested in the tile set itself (which he calls the China Labyrinth</a>, even though they have nothing to do with China) than the game. He made other games with the tile set which you can find by browsing his site. In code and my handwritten notes, I call the tile set Bustan: Arabic for Garden (hence the name of this article). Variety, branches, paths, and fences all make gardens seem like an appropriate metaphor for what is going on. This is the sample game Freeling posted on his website: I am writing this post as a way to organize my thoughts on how to represent the game in code. It is a semi organized, semi-chronologically-sorted brain dump: I am writing it as I iterate over the code. I will be talking mostly about how to represent the game in code form, leaving the rules for later. The end result can be found in the git history of the bustan</code> repository</a>. Tiles</h2> The most intriguing feature of Dominions, for me, is the tiles. There are many tile sets and game piece sets in the world of games: chess, dominoes, mahjong, playing cards, tarot. Dominions tiles are such a set. Each tile defines how it connects to its neighbors. They do not rotate, but they can be flipped to change controller (which is how capture happens in the game). A full set of tiles, pictured above, has 64 shapes. A clever Id/numbering scheme for the tiles is described by Freeling. Each side is assigned a power of two. So each piece unique id is simply the sum of which sides it connects. Binary system! Each player starts the game with the full set of tiles, minus the Blank, the Zero-Tile, as it does not have a role in the game. So each tile has also, an owner. During the game, control of tiles can be flipped from the owner to the opponent. After all one wins the game by controlling more tiles. All this neatly fits into a u8</code>! And Odin has bit-sets native into the language. When I was asking questions in Odin's discord, Ginger Bill (BDFL of Odin) suggested this structure. // 0b 0 0 _ 0 0 0 _ 0 0 0 // | | | | | | | | // | | | | | | | Top_Right // | | | | | | Right // | | | | | Btm_Right // | | | | Btm_Left // | | | Left // | | Top_Left // | Owner_Is_Host // Controller_Is_Host // // the two players are Guest and Host. Guest starts. Tile_Flag :: enum u8 { Top_Rght, Right, Btm_Right, Btm_Left, Left, Top_Left, Owner_Is_Host, Controller_Is_Host, } Tile :: distinct bit_set[Tile_Flag;u8]</code></pre> The game starts with 126 unique tiles. Guest starts with the tiles 0b00_000_001</code> to 0b00_111_111</code> (or neatly in Octal: 0o001</code> to 0o077</code>). Host starts with the tiles 0b11_000_001</code> to 0b11_111_111</code> (0o301</code> to 0o377</code>). The highest bit may flip during the game. // ^ is the pointer operator in Odin. // ~ is bitwise XOR. tile_flip:: proc(t: ^Tile) { t^ ~= Tile{.Controller_Is_Host} }</code></pre>Board Representation</h2> Here is where I go back and forth into reading Red Blob Games's excellent guide to Hexagonal boards</a>. The board of Dominions is a 9-sided hexagon. 217 cells. Ignoring tile placement restrictions for now, how to actually place tiles onto the board? I oscillated (heh) between a few ideas, but eventually settled on a giant big array of [217]Tile</code>, where an empty cell has the value 0</code>1</a>. To calculate offsets, I adapted the functions declared in the Red Blob article, and started with this neat loop (N</code>, CENTER</code>, and CELL_COUNT</code> are compile-time constants based on the board's size): Board :: [CELL_COUNT]Tile board_get_tile :: proc(back: ^[CELL_COUNT]Tile, hex: Hex) -> (^Tile, bool) { if hex_distance(hex, CENTER) > N { return nil, false } // From RedBlob on representing Hexagonal boards with an array of arrays: // Store Hex(q, r) at array[r][q - max(0, N-r)]. Row r size is 2*N+1 - abs(N-r). // this loop calculates r's offset. q, r := hex[0], hex[1] r_len := 0 for r_idx in 0 ..< r { r_len += 2 * int(N) + 1 - abs(int(N - r_idx)) } idx := r_len + int(q - max(0, N - r)) return &back[idx], true }</code></pre> But having this loop run every time I make a look up is .. uncertain at best. So I decided to hardcode the row offsets: board_get_tile :: proc(back: ^[CELL_COUNT]Tile, hex: Hex) -> (^Tile, bool) { if hex_distance(hex, CENTER) > N { return nil, false } q, r := hex[0], hex[1] r_len := 0 switch r { case 0: case 1: r_len = 9 case 2: r_len = 19 case 3: r_len = 30 case 4: r_len = 42 case 5: r_len = 55 case 6: r_len = 69 case 7: r_len = 84 case 8: r_len = 100 case 9: r_len = 117 case 10: r_len = 133 case 11: r_len = 148 case 12: r_len = 162 case 13: r_len = 175 case 14: r_len = 187 case 15: r_len = 198 case 16: r_len = 208 } idx := r_len + int(q - max(0, N - r)) return &back[idx], true }</code></pre> This has the advantage of being "clean", with no wasted place in the array for unused coordinates.2</a> Cell lookup is O(1): when checking a cell, it is imperative to check its neighbors quickly to verify legal moves. A separate data structure would be needed to track the tiles in player's hands (not in play yet). An alternative idea is to represent Tile</code> itself by a struct, that holds data whether the tile is in play, and if so its location on the board. However, the only way to quickly query the tile's neighbors is to iterate over every tile at every check to know where it is. Might as well stuff them all in an array. Hand</h2> Following the same idea from representing the board, Hands are better implemented as a fixed array. Initializing a full set of hands for both players by transmuting a u8</code> into a Tile</code>. That is why I wanted Tile</code>s to be u8</code>s to begin with!! HAND_SIZE :: 63 Hand :: distinct [HAND_SIZE]Tile hands_init :: proc() -> (guest: Hand, host: Hand) { for i in 0 ..< u8(HAND_SIZE) { guest[i] = transmute(Tile)(i + 0o001) host[i] = transmute(Tile)(i + 0o301) } return }</code></pre> Also, to check the legality of move, the player has to have the tile available to play, so I defined a couple of helper methods</del> procedures. hand_has_tile :: proc(hand: Hand, id: u8) -> bool { assert(0 < id, "Blank Tile is not playable") assert(id <= HAND_SIZE, "Tile is impossible") return !tile_is_empty(hand[id - 1]) } hand_get_tile :: proc(hand: ^Hand, id: u8) -> (Tile, bool) { if !hand_has_tile(hand^, id) do return nil, false ret := hand[id - 1] hand[id - 1] = {} return ret, true }</code></pre> About here, I thought I would like to test my logic. So I took advantage of Odin's testing framework with a couple of simple asserts. Note that I put these in the same file as the procedures they test, and they worked just fine. They are all prefixed with test_*</code> not to pollute the autocompletion too much. import "core:testing" @(test) test_hand_get_tile :: proc(t: ^testing.T) { w, _ := hands_init() tile: Tile ok: bool // should be successful tile, ok = hand_get_tile(&w, 63) testing.expect(t, !tile_is_empty(tile)) testing.expect(t, ok) // should fail. tile was emptied tile, ok = hand_get_tile(&w, 63) testing.expect(t, tile_is_empty(tile)) testing.expect(t, !ok) } // other tests go here</code></pre> After that, I figured I would make sure for every weird logic I make I'd have to write a bunch of tests to .. test it, and sanity check my code. The Game Object - First Draft</h2> Now that I have the basic, naïve framework of the game's representation, I started looking into it from the other side: top down. Looking at different game libraries, to see which API they provide and how they allow their users to interact with the internal rules engine. The shakmaty crate</a> has been of great help in the past, as well as the goban crate</a>. JaniM from the Rust discord also shared with me their Variant Go Server</a>. I also looked back at my Havannah implementation</a> which I did last year to remember how I structured things. (Maybe I'd tool both into WASM modules. How about that?) Most of these structure their API around a specific object: the Game object, which is interacted with by querying legal moves, making moves, and, optionally, undoing moves. That's it. The Game object tracks the state of all elements in the game. The most straightforward way to do that is a struct. I do not really know if this is the "optimal" arrangement of fields, but I am doing what makes sense to me. Player :: enum u8 { Guest, // White. Goes first. Host, // Black } Status :: enum u8 { Ongoing, Guest_Win, Host_Win, Tie, } // The Object of attention Game :: struct { board: Board, to_play: Player, status: Status, guest_hand, host_hand: Hand, // ... more to come }</code></pre> To inquire about moves, a Move</code> struct is needed3</a>. A move is simply a placement of a Tile</code> on Hex</code>. Whose tile and whose turn are ideally tracked and verified by the Game object. Move :: struct { hex: Hex, tile: Tile, }</code></pre> This misses one big thing: Groups. Dominions is a game of territory, based on Go. Tiles together make Groups. Groups have liberties, which they live and die of. Groups capture other Groups. The winner is the player with bigger Groups. Groups are important. Bitboards</h2> Before talking about Groups, Bitboards</a> warrant an introduction. Bitboards are, at their core, based on a simple observations: a chessboard has 64 squares, and a u64</code> has 64 bits. By tying each bit address to a specific square, any binary value can be mapped. In chess implementations: There is a bitboard (read: a u64</code>) to mark where all the White pieces are. There is another bitboard to mark where all the squares the Queen sees are. The other advantage of bitboards is that, since they are just bits, they have bit operations. With a bitboard showing the white pieces are and a bitboard showing where black's pieces can move next turn: just AND</code> them together and there is now a new bitboard of which pieces of white are under attack. They are small, simple integers, and operating on them is as easy as integers. Unfortunately, however, the Dominions board is decidedly not 64 cells, or any such convenient number. It is 217 cells. It would be possible to represent the whole board with a 217bit integer, should it exist, but the largest bit set Odin provides is 128 bits. But have no fear! An array of 7 bit sets can solve the problem, as 7 u32</code> integers can fit the needed 217 bits and more (namely 224 bits). This is as small as can be. Thanks to Odin's array programming (which is also taken advantage of for Hex math), using bitwise operations on these bitboards is as easy as they are on usual bit-sets. The additional 7 bits can be used for metadata, as well. Bitboard :: distinct [7]bit_set[0 ..< 32;u32] // 7 * 32 = 224 PLAY_AREA: Bitboard : {~{}, ~{}, ~{}, ~{}, ~{}, ~{}, ~{25, 26, 27, 28, 29, 30, 31}} DATA_AREA: Bitboard : {{}, {}, {}, {}, {}, {}, {25, 26, 27, 28, 29, 30, 31}} // Extra bits for metadata // helper procedure for common maths @(private = "file") bit_to_col_row :: proc(bit: int) -> (col, row: int) { col = bit % 32 row = bit / 32 return } bb_set_bit :: proc(bb: ^Bitboard, bit: int) { // assert here to allow an unchecked version for meta data assert(0 <= bit && bit < CELL_COUNT) col, row := bit_to_col_row(bit) bb^[row] |= {col} } bb_get_bit :: proc(bb: Bitboard, bit: int) -> bool { col, row := bit_to_col_row(bit) return col in bb[row] }</code></pre> Not to get ahead of the topic, but merging two of these bitboards (bitwise OR</code>) is now as simple as this: group_capture :: proc(blessed, cursed: ^Group) { // ----- snipped blessed.tiles |= cursed.tiles // `tiles` field is a Bitboard // ----- snipped }</code></pre> As an addition, I implemented an iterator over the set bits in Bitboard</code>. Odin has nice syntax to iterate over the set flags in a bit-set, but an array of bit-sets presents a logistical challenge. I hacked at it for a couple of hours and came up with the following. Odin's implementation of iterators is, all considered, fairly easy. // The State machine Bitboard_Iterator :: struct { bb: Bitboard, next: int, } bb_make_iter :: proc(bb: Bitboard) -> (it: Bitboard_Iterator) { it.bb = bb & PLAY_AREA return } // The iterator procedure bb_iter :: proc(it: ^Bitboard_Iterator) -> (item: Hex, idx: int, ok: bool) { for i in it.next ..< CELL_COUNT { if bb_get_bit(it.bb, i) { item = hex_from_index(i) idx = i it.next = i + 1 ok = true return } } return } // And it is used such: bbi := bb_make_iter(bitboard_from_somewhere) for hex, idx in bb_iter(&bbi) { // do something with this hex or its corresponding index }</code></pre> With the help of the Hex to Index math mentioned earlier in Board</code> (and I have by now extracted it to another procedure), each bit in the bitboard can be quickly mapped to its corresponding Hex. Which would bring the topic back to Groups, but there is another detour. Slotmaps</h2> In my implementation of Havannah (linked earlier), I used the slotmap</code> Rust crate to track groups. Odin does not have a slotmap4</a> in its core library. There is a sample showcase implementation by Ginger Bill</a>, but I wanted to try my hand at this FFI thing. With help from the Rust and Odin discords to navigate the FFI of both languages, I did the following, and it works! Almost statically type checked from both sides, too. use slotmap::{DefaultKey, Key, KeyData, SlotMap}; // The Rust code does not need to manipulate the object in anyway. // Type erased `*mut c_void` is all Rust knows: only a pointer. type SmItem = *mut core::ffi::c_void; type SmPtr = *mut SlotMap<DefaultKey, SmItem>; #[no_mangle] pub extern "C" fn slotmap_init() -> SmPtr { let sm = Box::new(SlotMap::<_, SmItem>::new()); Box::into_raw(sm) } #[no_mangle] pub unsafe extern "C" fn slotmap_destroy(sm: SmPtr) { _ = unsafe { Box::from_raw(sm) }; } #[no_mangle] pub unsafe extern "C" fn slotmap_insert(sm: SmPtr, item: SmItem) -> u64 { let Some(sm) = (unsafe { sm.as_mut() }) else { return 0; }; let handle = sm.insert(item); handle.data().as_ffi() } #[no_mangle] pub unsafe extern "C" fn slotmap_contains_key(sm: SmPtr, key: u64) -> bool { let Some(sm) = (unsafe { sm.as_mut() }) else { return false; }; let key = DefaultKey::from(KeyData::from_ffi(key)); sm.contains_key(key) } #[no_mangle] pub unsafe extern "C" fn slotmap_get(sm: SmPtr, key: u64) -> SmItem { let Some(sm) = (unsafe { sm.as_mut() }) else { return core::ptr::null_mut(); }; let key = DefaultKey::from(KeyData::from_ffi(key)); let ret = sm.get(key); *ret.unwrap_or(&core::ptr::null_mut()) } #[no_mangle] pub unsafe extern "C" fn slotmap_remove(sm: SmPtr, key: u64) -> SmItem { let Some(sm) = (unsafe { sm.as_mut() }) else { return core::ptr::null_mut(); }; let key = DefaultKey::from(KeyData::from_ffi(key)); sm.remove(key).unwrap_or(core::ptr::null_mut()) }</code></pre> And after compiling this to a staticlib</code>, do this from the Odin side: foreign import slotmap "deps/slotmap/libslotmap.a" import "core:c" Slot_Map :: distinct rawptr Sm_Key :: distinct c.uint64_t Sm_Item :: ^Group foreign slotmap { slotmap_init :: proc() -> Slot_Map --- slotmap_destroy :: proc(sm: Slot_Map) --- slotmap_insert :: proc(sm: Slot_Map, item: Sm_Item) -> Sm_Key --- slotmap_contains_key :: proc(sm: Slot_Map, key: Sm_Key) -> c.bool --- slotmap_get :: proc(sm: Slot_Map, key: Sm_Key) -> Sm_Item --- slotmap_remove :: proc(sm: Slot_Map, key: Sm_Key) -> SmItem --- }</code></pre> Odin's type system allows a rawptr</code> to really be cast to .. any pointer. So here all the types I know are the same, regardless of C's type erasure, are marked with the same alias. Even though, as far as the C ABI is concerned, both Slot_Map</code> and Sm_Item</code> are rawptr</code>s, but I know the difference. Conveniently, Sm_Item</code> can now be easily dereferenced to get the underlying Group without casting. Having done that, and as much as I am pleased with myself for getting this to work, I do not like that the keys are u64</code>s. They are large: much larger than any number of groups/indices required. Even in the unlikely event of each tile placed forming its own group, there would be a maximum of 126 groups (2 x 63). Eight bits would be enough to have a unique key for every possible group in the game. Something to optimize later, perhaps? Groups, and Game Object - Second Draft</h2> A group is, simply, two bitboards: Group :: struct { tiles: Bitboard, liberties: Bitboard, }</code></pre> Each group keeps track of its location (which hexes it occupies) and its liberties. A group's size and liberty count is calculated by calculating the cardinality of the respective bitboards. bb_card :: proc(bb: Bitboard) -> (ret: int) { temp := bb & PLAY_AREA for i in 0 ..< 7 { ret += card(temp[i]) } return } group_size :: proc(grp: ^Group) -> int { return bb_card(grp.tiles) } group_life :: proc(grp: ^Group) -> int { return bb_card(grp.liberties) }</code></pre> To track groups, two fields are added to the Game Object: one for each player. Additionally, a "Group Map" of sorts might be needed to quickly look up the group any cell belongs to. (This is the initial idea, but keeping all these things in sync seems daunting. There could be a better design that is only revealed with a concrete implementation.) Revisiting the Game</code> struct from before: Game :: struct { board: Board, to_play: Player, status: Status, last_move: Maybe(Move), // <-- I will get to that guest_hand, host_hand: Hand, groups_map: [CELL_COUNT]Sm_Key, guest_grps, host_grps: Slot_Map, }</code></pre> Note that the score is not recorded here. As with many aspects of this design, I am currently unsure if it something should be tracked and updated individually with each move, or something calculated from the game state at any given moment. There is also no history tracking, which needs lists of Move</code>s and past Board</code>s and Hand</code>s. The slotmap, being currently the only heap allocation, complicates undoing moves trivially, so a new Game</code> object would be constructed from past raw data of boards and games. I decided to ignore history tracking for now. // A player's territory consists of the number of their pieces // on the board minus the number of pieces they didn't place. game_get_score :: proc(game: ^Game) -> (guest, host: int) { for tile in game.board { (!tile_is_empty(tile)) or_continue if .Controller_Is_Host in tile { host += 1 } else { guest += 1 } } for tile in game.guest_hand { if !tile_is_empty(tile) do guest -= 1 } for tile in game.host_hand { if !tile_is_empty(tile) do host -= 1 } return }</code></pre> Now the game state and its metadata at any given point are neatly encoded, it is time to start thinking about how to make moves, and how to check their legality. The Game Rules</h2> So far, I made no mention of the game's rules, only talking about its physical properties: the tiles, the board, the game zones. But players cannot place Tiles where they choose: here are the rules summarized for your convenience: Tiles</h3> Each Tile has a specific set of sides it can connect to, encoded, visually, on the Tile itself.</li> Each Tile must match its neighboring Tiles in connections. Connected sides must match and Separated sides must match.</li> Board edges are considered neutral Separateds.</li> </ul> Groups</h3> A Group is one or more Tiles of the same color (Controller) that are connected together. Adjacencies on Separated sides do not count.</li> The Liberties of a Group are its Connected sides border empty cells.</li> If a Group loses its last Liberty, it is captured, and all its tiles are flipped (switched Controller).</li> </ul> The Structure</h3> The Structure is the collective name of all the connections together, in the whole board, regardless of Controller.</li> Disconnected parts of the Structure are called Sections, created as Tiles are placed on Separated adjacencies. A single Section can include multiple Groups (but a Group cannot belong to multiple Sections).</li> </ul> Placement</h3> Finally Bears repeating: A Tile can only be placed where it matches its neighbours in Connected and Separated sides.</li> Firstly, the Guest, first player, starts by placing any Tile anywhere.</li> Afterwards, a Tile can only be placed adjacent to an enemy Tile, with one exception.</li> A Tile can be placed where it is not adjacent to an enemy Tile if and only if it is extending a Group that is a whole Section.</li> Suicide is legal. Player may place a Tile so it has no Liberties (by matching its Liberties to enemy Tiles), and is therefore immediately captured.</li> Lastly, it is illegal to cause Oscillation: a Section which has no Liberties. ^{5</a></li>} A Player may, instead of placing a Tile, pass the turn instead. Or if they have no legal moves, they must.</li> </ul> Scoring</h3> The Game ends when both players pass consecutively.</li> The winner is the Player with the highest score.</li> The score is the amount of Tiles controlled on the board minus the Tiles still in Hand.</li> </ul> Moves, and Game Object - Third Draft</h2> Back to code. As a refresher, here is Move</code>: Move :: struct { hex: Hex, tile: Tile, }</code></pre> That is all a Move</code> is: the player whose turn it is (as tracked by Game</code>) places a Tile</code> from their hand into a Hex</code>, and from there follows that groups get updated and tiles get flipped and game progresses. But as a player may (or may have to) pass the turn, and as the game ends with two successive passes, last_move</code> is registered in Game</code> to track whether the game is about to end.6</a> To play the game, Game</code> needs to have the following: A way to make the move</li> A way to query if a move is legal</li> A way to query the list of legal moves</li> </ul> Perhaps the most straightforward way to do it is to have game_make_move</code> do all the work. It updates the game state, naturally, but also (based on said game state) update an embedded list of legal moves in Game</code> itself. And then to check if a move is legal, simply query the embedded list of moves. Here is what Game</code>, and the skeleton of game_make_move</code>, look like now: Game :: struct { board: Board, to_play: Player, status: Status, last_move: Maybe(Move), guest_hand, host_hand: Hand, groups_map: [CELL_COUNT]SmKey, guest_grps, host_grps: SlotMap, legal_moves: [dynamic]Move, } game_make_move :: proc(game: ^Game, try: Maybe(Move)) -> bool { (game.status == .Ongoing) or_return // Game is over. move, not_pass := try.? // legal move check (!not_pass || slice.contains(game.legal_moves[:], move) || board_is_empty(&game.board)) or_return defer if game.status == .Ongoing { switch game.to_play { case .Guest: game.to_play = .Host case .Host: game.to_play = .Guest } game.last_move = move game_regen_legal_moves(game) } if !not_pass { // Pass if game.last_move == nil { // Game ends guest, host := game_get_score(game) if guest > host do game.status = .Guest_Win else if guest < host do game.status = .Host_Win else do game.status = .Tie } return true } active_hand: ^Hand switch game.to_play { case .Guest: active_hand = &game.guest_hand case .Host: active_hand = &game.host_hand } // Make move. Already known to be legal!! hand_tile, removed := hand_remove_tile(active_hand, move.tile) assert(removed) // but verify game.board[hex_to_index(move.hex)] = hand_tile move.tile = hand_tile // probably superfluous. // TODO: update game state return true } @(private) game_regen_legal_moves :: proc(game: ^Game) { clear(&game.legal_moves) // TODO: build them again }</code></pre> The two TODO</code>s left are probably the meat of this whole engine. It is clearer to write them out in steps, in English, before encoding them into code. Updating the Game State</h2> The key observation, I think, is that the game state change starts from the where the last move is played. No need for a global search process, but only the exact Hex being played and the surrounding groups. group_section_init</code></h3> The simplest, and shortest, game state change that is not a Pass, is a Tile that starts its own Section. This happens in two occasions: the first move, and whenever a Tile is only adjacent to other tiles via its Separated sides. This Tile by definition also starts a new Group. This procedure assumes it is only called when a move is legal. @(private) group_section_init :: proc(move: Move, game: ^Game) -> (ret: Group, ok: bool = true) { // Check that all Connected sides connect to empty tiles. // AND find Liberties for flag in move.tile & CONNECTION_FLAGS { neighbor := move.hex + flag_dir(flag) t, in_bounds := board_get_tile(&game.board, neighbor) if in_bounds { tile_is_empty(t^) or_return bb_set_bit(&ret.liberties, hex_to_index(neighbor)) } } bb_set_bit(&ret.tiles, hex_to_index(move.hex)) // CELL_COUNT is where the Extended flag is. bb_set_bit_unchecked(&ret.liberties, CELL_COUNT) return }</code></pre> And in game_make_move</code> : if grp, ok := group_section_init(move, game); ok { grp := new_clone(grp) key := slotmap_insert(game.host_grps, grp) game.groups_map[hex_to_index(move.hex)] = key }</code></pre>group_attach_to_friendlies</code></h3> The second easiest to deal with is a Tile that only connects to Groups of its own color. But there are a few variations Extension: The Tile may only extend one Group</li> Merge: The Tile may merge two friendly Groups into one.</li> Suicide: The Tile may consume the remaining Liberties of the Group(s) it connects to, Suiciding the whole Group. In which case the newly formed Group is captured (flipped) as a whole, and merged with the surrounding enemy Group(s).</li> </ul> The last case is interesting. Note that Move legality is checked earlier, so on principle, it is known that the newly formed enemy Group has at least one Liberty left. A funny observation here is that it is impossible for a Tile that only connects to friendlies to Capture an enemy Group, but it is entirely possible that is causes its own Group to be immediately captured. An easy check to whether this move is a Suicide is to check whether the new Tile has any Liberties of its own. For example: if it has 2 Connections and is connected to a friendly Group from only one side of the two, and the other Connection is to an empty cell, then it is obviously not a Suicide. Ergo, the procedure for extension or merging is as follows. It turned way longer than I expected. @(private) group_extend_or_merge :: proc(move: Move, game: ^Game) -> (ok: bool = true) { // Bug tracker tile_liberties_count := card(move.tile & CONNECTION_FLAGS) // Friendliness tracker tile_control := move.tile & {.Controller_Is_Host} // Scratchpad: Found Groups nbr_friend_grps: [6]Sm_Key nfg_cursor: int // Scratchpad: New Liberties new_libs: [6]Hex libs_cursor: int for flag in move.tile & CONNECTION_FLAGS { neighbor := move.hex + flag_dir(flag) nbr_tile := board_get_tile(&game.board, neighbor) or_continue if tile_is_empty(nbr_tile^) { new_libs[libs_cursor] = neighbor libs_cursor += 1 } else if nbr_tile^ & {.Controller_Is_Host} == tile_control { // Same Controller tile_liberties_count -= 1 // record Group of neighbor tile. key := game.groups_map[hex_to_index(neighbor)] if !slice.contains(nbr_friend_grps[:], key) { nbr_friend_grps[nfg_cursor] = key nfg_cursor += 1 } } else { // Different Controller return false } } assert( tile_liberties_count >= 0, "if this is broken there is a legality bug" ) assert( nfg_cursor > 0, "This proc should not be called with no friendly neighbors" ) // and if this is false then this might be a Suicide (tile_liberties_count > 0) or_return // == Are we the Baddies? friendly_grps: Slot_Map if tile_control == {} { friendly_grps = game.guest_grps } else { friendly_grps = game.host_grps } // == Identify first group blessed_key := nbr_friend_grps[0] assert(slotmap_contains_key(friendly_grps, blessed_key)) blessed_grp := slotmap_get(friendly_grps, blessed_key) bb_set_bit(&blessed_grp.tiles, hex_to_index(move.hex)) // == Merge other groups with first group for i in 1 ..< nfg_cursor { assert(slotmap_contains_key(friendly_grps, nbr_friend_grps[i])) grp := slotmap_remove(friendly_grps, nbr_friend_grps[i]) defer free(grp) blessed_grp.tiles |= grp.tiles // Only if both Groups are extendable is the new one extendable play_area := PLAY_AREA & (blessed_grp.liberties | grp.liberties) data_area := DATA_AREA & (blessed_grp.liberties & grp.liberties) blessed_grp.liberties = play_area | data_area } // == Update liberties bb_unset_bit(&blessed_grp.liberties, hex_to_index(move.hex)) for l in 0 ..< libs_cursor { bb_set_bit(&blessed_grp.liberties, hex_to_index(new_libs[l])) } // == Update the groupmap bbi := bb_make_iter(blessed_grp.tiles) for _, idx in bb_iter(&bbi) { game.groups_map[idx] = blessed_key } // this should be all. return }</code></pre> Now, what to do if it is a potential suicide? First of all, it is not possible to know whether it is a suicide or not without doing everything already done in group_extend_or_merge</code> anyway. If it were a Suicide, the final, coalesced Group would have no Liberties, and therefore it would be an easy decision to simply flip its Controller and swap its allegiance. The problem lies in how to merge it with its capturing Group(s). It needs to be merged to maintain an accurate count of Liberties, as the count of Liberties is clearly being used to test whether it needs to be captured or not!! (Not in this move. but in subsequent ones.) The naïve approach is to iterate over every Tile member of the Group and every Connection of said Tile, and compile a list of opponent-controlled neighbors, then check the Group membership of those Tiles, (similarly to how nbr_friend_grps</code> is used,) then do the merging routine again, but this time for the opponent. The other naïve approach is to add a third Bitboard</code> to Group</code> to track its surrounding enemies and just iterate over those. Both solutions bother me for different reasons. The first seems like a time and CPU cycle waste (as I am already iterating over the same Tile's flags multiple times ..), and the second seems adds an extra knob to keep track of. Rethinking Groups</h2> One idea to explore is to change Group</code>'s representation entirely, away from Bitboards. (Yes I spent a lot of time talking about Bitboards, and they're really cool, but hear me out). Throughout the program, the state of the game has been tracked through a number of [CELL_COUNT]</code> arrays, of variable things, with trivial conversion from a human-readable Hex</code> to an index within these arrays. So what's another [CELL_COUNT]</code> array? But what would it be an array of? Currently, as far as a Group is concerned, a location can be one of three states: Either a member Tile, a Liberty, or an enemy Connection. That is an enum! More things can be added to it later as needed. Hex_State :: enum u8 { Empty = 0b000, // numbers chosen for a reason Liberty = 0b001, Enemy_Connection = 0b011, Member_Tile = 0b111, }</code></pre> And this how the Group</code> using this enum would look like: Rethought_Group :: struct { state: [CELL_COUNT]Hex_State, extendable: bool, // sadly no clean niche to hide that }</code></pre> Much cleaner! Surprisingly, Odin allows bitwise OR over enumerations.^{7</a> If the resulting value has no tag assigned, it becomes a BAD_ENUM_VALUE</code> and may potentially wreck the program. But if the numbers are assigned appropriately, it can be made to always have a valid value.} Thinking through this, it is clear that .Empty</code> with any other tag should be, well, that other tag. .Liberty</code>, being essentially an empty cell as well, with any of the other two tags should produce the other tag. .Member_Tile</code> and .Enemy_Connection</code> overlap when capturing groups, so Enemies should be converted to Members. Here is the printed OR</code> table: Empty Liberty Enemy Member Empty Empty Liberty Enemy Member Liberty ------> Liberty Enemy Member Enemy ----------------> Enemy Member Member ------------------------> Member</code></pre> Great. Looks good to me. Let's roll with it. This is how Group</code> works now. Should I need more states I shall think of other clever numbers to use. Then follow the compiler's errors about missing fields and correct those as needed. Compiler driven development !! Back to Updating Game State</h2> Back to group_attach_to_friendlies</code></h3> This above change makes the merging process much simpler. It also allowed me to delete the entire bitboard.odin</code> file! Following the compiler's errors leads me to this version of group_extend_or_merge</code>: @(private) group_extend_or_merge :: proc(move: Move, game: ^Game) -> (ok: bool = true) { // ----- snip: same as before, for now. // == Identify first group blessed_key := nbr_friend_grps[0] assert(slotmap_contains_key(friendly_grps, blessed_key)) blessed_grp := slotmap_get(friendly_grps, blessed_key) blessed_grp.state[hex_to_index(move.hex)] = .Member_Tile // == Merge other groups with first group for i in 1 ..< nfg_cursor { assert(slotmap_contains_key(friendly_grps, nbr_friend_grps[i])) grp := slotmap_remove(friendly_grps, nbr_friend_grps[i]) defer free(grp) blessed_grp.state |= grp.state blessed_grp.extendable &= grp.extendable } // == Update liberties for l in 0 ..< libs_cursor { blessed_grp.state[hex_to_index(new_libs[l])] |= .Liberty } // == Update the groupmap for slot, idx in blessed_grp.state { if slot == .Member_Tile do game.groups_map[idx] = blessed_key } return }</code></pre> Now, merging groups membership and liberties also merges their enemy connections as well. No additional bookkeeping! The check for potential Suicide earlier can now be removed, and writing this (larger and larger) procedure can continue: // ----- snip: same as before + declaring pointers to enemy groups // == Update liberties for l in 0 ..< libs_cursor { blessed_grp.state[hex_to_index(new_libs[l])] |= .Liberty } if tile_liberties_count == 0 // Potential Suicide && group_life(blessed_grp) == 0 // Definite Suicide { // no insertion into the enemy slotmap .. there is merging to be done! cursed_grp := slotmap_remove(friendly_grps, blessed_key) defer free(cursed_grp) // Scratchpad nbr_enemy_grps := make([dynamic]Sm_Key) defer delete(nbr_enemy_grps) for loc, idx in cursed_grp.state { #partial switch loc { case .Member_Tile: tile_flip(&game.board[idx]) case .Enemy_Connection: key := game.groups_map[idx] if !slice.contains(nbr_enemy_grps[:], key) { append(&nbr_enemy_grps, key) } } } // == same steps as before assert(len(nbr_enemy_grps) > 0) // or there is Oscillation blessed_key = nbr_enemy_grps[0] assert(slotmap_contains_key(enemy_grps, blessed_key)) blessed_grp = slotmap_get(enemy_grps, blessed_key) blessed_grp.state |= cursed_grp.state for i in 1 ..< len(nbr_enemy_grps) { assert(slotmap_contains_key(enemy_grps, nbr_enemy_grps[i])) grp := slotmap_remove(enemy_grps, nbr_enemy_grps[i]) defer free(grp) blessed_grp.state |= grp.state } // check if blessed_grp is extendable extendable := true for loc in blessed_grp.state { if loc == .Enemy_Connection { extendable = false break } } blessed_grp.extendable = extendable } // == Update the groupmap for slot, idx in blessed_grp.state { if slot == .Member_Tile do game.groups_map[idx] = blessed_key } return }</code></pre> Change the procedure's name to group_attach_to_friendlies</code>, and now it is done. All this merging logic should really be refactored and DRY</code>'d, but I will let it be for now. This one procedure end up about 130 lines of code. group_attach_to_enemies</code></h3> Similarly to attaching to friendlies, this move can either be a nothing, a capture, or a suicide. The logic of the friendlies</code> procedure might need to be repeated in this one. And the checks done and the data collected would also need to be repeated. So why separate it at all? The idea was it would simplify handling, but it does not seem to do that. So, rethinking the move handling, allow me to try summarising the logic that actually needs to be done (this was revised in tandem with writing the code in the next section): Create these trackers: Liberties of the newly placed tile,</li> Neighboring, connected Friendlies (tracking Groups), and</li> Neighboring, connected Enemies (tracking locations).</li> </ol> </li> Iterate over all Connected Sides of the newly placed tile, and fill in the trackers as needed,</li> For every Friendly connected Group, merge them together. (If none are connected, the Tile starts its own Group with marked Liberties and Enemy neighbors.) Mark this group as "Blessed".</li> Iterate over neighboring Enemy Groups, if any have a Liberty count of 0: ^{8</a> They are flipped and merged with the Blessed Group.</li> Iterate over connections of the Blessed Group, and merge with it any friendly Groups found.</li> Move is over. (This is because legal moves are already screened, or a check for Oscillation would be needed.)</li> </ol> </li> The Blessed Group is checked for Liberty count. If it is 0, it is captured (flipped) and merged with its surrounding Enemy Groups.</li> Done</li> </ol> Note that I am assuming here that this is not a recursive operation. Here is the assumption: A new Tile placement that has no liberties but takes away the last liberty of an enemy Group captures it. There is no need to check if the surrounding friendly Groups (that surrounded the surrounding Enemy Groups) would also have no Liberties, because if they had no Liberties they would not exist! A lot of weight is placed right now on the correctness of game_regen_legal_moves</code>, which is still delayed for later. game_update_state_inner</code> - Second Draft</h3> A monstrous 230-ish lines of code which could really use some refactoring. This drags on but I made my best to comment my thoughts throughout. Written without Tests, as I don't actually know how to test this yet @(private) game_update_state_inner :: proc(move: Move, game: ^Game) { // Bug tracker tile_liberties := card(move.tile & CONNECTION_FLAGS) tile_liberties_countdown := tile_liberties // Friendliness tracker tile_control := move.tile & {.Controller_Is_Host} // Scratchpad: Found friendly Groups nbr_friend_grps: [6]Sm_Key nfg_counter: uint // Scratchpad: Found Enemy Groups nbr_enemy_tiles: [6]Hex net_counter: uint // Scratchpad: New Liberties new_libs: [6]Hex libs_counter: uint for flag in move.tile & CONNECTION_FLAGS { neighbor := move.hex + flag_dir(flag) nbr_tile := board_get_tile(&game.board, neighbor) or_continue if tile_is_empty(nbr_tile^) { new_libs[libs_counter] = neighbor libs_counter += 1 } else if nbr_tile^ & {.Controller_Is_Host} == tile_control { // Same Controller tile_liberties_countdown -= 1 // record Group of neighbor tile. key := game.groups_map[hex_to_index(neighbor)] if !slice.contains(nbr_friend_grps[:], key) { nbr_friend_grps[nfg_counter] = key nfg_counter += 1 } } else { // Different Controller tile_liberties_countdown -= 1 nbr_enemy_tiles[net_counter] = neighbor net_counter += 1 } } assert( tile_liberties_countdown >= 0, "if this is broken there is a legality bug" ) // == Are we the Baddies? friendly_grps, enemy_grps: Slot_Map if tile_control == {} { // Guest Controller friendly_grps = game.guest_grps enemy_grps = game.host_grps } else { // Host Controller friendly_grps = game.host_grps enemy_grps = game.guest_grps } // The placed Tile's Group blessed_key: Sm_Key blessed_grp: Sm_Item if nfg_counter == 0 { blessed_grp = new(Group) blessed_key = slotmap_insert(friendly_grps, blessed_grp) if net_counter == 0 { blessed_grp.extendable = true } } else { blessed_key = nbr_friend_grps[0] assert( slotmap_contains_key(friendly_grps, blessed_key), "Friendly slotmap does not have friendly Key", ) blessed_grp = slotmap_get(friendly_grps, blessed_key) // == Merge other groups with blessed group for i in 1 ..< nfg_counter { assert( slotmap_contains_key(friendly_grps, nbr_friend_grps[i]), "Friendly slotmap does not have friendly Key", ) temp_grp := slotmap_remove(friendly_grps, nbr_friend_grps[i]) defer free(temp_grp) blessed_grp.state |= temp_grp.state blessed_grp.extendable &= temp_grp.extendable } } blessed_grp.state[hex_to_index(move.hex)] |= .Member_Tile defer { // == Update the groupmap for slot, idx in blessed_grp.state { if slot == .Member_Tile do game.groups_map[idx] = blessed_key } } // == Update liberties for i in 0 ..< libs_counter { blessed_grp.state[hex_to_index(new_libs[i])] |= .Liberty } // == Update Enemy neighbors for blessed group for i in 0 ..< net_counter { blessed_grp.state[hex_to_index(nbr_enemy_tiles[i])] |= .Enemy_Connection } // == register surrounding Enemy Groups of blessed Group surrounding_enemy_grps := make([dynamic]Sm_Key) defer delete(surrounding_enemy_grps) for slot, idx in blessed_grp.state { (slot == .Enemy_Connection) or_continue key := game.groups_map[idx] if !slice.contains(surrounding_enemy_grps[:], key) { append(&surrounding_enemy_grps, key) } } // == if there are no surrounding enemy groups there is nothing to do if len(surrounding_enemy_grps) == 0 { assert( group_life(blessed_grp) > 0, "newly formed groups must have liberties or enemy connections", ) blessed_grp.extendable = true return } // == these are the friendly groups that surround the dead enemy groups. level_2_surrounding_friendlies := make([dynamic]Sm_Key) defer delete(level_2_surrounding_friendlies) // == go over surrounding enemy groups to see if they're dead. capture_occurance := false for key in surrounding_enemy_grps { assert( slotmap_contains_key(enemy_grps, key), "Enemy slotmap does not have enemy Key" ) temp_grp := slotmap_get(enemy_grps, key) temp_grp.state[hex_to_index(move.hex)] |= .Enemy_Connection // spoiler: there be bugs // Enemy Group is dead (group_life(temp_grp) == 0) or_continue capture_occurance = true cursed_grp := slotmap_remove(enemy_grps, key) defer free(cursed_grp) for slot, idx in cursed_grp.state { #partial switch slot { case .Member_Tile: tile_flip(&game.board[idx]) case .Enemy_Connection: key := game.groups_map[idx] if !slice.contains(level_2_surrounding_friendlies[:], key) { append(&level_2_surrounding_friendlies, key) } } } // CAPTURE blessed_grp.state |= cursed_grp.state blessed_grp.extendable &= cursed_grp.extendable } // == merge level 2 surrounding friendlies into blessed group for key in level_2_surrounding_friendlies { assert(slotmap_contains_key(friendly_grps, key)) temp_grp := slotmap_remove(friendly_grps, key) defer free(temp_grp) blessed_grp.state |= temp_grp.state blessed_grp.extendable &= temp_grp.extendable } // == if there is a capture, it is done. if capture_occurance do return // == if blessed group's liberties larger than 0, it is done capturing if group_life(blessed_grp) > 0 do return // == Now the blessed group has converted. cursed_grp := slotmap_remove(friendly_grps, blessed_key) defer free(cursed_grp) new_family := make([dynamic]Sm_Key) defer delete(new_family) for loc, idx in blessed_grp.state { #partial switch loc { case .Member_Tile: tile_flip(&game.board[idx]) case .Enemy_Connection: key := game.groups_map[idx] if !slice.contains(new_family[:], key) { append(&new_family, key) } } } assert(len(new_family) > 0, "Oscillation") blessed_key = new_family[0] assert( slotmap_contains_key(enemy_grps, blessed_key), "Enemy key is not in enemy map" ) blessed_grp = slotmap_get(enemy_grps, blessed_key) blessed_grp.state |= cursed_grp.state for i in 1 ..< len(new_family) { assert(slotmap_contains_key(enemy_grps, new_family[i])) temp_grp := slotmap_remove(enemy_grps, new_family[i]) defer free(temp_grp) blessed_grp.state |= temp_grp.state } // check if new blessed group is extendable extendable := true for loc in blessed_grp.state { if loc == .Enemy_Connection { extendable = false break } } blessed_grp.extendable = extendable return }</code></pre> I attempted refactoring some of it out but I am not sure the result is necessarily better. I also used the core:container/small_array</code> library (from the Odin core lib) to smooth some of the logic. game_regen_legal_moves</code></h2> This is the current state of this procedure, which a lot is riding on: @(private) game_regen_legal_moves :: proc(game: ^Game) { clear(&game.legal_moves) // todo: build them again }</code></pre> This is perhaps an easier problem than it presented to me at first. The rules have a harsh restriction to placement: any Tile placed MUST be placed next to an enemy Tile, unless it extends a Group which makes its own Section (which is already being tracked by the extendable</code> field in Group</code>. Assuming the logic, of course, is correct.) The naïve, crude approach is as follows: Iterate over every empty Hex</code>, and if it has an enemy neighbor, or if it has a friendly neighbor that is part of an extendable Group and it is that Group's liberty, add it to the candidates. Then for each Hex</code> of the candidates, iterate over the friendly Hand</code>, and add a Move</code> where the Tile</code> fits. There might be a more efficient way but this will do for now. (Yes this does not take Oscillation into account, yet?) game_regen_legal_moves :: proc(game: ^Game) { clear(&game.legal_moves) // == Are we the Baddies? friendly_grps: Slot_Map friendly_hand: ^Hand enemy_grps: Slot_Map switch game.to_play { case .Guest: friendly_grps = game.guest_grps friendly_hand = &game.guest_hand enemy_grps = game.host_grps case .Host: friendly_grps = game.host_grps friendly_hand = &game.host_hand enemy_grps = game.guest_grps } // get the hexes allowed to be played in playable_hexes := make([dynamic]Hex) defer delete(playable_hexes) outer: for key, idx in game.groups_map { // Hex must be empty. Hope Slotmap does not give a Key of 0. (key == 0) or_continue hex := hex_from_index(idx) for flag in CONNECTION_FLAGS { nbr_hex := hex + flag_dir(flag) nbr_idx := hex_to_index(nbr_hex) or_continue nbr_key := game.groups_map[nbr_idx] if nbr_key == 0 do continue if slotmap_contains_key(enemy_grps, nbr_key) { append(&playable_hexes, hex) continue outer } else if slotmap_contains_key(friendly_grps, nbr_key) { grp := slotmap_get(friendly_grps, nbr_key) if grp.extendable && grp.state[idx] == .Liberty { append(&playable_hexes, hex) continue outer } } else { // This is essentially an assert. panic("key is not 0, is not in friendly groups, not in enemy groups, ??") } } } // fill Tiles to go with found hexes. candidate_moves := make([dynamic]Move) defer delete(candidate_moves) for hex in playable_hexes { idx := hex_to_index(hex) for tile in friendly_hand { if tile_is_empty(tile) do continue score := 0 // if score is 6, tile is playable. defer if score == 6 do append(&candidate_moves, Move{hex, tile}) for flag in CONNECTION_FLAGS { nbr_hex := hex + flag_dir(flag) nbr_idx, in_bounds := hex_to_index(nbr_hex) nbr_tile := game.board[nbr_idx] // nbr_idx == 0 when out ((!in_bounds && flag not_in tile) || (in_bounds && (tile_is_empty(nbr_tile) || /* so much rightward drift */ (flag in tile && flag_opposite(flag) in nbr_tile) || /* the line is 102 chars */ (flag not_in tile && flag_opposite(flag) not_in nbr_tile)))) or_break score += 1 if cond else 0 } } } // todo: deal with Oscillation }</code></pre>Oscillation</h3> Now, back to Oscillation. What's that again? When a Group has no Liberties, it is converted to the other side. If, upon conversion, it still has no Liberties, it flips again, and .. well .. oscillates. That's illegal! So Oscillation as a result of a Move can only happen if a Group has only one Liberty and the Move is on that Hex. So I will make a number of assumptions here: If the Move connects only to extendable Groups (Friendly or Enemy), and takes away their last Liberty, it is Oscillation.</li> If the Move connects to a usual Group (Friendly or Enemy), it is not Oscillation, but either a Capture or a Suicide. (If the surrounding Groups had no other Liberties of their own, they'd be Captured already).</li> </ol> I do not know if these assumptions would actually work out. A more sure approach is to actually just ... make the move, and should there be an Oscillation, reject it, and roll back the changes. While it is possible, it would make the list of legal moves wrong, as it might include these Oscillation moves. Encoding these rules in the loop above is perhaps the best option. It would save an allocation of candidate_moves</code>, and it is already iterating the edges and the neighbors. It could have its own tracker and score. This is how it currently looks: // ------- same as before, minus the temporary `candidate_moves` for hex in playable_hexes { idx := hex_to_index(hex) for tile in friendly_hand { if tile_is_empty(tile) do continue score := 0 // if score is 6, tile is playable. osc_pen := 0 // unless this is equal to Tile cardinality defer if score == 6 && osc_pen != card(tile & CONNECTION_FLAGS) { append(&game.legal_moves, Move{hex, tile}) } for flag in CONNECTION_FLAGS { nbr_hex := hex + flag_dir(flag) nbr_idx, in_bounds := hex_to_index(nbr_hex) nbr_tile := game.board[nbr_idx] // nbr_idx == 0 when out ((!in_bounds && flag not_in tile) || (in_bounds && (tile_is_empty(nbr_tile) || (flag in tile && flag_opposite(flag) in nbr_tile) || (flag not_in tile && flag_opposite(flag) not_in nbr_tile)))) or_break score += 1 // Only check for Oscillation if it takes away a Liberty. (in_bounds && flag_opposite(flag) in nbr_tile) or_continue nbr_key := game.groups_map[nbr_idx] if slotmap_contains_key(enemy_grps, nbr_key) { nbr_grp := slotmap_get(enemy_grps, nbr_key) if group_life(nbr_grp) == 1 && nbr_grp.extendable { osc_pen += 1 } } else if slotmap_contains_key(friendly_grps, nbr_key) { nbr_grp := slotmap_get(friendly_grps, nbr_key) if group_life(nbr_grp) == 1 && nbr_grp.extendable { osc_pen += 1 } } } } } }</code></pre> One tiny thing, this is flag_opposite</code> : flag_opposite :: proc(flag: Tile_Flag) -> (ret: Tile_Flag) { #partial switch flag { case .Top_Right: ret = .Btm_Left case .Right: ret = .Left case .Btm_Right: ret = .Top_Left case .Btm_Left: ret = .Top_Right case .Left: ret = .Right case .Top_Left: ret = .Btm_Right } return }</code></pre> I think it is done!! What's left? Oh, actually using it. First Bug</h2> Did not take long to find a bug. main :: proc() { game := game_init() defer game_destroy(game) ok: bool ok = game_make_move( &game, Move{hex = {0, 0}, tile = tile_from_id(63, .Guest)} ) fmt.printfln("%v", ok) ok = game_make_move( &game, Move{hex = {1, 0}, tile = tile_from_id(63, .Host)} ) fmt.printfln("%v", ok) fmt.printfln("%v", game.groups_map) }</code></pre> Both moves should be legal, and indeed both ok</code>s are true. However, going over the printed output of game.groups_map</code>, I noticed both of these tiles have the exact same Key. (And is in fact the same Key across multiple runs: 4294967297.) This is an artifact of how the slotmap</code> crate does its thing, as I found out. The problem is that so far I have been operating under the assumption that the Keys produced are, at least reasonably, different from each other, as I have been using this Key in no less than three places to know in which camp does a key belong. (One of them is game_destroy</code>, where it doesn't matter, but still!) The easy fix would be perhaps to annotate all three lookups with companion look ups to see the owner of the Tile</code> in the corresponding game.board</code> index. But this feels like a bandaid, that also adds another failure point. Trying different values I came across another bug that apparently triggered an assertion, but it was whether a slotmap</code> contains a key or not. So a fix for this first bug is needed before any procession. Rethinking the Need for slotmap</code></h2> Using a Rust crate for this functionality has a couple of pain points already: I have mentioned this before, but u64</code> Keys are huge. They are much larger than what is needed in this game. The amount of Groups the entire game cannot actually exceed 126. (If all Tiles were played and each Tile had its own Group, which is also impossible.)</li> Speed of access: the slotmaps are behind pointers and the Groups are behind pointers, and this is checked and accessed multiple times per move.</li> Compilation: while this set up works fine on my machine TM, compiling a shim Rust crate separately from the main Odin codebase is a bit more effort than what is usually needed. If this were to be set up for users, I would need to include a build system.</li> WASM: Add to that, while slotmap</code> itself can be compiled to WASM just fine, compiling two languages into one WASM module is .. not the easiest path forward.</li> Last but not least, undoing the Game state (something which is needed for engines) would be a lot easier if there were no pointers involved.</li> </ul> So now what? Is a slotmap-like data structure even the correct decision at all, actually? Slotmap's advantage is reusing existing allocations for deleted items, while not reusing the handles/indices. But if it lives all on the stack, say through a [CELL_COUNT]Group</code> array, it is not possible to grow memory dynamically as needed (since it is all statically allocated), and one would have to allocate for the worst case scenario anyway removing the main advantage there. One option is to put Groups in a fixed array, and use indices into that array as keys, and simply stop using that index whenever a Group dies. While a real game is never getting to 126 Groups at once, and there is no game database of played Dominions games to analyze and determine the maximum number of Groups in a real game is. Using Mindsports' applet</a>, I managed to get to about 92 individual living "groups" at the same time before I got bored. How would this look like, though? Separate arrays, one for each player? One array for both? If the second, how to distinguish between friendly and enemy groups storage? My current thinking is to borrow the handle/index++ idea and invent, essentially, my own handle. Honestly I just want an excuse to use Odin's bit fields. Group_Handle :: bit_field u8 { // name: Type | Size idx: u8 | 6, owner: Player | 1, valid: bool | 1, } </code></pre> The idx</code> here cannot, ever, be more than 63, which is all the address space needed for the worst case scenario, and a little bit more. The owner</code> field is one bit, either this player or that player. The last bit is a funny one: it is just to allow using the zero value of the key (where valid</code> is false), as a sentinel value for an empty slot. At first, I considered having the extendable</code> boolean I have had in Group</code>, but that just makes bookkeeping and updating Group</code> status that much harder. 9</a> So now how about this? Rethought_Group :: struct { state: [CELL_COUNT]Hex_State, using status: bit_field u8 { extendable: bool | 1, alive: bool | 1, _padding: u8 | 6, // <- ? }, } Group_Store :: // todo</code></pre> The field _padding</code> is there to suppress a compiler warning about boolean-only bit fields being a terrible idea. Mind you Rethought_Group</code> has two Odin features Ginger Bill dislikes using: using</code> and bool</code>-only bit_field</code>s, and uses them together, with an anonymous bit_field</code> to boot. (I am actually pleasantly surprised this compiles and works at all.) My goal here is to essentially cram both of extendable</code> and alive</code> booleans into one byte, and they do not really make sense to me as their own type, and keep the nice ergonomics of using them. The advised/Ginger Bill-authorized way to write this is the following: Group_Status :: enum u8 { Extendable, Alive, } Rethought_Group :: struct { state: [CELL_COUNT]Hex_State, status: bit_set[Group_Status; u8], } if .Extendable in grp.status { // etc</code></pre> But I have already been using Group</code> throughout, and I really like just writing this: if grp.extendable { // etc // AND grp.extendable = true</code></pre> But what is programming if not suppressing compiler warnings and ignoring the advice of more experienced practitioners? Especially the author of the language you are writing in? Back to business. The first thing needed is to reimplement the slotmap</code> API for the new Group_Store</code>. Here is the original: slotmap_init :: proc() -> Slot_Map --- slotmap_destroy :: proc(sm: Slot_Map) --- slotmap_insert :: proc(sm: Slot_Map, item: Sm_Item) -> Sm_Key --- slotmap_contains_key :: proc(sm: Slot_Map, key: Sm_Key) -> c.bool --- slotmap_get :: proc(sm: Slot_Map, key: Sm_Key) -> Sm_Item --- slotmap_remove :: proc(sm: Slot_Map, key: Sm_Key) -> Sm_Item ---</code></pre> init</code> and destroy</code> are perhaps not needed, as they exist to manage heap memory. contains_key</code> could be replaced by get</code> (and remove</code>) returning a success or failure value, as is idiomatic in Odin. To ascertain that Group_Store</code> can only return keys of the appropriate Player, some type system hackery would have been of use, but alas I could not make it work. So I replaced it instead with cursed using</code> hackery. (Rethought_Group</code> is just Group</code> here.) Group_Store :: struct { data: [HAND_SIZE]Group, using _meta: bit_field u8 { cursor: u8 | 7, player: Player | 1, } } store_insert :: proc(store: ^Group_Store, group: Group) -> Group_Handle { assert(group.alive, "trying to insert a dead group!") store.data[store.cursor] = group defer store.cursor += 1 return Group_Handle{idx = store.cursor, owner = store.player, valid = true} } store_get :: proc(store: ^Group_Store, key: Group_Handle) -> (ret: ^Group, ok: bool) { (key.valid && store.player == key.owner && key.idx < store.cursor && store.data[key.idx].alive) or_return return &store.data[key.idx], true } store_remove :: proc(store: ^Group_Store, key: Group_Handle) -> (ret: Group, ok: bool) { grp := store_get(store, player, key) or_return defer grp.alive = false return grp^, true }</code></pre> This covers it, maybe? Time to delete slotmap.odin</code> and get going with some Compiler-Driven Development. I will not reprint all the changed code here, as it would be boring and a slog. Suffice it to say I did an admirable job at it and all the calls to slotmap</code> were handled excellently and appropriately. Now time to check if that effort fixes that first bug. Here is the test code again (slightly edited): main :: proc() { game := game_init() defer game_destroy(game) ok: bool ok = game_make_move(&game, Move{hex = {0, 0}, tile = tile_from_id(63, .Guest)}) fmt.printfln("%v", ok) ok = game_make_move(&game, Move{hex = {1, 0}, tile = tile_from_id(63, .Host)}) fmt.printfln("%v", ok) for key in game.groups_map { fmt.printf("%v, ", transmute(u8)key) } }</code></pre> And voila! It gives two different keys of 128</code> and 192</code> (though I can't quite tell which is which) I caught a subtle bug while I was going through the code, which gave the need for a valid</code> field in the key, so that it guarantees no valid key is 0</code>. Testing with different pairs of first moves has gone swimmingly. Things get captured when they should and get rejected when they should. But testing a whole game is more difficult to verify looking at stdout</code> printed arrays of numbers. A different visualization is required. Visualizing the Board</h2> A hexagonal board does not quite lend itself to being nicely printed in the terminal. Even if the pieces were simple stones (or chess pieces, available nicely as Unicode code points), However, solutions as old as time, or at least as old as monospaced fonts, exist. # pointy representation # "Simply" shift the squares each row half a "square". # this is the coordinate system used in the code. -3 | | | | | -2 | | | | | | -1 | | | | | | | 0 | | | | | | | | 1 | | | | | | | 3 2 | | | | | | 2 3 | | | | | 1 -3 -2 -1 0 # or _pointier_, without coordinates. / \ / \ / \ / \ | | | | | / \ / \ / \ / \ / \ | | | | | | / \ / \ / \ / \ / \ / \ | | | | | | | / \ / \ / \ / \ / \ / \ / \ | | | | | | | | \ / \ / \ / \ / \ / \ / \ / | | | | | | | \ / \ / \ / \ / \ / \ / | | | | | | \ / \ / \ / \ / \ / | | | | | \ / \ / \ / \ / # flat representation # letter/number coordinates system __ __/ \__ __/ \__/ \__ __/ \__/ \__/ \__ / \__/ \__/ \__/ \ g\__/ \__/ \__/ \__/7 / \__/ \__/ \__/ \ f\__/ \__/ \__/ \__/6 / \__/ \__/ \__/ \ e\__/ \__/ \__/ \__/5 / \__/ \__/ \__/ \ d\__/ \__/ \__/ \__/4 c\__/ \__/ \__/3 b\__/ \__/2 a\__/1</code></pre> For my implementation of Havannah</a>, I went with the Flat ASCII representation. However, since the directions are defined as TOP_RIGHT</code>, RIGHT</code>, etc, the Pointy representation is more appropriate. The compact version above is also, well, compact. Perfect for quick visualizations and debugging, especially that the board size in Dominions is much larger. Rendering the board is the easy part. But pieces in Dominions are distinct from each other, and it is not feasible to just draw the Tile's shape in the terminal. Firstly, that would be a very complex endeavour, and secondly it would balloon the drawing's size. But each tile can be represented, is represented, by a unique number from 1 to 63, plus a flag for the Owner. Using octal numbers, this can be made much nicer. Octal numbers today are mostly useless. Every language supports them because, well, they cost nothing to add, and they have one niche use in Unix file permissions. The reason they are useful for Unix file permissions is that these come in sets of three flags. So each set can be represented neatly by one octal digit (which stands for three bits).10</a> It just so happens that the Tiles in this game are also divided in two sets of three bits: three bits for the Right-side connections, and three to the Left-side connections, and two bits for Owner and Controller, detailed earlier</a>. This makes it so the right digit represents the right-side connections, and the middle digit, to the left represents the left side connections. The rightmost digit is either 0</code> or 1</code> donating the Owner, and using ANSI magic, the tile is colored by the Controller's color. Neat, hah? While this is unusable for an actual playable application meant for humans (a Graphical interface is more suited), it is very useful for debugging the game's status at any given point. With some trial and error, this is the printing code, the result of which I do not know to show on the web without a screenshot. import "core:encoding/ansi" import "core:fmt" board_print :: proc(board: Board) { w_on_b :: ansi.CSI + ansi.FG_WHITE + ";" + ansi.BG_BLACK + ansi.SGR b_on_w :: ansi.CSI + ansi.FG_BLACK + ";" + ansi.BG_WHITE + ansi.SGR end :: ansi.CSI + ansi.RESET + ansi.SGR row := min(i8) for tile, idx in board { hex := hex_from_index(idx) if hex.y > row { row = hex.y if row != -N do fmt.println("|") for i in 0 ..< abs(row) { fmt.print(" ") } } if tile_is_empty(tile) { fmt.print("| ") } else if .Controller_Is_Host in tile { fmt.printf( "|" + w_on_b + "%3o" + end, tile & ~{.Controller_Is_Host} ) } else { fmt.printf("|" + b_on_w + "%3o" + end, tile) } } fmt.println("|") }</code></pre> Perfect. Every other pair of opening moves I tried works as expected. So now is the time for the Ultimate Test: transcribe the full example game! The Example Game</h2> The full list of moves for the example game is here, in the source code of the game's page</a>. Parsing the moves is a fun exercise, even if I will not use the same notation eventually, so I will do that. The first task is to know how the coordinates map from the Mindsports representation to mine. This one is straightforward, if a bit tricky. The letters are the rows, or the hex.y</code> coordinate, so A to Q</code> map to 8 to -8</code> (yes, backwards.) The columns map 1 to 17</code> to -8 to 8</code>. The center hex goes from i9</code> to {0, 0}</code>. The Tile numbers do not map as neatly. P63</code> is the same, but not the others. It is important to know how the directions map between both representations. Amusingly, I found out that the P1</code> tile is also the same, but the Mindsports representation advances counter-clockwise, while mine does it clockwise. So Top_Left</code>, above being 32</code>, is here marked as 2</code>, and so on. That gives us this: tile_mindsports_id :: proc(id: u8, player: Player) -> (ret: Tile) { if id & (1 << 0) > 0 do ret |= {.Top_Right} if id & (1 << 1) > 0 do ret |= {.Top_Left} if id & (1 << 2) > 0 do ret |= {.Left} if id & (1 << 3) > 0 do ret |= {.Btm_Left} if id & (1 << 4) > 0 do ret |= {.Btm_Right} if id & (1 << 5) > 0 do ret |= {.Right} switch player { case .Host: ret |= HOST_FLAGS case .Guest: } return }</code></pre> And to parse each move: import "core:strconv" move_mindsports_parse :: proc(str: string) -> (ret: Move, ok: bool) #optional_ok { // move format: B[P12p15] player: Player switch str[0] { case 'W': player = .Guest case 'B': player = .Host case: return {}, false } c := 3 tile: Tile for len := 2; len >= 0; len -= 1 { if len == 0 do return {}, false if id, ok := strconv.parse_uint(str[c:][:len]); ok { c += len tile = tile_mindsports_id(u8(id), player) break } } row := N - i8(str[c] - 'a') c += 1 col: i8 for len := 2; len >= 0; len -= 1 { if len == 0 do return {}, false if res, ok := strconv.parse_uint(str[c:][:len]); ok { col = i8(res) - N - 1 break } } return {tile = tile, hex = {col, row}}, true }</code></pre> Ok, now it is time to play through the game, where MOVE_LIST</code> is a defined constant: main :: proc() { game := game_init() defer game_destroy(game) moves := strings.split(MOVE_LIST, ";") for m, idx in moves { move: Move ok: bool move, ok = move_mindsports_parse(m) if !ok { fmt.println("MOVE PARSER BROKE") break } fmt.println(idx + 1, m) ok = game_make_move(&game, move) if !ok { fmt.println("COULD NOT MAKE MOVE") break } board_print(game.board) buffer: [2]u8 os.read(os.stdin, buffer[:]) // advancing manually } }</code></pre> Now it is possible to go through the game move by move and compare, manually, the results of this program vs the "canonical" game client. Second Bug</h2> Running the engine hit an assertion at move 44. All the moves prior to that, including Suicides (amusingly enough), worked fine, but move 44 has the first Capture in the game. The move right after this position, where the first player captures at g2</code>: (They are the same. You can even see the first player's suicides as black numbers starting with 0</code>). The capture logic can be found above</a>. The failure happens here: // == merge level 2 surrounding friendlies into blessed group for key in level_2_surrounding_friendlies { temp_grp, ok := store_remove(friendly_grps, key) assert(ok) // <--- right there. blessed_grp.state |= temp_grp.state blessed_grp.extendable &= temp_grp.extendable }</code></pre> I am not sure what this was supposed to accomplish, to be honest; and commenting it out, amusingly, progresses the game and resolves that move, until move 50 when it crashes. Unfortunately between both moves the state is all wrong. Either way, the assert that fails is slightly ahead of that loop. // == go over surrounding enemy groups to see if they're dead. capture_occurance := false for key in surrounding_enemy_grps { temp_grp, ok := store_get(enemy_grps, key) assert(ok, "Enemy slotmap does not have enemy Key") // <--- right here temp_grp.state[hex_to_index(move.hex)] |= .Enemy_Connection // --- snip</code></pre> This part seems too important to comment out, not to mention the all wrong state. Obviously, however, there is a problem in assigning and updating Group</code> keys. Probably some keys are persistent in group_map</code>, and are not getting updated as new Group</code>s happen. So obviously the next step is to update the board visualization to include both Tiles and Group IDs. Updated Board Visualization</h2> To print each row twice, each time with different data, two buffers are needed instead of printing directly. The two buffers are flushed out once per row.string</code>s in Odin are immutable, and to have a string buffer, one needs to dip one's toes into the Odin core library, and use a strings.Builder</code>. board_print :: proc(board: Board, grp_map: [CELL_COUNT]Group_Handle) { w_on_b :: ansi.CSI + ansi.BG_BLACK + ";" + ansi.FG_WHITE + ansi.SGR b_on_w :: ansi.CSI + ansi.BG_WHITE + ";" + ansi.FG_BLACK + ansi.SGR end :: ansi.CSI + ansi.RESET + ansi.SGR tiles_buffer := strings.builder_make() grps_buffer := strings.builder_make() defer strings.builder_destroy(&tiles_buffer) defer strings.builder_destroy(&grps_buffer) row := min(i8) for tile, idx in board { hex := hex_from_index(idx) if hex.y > row { row = hex.y if row != -N { fmt.sbprint(&tiles_buffer, "|") fmt.sbprint(&grps_buffer, "|") fmt.println(strings.to_string(tiles_buffer)) fmt.println(strings.to_string(grps_buffer)) strings.builder_reset(&tiles_buffer) strings.builder_reset(&grps_buffer) } for _ in 0 ..< abs(row) { fmt.sbprint(&tiles_buffer, " ") fmt.sbprint(&grps_buffer, " ") } } if tile_is_empty(tile) { fmt.sbprint(&tiles_buffer, "| ") fmt.sbprint(&grps_buffer, "| ") } else if .Controller_Is_Host in tile { fmt.sbprintf( &tiles_buffer, "|" + w_on_b + "%3o" + end, tile & ~{.Controller_Is_Host} ) fmt.sbprintf( &grps_buffer, "|" + w_on_b + "%3X" + end, transmute(u8)grp_map[idx] ) } else { fmt.sbprintf( &tiles_buffer, "|" + b_on_w + "%3o" + end, tile ) fmt.sbprintf( &grps_buffer, "|" + b_on_w + "%3X" + end, transmute(u8)grp_map[idx] ) } } }</code></pre> There you go, much better. Back to the Second Bug</h2> Honestly, I am not enjoying debugging this. I feel it would be easier to rewrite the whole thing with the learned assumptions. But persevere I must. Going back to move 44, where the first crash happened: All the groups that must be separate are separate, and all the ones that must be connected are connected. So, the easy assumption is that all the code paths triggered so far are correct. The offending next move, W[P56q10]</code> does the following, physically: It connects to two enemy groups, C7</code> and C8</code>. And to one friendly group, 88</code>.</li> Groups C7</code> and 88</code> both lose all liberties, as does the new placed Tile.</li> Enemy group C7</code> is captured, as it loses all liberties and is connected, from the other side, to another friendly group 87</code>.</li> The result of this is the newly placed tile, friendly groups 88</code>, 87</code>, and captured enemy group C7</code> must now all be unified into one new group, ideally should be 88</code>.</li> </ol> So what does the code actually do? Time to use a debugger. Setting up the Debugger</h3> Thankfully, due to dabbling in Rust, I already had LLDB installed as a VSCode extension. All is needed for it to work is to create a launch.json</code> file (that's almost filled up already), and have it work on an executable built with: $ odin build . -debug</code></pre> Using a template from the Odin Discord server and adapting it to macOS (from Windows), I got the following two files in the .vscode</code> directory: // launch.json { "version": "0.2.0", "configurations": [ { "type": "lldb", "request": "launch", "preLaunchTask": "Build", "name": "Debug", "program": "${workspaceFolder}/build/debug", "args": [], "cwd": "${workspaceFolder}", } ] }</code></pre>// tasks.json { "version": "2.0.0", "command": "", "args": [], "tasks": [ { "label": "mkdir", "type": "shell", "command": "mkdir -p build", // <-- Does not work on Windows. }, { "label": "build", "type": "shell", "command": "odin build . -debug -out:build/debug", "group": "build" }, { "label": "Build", // <-- launch task runs this "dependsOn": [ "mkdir", // which runs this "build" // then this ] } ] }</code></pre> Now all I need to do is set breakpoints and click the VSCode button that says Debug</code>. Yay for technology. But how to set a conditional breakpoint based on the move number when game_make_move</code> does not know the move number? Normally, one would either have to set the move number as a global variable, or pass it on as an explicit parameter into every procedure, both are equally annoying. In Odin, I can just use the context</code> system. The Context</code> struct, which is passed implicitly by default to every Odin procedure, contains a user_index</code> field which can be set to whatever. Inside game_make_move</code> or any of its inner procedures, I simply check for context.user_index</code>. // inside main context.user_index = idx ok = game_make_move(&game, move)</code></pre> // inside game_inner_update_state for key in level_2_surrounding_friendlies { if context.user_index >= 43 { fmt.println(context.user_index) // <-- set breakpoint here } temp_grp, ok := store_remove(friendly_grps, key) assert(ok) // <-- failing assertion blessed_grp.state |= temp_grp.state blessed_grp.extendable &= temp_grp.extendable }</code></pre> And that's how it looks when reached. Tedious Debugging</h3> This part is not cohesive writing. I will be listing my findings, but it might not be the easiest thing to follow. Going through the variables on the left of the debugger view, some things jump out. level_2_surrounding_friendlies</code> has invalid keys. (Literally, valid</code> is set to false.) What seems to be happening is that, earlier a bit, I am setting the just played tile as .Enemy_Connection</code> for the neighboring enemy group. But when when the key is looked up in game.groups_map</code>, it is still set to 0</code> as it is not added there yet. Adding blessed_key</code> to group_map</code> on creation passes the game_make_move</code> procedure fine, but trips on another assertion in game_regen_legal_moves</code>. Apparently, the new group is not in any group store. It is getting removed somehow. Using Odin magic like adding a loc := #caller_location</code> parameter to store_remove</code>, I can get it to print where that remove call was made. (And yes, I could have put in a breakpoint in there too.) import "core:fmt" store_remove :: proc( store: ^Group_Store, key: Group_Handle, loc := #caller_location // <-- Default value so call sites don't change. ) -> (ret: Group, ok: bool) { if key.idx == 8 && key.owner == .Guest { // <-- the offending group fmt.println("REMOVED HERE:", loc) } grp := store_get(store, key) or_return defer grp.alive = false return grp^, true }</code></pre> Running it quickly shows it is removed in that level_2_surrounding_friendlies</code> loop. Where the first assert failed. Adding a condition to check for equality with blessed_key</code> like the following, allows the game to hit no more asserts and continue to completion. Yay! for key in level_2_surrounding_friendlies { if key == blessed_key do continue temp_grp, ok := store_remove(friendly_grps, key) assert(ok) blessed_grp.state |= temp_grp.state blessed_grp.extendable &= temp_grp.extendable }</code></pre> Now that the example game runs to completion without hitting any assertions, time to check the state's correctness. The final position in the game seems to be all correct, so I will consider this implementation now bug free. Maybe for part 2. What's left?</h2> Work is not done on this by any means, even if I ascertained it is 100% bug free. (the legal move generator is not thoroughly tested really.) But I have spent a month, on and off, on this and my wife is frankly sick of me. So here is a list of what is left to implement in round 2: Undoing Moves: Necessary for engine implementations, and easy in theory. All is needed really is to copy the game's state into a dynamic array of previous game states. To save memory, only saving previous Board</code>s is required, as they actually have enough data to generate all the remaining fields.</li> An interactive interface. A way to interact with the game outside of writing procedure calls in main()</code> Either through a wasm module or a CLI interface, or a common interface either one can use.</li> Nicer, graphic visualization. The ASCII visualization is useful for debugging, but it is not actually playable. This game needs beautiful art and a beautiful Graphical User Interface.</li> </ol> Lessons Learned</h2> So throughout working on this article (for almost a whole month) I advanced my knowledge about the following: The Odin programming language,</li> Interop between Odin and Rust (and any other language really),</li> HTML and CSS and Zola templating (in setting up this website),</li> Basic usage of LLDB debugging,</li> </ol> So, this was a fun ride. But it is time to move on to something else. The code here is in the bustan</code> repository</a>. You can see the evolution of the code, and much of what is in this article, through the git history. Until later. As mentioned earlier, the Blank tile is not used in the game. This permits using a sentinel value of 0</code> (or really any value with the smallest six bits set to 0</code>) to mark an empty cell. Since Tiles are a u8</code> bit-set anyway, why waste memory on pointers (which are wider), or Maybe</code>, which is at least an extra byte in size? I am not thinking too hard about performance (I know nobody will use this), but it is an interesting constraint to keep in mind. ↩</a> </li> I should be careful not to index into Board</code> directly, however. ↩</a> </li> I love that move</code> is not a keyword here, which is really annoying in Rust. ↩</a> </li> Generational arena, generational handles, handle-based map, a rose by any other name. ↩</a> </li> It is Oscillation because the resulting Group has no Liberties, and therefore has no clear Controller, so it oscillates between both colors. This is way the Blank tile has no role in the game: it automatically oscillates. ↩</a> </li> Technically, only a record of whether the last move was a pass is needed, but last_move</code> is semantically clearer than a last_move_was_a_pass</code> (or pass_ends_the_game</code> or the_end_is_nigh</code>) boolean or a player_who_last_made_a_move</code> enum field. It may also be useful to highlight the last move in a GUI. ↩</a> </li> Rust would totally yell at me. Then tell me to implement the trait to define the behavior myself. ↩</a> </li> Freeling does not specify in which order captures are processed. I am assuming here the order is the same as Go. Anyway, all this needs to be verified later once (if?) the engine is implemented. ↩</a> </li> I have been thinking of why Groups need to own their own data anyway? Group membership is clearly delineated in Game.group_map</code> as it currently is (which would become [CELL_COUNT]Group_Handle</code> under the new regime.) Liberties and Enemy Connections can be calculated and collected with a combination of game.groups_map</code> and, well, game.board</code>. But that's another day's battle. ↩</a> </li> A fuller explanation of file permissions, as irrelevant as they are to this article, can be found here</a>. ↩</a> </li> </ol> </section>}

النشيد السوري

The Unbearable Frustration of Figuring Out APIs

First Steps</h2>

A Dumb Introduction to z3

Define Zig Enums from Data

LLMs in the Building Industry

For Potential Homeowners</h2>

For Engineering Offices</h2>
There are many different professions working in a regular Engineering 2</a></sup> practice. This is an overview following the different stages of a project.</p>

Writing a C Compiler, in Zig

Chapter 10: Linking

Chapter 9: Functions

Chapter 8: Loops

Chapter 7: Blocks

Lessons Learned</h2>
The most fun I had here was experimenting with the AST pretty printer. Cracking the nut of the duplicate new line problem.</p>

Or pull a Python and output `...</code> or pass</code>. ↩</a></p> </li> </ol> </section>`

Chapter 6: Conditions

Chapter 5: Variables

Lexer and Parser</h2>
One new token, `=</code>. Adding this is trivial so I am sparing you the details. There is one thing to update that, while belongs to the parser, is placed in lexer.zig</code>. The precedence chart.</p>`

Chapter 4: Logic

Parser</h2>
The parser is a relatively simpler affair. More binary operations with different precedence levels. None of them are right associative, so nothing but updating the AST and the various `switch</code> statements. Nothing much to write about.</p>`

Chapter 3: Binary

Extra Credit</h2>
The Book has an extra credit section for doing bitwise operations. I am skipping extra credit this run. They are useful and interesting but also more churn in the following chapters. This is the last time you will hear me talk of it.</p>

Chapter 2: Unary

Chapter 1: Intro

Placemaking Syria

Parsing Hearthstone Deck Codes

The Hanging Gardens Problem

Square Gardens</h2>
</p>
Phrasing the problem in terms of Squares, and 2D connections, rather than Cubes, shrinks the problem space by an order of magnitude. Plus, there is a known solution given by Freeling</a>!</p>

My Ideal Code Editor

Has</h2>
However, MICE has these nice features:</p>

The Garden of Odin

The Game Rules</h2>
So far, I made no mention of the game's rules, only talking about its physical properties: the tiles, the board, the game zones. But players cannot place Tiles where they choose: here are the rules summarized for your convenience:</p>
</p>

Abdul Rahman Sibahi

Adding WASM Plugins to Your App

النشيد السوري الجديد

Abdul Rahman Sibahi

Adding WASM Plugins to Your App

What is Left</h2> This article has gone long enough. Now all the hard work is done, all that remains is small details like the following:</p>

النشيد السوري الجديد

النشيد السوري

The Unbearable Frustration of Figuring Out APIs

First Steps</h2>

Async Woes</h2> Running this with swift run -q MyCLI 你好</code> gives the following message:</p>

Final Code</h2> This is the full final code, with some proper error names thrown in for good measure.</p>

A Dumb Introduction to z3

A Simple Equation</h2> Let's start with what might be the simplest, dumbest equation. Solve for x</code>:</p>

A Jump in Complexity</h2> In school, jumping from solving equations with a single variable to equations with two variables was a real jump on complexity. Everything was doubled! Here is a pair of equations we will try to solve next:</p>

Multiple Solutions</h2> As I am sure you know from your high school math, some equations have multiple solutions. Here is a simple one.</p>

Define Zig Enums from Data

The Library</h2> Here is the cool stuff, if I may. Since this for defining a normal, non-generic type, we shall start it thus. All the code in the following snippets will be in there.</p>

LLMs in the Building Industry

For Potential Homeowners</h2>

For Engineering Offices</h2> There are many different professions working in a regular Engineering2</a></sup> practice. This is an overview following the different stages of a project.</p>

Writing a C Compiler, in Zig

Chapter 10: Linking

Lexer</h2> The Book spends 15 pages talking about linkage and static variabeles and tentative declarations, etc, before it gets to the lexer. Only two new keywrods are needed: static</code> and extern</code>. This is the current keyword static map:</p>

Next Thing to Do</h2> I am undecided yet. Either go for Part 3, which is about optimizations, or do Part 1 again in another language, like Swift or .. I dunno .. TypeScript? Maybe I will start working on my own language, or working on a compiler for Hare. Or maybe just go find a job.</p>

Chapter 9: Functions

Code Emission</h2> The changes here are pretty small. The new instructions are one thing, but there are new registers and new register sizes for push</code>, which takes 8-width registers. The logic for printing registers got pretty large so I put it in its own function.</p>

Chapter 8: Loops

Lexer, AST and Parser</h2> The lexer is just a few new keywords. Been there, done that. Changes to the AST are more interesting. No less than five new statements, one for every new keyword. Here is the new Stmt</code> without further ado:</p>

Fixing the Formatting Problem</h2> As mentioned before, there are four options, on different levels of unhingedness.</p>

Lessons Learned</h2> Loops are interesting</li> Do not add a for</code> loop to my language.</li> To love Rust more. Rust would never</em> have let me segfault!</li>

Chapter 7: Blocks

Semantic Analysis</h2> I wish this were as easy as filling up the blanks, but that is actually where the meat of this chapter is. Because C allows shadowing in inner scopes. The following is perfectly valid C code.</p>

Lessons Learned</h2> The most fun I had here was experimenting with the AST pretty printer. Cracking the nut of the duplicate new line problem.</p> Or pull a Python and output ...</code> or pass</code>. ↩</a></p> </li> </ol> </section>

Chapter 6: Conditions

AST</h2> One new statement and one new expression today. Compound statements survive another day without being implemented.</p>

Chapter 5: Variables

Lexer and Parser</h2> One new token, =</code>. Adding this is trivial so I am sparing you the details. There is one thing to update that, while belongs to the parser, is placed in lexer.zig</code>. The precedence chart.</p>

New Statements and Expressions!</h3> Two new expressions:</p>

Chapter 4: Logic

Parser</h2> The parser is a relatively simpler affair. More binary operations with different precedence levels. None of them are right associative, so nothing but updating the AST and the various switch</code> statements. Nothing much to write about.</p>

Internal Representation</h2> The IR this chapter is a lot more interesting than usual. Since &&</code> and ||</code> are short circuting operators, it means now is the time have jumps and labels and goto</code>s and jump-if-zeros. With conditionals, anything is possible.</p>

Chapter 3: Binary

Extra Credit</h2> The Book has an extra credit section for doing bitwise operations. I am skipping extra credit this run. They are useful and interesting but also more churn in the following chapters. This is the last time you will hear me talk of it.</p>

Lessons Learned</h2> Implemented a macro</del> comptime</code> function.</li> Closures at home.</li>

Chapter 2: Unary

Lexer</h2> There are three tokens to add in this chapter: ~</code>, -</code>, and --</code>. The decrement operator is only being tokenized here to reject illegal C syntax like return --2</code>, but otherwise will not be implemented.</p>

Codegen</h2> The first task here is almost invariably to update the assembly AST to include more instructions. Again deviating from the book, I will splat the unary operator into the instructions. This will totally bite me later. The updated parts of the new AST look as follows:</p>

For Engineering Offices</h2>
There are many different professions working in a regular Engineering 2</a></sup> practice. This is an overview following the different stages of a project.</p>

Lessons Learned</h2>
The most fun I had here was experimenting with the AST pretty printer. Cracking the nut of the duplicate new line problem.</p>

Or pull a Python and output `...</code> or pass</code>. ↩</a></p> </li> </ol> </section>`

Lexer and Parser</h2>
One new token, `=</code>. Adding this is trivial so I am sparing you the details. There is one thing to update that, while belongs to the parser, is placed in lexer.zig</code>. The precedence chart.</p>`

Parser</h2>
The parser is a relatively simpler affair. More binary operations with different precedence levels. None of them are right associative, so nothing but updating the AST and the various `switch</code> statements. Nothing much to write about.</p>`

Extra Credit</h2>
The Book has an extra credit section for doing bitwise operations. I am skipping extra credit this run. They are useful and interesting but also more churn in the following chapters. This is the last time you will hear me talk of it.</p>