Fuzzing is the art of testing through randomized inputs. It attempts to minimize the inputs required to produce a failure. I'm applying fuzzing to my HTTP parser to catch any crashes that mutation to valid requests might cause.

Case Study #1

AFL++ discovered that I had several out of bounds index issues related to parsing the request line. For one of them, it took the following valid request:

GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.88.1
Accept: */*

And, mutated it by removing the target from the request line:

GET HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.88.1
Accept: */*

Which illuminated the following bug:

$ zig build test
run test: error: thread 4165830 panic: index out of bounds: index 0, len 0
src/request.zig:277:27: 0x25f5a5 in parse (test)
                if (target[0] == '/') {

I failed to check the length of the target string before indexing into it to determine whether it was in absolute or origin form. One simple patch later, and it's fixed:

diff --git a/src/request.zig b/src/request.zig
index 0df8602..46b6573 100644
--- a/src/request.zig
+++ b/src/request.zig
@@ -260,6 +260,8 @@ pub fn Request(
                     return error.HttpNotImplemented;
                 const target = iter.next() orelse
                     return error.HttpBadRequest;
+                if (target.len == 0)
+                    return error.HttpBadRequest;
                 const version_str = iter.next() orelse
                     return error.HttpBadRequest;
                 if (version_str.len == 0)

This was run without any of AFL++'s fancy binary instrumentation. It was using the -n parameter which as I understand it blindly mutates the data according to some algorithm which had no knowledge of the internal structure of the program. Which leads me to the next section.

Woes w/ AFL++

AFL++ seems to be an amazing tool, but damn did I shoot myself in the foot by deviating from the norm. AFL++ normally instruments the software under test by providing their own compiler/linker which is meant to be used in the program's build process. For most C/C++ programs that seems to be as simple as setting an environment variable CC=afl-clang-fast or similar.

The issue for me is that I want to use it with Zig. There's been some prior work on getting Zig to play nicely with AFL++¹ which uses AFL++'s linker to instrument the executable after Zig has compiled it. However, that no longer seems to be entirely working. From my initial look, it appears to be something to do with LLVM 17. Zig has updated LLVM to the bleeding edge while AFL++ is taking a more conservative approach.

In any case, the normal path does not work for Zig. There remains a few options. AFL++ can use -n for normal fuzzing without instrumentation. Their documentation indicates it will perform much worse than instrumented code.

Another option is to use Qemu mode. Qemu is another amazing project which does emulation/virtualization. In this case, AFL++ seems to be using it to give visibility into the code paths which might be traversed by the software under test. I'm still a bit fuzzy on what that means, but I want it.

Qemu Mode Debugging

I fired up afl-fuzz -Q -i src/data/request/ -o qemu-fuzz -- zig-cache/o/9f2296761a08d6864bb1eb7120015cd1/fuzz, and was immediately told I needed another binary afl-qemu-trace. Turns out Debian's AFL++ build doesn't have Qemu mode. As an alternative, I launched AFL++'s official docker container which thankfully does have Qemu mode.

$ afl-fuzz -Q -i src/data/request/ -o qemu-fuzz -- zig-cache/o/9f2296761a08d6864bb1eb7120015cd1/fuzz
...

[-] PROGRAM ABORT : Program 'afl-qemu-trace' not found or not executable
         Location : find_binary(), src/afl-common.c:361

Running again gave me another error report about needing at least one non-crashing test case for AFL++ to start. I already know each of those test cases are valid and do not cause crashes because they are all in my handcrafted test suite. So, it must be something else.

Turns out, it's Qemu, not my program:

$ AFL_DEBUG=1 afl-fuzz -Q -i src/data/request/ -o qemu-fuzz -- zig-cache/o/9f2296761a08d6864bb1eb7120015cd1/fuzz
...
[*] Attempting dry run with 'id:000000,time:0,execs:0,orig:chunked-transfer-encoding.http'...
[*] Spinning up the fork server...
AFL forkserver entrypoint: 0x2354f0
Debug: Sending status c201ffff
[+] All right - fork server is up.
[*] Extended forkserver functions received (c201ffff).
[*] Target map size: 65536
[D] DEBUG: calibration stage 1/7
qemu: uncaught target signal 4 (Illegal instruction) - core dumped

There's an illegal instruction. This beautiful person gave me the clue I needed on that one. I want to know which instruction caused the signal, and it turns out GDB will give you the address when it detects SIGILL. Further, AFL++/Qemu provides a mechanism for attaching to the program using GBD.

In one console (inside the container for me):

$ afl-qemu-trace -g 5555 zig-cache/o/9f2296761a08d6864bb1eb7120015cd1/fuzz

In parallel, in another:

$ gdb
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote localhost:5555
Remote debugging using localhost:5555
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x00000000002354f0 in ?? ()
(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
0x00000000003210ce in ?? ()
(gdb)

0x00000000003210ce is the culprit. Now, to identify what that instruction is:

$ objdump -D zig-cache/o/9f2296761a08d6864bb1eb7120015cd1/fuzz | grep 3210ce
  3210ce:       c4 e2 79 78 c0          vpbroadcastb %xmm0,%xmm0

vpbroadcastb, an vector instruction (AVX2 (maybe?)). Qemu doesn't seem to support AVX2, so I need to disable that somehow. I tried disabling it in the Zig build process: zig build fuzz -Dcpu=native-avx-avx2, but that didn't get rid of the illegal instruction. It just moved it to a different address.

That's where I am with that right now.

The zig compiler can be told to build an executable which uses an instruction set which is more compatible using -Dcpu=baseline. Building the fuzzer with that allows AFL++ to run against zig executables while in Qemu mode... And, this found another bug! This time it's not in my code. It's in the standard library. ziglang/zig#17867.