# Fuzzing Excellent playlist: {% embed url="" %} ## Coverage-Guided Fuzzing Takes a corpus of inputs. Then it picks an input and does a random mutation on it. See if this mutated test covers new code; if so, store it in the corpus. Repeat the process in a loop. API fuzzing: {% embed url="" %} API Fuzzing {% endembed %} **When should I use coverage guided fuzzing?** Coverage guided fuzzing is recommended as it is generally the most effective. This works best when: * The target is self-contained. * The target is deterministic. * The target can execute dozens or more times per second (ideally hundreds or more). ### **Coverage** **What should the coverage look like?** * You want to cover as many lines as possible, * You want to cover those lines as uniformly as possible (not only cover some line once) * You don’t need to instrument things you don’t care about (well-tested 3rd party libraries, etc **How do I deal with coverage gaps?** * Edit target code to take a branch/generate structured data (least amount of work) * In API testing, you can edit the harness to make code reachable or more likely to be reached. (medium amount of work, but doesn’t affect the tested API) * Add a test case to reach it directly (most amount of work) * Nop out/avoid some large sections of code that are just noise (if you’re fuzzing ffmpeg, only fuzz one format) **I keep hitting DoS bugs** You can just patch around uninteresting bugs until you find an RCE. **Reporting** You can put the main function in a try-catch block, because you often don’t care about the exceptions the application knows to throw. You care about things that create a crash, like segfaults and stuff. ### **Sanitization** Just detecting crashes isn’t good enough, since not all memory corruptions produce a crash (like if a single byte of memory is corrupted). Instead, you should use something like ASan (Address Sanitizer) to detect memory corruption. However, it can be a good idea to run LibFuzzer without sanitization first, as that way you’ll find the more critical bugs first. Otherwise, it’ll trip on a smaller issue (off-by-one heap overflow for example) and it won’t get to the bigger issue until you patch it *Note: LeakSanitizer for memory leakage issues?* ### **Variant Analysis** Also known as **variant fuzzing**, **differential fuzzing**, and **bounded analysis**. {% embed url="" %} Timestamped explanation {% endembed %} This sort of fuzzing doesn’t look for crashes or infinite loops. Instead, it checks if the output of a function exceeds expected bounds. **Basically, it finds logic bugs** For example, let’s say there’s a language analysis function, which rates a Youtube comment based on its offensiveness, on a scale of 0-9. If we do variant analysis on that function, then we would run LibFuzzer for example, and check the function’s output. If the output is less than 0 or more than 9, then we have hit a logic bug. **What is it useful for?** * Finding bugs in memory-safe languages (golang, javascript) for certain functions * Fuzzing Ethereum Solidity smart contracts * There have been a few times where a coin operator has created a contract with bugs in it. * You can use this fuzzing method to find logical errors, where you can give yourself too much money. ### **AFL** **Benefits over LibFuzzer:** * Easier to set up * Can be more reliable or work in more difficult situations Negatives: * Might catche less bugs * Doesn’t have memory leak detection instrumentation, for example. * **It does have address sanitation**, though, which is good. * With LibFuzzer you can easily target individual functions because you have to write an LLVMTestOneInput entry function. **Setting Up** Example of setting up fuzzing for[ duktape javascript engine](https://github.com/svaarala/duktape/releases/tag/v2.0.3) For normal compilation, you can just run `make -f Makefile.cmdline` Create a directory `in` w\`ith a corpus of a few good examples (of javascript examples, in this case). I added 2 examples in files 000 and 001: 1. `1 + 2` 2. `var num = 2; num * 4` Also make an `out` dir. Then, in the makefile, you’ll need to specify the afl compiler as the compiler of the program, and compile an AFL version of the program. There’s a segment that explains this in the AFL github page. And all we have to change in the Makefile.cmdline is to change: ``` CC = gcc ``` To: ``` CC = afl-clang ``` Then, make the file: ``` Make -f Makefile.cmdline ``` And run the fuzzer (`./duk` is the executable): ``` afl-fuzz -i in/ -o out/ -- ./duk ``` #### **AFL Workshop** {% embed url="" %} Use AFL++, it's maintained. Original isn't. Useful documentation is in AFL++ documentation: * docs/life\_pro\_tips.txt * docs/README.md * llvm\_mode/README.md ***Quickstart*** You have an inputs corpus in a folder (and make sure to add a valid input). ``` mkdir inputs ``` Compile the program with AFL: ``` CC=afl-clang-fast AFL_HARDEN=1 make ``` AFL\_HARDEN adds code hardening options Then fuzz it: ``` afl-fuzz -i inputs -o out ./vulnerable ``` ***Test Harnesses*** {% embed url="" %} Harness requirements: * Runs in the foreground * Processes input from stdin or from a specified file * Exits cleanly * Skips checksums and cryptographic integrity tests (or use a postprocessor), because the fuzzer will never get through those Recommended: * If you're going beyond simple program crashes, then the test harness should crash if the program is in an invalid state * Use AFL's deferred forkserver + persistent mode for speed *Simple Harness* This harness only tests one of the two functions Important points: * Define the size of the input buffer to limit the fuzzer * Initialize the input buffer with zeroes to make results reproducible! ``` #include #include #include #include "library.h" // fixed size buffer based on assumptions about the maximum size that is likely necessary to exercise all aspects of the target function #define SIZE 50 int main() { // make sure buffer is initialized to eliminate variable behaviour that isn't dependent on the input. char input[SIZE] = {0}; ssize_t length; length = read(STDIN_FILENO, input, SIZE); lib_echo(input, length); } ``` Compile and run like this: ``` AFL_HARDEN=1 afl-clang-fast harness.c library.c -o harness afl-fuzz -i in -o out ./harness ``` *Integer Inputs* This harness uses two 8-byte integer inputs if you specify the "mul" argument: ``` #include #include #include #include "library.h" // fixed size buffer based on assumptions about the maximum size that is likely necessary to exercise all aspects of the target function #define SIZE 100 int main(int argc, char* argv[]) { if((argc == 2) && strcmp(argv[1], "echo") == 0) { // make sure buffer is initialized to eliminate variable behaviour that isn't dependent on the input. char input[SIZE] = {0}; ssize_t length; length = read(STDIN_FILENO, input, SIZE); lib_echo(input, length); } else if ((argc == 2) && strcmp(argv[1], "mul") == 0) { int a,b = 0; read(STDIN_FILENO, &a, 4); read(STDIN_FILENO, &b, 4); printf("%d\n", lib_mul(a,b)); } else { printf("Usage: %s mul|echo\n", argv[0]); } } ``` Run it like this: ``` afl-fuzz -i in -o out ./harness mul ``` *Reading from files* The "@@" symbol specifies that the input is a filename. ``` afl-fuzz -i in -o out ./harness @@ ``` #### **LLVM Mode** AFL comes with the afl-gcc and afl-clang compilers by default. Where possible, you should prefer to use the afl-clang-fast compiler, because then you can use LLVM mode. However note that trying to compile a program with clang instead of gcc can sometimes fail. LLVM mode allows you to significantly speed up your execution for most targets because of multiple reasons: * [Deferred forkserver](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#3-deferred-initialization) * Tell AFL where to start each new process * Is especially useful for processes with an expensive setup phase that isn’t dependent on input * [Persistent mode](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md) * Don’t fork for each run, just loop around this bit of code * Requires that the program’s state can be completely reset so that multiple calls can be performed without resource leaks * [Shared Memory Fuzzing](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#3-deferred-initialization) *Note:* [*This documentation page*](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.lto.md) *says that afl-clang-lto is the best compiler to use because the instrumentation doesn't have collisions, and therefore gets better coverage. However, it's a work in progress and you might run into issues during compilation.* If there's a memory corruption that starts making it so that the program's state isn't completely reset and you're not using ASan to detect it, then you can use first persistent mode to fuzz to a certain extent to get a good corpus, and then fuzz in normal mode for consistency. Running in persistent mode might cause stability to lower, this is normal. As long as stability is over 90 percent, everything is fine. The [user manual](https://afl-1.readthedocs.io/en/latest/user_guide.html#path-geometry) goes into a bit more depth on this.
**Seed Corpus** Tips: * Find real inputs that exercise as much of the target as possible. * Keep your seed files small (under 1kB is ideal). * Don’t put in redundant files. Use afl-tmin and afl-cmin to minimize your seed corpus. Also when your fuzzing job has been running for a while (after the first cycle of the master fuzzer for example), you can also take your test corpus, minimize that and fuzz some more. You can use afl-ptmin to parallellize this process. {% embed url="" %} **Sanitizers** Sanitizers turn tiny issues into crashes which AFL can detect. Sanitizers are mutually incompatible, so run multiple fuzzing instances with different sanitizers * UBSan * MSan * ASan * TSan Use AFL\_HARDEN=1 to harden your program (fortify\_source and stack protection) and therefore catch more issues ***ASan*** ASan can require **a lot of memory** – so much that using it on a 64-bit system can be **infeasible** if the program isn't robust and doesn't enforce reasonable memory limits. Read the [documentation](https://github.com/mirrorer/afl/blob/master/docs/notes_for_asan.txt) for workarounds to this. Here are some options: * Compile as 32-bit * Gauge memory limits using recidivm * **Use cgroups** to limit available memory *Note: I haven't been able to get the cgroups script to work in the afl++ docker container.* ``` /AFLplusplus/utils/asan_cgroups/limit_memory.sh -u username_here afl-fuzz -i in -o out ./target ``` #### **Parallel Fuzzing** AFL runs on one core, so it makes sense to run one per core. However it’s not trivial. Multicore: * One main instance. -M * One secondary instance per core. -S * All share the same output directory. -o Multi Machine: * Lots of secondary instances * You need to sync state directories periodically *Tip: Run fuzzers with differing behaviours (sanitizers; fuzzing algorithms)* For large-scale serious fuzzing, it makes sense to use a fuzzing service that allows you to scale up massively. That way you don’t have to hack together multi-machine fuzzing yourself. * ClusterFuzz * OneFuzz #### **Dictionaries** Very useful for helping the fuzzer get through difficult parts where certain values are expected. Specifies interesting tokens the fuzzer can use to access paths it otherwise might not stumble upon. Sources: * “dictionaries” folder * Automatically generated with afl-clang-lto (llvm11+) * Statically finds string comparisons while compiling * This feature is called [autodictionary](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.lto.md#autodictionary-feature) * [libFuzzer’s collection](https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/fuzzers/dicts) * Automatically generated with libtokencap * At runtime, looks at calls to strcmp and memcmp and remembers tokens * Source code review Other similar useful features: * [Iaf-intel](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.laf-intel.md) * Works by splitting larger comparisons down to smaller ones so the fuzzer can traverse paths and get further. * Use `export AFL_LLVM_LAF_ALL=1` to enable all Iaf-intel passes **Beyond memory corruption** You can add checks to detect if bad things have happened, e.g: * Send input into two different math/crypto libraries and asserts that their output is identical * assert that result of BN\_sqr(x) == BN\_mul(x,x) - CVE-2014-3570 in OpenSSL * assert if any filesystem changes have occurred – CVE-2014-6271 in Bash (shellshock) If something bad has happened, then just crash the program manually using assert. #### **Miscellaneous Tips** Network targets: * Write a harness around the target function * Use inetd mode if your target supports it * Intercept network system calls, e.g using preenv The queue directory contains a corpus of test cases that exercise your program * You can run it through CoverageSanitizer or gcov and see what isn’t hit Resume a fuzzing run with `-i-` * You can recompile the binary and then do this #### **Triage Tips** * ASan traces can be helpful * Use afl-tmin to minimise and simplify test cases while retaining the original control flow * “Unique” bugs aren’t unique. They might be the same bug, just through a different control flow. Fix them as you go along and carry on fuzzing. * Memory limits can give false positives Use crashwalk to get a head start on triaging crashes and finding which ones are useful for exploitation. However, note that if you're using ASan, then this probably isn't always helpful, since ASan just does SigAbort when it finds something, and crashwalk doesn't know what to do with it. If you turn off ASan, then the program might not crash at all. {% embed url="" %}
{% endembed %} #### **When to Stop** Options: * Never - Fuzz as part of CI * When the cycles counter is green * Last new path was found many cycles ago * Pending paths is zero If you want to stop earlier: * Cycles counter is blue * Been running for a while * Look at the output of afl-plot Remember to check code coverage. * See which parts of the program are hit, and whether all the important functions are hit. You can check for code coverage using afl-cov. {% embed url="" %}
{% endembed %} #### **Examples** ***LibXML (Optimization)*** {% embed url="" %} We're compiling using afl-clang-fast here, but note that lto is usually better. We're using ASAN as the sanitizer. We're fuzzing the xmlReadMemory function. Instrumenting the library with ASAN: ``` CC=afl-clang-fast ./autogen.sh AFL_USE_ASAN=1 make -j 4 ``` Compiling the harness: ``` AFL_USE_ASAN=1 afl-clang-fast ./harness.c -I libxml2/include libxml2/.libs/libxml2.a -lz -lm -o fuzzer ``` Seed corpus: ``` echo "" > in/hi.xml ``` Fuzzing with an XML dictionary: ``` afl-fuzz -i in -o out -x ~/AFLplusplus/dictionaries/xml.dict ./fuzzer ``` Basic harness, no optimizations, 250 execs/sec, BAD: ``` #include "libxml/parser.h" #include "libxml/tree.h" #include #define SIZE 5000 int main(int argc, char **argv) { char buf[SIZE] = {0}; int len = read(STDIN_FILENO, buf, SIZE); xmlInitParser(); xmlDocPtr doc = xmlReadMemory((char *)buf, len, "https://mykter.com", NULL, 0); if (doc != NULL) { xmlFreeDoc(doc); } xmlCleanupParser(); return(0); } ``` **Important**: Note how the buffer is filled with zeros in main. This is good, because otherwise it would contain random data and the fuzzing might not be reproducible if there is a bug (stability will suffer). Let's start adding [persistent mode](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md) optimizations. Deferred initialization gives a negligible speedup (from 250 to 270 executions per second). Note that AFL does deferred initialization to the main function by default, but you can defer it even more (after xmlInitParser() in this case): ``` #include "libxml/parser.h" #include "libxml/tree.h" #include #define SIZE 5000 int main(int argc, char **argv) { xmlInitParser(); #ifdef __AFL_HAVE_MANUAL_CONTROL __AFL_INIT(); #endif char buf[SIZE] = {0}; int len = read(STDIN_FILENO, buf, SIZE); xmlDocPtr doc = xmlReadMemory((char *)buf, len, "https://mykter.com", NULL, 0); if (doc != NULL) { xmlFreeDoc(doc); } xmlCleanupParser(); return(0); } ``` Adding persistent mode via AFL\_LOOP gives \~10x speedup to \~2500 execs/sec: ``` #include "libxml/parser.h" #include "libxml/tree.h" #include #define SIZE 5000 int main(int argc, char **argv) { xmlInitParser(); #ifdef __AFL_HAVE_MANUAL_CONTROL __AFL_INIT(); #endif while (__AFL_LOOP(1000)) { char buf[SIZE] = {0}; int len = read(STDIN_FILENO, buf, SIZE); xmlDocPtr doc = xmlReadMemory((char *)buf, len, "https://mykter.com", NULL, 0); if (doc != NULL) { xmlFreeDoc(doc); } xmlCleanupParser(); } return(0); } ``` When we added shared memory fuzzing, we further increased the speed by \~50%. In this case, we got rid of the memory limitation, but you can easily just program one in by returning when the "len" variable is higher than what you want. ``` #include "libxml/parser.h" #include "libxml/tree.h" #include __AFL_FUZZ_INIT(); int main(int argc, char **argv) { xmlInitParser(); #ifdef __AFL_HAVE_MANUAL_CONTROL __AFL_INIT(); #endif unsigned char *buf = __AFL_FUZZ_TESTCASE_BUF; while (__AFL_LOOP(1000)) { int len = __AFL_FUZZ_TESTCASE_LEN; xmlDocPtr doc = xmlReadMemory((char *)buf, len, "https://mykter.com", NULL, 0); if (doc != NULL) { xmlFreeDoc(doc); } xmlCleanupParser(); } return(0); } ``` We further increased speed using [parallellization](https://github.com/mirrorer/afl/blob/master/docs/parallel_fuzzing.txt). We ran 4 fuzzers in parallel and the fuzz speed increased by 4x, since each fuzzer runs on 1 thread. ``` mkdir sync_dir afl-fuzz -i in -o sync_dir -M fuzzer01 -x ~/AFLplusplus/dictionaries/xml.dict ./fuzzer afl-fuzz -i in -o sync_dir -S fuzzer02 -x ~/AFLplusplus/dictionaries/xml.dict ./fuzzer afl-fuzz -i in -o sync_dir -S fuzzer03 -x ~/AFLplusplus/dictionaries/xml.dict ./fuzzer afl-fuzz -i in -o sync_dir -S fuzzer04 -x ~/AFLplusplus/dictionaries/xml.dict ./fuzzer ``` When doing parallel fuzzing, you can check up on your progress using the afl-whatsup tool: ``` afl-whatsup sync_dir/ ``` After 44 minutes of total running time (11 minutes per core), we have 48 locally unique crashes
### **LibFuzzer** It’s said to be recommended to use LibFuzzer over AFL and honggfuzz (or use multiple in conjunction), as it will find the most bugs. But at the same time, it’s a bit more difficult to set up. Unlike AFL, LibFuzzer doesn’t do forking + feeding data to stdin. * It fuzzes in-process persistently by default (though AFL added support for this) * State can accumulate (sometimes good, sometimes bad) You have to be careful about state accumulation. Similar fuzzers: * Honggfuzz {% embed url="" %} Excellent LibFuzzer tutorial {% endembed %} The above tutorial contains instructions about: * Setting up libfuzzer * Running it on multiple cores * Minimizing the corpus * Finding RCEs which are blocked by DoS vulns * Coverage visualization * And more #### **Running It** This demonstrates how to run LibFuzzer on a very simple [fuzzme](https://github.com/google/fuzzing/blob/master/tutorial/libFuzzer/fuzz_me.cc), which already includes the LLVMTestOneInput function. Compile it with ASan and fuzzer options: ``` clang++ -g -fsanitize=address,fuzzer ./fuzz_me.cc ``` Then run it, and it will find a heap buffer overflow. ### **Honggfuzz** Similar to LibFuzzer. May find some bugs that libfuzzer won’t and may miss a lot of bugs that LibFuzzer won’t. It’s a bit of a mix of libfuzzer and AFL in its usage. First you have to compile it with instrumentation: ``` hfuzz-clang++ -o fuzz_me.hfuzz -g fuzz_me.cc ``` Then, you have to run it with honggfuzz: ``` honggfuzz -P --output ./out/ -i ./in/ -- ./hfuzz.out ``` * `-P` is for persistent mode * `--output` saves the generated corpus in the specified directory * `-i` specifies where the input corpus should be taken from ## Black Box Fuzzing A blackbox fuzzer generates inputs for a target program without knowledge of its internal behaviour or implementation. A blackbox fuzzer may generate inputs from scratch, or rely on a static corpus of valid input files to base mutations on. Unlike coverage guided fuzzing, the corpus does not grow here. Also, if you don’t have coverage guiding or sanitization (bug detection), then it will probably miss a lot of bugs. ### **Radamsa** Pretty easy and beginner-friendly tool. Radamsa accepts input and tries to generate interesting test cases from that. {% embed url="" %} You’ll probably want to write a script to continuously use radamsa on a program and detect segfaults and such. Here’s an example of using it on Exploit Exercises Phoenix Format 5: ``` #!/bin/bash while true; do inp="$(echo "hello" | radamsa)" echo "$inp" | /opt/phoenix/i486/stack-five > /dev/null 2>&1 if [ $? -gt 127 ]; then echo "$inp" > crash-`date +s%.%N`.txt echo "Crash found!" fi done ``` ## Fuzzing Closed-Source Software {% embed url="" %} This section covers difficulties in fuzzing programs, which do not include the source code, and ways to overcome that issue. Approaches: * Black box fuzzing - doesn’t find a lot of bugs * Adding instrumentation (coverage tracking, etc) by dynamically rewriting - simple to implement, but 10-100x slower * AFL can do this by running executables under QEMU, and QEMU provides tracing and everything for you. You can even use it to fuzz ARM binaries from an x86 system. * Adding instrumentation by **statically rewriting** - more complex analysis, but much more performant. Needs to fully model program control flow without executing each path, so can be buggy (for example with stripped binaries). ## Symbolic Execution Symbolic execution executes a program, but uses symbolic values (formulae) instead of normal variable names. When encountering a path, it takes both paths (and adjusts the possible values accordingly for each path). Thus it can theoretically traverse every single branch of a program and find everything. Practically, it has a number of practical limitations, however: * Large computing power demand * State explosion * Difficult to set up {% embed url="" %} {% embed url="" %}
{% embed url="" %} OpenSecurityTraining course {% endembed %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://heinosass.gitbook.io/leet-sheet/binary-exploitation/fuzzing.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.