Fuzzing — Breaking Things with Random Inputs

Fuzzers

Fuzzer is a base class for fuzzers, with RandomFuzzer as a simple instantiation. The fuzz() method of Fuzzer objects and returns a string with a generated input.

>>> random_fuzzer = RandomFuzzer()
>>> random_fuzzer.fuzz()

Output: %$<1&<%+=!"83?+)9:++9138 42/ "7;0-,)06 "1(2;6>?99$%7!!*#96=>2&-/(5*)=$;0$$+;<12"?30&

The RandomFuzzer constructor allows a number of keyword arguments:

>>> print(RandomFuzzer.__init__.__doc__)

Output: Produce strings of min_length to max_length characters in the range [char_start, char_start + char_range)

>>> random_fuzzer = RandomFuzzer(min_length=10, max_length=20, char_start=65, char_range=26)
>>> random_fuzzer.fuzz()

Output: 'XGZVDDPZOOW'

Runners

A Fuzzer can be paired with a Runner which takes the fuzzed strings as input. Its result is a class-specific status and an outcome (PASS, FAIL, or UNRESOLVED). A PrintRunner will simply print out the given input and return a PASS outcome:

>>> print_runner = PrintRunner()
>>> random_fuzzer.run(print_runner)

Outcome: ('EQYGAXPTVPJGTYHXFJ', 'UNRESOLVED')

A ProgramRunner will feed the generated input into an external program. Its result is a pair of the program status (a CompletedProcess instance) and an outcome (PASS, FAIL or UNRESOLVED):

>>> cat = ProgramRunner('cat')
>>> random_fuzzer.run(cat)

Outcome: (CompletedProcess(args='cat', returncode=0, stdout='BZOQTXFBTEOVYX', stderr=''), 'PASS')

Creating Input Files

basename = "input.txt"
tempdir = tempfile.mkdtemp()
FILE = os.path.oin(tempdir, basename)
print(FILE)

Output: /var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/tmp33gvbu2n/input.txt

Let’s open it. The Python open() function opens a file into which we can then write arbitrary contents. Very commonly, it is used with the with statement, which ensures that the file is closed as soon as it is no longer needed.

data = fuzzer()
with open(FILE, "w") as f:
    f.write(data)

The file can be verified by reading its contents:

contents = open(FILE).read()
print(contents)
assert(contents == data)

Output: <?6&" !3'7-5>18%55*,5

Invoking External Programs

With that input file, we can invoke a program on it.

We will use the bc calculator program, which takes an arithmetic expression and evaluates it.

To invoke bc let’s use the Python subprocess module. This is how it works:

program = "bc"
with open(FILE, "w") as f:
    f.write("2 + 2\n")
result = subprocess.run([program, FILE],
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True)

Using result, we can check the program’s output. In the case of bc, this is the result:

result.stdout

Output: 4\n

The status may also be checked, a value of 0 indicating the program terminated correctly.

result.returncode

Output: 0

Any error messages would be available in results.stderr:

results.stderr

Output: ''

Any program is able to go through this process, however, you should be careful as you could change or even damage your system.

Buffer Overflows

Buffer overflows are triggered when going over a program’s built-in maximum lengths for inputs and input elements.

Buffer overflow behavior can be easily simulated in a Python function:

def crash_if_too_long(s):
    buffer = "Thursday"
    if len(s) > len(buffer):
        raise ValueError

This leads to a crash very quickly.

trials = 100
with ExpectError():
    for i in range(trials):
        s = fuzzer()
        crash_if_too_long(s)

Output: Traceback (most recent call last): File "/var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/ipykernel_1932/292568387.py", line 5, in <module> crash_if_too_long(s) File "/var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/ipykernel_1932/2784561514.py", line 4, in crash_if_too_long raise ValueError ValueError (expected)

The with ExpectError() line in the above code ensures that the error message is printed, yet execution continues; this is to differentiate this “expected” error from “unexpected” errors in other code examples.

Missing Error Checks

Many programming languages do not have exceptions, but instead have functions return special error codes in exceptional circumstances. The C function getchar(), for instance, normally returns a character from the standard input; if no input is available anymore, it returns the special value EOF(end of file).

In a situation where, for example, getchar() reaches and scans a space character, getchar() could return EOF, and keep returning EOF when called again. This would lead to an infinite loop. However, if a line with the with ExpectTimeout() was implemented, the code would be interrupted after a set amount of time, printing an error message.

Rogue Numbers

When fuzzing, it is easy to generate uncommon values in the input, which can lead to a load of interesting behavior. The value could exceed program memory, lead to a crash, or an array of other unexpected outcomes.

It is important to use these rogue numbers because it is much better to quickly fail the program than to allow it to destroy something later down the line.

Generic Checkers

Generic checkers detect common problems. For example, AddressSanitizer detects invalid memory accesses in C and C++ programs. Outputs can also be checked for leaked secret information.

Program-Specific Checkers

Program-specific checkers use assertions to verify rules about a program’s data or results. The chapter’s airport-code example checks for three uppercase letters. Here is a simplified version:

def check_airport_code(code):
    assert len(code) == 3, "Expected three characters"
    assert all(c.isalpha() and c.isupper() for c in code), "Expected uppercase letters"

for code in ["JFK", "jfK"]:
    try:
        check_airport_code(code)
        print(code, "PASS")
    except AssertionError as error:
        print(code, "FAIL:", error)

JFK PASS
jfK FAIL: Expected uppercase letters

JFK passes both checks. jfK fails because it contains lowercase letters.

Static Code Checkers

Static checkers inspect code without running it. For example, mypy can detect type mismatches, but it cannot check every program-specific rule.