Testing

In his 1972 essay "The Humble Programmer," Edsger W. Dijkstra wrote that "program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence." That does not mean we should not try to test as much as we can!

Correctness in our programs is the extent to which our code does what we intend it to do. Auto is designed with a high degree of concern about correctness -- the type system, ownership rules, and borrowing checker catch many bugs at compile time. But the type system cannot catch everything. That is where automated tests come in.

Auto's testing system is closely aligned with Rust's, since Auto transpiles to Rust and runs tests using the same test runner. This chapter covers the testing mechananics available to you in Auto.

Why Testing Matters

Tests are functions that verify non-test code is functioning as expected. Test functions typically perform three actions:

1. Set up any needed data or state.
2. Run the code you want to test.
3. Assert that the results are what you expect.

Writing tests has concrete benefits. Tests catch bugs early -- when a test fails, you know exactly which behavior broke. Tests let you refactor with confidence -- if all tests pass after a change, you probably did not break anything. Tests serve as documentation -- reading a test tells you what a function is supposed to do.

In Auto, tests are particularly valuable because they run on the transpiled Rust code. If your Auto code compiles and all tests pass, you have strong evidence that the code is correct on all five targets: Rust, Python, C, TypeScript, and Auto itself.

Your First Test

A test in Auto is a function annotated with #[test]. When you run automan test, the transpiler compiles your code to Rust and runs cargo test, which builds a test runner binary that executes the annotated functions.

A test function using `assert_eq`

The #[test] attribute marks the function as a test. The test runner finds all functions with this attribute and runs them as separate test cases. If a test function returns without panicking, the test passes. If it panics (or an assertion fails), the test fails.

The assert_eq macro checks that two values are equal. If they are not, the test panics with a message showing both the expected and actual values. This makes debugging much easier than a simple assert -- you can see exactly what went wrong.

In Rust, assert_eq! uses the ! macro suffix and requires a semicolon. In Auto, assert_eq is called without ! and without a semicolon. The transpiler adds both when generating Rust code.

In Python, assert_eq transpiles to a plain assert result == 4 statement. In C, it becomes an if check with fprintf and __builtin_trap. In TypeScript, it becomes an if check that throws an Error. Each target uses its idiomatic assertion mechanism.

Assert Macros

Auto provides three assertion macros that cover the most common testing needs:

• assert_eq(a, b) -- Passes if a equals b. Fails with both values displayed.
• assert_ne(a, b) -- Passes if a does not equal b. Fails with both values displayed.
• assert!(condition) -- Passes if condition is true. Fails with no extra context.

Testing string operations with multiple assert macros

This test demonstrates all three macros:

1. assert_eq(s.len(), 5) -- Verifies that the string length is exactly 5. If the string were "hi" (length 2), this would fail and show both values: assertion failed: left == right, left: 2, right: 5.

2. assert_ne(s, "world") -- Verifies that the string is not "world". This is useful when you want to confirm that a value is different from a specific sentinel or error value.

3. assert!(s.contains("ell")) -- Verifies that a boolean condition is true. This is the most general assertion -- you can use it with any expression that returns a boolean. If the condition is false, the test fails with a generic message (no values displayed, since assert! does not know which values to show).

A good practice is to use assert_eq and assert_ne whenever possible, since they display both values on failure. Reserve assert! for conditions where the values are obvious from context or when testing boolean-returning methods like .contains().

In Python, s.contains("ell") transpiles to the idiomatic in operator: "ell" in s. In C, string operations use strlen, strcmp, and strstr from <string.h>. In TypeScript, .contains() maps to .includes().

Testing Error Cases

Many functions return !T (Result) to indicate success or failure. Testing these functions means testing both the success path and the error path. Auto makes this natural by combining !T with pattern matching using is.

Testing both success and error paths of a `!T` function

Let us walk through the key ideas:

1. fn divide(a float, b float) !float -- The !float return type means this function returns a Result<float, String>. On success, it returns the quotient. On failure (division by zero), it returns an error string.

2. if b == 0.0 { "division by zero" } else { a / b } -- In Auto, a bare string in an error-returning function implicitly becomes Err("division by zero"). A bare value becomes Ok(a / b). This is Auto's concise error handling syntax. The transpiler expands these into explicit Ok(...) and Err(...) constructors in Rust.

3. test_divide_by_zero -- This test calls divide(10.0, 0.0) and pattern matches on the result. If we get Ok(_), the test fails with assert!(false) -- we expected an error, not a success. If we get Err(e), the test passes. The _ in Ok(_) means "I do not care about the value" (since there should not be one in the error case).

4. test_divide_success -- This test calls divide(10.0, 2.0) and verifies the result. If we get Ok(v), we check that v == 5.0 using assert_eq. If we get Err(_), the test fails with assert!(false) -- the division should have succeeded.

The pattern of testing both branches of a !T result is fundamental. Every function that can fail should have at least two tests: one for the success path and one for each error path.

In Python, the !T type transpiles to a custom Result class with an ok flag. In C, it becomes a struct with a boolean discriminant and a union for the ok value and error value. In TypeScript, it becomes a discriminated union type with { ok: true; value: T } and { ok: false; error: string } shapes.

Integration Tests

Unit tests verify individual functions in isolation. Integration tests verify that multiple components work together correctly. They test entire workflows -- create an object, call several methods, check the final state.

Testing a multi-step workflow with a Calculator type

Key observations:

1. type Calculator { total float } -- A simple type with one field. The total field tracks the running sum.

2. fn new() Calculator -- A static constructor that creates a Calculator with total set to 0.0. In Auto, fn new() inside an ext block is a static method (no self parameter).

3. fn add(&self, value float) -- A method that modifies self.total. The &self parameter tells the compiler this method may mutate the receiver. In Rust, this transpiles to &mut self. In Python and TypeScript, self (or this) is implicit.

4. fn clear(&self) -- Resets total back to 0.0. This is another mutating method.

5. The test -- Creates a calculator, adds two values, checks the total, clears it, and checks that the total is back to zero. This is a multi-step workflow test. It verifies that add and clear work correctly both individually and in combination.

The key difference between unit tests and integration tests is scope. A unit test for add might test one addition: calc.add(5.0); assert_eq(calc.total, 5.0). An integration test exercises the full workflow: create, add multiple times, check, clear, check again. Both are valuable.

In C, the &self methods become functions that take a pointer to the struct (Calculator* self). The caller must pass &calc explicitly. In TypeScript, Calculator.new() is a static factory method that returns a new instance.

Test Organization

As your project grows, you need a strategy for organizing tests. Auto follows the same conventions as Rust: unit tests live alongside the code they test, and integration tests live in a separate tests/ directory.

You can also use spec (from Chapter 13) to define the interface you want to test, and write tests against that interface. This gives you flexibility to swap implementations without changing tests.

Organizing tests with a spec

Here is how test organization works in Auto:

1. spec MathUtils -- Defines the contract: any implementation of MathUtils must provide an add function. Tests can be written against this spec, so they work with any implementation.

2. ext MathUtils -- Provides the default implementation. The add function simply returns a + b. If you later need a version that logs calls or handles overflow, you can swap the implementation without changing the tests.

3. #[test] fn test_add() -- The test function calls MathUtils.add and verifies the result. Because the test uses the spec name, it works with any implementation that conforms to the spec.

4. fn main() -- The main function is ignored by the test runner. Only functions marked with #[test] are executed when you run automan test.

Running Tests

Run all tests with:

$ automan test

Run a single test by name:

$ automan test test_add

Run tests whose names match a pattern:

$ automan test divide

Show output from passing tests:

$ automan test -- --show-output

Run tests one at a time (useful for debugging):

$ automan test -- --test-threads=1

Test Syntax Quick Reference

Auto	Rust	Purpose
#[test]	#[test]	Mark function as a test
assert_eq(a, b)	assert_eq!(a, b)	Assert equality
assert_ne(a, b)	assert_ne!(a, b)	Assert inequality
assert!(cond)	assert!(cond)	Assert condition is true
fn test() !	fn test() -> Result<(), String>	Return errors instead of panicking

Summary

This chapter covered Auto's testing system. Here is what we learned:

Concept	Syntax	Purpose
Test attribute	#[test]	Mark a function as a test
Equality assertion	assert_eq(a, b)	Verify two values are equal
Inequality assertion	assert_ne(a, b)	Verify two values differ
Boolean assertion	assert!(cond)	Verify a condition is true
Error testing	is result { Ok => ..., Err => ... }	Test both paths of !T
Run tests	automan test	Execute all test functions

Key takeaways:

1. Tests are functions with #[test] -- The test runner discovers and executes them. A test passes if it returns without panicking.

2. Use assert_eq and assert_ne preferentially -- They display both values on failure, making debugging easier. Use assert! only when the values are self-evident.

3. Test both paths of !T -- Every function that can fail should have at least one test for the success path and one for each error path.

4. Integration tests exercise workflows -- Test that multiple components work together: create, operate, verify, reset, verify again.

5. spec enables testable interfaces -- Write tests against specs so they work with any conforming implementation.

6. automan test runs everything -- It transpiles to Rust and invokes cargo test, giving you the full power of Rust's test runner.

Testing is not just about finding bugs. It is about having confidence that your code works correctly, today and after every future change. The small investment of writing tests pays for itself many times over in reduced debugging time and increased confidence in refactoring.