References and Pointers

A pointer is a general concept for a variable that contains an address in memory. This address refers to, or "points at," some other data. The most common kind of pointer in Auto is a reference, which you learned about in Chapter 4. References borrow the value they point to. They don't have any special capabilities other than referring to data, and they have no overhead.

Smart pointers, on the other hand, are data structures that act like a pointer but also have additional metadata and capabilities. The concept of smart pointers isn't unique to Rust: smart pointers originated in C++ and exist in other languages as well. Rust has a variety of smart pointers defined in the standard library that provide functionality beyond that provided by references. Auto shares many of these concepts through its interop with Rust.

In Auto, with its concept of ownership and implicit move, there is an additional difference between references and smart pointers: while references only borrow data, in many cases smart pointers own the data they point to.

Smart pointers are usually implemented using types. Unlike an ordinary type, smart pointers implement specs that provide pointer-like behavior. The Deref spec allows an instance of the smart pointer type to behave like a reference so that you can write your code to work with either references or smart pointers. The Drop spec allows you to customize the code that's run when an instance of the smart pointer goes out of scope. In this chapter, we'll discuss both of these specs and demonstrate why they're important to smart pointers.

Given that the smart pointer pattern is a general design pattern used frequently, this chapter won't cover every existing smart pointer. Many libraries have their own smart pointers, and you can even write your own. We'll cover the most common smart pointer patterns:

• Heap allocation, for allocating values on the heap
• Reference counting, a type that enables multiple ownership
• Interior mutability, a type that enforces the borrowing rules at runtime instead of compile time

In addition, we'll cover the interior mutability pattern where an immutable type exposes an API for mutating an interior value. We'll also discuss reference cycles: how they can leak memory and how to prevent them.

Note: Auto's smart pointer types leverage Rust's standard library through the use.rust mechanism. While Auto has its own memory model (implicit move and AutoFree), the smart pointer patterns described here work identically when using Rust interop. Native Auto smart pointer APIs are planned for future releases.

Let's dive in!

Using Heap Allocation to Point to Data

The most straightforward smart pointer is a box, which in Rust is written Box<T>. Boxes allow you to store data on the heap rather than the stack. What remains on the stack is the pointer to the heap data. Refer to Chapter 4 to review the difference between the stack and the heap.

Boxes don't have performance overhead, other than storing their data on the heap instead of on the stack. But they don't have many extra capabilities either. You'll use them most often in these situations:

• When you have a type whose size can't be known at compile time, and you want to use a value of that type in a context that requires an exact size
• When you have a large amount of data, and you want to transfer ownership but ensure that the data won't be copied when you do so
• When you want to own a value, and you care only that it's a type that implements a particular spec rather than being of a specific type

Storing Data on the Heap

Listing 15-1 shows how to use a box to store an int value on the heap.

Storing an `int` value on the heap using a box

We define the variable b to have the value of a Box that points to the value 5, which is allocated on the heap. This program will print b = 5; in this case, we can access the data in the box similarly to how we would if this data were on the stack. Just like any owned value, when a box goes out of scope, as b does at the end of main, it will be deallocated. The deallocation happens both for the box (stored on the stack) and the data it points to (stored on the heap).

Enabling Recursive Types with Boxes

A value of a recursive type can have another value of the same type as part of itself. Recursive types pose an issue because the compiler needs to know at compile time how much space a type takes up. However, the nesting of values of recursive types could theoretically continue infinitely, so the compiler can't know how much space the value needs. Because boxes have a known size, we can enable recursive types by inserting a box in the recursive type definition.

As an example of a recursive type, let's explore the cons list — a data type commonly found in functional programming languages.

Attempting to define a recursive enum (won

The error shows this type "has infinite size." The reason is that we've defined List with a variant that is recursive: it holds another value of itself directly. As a result, the compiler can't figure out how much space it needs to store a List value.

Because a Box<T> is a pointer, the compiler always knows how much space a Box<T> needs: a pointer's size doesn't change based on the amount of data it's pointing to. This means we can put a Box<T> inside the Cons variant instead of another List value directly.

Definition of `List` using `Box<T>` for a known size

The Cons variant needs the size of an int plus the space to store the box's pointer data. The Nil variant stores no values, so it needs less space on the stack than the Cons variant. We now know that any List value will take up the size of an int plus the size of a box's pointer data. By using a box, we've broken the infinite, recursive chain, so the compiler can figure out the size it needs to store a List value.

Treating Smart Pointers Like Regular References

Implementing the Deref spec allows you to customize the behavior of the dereference operator *. By implementing Deref in such a way that a smart pointer can be treated like a regular reference, you can write code that operates on references and use that code with smart pointers too.

Following the Pointer to the Value

A regular reference is a type of pointer, and one way to think of a pointer is as an arrow to a value stored somewhere else. In Listing 15-4, we create a reference to an int value and then use the dereference operator to follow the reference to the value.

Using the dereference operator to follow a reference

The variable x holds an int value 5. We set y equal to a reference to x. We can assert that x is equal to 5. However, if we want to make an assertion about the value in y, we have to use *y to follow the reference to the value it's pointing to.

Using Box<T> Like a Reference

We can rewrite the code in Listing 15-4 to use a Box<T> instead of a reference; the dereference operator used on the Box<T> functions in the same way as the dereference operator used on the reference.

Using the dereference operator on a `Box<int>`

Implementing the Deref Spec

To enable dereferencing with the * operator, a type implements the Deref spec. The Deref spec requires implementing a deref method that borrows self and returns a reference to the inner data.

Implementing `Deref` on `MyBox<T>`

The type Target = T syntax defines an associated type for the Deref spec. We fill in the body of the deref method with &.0 so that deref returns a reference to the value we want to access with the * operator.

Without the Deref spec, the compiler can only dereference & references. The deref method gives the compiler the ability to take a value of any type that implements Deref and call the deref method to get a reference that it knows how to dereference.

When we entered *y, behind the scenes the compiler actually ran *(y.deref()). This feature lets us write code that functions identically whether we have a regular reference or a type that implements Deref.

Deref Coercion

Deref coercion converts a reference to a type that implements the Deref spec into a reference to another type. For example, deref coercion can convert &String to &str because String implements Deref such that it returns &str. Deref coercion happens automatically when we pass a reference to a particular type's value as a function argument that doesn't match the parameter type.

Deref coercion in action

Here we're calling the hello function with the argument &m, which is a reference to a MyBox<String> value. Because MyBox<T> implements Deref, the compiler can turn &MyBox<String> into &String by calling deref. The standard library provides an implementation of Deref on String that returns a string slice, and the compiler calls deref again to turn the &String into a &str, which matches the hello function's definition.

Deref coercion also works with mutable references:

1. From &T to &U when T: Deref<Target=U>
2. From &mut T to &mut U when T: DerefMut<Target=U>
3. From &mut T to &U when T: Deref<Target=U>

The third case is trickier: the compiler will also coerce a mutable reference to an immutable one. But the reverse is not possible: immutable references will never coerce to mutable references.

Running Code on Cleanup with the Drop Spec

The second spec important to the smart pointer pattern is Drop, which lets you customize what happens when a value is about to go out of scope. You can provide an implementation for the Drop spec on any type, and that code can be used to release resources like files or network connections.

In some languages, the programmer must call code to free memory or resources every time they finish using an instance. If the programmer forgets, the system might become overloaded and crash. In Auto, you can specify that a particular bit of code be run whenever a value goes out of scope, and the compiler will insert this code automatically. As a result, you don't need to be careful about placing cleanup code everywhere in your program — you still won't leak resources!

A type that implements the `Drop` spec

When we run this program, we'll see the following output:

CustomSmartPointers created
Dropping CustomSmartPointer with data `other stuff`!
Dropping CustomSmartPointer with data `my stuff`!

The compiler automatically called drop for us when our instances went out of scope. Variables are dropped in the reverse order of their creation, so d was dropped before c.

Dropping a Value Early with drop

Occasionally, you might want to clean up a value early. One example is when using smart pointers that manage locks: you might want to force the drop method that releases the lock so that other code in the same scope can acquire the lock. Auto doesn't let you call the Drop spec's drop method manually; instead, you use the drop() function provided by the standard library.

Calling `drop()` to explicitly drop a value before it goes out of scope

Running this code will print:

CustomSmartPointer created
Dropping CustomSmartPointer with data `some data`!
CustomSmartPointer dropped before the end of main

The Dropping message appears between the two print statements, showing that c is dropped at that point. With the Drop spec and Auto's ownership system, you don't have to remember to clean up — Auto does it automatically.

Reference Counting: Rc<T>

In the majority of cases, ownership is clear: you know exactly which variable owns a given value. However, there are cases when a single value might have multiple owners. For example, in graph data structures, multiple edges might point to the same node, and that node is conceptually owned by all of the edges that point to it.

You have to enable multiple ownership explicitly by using the Rc<T> type, which is an abbreviation for reference counting. The Rc<T> type keeps track of the number of references to a value to determine whether or not the value is still in use. If there are zero references to a value, the value can be cleaned up without any references becoming invalid.

Note: Rc<T> is only for use in single-threaded scenarios. When we discuss concurrency in Chapter 16, we'll cover how to do reference counting in multithreaded programs.

Sharing Data with Rc<T>

Let's return to our cons list example. This time, we'll create two lists that both share ownership of a third list.

Using `Rc<T>` to share ownership between two lists

When we create b, instead of taking ownership of a, we clone the Rc<List> that a is holding, thereby increasing the number of references from one to two and letting a and b share ownership of the data. We also clone a when creating c, increasing the number of references from two to three.

Cloning to Increase the Reference Count

Printing the reference count

This code prints:

count after creating a = 1
count after creating b = 2
count after creating c = 3
count after c goes out of scope = 2

We can see that the Rc<List> in a has an initial reference count of 1; then, each time we call clone, the count goes up by 1. When c goes out of scope, the count goes down by 1. We don't have to call a function to decrease the reference count — the Drop implementation decreases it automatically when an Rc<T> value goes out of scope.

RefCell<T> and the Interior Mutability Pattern

Interior mutability is a design pattern that allows you to mutate data even when there are immutable references to that data; normally, this action is disallowed by the borrowing rules. To mutate data, the pattern uses sys (equivalent to Rust's unsafe) code inside a data structure to bend the usual rules that govern mutation and borrowing. We will discuss sys more in Chapter 20.

Enforcing Borrowing Rules at Runtime

Unlike Rc<T>, the RefCell<T> type represents single ownership over the data it holds. Recall the borrowing rules you learned in Chapter 4:

• At any given time, you can have either one mutable reference or any number of immutable references (but not both).
• References must always be valid.

With references and Box<T>, the borrowing rules' invariants are enforced at compile time. With RefCell<T>, these invariants are enforced at runtime. With references, if you break these rules, you'll get a compiler error. With RefCell<T>, if you break these rules, your program will panic and exit.

Here is a recap of the reasons to choose Box<T>, Rc<T>, or RefCell<T>:

Type	Owners	Borrow checking	Mutability
Box<T>	Single	Compile time	Immutable or mutable
Rc<T>	Multiple	Compile time	Immutable only
RefCell<T>	Single	Runtime	Immutable or mutable

Using Interior Mutability: Mock Objects

A practical use case for RefCell<T> is creating mock objects during testing. Here's an example using a Messenger spec:

The sent_messages field is now of type RefCell<List<String>>. The send method calls borrow_mut on the RefCell to get a mutable reference to the inner list. Even though send takes an immutable reference to self, we can still mutate the inner data through RefCell<T>.

Combining Rc<T> and RefCell<T>

A common way to use RefCell<T> is in combination with Rc<T>. Recall that Rc<T> lets you have multiple owners of some data, but it only gives immutable access to that data. If you have an Rc<T> that holds a RefCell<T>, you can get a value that can have multiple owners and that you can mutate.

Using `Rc<RefCell<int>>` to create a `List` we can mutate

When we print a, b, and c, we can see that they all have the modified value of 15 rather than 5. By using RefCell<T>, we have an outwardly immutable List value, but we can modify the data when we need to.

Reference Cycles Can Leak Memory

Auto's memory safety guarantees make it difficult, but not impossible, to accidentally create memory that is never cleaned up (known as a memory leak). Preventing memory leaks entirely is not one of Auto's guarantees. We can see that memory leaks are possible by using Rc<T> and RefCell<T>: it's possible to create references where items refer to each other in a cycle. This creates memory leaks because the reference count of each item in the cycle will never reach 0, and the values will never be dropped.

Creating a Reference Cycle

Creating a reference cycle of two `List` values pointing to each other

The reference count of both a and b is 2 after we create the cycle. At the end of main, the Rc<List> instances can't be dropped because their reference counts never reach 0. The memory will remain uncollected.

Preventing Reference Cycles with Weak<T>

You can create a weak reference to the value within an Rc<T> by calling Rc::downgrade. Weak references don't express an ownership relationship, and their count doesn't affect when an Rc<T> instance is cleaned up.

A tree structure using `Weak<T>` for parent references

When the inner scope ends, branch goes out of scope and the strong count of the Rc<Node> decreases to 0, so its Node is dropped. The weak count of 1 from leaf.parent has no bearing on whether the Node is dropped, so we don't get any memory leaks!

If we try to access the parent of leaf after the end of the scope, we'll get None again. All of the logic that manages the counts and value dropping is built into Rc<T> and Weak<T> and their implementations of the Drop spec.

Smart Pointers Quick Reference

Feature	Auto	Rust
Heap allocation	Box.new(v) via use.rust	Box::new(v)
Reference counting	Rc.new(v) via use.rust	Rc::new(v)
Interior mutability	RefCell.new(v) via use.rust	RefCell::new(v)
Dereference spec	spec Deref	impl Deref
Cleanup spec	spec Drop	impl Drop
Weak reference	Rc.downgrade(&v)	Rc::downgrade(&v)
Shared + mutable	Rc<RefCell<T>>	Rc<RefCell<T>>

Summary

This chapter covered how to use smart pointers to make different guarantees and trade-offs from those Auto makes by default with regular references:

1. Box<T> — heap allocation with single ownership, enables recursive types
2. Deref spec — treating smart pointers like regular references, with automatic deref coercion
3. Drop spec — running cleanup code automatically when values go out of scope
4. Rc<T> — reference counting for multiple owners of the same data
5. RefCell<T> — interior mutability with runtime borrow checking
6. Weak<T> — preventing reference cycles that can cause memory leaks

In the next chapter, we'll explore Auto's concurrency model, which uses the actor pattern rather than Rust's thread-based model.