This release is almost entirely clusters, and it carries two compatibility notes worth reading before you upgrade.
The first one is about the package manager. scua-pkg
refuses any toolchain newer than the one it knows about, and it does not
know about 0.16 yet, so scua add, vendor and
the rest stay refused until a scua-pkg release supports it.
If you use those commands day to day, stay on 0.15 until that lands. A
project that already has its deps/ runs normally, and
nothing in the language is affected.
The second is that durable cluster snapshots have a new format. A 0.16 node reads a 0.15 snapshot. A 0.15 node handed a 0.16 snapshot starts empty instead of loading it, which is deliberate. The new field records which stale writes to refuse, and loading the file without it would drop that protection with nothing to show for it. Upgrade a cluster together, or expect a node that gets rolled back to refill from its peers.
The fixes are what made the format change necessary. A restarted node could accept a deleted key back: the record of what had already been cleaned up was never written to disk, so restarting reopened exactly the window it exists to close. A replica that had been offline could hand back a value you deleted, and it would come back. Cleaning up a deleted key also failed to mark the store as needing a save, so a node whose last action was a delete could keep writing out its previous state indefinitely and reload the deleted key on restart. Those two compounded: the protection was going into a file that was not being written.
Before those, a cluster could stop reclaiming deleted keys at all. A node that accepted a connection handed over its whole store before it knew who it was talking to, then cleaned up a deleted key it had just sent, on the grounds that nobody it knew about still held one. The node on the other end had that deletion refused when it tried to hand it back, because the protection against stale writes did not distinguish a delete from a value, and a delete cannot bring anything back. With the two disagreeing and neither able to accept the other's copy, the repair path that exists for this stayed quiet, because a node with nothing left to send sent nothing at all rather than announcing what it had. A node that listed its own address as a seed, which is what the obvious configuration does, could also dial itself in a loop. Those three landed just after 0.15.0 was cut and did not make it into that build, so this is the first release carrying them.
A cluster could also stop reclaiming space forever. A node that had been fenced off, refused for sending a value older than a delete everyone else had already cleaned up, kept the whole cluster's cleanup pinned behind it. One late node meant nobody could collect anything. Fenced nodes are now skipped when working out what is safe to collect, and a node rejoins the count as soon as it agrees again.
Under load, handling one peer's message could disconnect a different
peer, after which the node kept writing into memory belonging to the
peer that had gone. That surfaced as a crash or as a frame delivered to
the wrong node. A member with no connection was marked as probed without
a probe ever being sent, so it stayed listed as alive indefinitely.
Rejoining after a failed start could free the same address twice.
cluster.stats now reports rejected_skew, which
counts writes discarded because the sending node's clock was too far
ahead of yours. That was previously counted nowhere, so a node with a
bad clock could have every write it sent dropped, indefinitely, with
nothing to look at.
Most of these came out of running a real five node mesh in Docker with faults injected against it, which is also how we found that the first version of the snapshot fix did not work. It wrote the right field into a file that never got saved.
One fix outside clusters: sys.exit inside
spawn stops that task now. It used to abort the task and
then run it again from the top, over and over, producing millions of
repeats of whatever it printed and freezing every other task in the
program behind it, because the scheduler never got a chance to move time
forward. The exit code is still discarded when you call it from inside
spawn, so if you need a run to fail, return the outcome and
exit from the top level.
The changelog has the full accounting.