What’s going on in libcluster? (Elixir library overview)

The README and documentation are pretty good.

They cover the features of the library and provide a mini “getting started” tutorial.

But since we’ve all already read that, we can skip to opening the lib/ directory and scanning the module names on Elixir projects.

It looks like there’s a general “strategy” module / behavior, and then a bunch of modules defining implementations for these strategies, like kubernetes.

Diving straight in to the kubernetes.

ex file is a good place to start.

Jumping to the middle will give us some clues on how to bridge the theory in the code (which is down a level, in the strategy.

ex behavior), with how the strategy gets configured and booted (which is up a level in the supervisor.



ex defines a GenServer.

The GenServer keeps track of the list of nodes in the cluster in its state.

When the GenServer boots it calls load/2.

To load the nodes into state, it performs 4ish actions:Get’s the nodes currently in the k8s cluster (source of truth for this strategy).

Disconnects any nodes that are no longer supposed to be connected.

Connects any new nodes that should be in the cluster but aren’t yet.

Sends a message to itself to re-evaluate the cluster on some time interval.

Or, in “other words” Kubernetes.

load/2 is doing this stuff:Let’s dig into the first step — getting the current nodes in the cluster.

This is going to be necessarily strategy-dependent.

We can get the gist from this stunningly rendered annotation provided by yours truly:Beautiful handwriting, JoeyOkay, so we got the available nodes running in k8s by making an API call that queried across some of the meta data available in our k8s implementation.

The second and third steps are disconnecting and connecting the right nodes.

To find out what’s going on there, we need to head to the Behavior itself, strategy.


The function signature’s are pretty similar for both the connect and disconnect functions.

They take a topology, a {module, function, argument} tuple to describe how to connect to nodes, a {module, function, argument} tuple to describe how to retrieve a list of currently connected nodes, and finally a list of nodes that we need to disconnect from the cluster, if they’re currently connected.

The apply(mfa) tuple strategy is used so that libcluster can be extended to support projects that don’t use “normal” distributed Erlang.

In those cases you can prescribe custom mechanisms in the config file.

But all we care about is the standard Erlang strategy, which is included by default when the supervisor boots in the connect/disconnect/list_nodes mfa’s:At this point, we can see that we’re getting both the cluster of nodes that exists in kubernetes via our private get_nodes/1 and the list of nodes that our erlang runtime is currently connected to, via our mfa tuple or :erlang.


In this strategy k8s is our source of truth.

This gives us a node diff, and we can instruct Erlang to remove any nodes currently connected that shouldn’t be, and to establish a connection to new nodes.

This is the MagicThis sequence, and the fact that its repeated every X milliseconds, is the magic between the “self-healing” features of libcluster.

We look up the nodes that should be in the cluster, then prune dead nodes that are no longer in the cluster, and establish connections to the new nodes that should be.

It’s worth noting that distributed Erlang by itself is capable of maintaining its node-list via the Erlang Port Mapper Daemon, and handles nodes that go down naturally in the runtime.

In fact the only thing that’s implemented in the epmd.

ex strategy is the very initial cluster connection, based on the config file.

The rest is left to Erlang ❤.

Thanks for staying tuned and (hopefully awake).

One thing I learned from this library was the method of using an MFA tuple to let library user’s specify which Module.

functions(arguments) are actually used in a module via a config file.

Pretty neat!¹Awesome tangent, Erlang nodes will basically mock distress signals from missing nodes, so even if a node blows up so catastrophically that it can’t get a warning message out on the network, the network acts like it did.

²This problem isn’t isolated to Erlang.

Lot’s and lot’s of people decided that distributing stuff is a great idea and have spent billions reinventing Erlang/OTP into “microservices” [haters gunna hate].

³I love this helpful guide on “how to read”.

. More details

Leave a Reply