Application Code Upgrades in Elixir

In this third and final part of my series about production-code-upgrades-in-elixir, we will look at what happens during an application upgrade.

Let's get going!

Set-up: Create an Appup File in Elixir

As with a single module, we need new compiled code for a fresh version of an application.

But an application can consist of many modules and have running processes. So we need a scenario to specify what to upgrade and how to do it.

These scenarios are called application upgrade files (.appup files).

Let's try to make an appup file for our application. Imagine that we want to upgrade our counters so that we can specify counter increment speed.

Prepare a New Version of the Application in Elixir

First, we'll add our application to git and tag the old version:

1git add .
2git commit -m 'initial commit'
3git tag 'v0.1.0'

Now we should make a new version.

In mix.exs, update the version to 0.1.1:

1...
2 def project do
3    [
4      app: :our_new_app,
5      version: "0.1.1",
6...

We want interval to be an external parameter, not a module attribute. Make the following changes in lib/our_new_app/counter.ex:

1defmodule OurNewApp.Counter do
2  ...
3  def start_link({start_from, interval}) do
4    GenServer.start_link(__MODULE__, {start_from, interval})
5  end
6
7  ...
8
9  def init({start_from, interval}) do
10    Process.flag(:trap_exit, true)
11
12    st = %{
13      current: start_from,
14      timer: :erlang.start_timer(interval, self(), :tick),
15      terminator: nil,
16      interval: interval
17    }
18
19    {:ok, st}
20  end
21
22  ...
23
24  def handle_info({:timeout, _timer_ref, :tick}, st) do
25    :erlang.cancel_timer(st.timer)
26
27    new_current = st.current + 1
28
29    if st.terminator && rem(new_current, 10) == 0 do
30      # we are terminating
31      GenServer.reply(st.terminator, :ok)
32      {:stop, :normal, %{st | current: new_current, timer: nil}}
33    else
34      new_timer = :erlang.start_timer(st.interval, self(), :tick)
35      {:noreply, %{st | current: new_current, timer: new_timer}}
36    end
37  end
38
39  ...
40end

Now add a code_change callback to the same file:

1defmodule OurNewApp.Counter do
2  ...
3
4  def code_change(_old_vsn, st, new_interval) do
5    {:ok, Map.put(st, :interval, new_interval)}
6  end
7end

And change supervisor specs in lib/our_new_app/counter_sup.ex:

1  @impl true
2  def init(start_numbers) do
3    children =
4      for start_number <- start_numbers do
5        # We can't just use `{OurNewApp.Counter, start_number}`
6        # because we need different id's for children
7
8        Supervisor.child_spec({OurNewApp.Counter, {start_number, 200}}, id: start_number)
9      end
10
11    Supervisor.init(children, strategy: :one_for_one)
12  end

Let's construct our_new_app.appup file. We need to update our supervision specs and pass a new tick interval (250) to change_code callback:

1{
2    "0.1.1",
3    [{"0.1.0", [
4        {update, 'Elixir.OurNewApp.CounterSup', supervisor},
5        {update, 'Elixir.OurNewApp.Counter', {advanced, 250}}
6    ]}],
7    [{"0.1.0", []}]
8}.

Note that this file is written in Erlang syntax.

Now tag your new version:

1git add .
2git commit -m 'added customizable intervals'
3git tag 'v0.1.1'

Try to upgrade 0.1.0 to 0.1.1.

Checkout, compile, and run 0.1.0 version:

1git checkout v0.1.0
2iex -S mix

In a separate shell and different directory, checkout the new version and put the appup file in the appropriate place:

1...
2git checkout v0.1.0
3mix compile
4cp our_new_app.appup _build/dev/lib/our_new_app/ebin

Run the Application Upgrade in Elixir

We are ready to upgrade the application. In the running iex session, do the following:

1iex(1)> Application.spec(:our_new_app)
2[
3  description: 'our_new_app',
4  id: [],
5  vsn: '0.1.0',
6  modules: [OurNewApp, OurNewApp.Application, OurNewApp.Counter,
7   OurNewApp.CounterSup],
8  maxP: :infinity,
9  maxT: :infinity,
10  registered: [],
11  included_applications: [],
12  applications: [:kernel, :stdlib, :elixir, :logger],
13  mod: {OurNewApp.Application, []},
14  start_phases: :undefined
15]

You can see that the old application is running.

Run an "orphaned" counter outside the supervision tree (we will need it later):

1iex(2)> {:ok, pid} = OurNewApp.Counter.start_link(30000)
2{:ok, #PID<0.147.0>}

Check that your appup file is correct and the OTP knows how to upgrade your application:

1iex(3)> :release_handler.upgrade_script(:our_new_app, '/path/to/new/version/of/our_new_app/_build/dev/lib/our_new_app/')
2{:ok, '0.1.1',
3 [
4   {:load_object_code,
5    {:our_new_app, '0.1.1', [OurNewApp.CounterSup, OurNewApp.Counter]}},
6   :point_of_no_return,
7   {:suspend, [OurNewApp.CounterSup]},
8   {:load, {OurNewApp.CounterSup, :brutal_purge, :brutal_purge}},
9   {:code_change, :up, [{OurNewApp.CounterSup, []}]},
10   {:resume, [OurNewApp.CounterSup]},
11   {:suspend, [OurNewApp.Counter]},
12   {:load, {OurNewApp.Counter, :brutal_purge, :brutal_purge}},
13   {:code_change, :up, [{OurNewApp.Counter, 250}]},
14   {:resume, [OurNewApp.Counter]}
15 ]}

Check your counter processes and their pids:

1iex(4)> Supervisor.which_children(OurNewApp.CounterSup)
2[
3  {20000, #PID<0.143.0>, :worker, [OurNewApp.Counter]},
4  {10000, #PID<0.142.0>, :worker, [OurNewApp.Counter]}
5]

Upgrade the application!

1iex(5)> :release_handler.upgrade_app(:our_new_app, '/path/to/new/version/of/our_new_app/_build/dev/lib/our_new_app/')
2{:ok, []}
3iex(6)>
402:46:48.286 [info]  terminating with {{:badkey, :interval, %{current: 33478, terminator: nil, timer: #Reference<0.948322908.3672375303.146224>}}, [{OurNewApp.Counter, :handle_info, 2, [file: 'lib/our_new_app/counter.ex', line: 52]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 680]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 756]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}, counter is 33478
5
602:46:48.291 [error] GenServer #PID<0.147.0> terminating
7...

Let's see what happened:

1iex(1)> Application.spec(:our_new_app)
2[
3  description: 'our_new_app',
4  id: [],
5  vsn: '0.1.1',
6  modules: [OurNewApp, OurNewApp.Application, OurNewApp.Counter,
7   OurNewApp.CounterSup],
8  maxP: :infinity,
9  maxT: :infinity,
10  registered: [],
11  included_applications: [],
12  applications: [:kernel, :stdlib, :elixir, :logger],
13  mod: {OurNewApp.Application, []},
14  start_phases: :undefined
15]
16iex(2)> Supervisor.which_children(OurNewApp.CounterSup)
17[
18  {20000, #PID<0.143.0>, :worker, [OurNewApp.Counter]},
19  {10000, #PID<0.142.0>, :worker, [OurNewApp.Counter]}
20]
21iex(3)> pids = for {_, pid, _, _} <- Supervisor.which_children(OurNewApp.CounterSup), do: pid
22[#PID<0.143.0>, #PID<0.142.0>]
23iex(4)> OurNewApp.Counter.get(Enum.at(pids, 0))
2427225
25iex(5)> OurNewApp.Counter.get(Enum.at(pids, 1))
2617235
27iex(6)> :sys.get_state(Enum.at(pids, 0))
28%{
29  current: 27468,
30  interval: 250,
31  terminator: nil,
32  timer: #Reference<0.948322908.3672375311.146315>
33}
34iex(7)> :sys.get_state(Enum.at(pids, 1))
35%{
36  current: 17476,
37  interval: 250,
38  terminator: nil,
39  timer: #Reference<0.948322908.3672375311.146330>
40}
41iex(8)> :sys.get_state(OurNewApp.CounterSup)
42{:state, {:local, OurNewApp.CounterSup}, :one_for_one,
43 {[20000, 10000],
44  %{
45    10000 => {:child, #PID<0.142.0>, 10000,
46     {OurNewApp.Counter, :start_link, [{10000, 200}]}, :permanent, 5000,
47     :worker, [OurNewApp.Counter]},
48    20000 => {:child, #PID<0.143.0>, 20000,
49     {OurNewApp.Counter, :start_link, [{20000, 200}]}, :permanent, 5000,
50     :worker, [OurNewApp.Counter]}
51  }}, :undefined, 3, 5, [], 0, OurNewApp.CounterSup, [10000, 20000]}
52

Our upgrade process was successfully completed.

You can see that:

The new version of the application is now running.
Our child counters are alive — they updated their state and are functioning.
From the internal state of OurNewApp.CounterSup, we see those child specifications for our counters updated too! Now, if they die, they will properly restart.

But what about the error GenServer #PID<0.147.0> terminating?

Recall that #PID<0.147.0> is the pid of our "orphaned" counter, which was running outside of the supervision tree. As the application upgrade process traverses the supervision tree and updates processes, the "orphaned" counter's state was not updated. But the code of the OurNewApp.Counter did update, so the "orphaned" counter process died: new code met its old state.

We've seen how to upgrade a single running application. We needed only two special tools for that: an .appup file and :release_handler.upgrade_app/2 function. It was also crucial for us to follow OTP principles.

Wrap-up and What to Learn Next

I hope you've enjoyed this whirlwind ride through production-code-upgrades-in-elixir! We started with my guide to hot code reloading, followed by the best use of supervisors when building applications.

This final article has demonstrated how following OTP principles can show us the way to powerful application code upgrades.

It's worth noting that the application code upgrade I've demonstrated here still has disadvantages. The critical issue is that if the whole OS beam process restarts, this will load our application's old code (unless we take some action).

How can we handle this potential problem, I hear you ask? With so-called release upgrades. This awesome article from 'Learn you some Erlang' is a good starting point to dive into release upgrades.

Thanks again for taking the time to read this series — and happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Our guest author Ilya is an Elixir/Erlang/Python developer and a tech leader at FunBox. His main occupation is bootstrapping new projects from both human and technological perspectives. Reach out via his Twitter for interesting discussions or consultancy.