Applying my newfound knowledge

A few days ago, I left a test case in a failing state which made it very clear where I need to jump in next for this project. The test case is checking that the game session process is no longer alive once the game reaches a game over state. After a deep dive yesterday into how exit signals work for supervised GenServer processes, I decide to simply have the game session process tell its supervisor to shut the process down.

I start with a naïve change and run the test to see if I’m working in the right direction. I update the end_round/2 function which is called by the two handle_info callbacks used to signal the end of a round, one manually and the other based on a timer.

defmodule Minotaur.GameEngine.SessionServer do
  # ...

  def handle_call(:end_round, _, {game, join_code}) do
    {:ok, next_round_game} = end_round(game, join_code)
    {:reply, next_round_game, {next_round_game, join_code}}
  end

  def handle_info({:round_timer_expired, round}, {game, join_code}) do
    if game.round == round do
      {:ok, next_round_game} = end_round(game, join_code)
      {:noreply, {next_round_game, join_code}}
    else
      {:noreply, {game, join_code}}
    end
  end

  defp end_round(game, join_code) do
    timer_end_time = get_timer_end_time(@round_time_ms)
    {:ok, game} = Session.end_round(game, timer_end_time)
    notify_next_round_started(game)
    set_round_timer(game, @round_time_ms)
    update_summary_record(game, join_code)

    case game do
      %{status: :concluded} ->
        DynamicSupervisor.terminate_child(SessionSupervisor, self())
        {:ok, game}

      _ ->
        {:ok, game}
    end
  end

This change has a failure in the test since the process that is handling the :end_round message is immediately getting a shutdown signal which causes it to stop replying. I need to have the process add a message to its own inbox telling it to shut down. This will allow the current current message to finish processing and send a reply.

defmodule Minotaur.GameEngine.SessionServer do
  # …

  def handle_info(:end_session, state) do
    DynamicSupervisor.terminate_child(SessionSupervisor, self())

    {:noreply, state}
  end

  defp end_round(game, join_code) do
    timer_end_time = get_timer_end_time(@round_time_ms)
    {:ok, game} = Session.end_round(game, timer_end_time)
    notify_next_round_started(game)
    set_round_timer(game, @round_time_ms)
    update_summary_record(game, join_code)

    case game do
      %{status: :concluded} ->
        send(self(), :end_session)
        {:ok, game}

      _ ->
        {:ok, game}
    end
  end
end

Unfortunately, the terminate callback for the game session process is not being called with this logic. Eventually, I learn that this is due to calling terminate_child from the same process that is being terminated which the Elixir runtime considers a self-exit and is not following the same shutdown process as an external process sending the exit signal. This is a simple fix by having a one-off task process make the terminate_child call instead.

defmodule Minotaur.GameEngine.SessionServer do
  # …

  def handle_info(:end_session, state) do
    session_pid = self()

    Task.start(fn ->
      DynamicSupervisor.terminate_child(SessionSupervisor, session_pid)
    end)

    {:noreply, state}
  end

  # …
end

With this change, the terminate callback is called when the process is shutting down which will be important in a future step. The test case is now passing, but only intermittently. Sometimes the call to GameEngine.get_game returns the game state since the process hasn’t started to shutdown yet. Other times, the call results in an exception caused by the game session process shutting down while processing the get_game call. I update the test with a previously used wait_until helper function which will retry the assertion until it passes or the timeout is reached.

defmodule Minotaur.GameEngineTest do
  # …

  describe "round ends with 1 PC alive" do
    setup [:create_game_near_end, :start_game_session, :p1_wins]

    test "game status is concluded", %{game: game} do
      assert :concluded == game.status
    end

    test "game session process is stopped", %{game: game} do
      wait_until(500, fn ->
        assert {:error, :game_not_alive} = GameEngine.get_game(game.id)
      end)
    end
  end
end

I also update GameEngine.get_game/1 to catch this shutdown state since there are several places where this function could be called when the end of the game is reached.

defmodule Minotaur.GameEngine do
  # …

  def get_game(game_id) do
    game = GenServer.call(via_tuple(game_id), :get_game)
    {:ok, game}
  catch
    :exit, {:noproc, _} ->
      {:error, :game_not_alive}

    :exit, {:shutdown, _} ->
      {:error, :game_not_alive}
  end
end

A final change to end_round is to move the call to set_round_timer inside the case block for a non-concluded game so the round timer won’t continue for concluded games. This change might not be too important since the whole process is about to be shut down, but it feels right to clean up anyway. The call to notify_next_round_started will broadcast to all subscribers of this game session topic that the next round has started which is currently still needed for game over states otherwise any players who were eliminated or won on the final round will never get an updated state before the game session process is stopped. I may add a separate notification in the future to broadcast the final state for game over conditions.

Handling terminate cases

The terminate callback for game session processes currently stashes the GenServer state to a cluster-replicated data store so state can be restored after a rolling deploy. Now that the game session processes are stopped once a game over is reached, the terminate will stash state for a game that should never be started again. I need to update this logic to distinguish between these different shutdown cases.

I update the terminate callback by branching on the game status and only stash state when the game is still active.

defmodule Minotaur.GameEngine.SessionServer do
  # …

  def terminate(reason, {game, _join_code}) do
    case game.status do
      :active ->
        Minotaur.StateHandoff.stash("game_#{game.id}", game)
        Logger.info("Game Session #{game.id} stopping due to: #{reason}")

      status ->
        Logger.info("Game Session #{game.id} stopping with #{status} status")
    end
  end
end

This closes out the feature for proper game session cleanup after a game is concluded. I’ll consult my to-do list for what feature to work on next time.