Repository navigation
phase transition race condition analysis
There is a race condition in the phase transition logic in ServerGame.TransitionToPhase() that can cause bots and clients to send commands before the server has fully initialized the new phase.
In ServerGame.TransitionToPhase() (line 84-93):
private void TransitionToPhase(IGamePhase newPhase)
{
if (_currentPhase is GamePhase currentGamePhase)
{
currentGamePhase.Exit();
}
_currentPhase = newPhase;
SetPhase(newPhase.Name); // Step 3: Publishes ChangePhaseCommand to clients
_currentPhase.Enter(); // Step 4: Initializes phase on server
}Timeline of events:
-
Server: Calls
SetPhase()which publishesChangePhaseCommandto all clients -
Clients: Receive
ChangePhaseCommandimmediately -
Clients: Set
TurnPhaseproperty, triggering phase change notifications -
Bots: Receive phase change notification via
PhaseChangesobservable - Bots: Update decision engine for new phase
-
Clients: For
EndPhase, setActivePlayerlocally (line 106 inClientGame.cs) - Bots: Receive active player change notification
- Bots: Start making decisions and sending commands back to server
-
Server: Still hasn't called
phase.Enter()yet!⚠️
The problem: Bots can send commands (e.g., StartupUnitCommand, ShutdownUnitCommand, TurnEndedCommand) before the server has completed phase initialization via Enter().
EndPhase.Enter() (line 13-23):
- Clears
_playersEndedTurnHashSet - Processes consciousness recovery rolls
- Checks victory conditions
Issue: If a bot sends TurnEndedCommand before _playersEndedTurn is cleared, the HashSet might contain stale data from the previous turn, causing incorrect "all players ended turn" detection.
MainGamePhase.Enter() (line 15-19):
- Calculates turn order from initiative
- Calls
SetNextPlayerActive()which sets the active player and publishesChangeActivePlayerCommand
Issue: If a bot sends a command before SetNextPlayerActive() is called, the server's ActivePlayer might be null or incorrect, causing command validation to fail (line 36 checks playerId == Game.ActivePlayer?.Id).
DeploymentPhase.Enter() (line 11-15):
- Randomizes deployment order
- Sets next deploying player
Issue: If a bot sends DeployUnitCommand before deployment order is established, the ActivePlayer won't be set correctly.
One might think: "Just call Enter() before SetPhase()". However, this creates a different race condition:
If we call Enter() before SetPhase():
- Server calls
newPhase.Enter() - For server-driven phases,
Enter()callsSetActivePlayer(), which publishesChangeActivePlayerCommand - Clients receive
ChangeActivePlayerCommandwhile still in the old phase - Bots receive active player change notification
- Bots call
MakeDecision()with the OLD decision engine (e.g.,WeaponsEngineinstead ofMovementEngine) - Server publishes
ChangePhaseCommand - Clients update phase, bots update decision engine - but they already made a wrong decision!
Critical insight: Bots subscribe to both PhaseChanges and ActivePlayerChanges. They need to receive ChangePhaseCommand FIRST to update their decision engine, THEN receive ChangeActivePlayerCommand to make decisions with the correct engine.
For correct operation:
-
ChangePhaseCommandmust arrive at clients FIRST - Then
ChangeActivePlayerCommand(and any other phase initialization commands) - Server must complete
Enter()before processing any commands from clients/bots
Approach: Two-Stage Phase Transition Protocol with StartPhaseCommand
The key insight is that we need to satisfy two requirements:
- Server must complete
Enter()before clients can send commands back - Clients must not trigger bot decision-making until server is ready
The solution uses a two-stage protocol:
-
Stage 1: Server publishes
ChangePhaseCommandand callsEnter()(as before) -
Stage 2: Server publishes
StartPhaseCommandat the END ofEnter()to signal completion
Introduce a new StartPhaseCommand that signals when phase initialization is complete:
1. Create StartPhaseCommand (in src/MakaMek.Core/Data/Game/Commands/Server/StartPhaseCommand.cs):
public record struct StartPhaseCommand : IGameCommand
{
public required PhaseNames Phase { get; init; }
public required Guid GameOriginId { get; set; }
public DateTime Timestamp { get; set; }
public string Render(ILocalizationService localizationService, IGame game)
{
var localizedTemplate = localizationService.GetString("Command_StartPhase");
return string.Format(localizedTemplate, Phase);
}
}2. Modify EndPhase.Enter() to publish StartPhaseCommand at the end:
public override void Enter()
{
// Clear the set of players who have ended their turn
_playersEndedTurn.Clear();
// Process consciousness recovery rolls for unconscious pilots
ProcessConsciousnessRecoveryRolls();
// Check for victory conditions
CheckVictoryConditions();
// Publish StartPhaseCommand to signal that phase initialization is complete
Game.CommandPublisher.PublishCommand(new StartPhaseCommand
{
GameOriginId = Game.Id,
Phase = PhaseNames.End
});
}3. Update ClientGame.HandleCommand() to handle StartPhaseCommand:
case ChangePhaseCommand phaseCommand:
TurnPhase = phaseCommand.Phase;
// When entering the End phase, clear the players who ended turn
// Note: ActivePlayer is set when StartPhaseCommand is received to avoid race condition
if (phaseCommand.Phase == PhaseNames.End)
{
_playersEndedTurn.Clear();
}
break;
case StartPhaseCommand startPhaseCommand:
// When the End phase is fully initialized on the server, set the first alive local player as active
// This ensures the server has completed phase initialization before bots start making decisions
if (startPhaseCommand.Phase == PhaseNames.End)
{
ActivePlayer = AlivePlayers.FirstOrDefault(p => _localPlayers.ContainsKey(p.Id));
}
break;-
Server publishes
ChangePhaseCommand(as before) - clients update their phase -
Server calls
Enter()- phase is fully initialized (clears state, processes consciousness rolls, etc.) -
Server publishes
StartPhaseCommandat the end ofEnter()- signals initialization is complete -
Clients receive
ChangePhaseCommand- setTurnPhase, but DON'T setActivePlayeryet for EndPhase -
Clients receive other commands from
Enter()(consciousness rolls, etc.) -
Clients receive
StartPhaseCommand- NOW setActivePlayerfor EndPhase -
Bots receive
ActivePlayerchange - make decisions with correct decision engine -
Server is ready -
Enter()has completed, server can process incoming commands
For a transition to EndPhase:
Server side:
-
TransitionToPhase()callsSetPhase()→ publishesChangePhaseCommand -
TransitionToPhase()callsEndPhase.Enter() -
Enter()clears_playersEndedTurn -
Enter()callsProcessConsciousnessRecoveryRolls()→ publishesPilotConsciousnessRollCommand(if any) -
Enter()callsCheckVictoryConditions()→ might publishGameEndedCommand -
Enter()publishesStartPhaseCommandat the end - Server is now ready to process commands
Client side:
- Receives
ChangePhaseCommand→ setsTurnPhase = PhaseNames.End, clears_playersEndedTurn - Receives
PilotConsciousnessRollCommand(if any) → updates pilot consciousness state - Receives
StartPhaseCommand→ setsActivePlayerto first alive local player - Bots receive
ActivePlayerchange → callMakeDecision()withEndPhaseEngine - Bots send commands (
StartupUnitCommand,ShutdownUnitCommand,TurnEndedCommand) - Server processes commands (phase is fully initialized)
This solution is currently implemented only for EndPhase as a focused fix for the most critical race condition. The same pattern can be extended to other phases (MainGamePhase, DeploymentPhase) if needed in the future.
- Implement a deferred publishing mechanism in
CommandPublisherto queue commands duringEnter() - Publish
ChangePhaseCommandfirst, then publish all queued commands -
Rejected: More complex than the two-stage protocol, requires changes to
CommandPublisherinfrastructure -
Issue: The RX transport processes commands synchronously, so even with deferred publishing, clients could react before the server finishes
Enter()
- Call
Enter()before publishingChangePhaseCommand -
Rejected: Creates a different race condition where
ChangeActivePlayerCommandarrives beforeChangePhaseCommand - Issue: Bots would react to active player change with the wrong decision engine
- Add artificial delay before bots react to phase changes
- Rejected: Hacky, unreliable, doesn't fix the root cause
-
src/MakaMek.Core/Data/Game/Commands/Server/StartPhaseCommand.cs- NEW- Created new command to signal phase initialization completion
- Follows same pattern as
ChangePhaseCommand
-
src/MakaMek.Core/Models/Game/Phases/EndPhase.cs- MODIFIED- Added
StartPhaseCommandpublishing at the end ofEnter()method - Signals that phase initialization is complete
- Added
-
src/MakaMek.Core/Models/Game/ClientGame.cs- MODIFIED- Added handler for
StartPhaseCommand - Moved
ActivePlayersetting for EndPhase fromChangePhaseCommandhandler toStartPhaseCommandhandler - Added comment explaining the race condition prevention
- Added handler for
-
tests/MakaMek.Core.Tests/Models/Game/Phases/EndPhaseTests.cs- MODIFIED- Added test
Enter_ShouldPublishStartPhaseCommand_WhenPhaseInitializationCompletes - Added test
Enter_ShouldPublishStartPhaseCommandAfterOtherInitialization - Verifies that
StartPhaseCommandis published with correct properties - Verifies that
StartPhaseCommandis published last (after consciousness rolls, etc.)
- Added test
All 23 EndPhase tests pass, including the new tests for StartPhaseCommand:
- ✅
Enter_ShouldPublishStartPhaseCommand_WhenPhaseInitializationCompletes - ✅
Enter_ShouldPublishStartPhaseCommandAfterOtherInitialization
This solution is currently implemented only for EndPhase. If the race condition becomes an issue for other phases, the same pattern can be applied:
For MainGamePhase (Movement, WeaponsAttack, PhysicalAttack):
- Modify
MainGamePhase.Enter()to publishStartPhaseCommandat the end - Update
ClientGame.HandleCommand()to handleStartPhaseCommandfor these phases - Consider whether
ChangeActivePlayerCommandshould be deferred until afterStartPhaseCommand
For DeploymentPhase:
- Modify
DeploymentPhase.Enter()to publishStartPhaseCommandat the end - Update
ClientGame.HandleCommand()to handleStartPhaseCommandfor deployment - Consider whether deployment order randomization should be signaled to clients
The implemented two-stage phase transition protocol using StartPhaseCommand successfully resolves the race condition for EndPhase:
- ✅ Server completes initialization -
Enter()finishes before clients setActivePlayer - ✅ Bots react at the right time -
ActivePlayerchange happens afterStartPhaseCommand - ✅ Simple and focused - Only 3 files modified, minimal complexity
- ✅ Well tested - 2 new tests verify correct behavior
- ✅ Extensible - Same pattern can be applied to other phases if needed
This approach is simpler than deferred publishing, doesn't require infrastructure changes, and directly addresses the root cause of the race condition by delaying bot activation until the server is ready.