Server process exited without calling ProcessEnding()

Hi there. So I’ve been trying to track down this issue over time through experimentation without much to show for it. My theory is that there is some subtlety to the order or timing of server API calls related to shutting down that I’m missing. I’m using the .Net Server SDK within Unity 5.6.6f2.

On Linux instances I see:

SERVER_PROCESS_CRASHED - Server process exited without calling ProcessEnding(), exitCode(137)

On Windows I see:

SERVER_PROCESS_FORCE_TERMINATED - Server process did not cleanly exit within 30 seconds of OnProcessTerminate(), exitCode(-1)

In my logs from the server instances I see output from my call to GameLiftServerAPI.ProcessEnding() followed by my call to GameLiftServerAPI.Destroy(). I then call Unity’s Application.Quit() method to attempt to cleanly shutdown and upon Unity’s OnApplicationQuit event handlers firing I then attempt to set the exit code via System.Environment.Exit( 0 ). I suspect my attempt to use Unity’s quit functionality in conjunction with System.Environment.Exit( 0 ) may be a questionable approach but regardless I find the Linux instance’s fleet event stating that ProcessEnding() was not called puzzling. Thoughts? Thanks!

Hi @HighScoreHero, It sounds like you’re doing the right thing. Can you provide an example fleet-id and region where you’ve experience these errors?

Thanks,

Ben

Sure thing. Fleet fleet-150a20e1-e992-45f7-9b6a-0de48a82bb56 in us-west-2

Were you able to view that fleet? Any thoughts or insights are appreciated.

So this will sound bad…but I don’t actually have access to a Linux environment to try it locally (relying on the beautiful cross platform dream of Unity). However, I have run a Windows version of my server and verified its error code (cmd %errorlevel%) to be 0. That wasn’t running GameLift local though. I’ll try that as soon as I get a moment.

I’m definitely supposed to be calling the GameLift Destroy API before I exit, right?

Hi @HighScoreHero, Sorry for the delay. We’re seeing your ProcessEnding() call properly. The issue is the exit code we’re detecting from the process. I can see you’re attempting to exit with 0, which would be a successful clean shutdown, but we’re seeing the process exiting with exit code 137.

Process shutdown with exit code: 137

Can you try shutting down locally to see if you’re getting the same exit code?

Thanks,

Ben

Hey there. Finally got a chance to run my server against GameLift Local again. It’s under Windows and not Linux like I’m running in live but still interesting. It looks like GameLift Local is actually throw exceptions…and in doing so possibly preventing my server process from ending? Any thoughts?

7505-gamelift-local-log.txt (17.8 KB)gamelift-local-log.txt

Hi @HighScoreHero,

I’m looking into this now and hope to have an answer for you soon.

Much appreciated!

Hi @HighScoreHero , even Im getting the same logs in Events as ServerProcess didnot cleanly exit, sometimes. Can you please tell us how to overcome it.

Hi @Tejaswini. I don’t actually have a resolution for this issue yet. However, it’s hopefully some consolation that this does not appear to have an impact on the operation of fleets (I’ve been running fleets in a live environment for weeks without major issue). My theory is that this issue might be related to a crash on termination that I can replicate with GameLift Local. That issue has been reported to and replicated by the folks at GameLift.

Has this been resolved? I’ve updated to the latest server SDK for .NET 4.5 but I’m running into the same issue.

I’m seeing this issue about every 5 minutes in the event logs… Is this something others are seeing? I’ve run my game server locally on windows without issues for multiple hours so I’m not sure why that would be happening…

If you are calling ProcessEnding and then calling Destroy there appears to be a race condition that can cause your process to be incorrectly marked as crashed.

Short term fix is to avoid that call to Destroy. A fix is in the works, but no ETA at this time.

1 Like

Hi, I have same issue and avoiding calling Destroy doesn’t help me. My server process log says ProcessEnding() is succssfully called but fleet event log shows “Server process exited without calling ProcessEnding(), exitCode(137)”. In local gamelift, exit code is 0. Is there any solution for this?

(Unity 2019.1, GameLift Server SDK 3.3.0.)

I found the example in Unity says to call:

void OnApplicationQuit()
{
    // Make sure to call GameLiftServerAPI.Destroy() when the application quits,
    // this resets the local connection with GameLift's agent.
    if(!m_ProcessEnding)
        GameLiftServerAPI.Destroy();
}

If you just remove this bit of code, seems to terminate without errors