SERVER_PROCESS_FORCE_TERMINATED event when manually scaling down fleet capacity

Hi everyone, I recently uploaded a simple Unreal Engine 5 Early Access default project build onto GameLift, created a fleet, and was able to get clients to connect to server processes running on the instance in the fleet via hardcoded IP addresses and ports just for testing purposes. However, I noticed that when I manually scaled down the fleet’s instances, I get a SERVER_PROCESS_FORCE_TERMINATED event with the message “Server process did not cleanly exit within 30 seconds of OnProcessTerminate(), exitCode(139)…” for every running server process in the fleet.

The onProcessTerminate callback function in the ProcessParameters object passed to ProcessReady is defined, and that callback function calls ProcessEnding and cleanly shuts down the server using the Unreal FGenericPlatformMisc::RequestExit(false) function afterwards. It has been a little while since I’ve played with GameLift so I’m not sure if Unreal Engine has changed or if I’m doing something else wrong. Note that this happens with both Windows and Linux builds of the same project uploaded to GameLift.

For debugging purposes, here’s a fleet id: fleet-fa5303b3-6e88-4a54-a67e-a4f789de3ea0

Thank you to anyone who responds in advance!

Can you try to add a call to Application.Quit() or whatever the equivalent is for Unreal? I know this was an issue i ran into with Custom GameLift Unity server builds:

Hey, appreciate the reply. I believe that FGenericPlatformMisc::RequestExit(false) is the Unreal version of Application.Quit() and I call it after ProcessEnding, so I’m surprised that this is happening since it wasn’t happening a year ago when I was playing around with GameLift.

Hi Chris, from the UE docs on RequestExit, i see:

Force: If true, perform immediate exit (dangerous because config code isn't flushed, etc). If false, request clean main-loop exit from the platform specific code.

I’m wondering if you are seeing the SERVER_PROCESS_FORCE_TERMINATED fleet events because your process didn’t end in 30 seconds due to some time-confusing cleanup in the main loop. Could you try setting Force to true (i.e. FGenericPlatformMisc::RequestExit(true)) to see if the abnormal fleet events go away?

If so, does UE have a callback that triggers your custom code prior to ending the process? ProcessEnding might be best called there. How about SetGracefulTerminationHandler? FGenericPlatformMisc::SetGracefulTerminationHandler | Unreal Engine Documentation

Also, you could use SetGracefulTerminationHandler to log the end time, and prior to RequestExit, log the start time, to calculate how long the termination process actually takes.

Hey, sorry for the late response. I’ll try out your suggestions soon and let you know how it goes.

Hey @JamesM_Aws, so I tried some of your suggestions. I passed true instead of false to GenericPlatformMisc::RequestExit so that the process would theoretically end quicker. That unfortunately did not work.

I also tried to move the call to ProcessReady to a function that is called when the process is shutting down. That function is UGameInstance::Shutdown, so I had to overwrite my custom game instance class’ Shutdown method to call ProcessReady. In this case, I still call FGenericPlatformMisc::RequestExit(true) almost right after the onProcessTerminate handler function is called. I experimented with false instead of true here as well. That unfortunately did not work either.

As for FGenericPlatformMisc::SetGracefulTerminationHandler, this function is already overwritten by Unreal Engine’s source code for various platforms (Windows, Linux, etc.), and is called automatically for us when the game starts to make sure that the game exits properly in response to a console interrupt or exit request. For example, for Windows servers, FWindowsPlatformMisc::SetGracefulTerminationHandler sets the handler function to be a function that calls WINAPI’s TerminateProcess function. Therefore, I don’t believe this function should be modified, but I may for the sake of trying out your logging idea. Although this function sets the function that gets called when an “exit” is requested, I’m trying to avoid editing engine code since that seems like it should be a last resort for an issue like this.

I want to try your logging idea but I’m not sure how to go about that exactly. Do you have any suggestions on how to go about that? Perhaps adding a log statement in Shutdown as well as before calling RequestExit like you mentioned? Also, if the server process is force terminated after 30 seconds anyway, is it even possible to get how long it takes the server process to truly shut down? And how would I get these logs since I don’t believe server logs are archived anywhere? I appreciate your suggestions so far, so thank you.