GameLift Local - Test ProcessEnding

Hi,

I’ve been getting “Server process exited without calling ProcessEnding(), exitCode(137)” and honestly I have no idea how. My .net core server code should be returning 0, the code in theory goes back to the Main and returns 0, but I also tried Environment.Exit(0), even forcing an exception… nothing seems to work, and the reason this is bothering me so is because I am afraid I am doing something very wrong, because after a while, the server is no longer able to allocate new matches. (I recycle the same instance for multiple matches as it is very simple to do in my case)

So, I wanted to test the shutting down process on GameLiftLocal to see what is going on, since honestly I’ve been struggling with debugging features in the system, however, I dont know how to, from the aws cli, scale down the local so it messages my server to shut down or something like that. I’ve been browsing but couldn’t find it anywhere in the documentation so I am starting to be afraid it is not possible.

Could someone give me some light on this regard?

Best wishes,
Allan

By the way, if someone could help me with the problem itself it would be appreciated. When looking at the logs which were honestly a pain to get lol, it seems like everything is going on as expected:

Init sdk is called
Ready for match is called
Terminate match is called
Ready for match is called again for a new match

Still, it seems like when 50 or so matches happen, the server stops being able to start new matches (I got 50 instances, which means they are no recycling as expected). I have no idea what is going on cause I dont have any logs that show me any issue…

Thanks for the attention,
Best,
Allan

Exit code 137 would probably indicate that you may be out of memory and the process is being killed.

I would check:

  • Metrics for your fleet; look for unhealthy process metrics ( ServerProcessAbnormalTerminations or PercentHealthyServerProcesses) : https://docs.aws.amazon.com/gamelift/latest/developerguide/monitoring-cloudwatch.html
  • Remote into instance to check check instance memory/cpu usage
  • Check your server logs on an instance to see what happened from your side, esp any health check responses; Check that you are not reporting health check failures for your process or failing to report health checks.
  • Ensure you have a logging statement for all the callbacks in ProcessReady so you can see order of events from GameLift.

GameLift could not be recycling your processes because the instance is out of memory or your processes are crashing when trying to host a new session. Although on a crash/unhealthy process GameLift should recycle the processes. If the instance becomes unresponsive, this may not be possible but the whole instance should be recycled then by GameLift and you should have capacity.

You can check your restart behaviour somewhat with GameLift local. GameLift local does not launch / manage your process in the same way as GameLift so somethings can’t be tested (like the forced termination paths).

You can at least confirm you are calling ProcessEnding and handling the response ie:

11:57:32,741  INFO || - [SDKListenerImpl] nioEventLoopGroup-3-1 - onProcessEnding received from: /127.0.0.1:50445 and ackRequest requested? true

See this thread: SERVER_PROCESS_FORCE_TERMINATED Event and Testing OnProcessTerminate which has lots of useful information.

If you are still having problems, if you provide your fleet-id and region, we can get the GameLift service to take a look and see what their logs say.

Thank you for the answer Pip. That thread indeed has a lot of useful information, I had read it before but I apparently had missed some of it for sure. I will run some tests and report back later, thank you!

As you are running 50 servers per instance, it may be too much for your instance type. If you haven’t carefully scaled to this number, consider scaling down number of concurrent servers or beefing up the instance type.

I would always start with a small number and then work out best way to safely scale up based on data (CPU/Memory usage on instances, failure rate on placing game sessions etc)