Game sessions are not reaching the ACTIVE state and logs are not appearing

When creating a game session through Unity C# script or through the AWS CLI for gamelift, game sessions make it to the ACTIVATING state, but never make it to the ACTIVE state. In the server code, there is GameLiftServerAPI.ActivateGameSession() under onStartGameSession callback. I have borrowed heavily from the Gamelift Unity sample. It runs well locally.

I am having problems getting visibility into the problem due to the inability to review logs. I have RDP’d into the deployed server and I also do not see any logs. I have set the logging in the server application to a valid path under @“C:\game\output_log.txt” , but no logs. The health looks good from the Gamelift console.

Is there a permissions issue with the logs?
Is there a way to view the Unity Debug.log statements through some viewer? Or will those be pushed into the c:\game\output_log.txt? I have also tried writing the logs to the deployment data directory with no log being created there either.

Any help would be appreciated!

Have you tried debugging the issue against GameLift local? https://docs.aws.amazon.com/gamelift/latest/developerguide/integration-testing-local.html - just want to verify what you mean by locally.

If the game session is in activating, that means GameLift is trying to create the session. But without logs from the server, this may be hard to track down.

I believe a Gamelift hosted binary does not have permission to write outside of its home directory, so you may be right wrt to permissions. Have you tried changing the path to?
C:\game<MyGame>\output_path.txt

Finally, once you’re on the instance, you could try manually launching from the console (assuming you log anything to the console as well) which could help you track down the logging issue.

Let me know if any of that helps

Thanks for the information. I think you are correct about the permissions. I changed the location of where Unity writes its player.log by adding the logFile launch parameter. So when creating the fleet I now have this configuration: -batchmode -nographics -logFile “c:\game\player.log”. With that I now have logging! And can now move forward solving my issue. So thanks!

Previous to the logFile change, the logs would try to write to the c:\Users\gl-user-server\AppData\LocalLow\DefaultApp directory, but must hit a permissions issue. If the app was run manually, as you suggested, it would run as the gl-user-remote and write the player.log successfully. But, I could not use this instance for any real debugging since the Whitewater application server (Tomcat it looks like) would recognize it as an invalid instance. But that seemed to be enough to prove it must be a permissions issue.

As well, when I say local, I am referring to GameLift local.

Now that I have logging, I can see this. My health check callback is being called just fine. But, my onStartGameSession is not being called at all…even with a valid create game session request.

Here’s the create game session request made from AWS CLI. Note that health check continues to get called, but not OnStartGameSession…both of which are parameters successfully sent into ProcessReady (which is successful) through the ProcessParameters.

aws gamelift create-game-session --fleet-id “fleet-131a7072-ef04-4025-a845-c972aaf46c42” --maximum-player-session-count “5”
{
“GameSession”: {
“GameSessionId”: “arn:aws:gamelift:us-west-1::gamesession/fleet-131a7072-ef04-4025-a845-c972aaf46c42/gsess-3ddff47e-43ee-4154-9f7e-c2f3a8dbf940”,
“FleetId”: “fleet-131a7072-ef04-4025-a845-c972aaf46c42”,
“CreationTime”: 1579545502.917,
“CurrentPlayerSessionCount”: 0,
“MaximumPlayerSessionCount”: 5,
“Status”: “ACTIVATING”,
“GameProperties”: [],
“IpAddress”: “52.53.183.150”,
“DnsName”: “ec2-52-53-183-150.us-west-1.compute.amazonaws.com”,
“Port”: 1935,
“PlayerSessionCreationPolicy”: “ACCEPT_ALL”
}
}

Great. Now we get into the fun part.

When you try to create a game session, GameLift looks for spare capacity on your queue/fleet. This means you need to have healthy available server processes to host your new session.

Firstly, look at your fleet metrics and ensure you have capacity for a new game session, you should see available game sessions (https://docs.amazonaws.cn/en_us/gamelift/latest/developerguide/monitoring-cloudwatch.html)

If you have capacity, you should be seeing calls to onStartGameSession and your server should call ActivateGameSession afterwards to change the state.

If you’re not seeing these calls then:

  • You could be starving/not ticking threads such that your callbacks are dying - try dialing everything down to a single process on a single instance (ie no multiprocess in play here)
    • Validate game session creation via GameLift local (which you’ve already done)
    • Ensure you aren’t inadvertently failing health checks (needs to pass true every 60s)
  • You do not have capacity on your fleet - ie you already have active game sessions, so there are no server processes to host your new call
    • Scale down fleet to zero and scale up if you have no mechanism to end a game session
    • Fleet metrics should show you this.
  • GameLift thinks your process is unhealthy
    • Ensure you have called InitSDK and there are no unexpected Fleet events for your fleet
    • Fleet metrics look good wrt to process health
    • Turn off any scale down you may have so GameLift is not terminating instances unexpectedly.
  • Theres an SDK problem communicating with your fleet
    • Events really aren’t arriving on your fleet

The last is rare, but as you’ve already provided a fleet and game session id, have enough for GameLift to investigate.

First I would ask you to look at your fleet metrics and ensure you have capacity and to double check your FleetEvents to ensure that GameLift is not marking your process as unhealthy (ie its starting up but then failing due to blocked threads, health checks, failure to call to InitSDK/ProcessReady correctly etc).

I did follow the guidance @Pip provided (thank you BTW!). But still have not made any conclusion why the onStartGameSession callback is not executing. And to re-confirm - in the local GameLift (using the GameLiftLocal.jar), there is no problem.

On AWS GameLift fleet metrics, I can see that there is (1) available game session. When I use the AWS CLI to create a game session (same procedure as with the local), I can see the metric go to (0) after the request. As well, all test cases have been for 1 process / 1 instance. The one notable thing (that may be nothing at all) is that there is a log message from Unity that it has created thread for ‘Enlighten’ which is a lighting feature. I’ve included the snippet of the log below…which includes initialization logging and up to three health check confirmations.

So, not sure if we are limited to a certain number of threads??? But, I don’t think that is the case, since health checks are succeeding at a constant rate before and after the tests. I see them every 60 seconds.

I can confirm InitSDK is called and ProcessReady is successfully called - or else health checks would not be called.

Logging Snippet

Mono path[0] = ‘C:/game/unity-server_Data/Managed’
Mono config path = ‘C:/game/MonoBleedingEdge/etc’
Initialize engine version: 2019.3.0f5 (44796c9d3c2c)
[Subsystems] Discovering subsystems at path C:/game/unity-server_Data/UnitySubsystems
GfxDevice: creating device client; threaded=0
d3d11: QueryInterface(IDXGIFactory5) failed (80004002).
Direct3D:
Version: Direct3D 11.0 [level 11.1]
Renderer: Microsoft Basic Render Driver (ID=0x8c)
Vendor:
VRAM: 256 MB
FMOD initialized on nosound output
Begin MonoManager ReloadAssembly

  • Completed reload, in 0.091 seconds
    Dedicated video D3D11 device creation failed (error: 0x887a0004). This device is used specifically for video decoding, we will use software video decoding instead.UnloadTime: 0.487076 ms
    :slight_smile: GAMELOGIC AWAKE
    (Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: GAMELIFT AWAKE
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: I AM SERVER
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: GAMELOGIC START
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: GAMELIFT START
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: SDK VERSION: 3.3.0
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: SERVER IS IN A GAMELIFT FLEET
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: PROCESSREADY SUCCESS.
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

Setting up 1 worker threads for Enlighten.
Thread -> id: 6c0 -> priority: 1
:slight_smile: GAMELIFT HEALTH CHECK REQUESTED (HEALTHY)
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: GAMELIFT HEALTH CHECK REQUESTED (HEALTHY)
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

:slight_smile: GAMELIFT HEALTH CHECK REQUESTED (HEALTHY)
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

I’ve asked the GameLift service team to look at your fleet to see what is going. Hopefully we will here from them soon.

Hi Pip,

I’ve been actively working on the local version of the app using the Gamelift.jar file and am coming back to the server and am still experiencing the issue.

Here’s the fleet id that is having the issue.
fleet-e060857c-c09a-4c3e-b0fd-64d5bbfa3b48

Thanks!

Problem solved!

I had compiled using the .NET 3.5 libraries. Compiling with the 4.5 libraries solved the issue.

I thought I had read somewhere that you must use the 3.5 version??? Not sure. Anyhow, I have had no issues using the 4.5 libraries at this point.

Thanks!