Gamelift instances slow disk I/O

Hello,

We are noticing what we think are very slow disk I/O operations on our gamelift instances for our first game session (and only first game session, later sessions are fine).

Profiling our game servers, we noticed, on the first game session, when our server loads our game’s map, a ~4 minutes loading time whereas it only takes a few seconds for the following game sessions.

Since this is 100% reproducible with the first game session and the second game session’s map loading isn’t abnormally long, we are very confident this is not an issue with our game or game engine but rather an issue with gamelift instances setup.

Reading some documentation about EC2, I stumbled across the notion of EBS volume pre-warming to reduce I/O latency.

However, there is very little information regarding if and how this applies to GameLift, at what step this pre-warming should be ran (I suppose it would be executed with the setup.sh script ?)…

Best regards

Gamelift will typically pre-warm the EBS instance volume. Can you let me know the OS and instance type you are seeing this on?

Its also worth noting on the first run of a game session on a new instance a bunch of things are downloaded and checked, so first session launch can be slightly longer. But the impact of this should be minimal and definitely not minutes.

If you can provide a Fleet id or game session id, we can quickly see if theres something obvious. However because this may involve server logs etc it may be better to open an AWS support ticket.

Hi @Pip, thank you for your answer.

We are seeing this issue on linux servers and the three game instance type we tested (c4.large, m4.large and m4.xlarge).

A game session you can observe this issue on is:
Game Session ARN: arn:aws:gamelift:eu-central-1::gamesession/fleet-84258709-3df2-43d5-be20-c1fa4dc8394b/yyvnxlsvbmfiyndcafsjqhci

Pre warming manually the EBS volume by following this tutorial seemed to fix the issue but I have to conduct additional tests today to confirm.

Thanks for the details. Team is looking at this and hope to have an update soon.

I can confirm the 4 min start up and it does look like pre-warming is not happening on these instances.

The short term solution is to pre-warm/initialize your game directory yourselves on launch, as I believe you are already trying to do:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

I expect someone from Gamelift to reach out to you directly to follow-up on this. If this does not happen, please open an AWS support ticket.

Here’s the solution given by GameLift technical support for anyone interested / having this issue:

“In order to programatically warm your Elsatic Block Store disks, our suggestion from our developers is to do this as part of the process launch itself. The process could boot up, start prewarming and then once complete, call ProcessReady() to register itself with GameLift. Subsequent processes that attempted to prewarm would complete quickly since those blocks would already be warm so this is something they would only need to do once per instance.”