When are logs stored

Hello,
I have a question about logs and how gamelift stores them, or more specifically when.
I know from the docs:
“When a game session is terminated, GameLift automatically stores the logs in Amazon S3 and retains them for 14 days.”
However I am curious at what point are they stored? Is there a time in which we know they will have been stored? I want to make sure that logs are stored and uploaded before they are cleared or recycled. I see some samples using things like ports to keep log file names unique to their server process, but if a process is terminated then is there any guarantee when those logs will be uploaded before gamelift starts another server process up to replace the terminated one? Any clarification or details here would be appreciated, thanks.

Bryan

Hi @Bryan

Logs are stored after the game session process is terminated (or more technically, when the socket connection with the onbox websocket server is disconnnected due to the process that initiated the connection via initSDK() being terminated).

When the process is terminated, the onbox agent will collect the logs, zip the files, upload the zip to S3, and delete the zip file.

Our solution architects have provided guidance on how to conduct log rotation on GameLift in the past, I’ll get some of them to help.

We also have plans to add this detail in the AWS documentation, and eventually, provide out-of-the-box solution to handle log rotation for customers after opt-in.

GLIFT-15554, GLIFT-14237

Thank you, that would be appreciated. I am concerted that with log rotations I may not be able to control the rotation happening while gamelift is gathering or zipping the log files.

I am currently outputting log files with the process port as part of the filename to keep all logs unique across the instance. My concern being one the process terminates gamelift will start up a process to replace it and it could use the same port and thus overwrite the log file that gamelift should archive.

The best solution I can think of so far is to append a date time stamp as well and have a step to remove logs older than some time period where it is reasonable to assume gamelift will be done with them. That being said I don’t love this approach as it requires making that assumption about how long the files may be in use after the session has terminated. It also requires the assumption of a max session length.

I would love to hear the official recommendation! I look forward to hearing from the team.

Thanks again!

Bryan

Hi @Bryan

My concern being one the process terminates gamelift will start up a process to replace it and it could use the same port and thus overwrite the log file that gamelift should archive.

The log file lifecycle is controlled by the developers, so you have full control on what happens to the log file when a new process with the same pid is spun up, i.e. to overwrite or to append the existing log file. You will also have control on when to delete the log files since gamelift doesn’t handle deleting it for you yet. (Again, we are looking into an opt-in approach to handle log deletion for you)

Your concern is certainly valid, but it’s also easy to resolve, i.e. you can add other unique identifiers in your log name besides pid, for instance, a UUID, or the start time of the process. This way, you can guarantee no collisions in your log files.

I would love to hear the official recommendation! I look forward to hearing from the team.

There are 2 recommendations for rotating log files on your host:

  1. Setup CloudWatch agent in install.sh. Here is the example on how to do that: https://github.com/aws-samples/aws-gamelift-and-serverless-backend-sample/tree/master/LinuxServerBuild. This will provide more real-time game session logging (i.e. you don’t have to wait for game session termination to receive logs), and you can additionally set it up to rotate log files for you: CloudWatch Logs agent reference - Amazon CloudWatch Logs. NOTE: you’ll incur additional CloudWatch cost with this approach, depends on how many game sessions you run and how big your logs are. (Another relevant post on CW Agent)
  2. As said above, after process termination, GameLift copies your logs to a temporary directory, zips up that directory, upload the zip, and delete the temporary directory as well as the zip. So, you really just need to wait for the file copy to complete, and you don’t have to wait for S3 upload to complete. So, you can setup some heuristics, e.g. prior to terminating the process, schedule another janitor process to delete the specified log files after N minutes. Adjust N based on your log file size. 5-10 minutes might be good enough for most cases.