How to SSH to instance and replace build in-place to iterate quickly

Hi folks,

Just want to share some knowledge on how to iterate game server development quickly with a single fleet.

The best practice is still to test out your server SDK integration with GameLift Local first, to make sure you can successfully respond to a heartbeat. Otherwise, your fleet might go into ERROR if you deploy an untested build.

However, GameLift Local is quite limited currently and doesn’t support all features obtainable from Server SDK, so, sometimes it might be necessary to test in the actual GameLift EC2 environment. But there is another problem, uploading an updated build build and getting a new fleet to go ACTIVE takes a long time (20-40 minutes depending on your build) and is not viable for quick iteration.

GameLift is currently working on revamping this process, in the mean time, here is a workaround:

  1. Create a fleet with port 22 open. It’s always a good idea to open port 22 for development, in case fleet is stuck in ACTIVATING and you cannot modify port settings until fleet is ACTIVE. Sample command:
aws gamelift create-fleet --region us-west-2 \
--name "TestFleet" --ec2-instance-type c5.large  \
--ec2-inbound-permissions FromPort=6250,ToPort=6280,IpRange="0.0.0.0/0",Protocol=TCP FromPort=22,ToPort=22,IpRange="0.0.0.0/0",Protocol=TCP \
--build-id <build-id> \
--runtime-configuration ServerProcesses="[{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6251 gameSessionLengthSeconds:10',ConcurrentExecutions=1},\
{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6252 gameSessionLengthSeconds:10',ConcurrentExecutions=1}]"
  1. Once the fleet is ACTIVE, run the following script (you can also do this after the fleet has been ACTIVATING for a while, or is stuck in ACTIVATING, so that EC2 instance is created)
if [ "$#" -ne 1 ]; then
    echo "Usage: ./sshhost.sh fleetid"
    exit 1
fi
  
FLEET_ID=$1
  
echo Fleet Id is $FLEET_ID
  
aws gamelift update-fleet-port-settings --fleet-id $FLEET_ID --inbound-permission-authorizations FromPort=22,ToPort=22,IpRange=0.0.0.0/0,Protocol=TCP 2> /dev/null
  
INSTANCE_ID=$(aws gamelift describe-instances --fleet-id $FLEET_ID --output text --query 'Instances[0].InstanceId' 2> /dev/null)
  
echo Instance Id is $INSTANCE_ID
  
IP_ADDRESS=$(aws gamelift describe-instances --fleet-id $FLEET_ID --output text --query 'Instances[0].IpAddress' 2> /dev/null)
  
echo IP Address is $IP_ADDRESS
  
rm -rf /tmp/gamelift.pem
  
aws gamelift get-instance-access --fleet-id $FLEET_ID --instance-id $INSTANCE_ID --output text --query 'InstanceAccess.Credentials.Secret'  2> /dev/null | tee /tmp/gamelift.pem
  
chmod 400 /tmp/gamelift.pem
  
echo SSHING to host at $IP_ADDRESS
  
ssh -i /tmp/gamelift.pem gl-user-remote@$IP_ADDRESS
  1. Run “sudo su” to assume the root access
  2. Run "rm -rf /local/game/``<FILES IN THE BUILD> ” to unlink and delete the old build (it’s important to run rm instead of mv to unlink the running process, see stack overflow post: Replacing a running executable in linux - Stack Overflow)
  3. Upload the new build to “/local/game” using SCP, sample command:
scp -i <path-to-gamelift.pem> <path-to-new-build> \
gl-user-remote@<ec2-ip-or-dns-name>:/local/game/<path-to-old-build>
  1. Run “chown gl-user-server:gl-user <FILES IN THE NEW BUILD>” , need to use “--resursive” if you have directories, this will set the new files to the right permission so that the GameLift onbox process manager (“AuxProxy”) can read them.
  2. Run "ps -efww | grep "sudo -H -E -u gl-user-server” to get the running process ids
  3. Run “kill -9 <PID>” to kill the running processes, GameLift onbox process manager (“AuxProxy”) will try to spin up a new process afterwards
  4. Poll “ps -efww | grep "sudo -H -E -u gl-user-server” until a new process shows up
  5. Create game sessions and connect the client to the new server build to test!
  6. Repeat step 4-10 to iterate on the server builds.
1 Like

Thanks for the tip!

It would be beneficial for us if fleet deployment could be made faster. We do continuous deployment on our development branch and do a full build of our game client, game server, and deploy a new fleet. This happens 20+ times a day for us. By far the longest step in our build process is waiting for the fleet to deploy, which is typically around 20 minutes in our case. We do this so we always have latest code & data deployed for our team to playtest.

Maybe we should convert to deploying development server builds in the manner you suggested instead to speed things up. Though that would mean our production deploys would have a different deployment code path from our development deploys. The production path wouldn’t be as well tested. I think there is value in having our development deploys be near identical to production.

So overall it would be nice if fleet deployment could be dramatically sped up to be something closer to the time to bring up the underlying EC2 resources. Which in my experience is usually measured in seconds for Linux instances compared to 20+ minutes for GameLift managed instances.

1 Like

Hi @ShinyMark

Completely understandable. Speeding up fleet deployment time is a very highly requested feature and we have plans to eliminate that gap. Keep your eyes peeled for announcements when we roll out improvements :slight_smile: