Queue Priority not working as expected

I can confirm we are facing the same issue.
I have a queue in ap-southeast-1.
I have 2 spot fleets, 1 each in ap-southeast-1 and us-west-1

I have set the ap-southeast-1 fleet as the first priority but every single placement on that fleet is flagged as “not viable” and is instead placed on us-west-1. This has been the case for the last 2 weeks straight. So it’s not a momentary issue with the regions.

What can be done to mitigate this behaviour ? Since the documentation says that priority should be followed but that isn’t happening.

Edit: Both fleets are healthy with no unusual events on the dashboard.
No latency data is used for the placement requests

I moved this to a new a topic, from: GameLift Queue Priority is broken!

If you can provide your fleet ids, I can get the GameLift service team to take a look.

I would suggest troubleshooting this a bit:

  • Ensure you have spare capacity in ap-southeast-1? Do your fleet metrics show that you have capacity in the region?
  • Can you place game sessions in the fleet directly?
  • Are you seeing any abnormal fleet events in ap-southeast-1? Check your fleet dashboard and ensure your processes are healthy.
  • Are you using player latency data at all? If so ensure you are passing the right values?
  • Have you tried changing instance types in ap-southeast-1? Try using different generations of the instance type ie c3.large or c4.large or c5.large or maybe briefly c5.xlarge, c4.xlarge etc to see if its the instance type being constrained etc.

ap-southeast-1: fleet-9d2060c6-c959-4137-afac-e4ea2707a2e3
us-west-1: fleet-045832b1-1468-4b80-b765-62ce4dfb9763

We are testing various configurations so the queue setup might not be consistent, but when we do have 2 SPOT targets in the queue, it doesn’t pick ap-southeast-1 even if prioritised.
If ap-southeast-1 is the only target, then it works perfectly fine.

We have tested this with c5.large and c5.xlarge. Currently the fleets are both c5.large.
As previously mentioned, no latency data in the requests.
Fleets were healthy and had sufficient capacity (even at 100% available game sessions).

Note to the team: 1 unhealthy process (out of 4) in ap-southeast-1 (part of our testing) was only after we switched to single destination in the queue.

Thanks I’ve let the GameLift service team know and hopefully they will have a reply soon.

BTW In the future if you have an AWS account with technical support (https://docs.aws.amazon.com/awssupport/latest/user/case-management.html) I would encourage you to open issues through that channel. It will be much faster and you can share details privately.

Hi!

What are the timestamps that you see these failures during (including the time zone)?

I just did 2 placements and both directed to us-west-1 as usual.
2020-09-24 15:27:06 UTC+0800 was using fleets as queue target
2020-09-24 15:29:26 UTC+0800 was using aliases as queue targets
On the queue metrics both placements were flagged as first choice not viable

Edit: these are the game session creation timestamps. Let me know if there’s any additional information required

Thanks. Do you still recommend opening a case through this channel ?

Hey Alex. Any updates on the why the queue priority isn’t working as expected ?

We generally recommend that you open a case with technical support as they provide a high tier of support for customers including discussing technical details and logs that you may not wish to discuss on forums. Additionally they are often able to address issues sooner as they have 24/7 support.

I’ve look at this a bit further and concluded that queues is actually working as expected. When considering which fleet to place onto (when not considering/using latencies) a queue evaluates the following:

  1. The order of the fleets in the queue
  2. The capacity of the fleets
  3. The viability of the fleets being placed onto (if the fleet is a spot type)

Both fleets here are SPOT, which means the have the potential to be “unviable”, meaning the instance type is too risky for gamelift to place sessions on it. During the times you are calling placement, the ap-southeast-1 fleet is using an instance type which is considered unviable at that time, and so the queue falls back to the us-west-1 fleet so it can safely place your session. There are a couple actions I would recommend here:

  1. In your queue always have one “fall back” ondemand type fleet so in the worst case event that all spot is unviable (very, very unlikely but possible) you can still place game sessions.
  2. If you want sessions in a specific queue to always land in one region (or be far more likely to land there) create multiple instance type fleets to lower your chances the instances you are using are going to be unviable. Several instance types can be made in fleets in the queue, and as long as 1 is viable you will place on spot instances in that region.

What about using latency data ?
Would that change the behaviour with cross-regional spot fleets ? and if so, how ?

The behavior is similar, might be easiest to think about Game Session Queue destinations as sorted and filtered separately.

If not using player latencies, the destinations are sorted by the order in the destination list (you specify the sorted order). Spot fleets used as destinations will be filtered out if the instance type in the region is deemed unviable by GameLift’s FleetIQ algorithm.

If using player latencies, the destinations are sorted per placement based on the player latencies in the placement. Spot fleets used as destinations will still be filtered out if the instance type in the region is deemed unviable. So we still sort first, filter second, decide on the placement third.

GameLift is looking at ways to make Spot utilization better, but currently the more options you have the better chance of finding a Spot viable combination. If your Queue has 3 destinations:

  • Spot Fleet with c5.large / ap-southeast-1
  • Spot Fleet with c5.large / us-east-1
  • OnDemand Fleet with c5.large / us-east-1

GameLift only has a chance to look at 2 Spot instance pools and if both aren’t viable, then 2/3 of the destinations will be filtered out and placements will be sent to the OnDemand Fleet. It’s a lot of resources to manage during early development, so we recommend ramping up your options as you get closer to load testing/launching your game:

  • Spot Fleet with c5.large / ap-southeast-1
  • Spot Fleet with c5.xlarge / ap-southeast-1
  • Spot Fleet with c5.large / ap-northeast-1
  • Spot Fleet with c5.xlarge / ap-northeast-1
  • Spot Fleet with c5.large / us-east-1
  • Spot Fleet with c5.xlarge / us-east-1
  • Spot Fleet with c5.large / us-west-2
  • Spot Fleet with c5.xlarge / us-west-2
  • OnDemand Fleet with c5.large / ap-southeast-1
  • OnDemand Fleet with c5.large / us-east-1

Now GameLift has 8 Spot pools as options, so if 4/8 are unviable, GameLift can still utilize the other 4. The key here is to span multiple instance types and multiple regions. Also remember to always have OnDemand capacity as a backup in case all Spot options in your Queue are considered unviable at any given time.

Hope that helps!

1 Like

Thanks. That was a very helpful answer !