Queue not placing on certain fleets

Hi there,

We’ve started testing our online with fleets in more regions around the world (spot instances for now, to lower costs for dev testing).

We’re using FlexMatch and passing latency info up for each ticket. The matchmaking configuration then resolves to a Queue with a single fleet from ~4 regions covering Australia/US/Europe. The idea is that the players will then be placed on the region for which they have the “best” collective latency.

However we just ran into a situation where the Queue seems to have refused to acknowledge that a Sydney fleet was set up, and instead placed us on Singapore (2 players, both with < 30ms to Sydney and ~100ms to Singapore).

Confused, I then set a latency policy on the Queue of < 80ms, hoping to avoid being matched on Singapore, however it just resulted in the Tickets remaining in PLACING indefinitely.

I then removed all other fleets from the Queue, leaving only the Sydney fleet, and we tried again, and got stuck on PLACING again.

Again, we’re using Spot instances for all fleets if that could affect anything (nothing in the Events history that would indicate the fleet was torn down, although could it have been marked for tearing down, and Queues don’t place on fleets marked like that?).

Some ARNs if that helps:

Sydney fleet: arn:aws:gamelift:ap-southeast-2:129023025233:fleet/fleet-313d90f4-0cbc-49a1-af88-12423d00a4dd
Queue: arn:aws:gamelift:us-east-1:129023025233:gamesessionqueue/QA-0-0-0-0
Matchmaking Config: us-east-1/QA-0-0-0-0-Deathmatch-Casual

And matchmaking ticket info from our server logs:

T06:36:53: Starting matchmaking for ticket: {
T06:36:53: "ConfigurationName": "QA-0-0-0-0-Deathmatch-Casual",
T06:36:53: "Players": [
T06:36:53: {
T06:36:53: "LatencyInMs": {
T06:36:53: "us-east-1": 242,
T06:36:53: "eu-west-1": 332,
T06:36:53: "ap-southeast-2": 21,
T06:36:53: "ap-southeast-1": 95
T06:36:53: },
T06:36:53: "PlayerAttributes": {
T06:36:53: "service": {
T06:36:53: "SL": [
T06:36:53: "STEAM"
T06:36:53: ]
T06:36:53: },
T06:36:53: "compatibleServices": {
T06:36:53: "SL": [
T06:36:53: "STEAM",
T06:36:53: "LIVE",
T06:36:53: "PS4",
T06:36:53: "NULL"
T06:36:53: ]
T06:36:53: },
T06:36:53: "blockList": {
T06:36:53: "SL": [
T06:36:53: "DummyHyperId"
T06:36:53: ]
T06:36:53: }
T06:36:53: },
T06:36:53: "PlayerId": "76561198034212758"
T06:36:53: }
T06:36:53: ]
T06:36:53: }
T06:36:54: Starting matchmaking for ticket: {
T06:36:54: "ConfigurationName": "QA-0-0-0-0-Deathmatch-Casual",
T06:36:54: "Players": [
T06:36:54: {
T06:36:54: "LatencyInMs": {
T06:36:54: "us-east-1": 245,
T06:36:54: "eu-west-1": 334,
T06:36:54: "ap-southeast-2": 27,
T06:36:54: "ap-southeast-1": 96
T06:36:54: },
T06:36:54: "PlayerAttributes": {
T06:36:54: "service": {
T06:36:54: "SL": [
T06:36:54: "STEAM"
T06:36:54: ]
T06:36:54: },
T06:36:54: "compatibleServices": {
T06:36:54: "SL": [
T06:36:54: "STEAM",
T06:36:54: "LIVE",
T06:36:54: "PS4",
T06:36:54: "NULL"
T06:36:54: ]
T06:36:54: },
T06:36:54: "blockList": {
T06:36:54: "SL": [
T06:36:54: "DummyHyperId"
T06:36:54: ]
T06:36:54: }
T06:36:54: },
T06:36:54: "PlayerId": "76561197996651007"
T06:36:54: }
T06:36:54: ]
T06:36:54: }
T06:52:17: Saving tickets: [ { TicketId: '4a941c91-da3a-4bf5-a520-dcf8468bdebf',
T06:52:17: ConfigurationName: 'QA-0-0-0-0-Deathmatch-Casual',
T06:52:17: Status: 'PLACING',
T06:52:17: StartTime: 2018-04-25T06:51:45.486Z,
T06:52:17: Players: [ [Object] ] },
T06:52:17: { TicketId: '07d4d0cc-aaa4-47e1-9436-01db7eef8c40',
T06:52:17: ConfigurationName: 'QA-0-0-0-0-Deathmatch-Casual',
T06:52:17: Status: 'PLACING',
T06:52:17: StartTime: 2018-04-25T06:51:46.072Z,
T06:52:17: Players: [ [Object] ] } ]

Cheers,
Geordie

Hey @geordie!

If you are only using Spot fleets, and FleetIQ has deemed that an instance is not viable, it will not place your game session there. We recommend that you use an on-demand instance in each region to ensure that in the event Spot fleets in your lowest latest region are not viable, you can still place into that region to maintain any latency requirements you may have.

We are currently working on a metrics update that will allow you to see the reason for why a Spot fleet was not chosen and plan to release that at some point in the future to help self-diagnose situations like this.

Could you add an on-demand fleet in your lowest latency region for testing purposes with the necessary capacity and let me know if that works?

Bruce

Hi Bruce,

Thanks for the info - if the problem is indeed that FleetIQ deems it not viable, would we expect that the fleet would become viable again automatically in the future? (eg. once the spot capacity is regained or market price changes etc.) Or is that fleet just totally useless now, and we’d need to recreate it (with no way of knowing).

I checked again this morning and it still didn’t place us on that fleet (haven’t spun up an on-demand fleet yet cause was testing the above, but will do so now).

Cheers,
Geordie

Yes. We continuously evaluate the viability of a Spot instance such that the moment it becomes viable, it will start placing game sessions.

Hey @geordie

I wanted to say, too, that you also can see timeouts if your fleet does not have available capacity.

I noticed that there were active game sessions in this fleet (313d90f4-0cbc-49a1-af88-12423d00a4dd), and I could see your CloudWatch metrics for IdleHosts drop to 0 for an extended period of time. As such, the Queue is going to get an out-of-capacity exception when it attempts to place the match into that region and then fall back to the next lowest latency region. If there is no capacity available in any region where it has fleets, the Queue will eventually time out.

Hope that helps!

Hi @Matchmaker,

Just wanted to let you know that this “queue not able to place on a fleet” issue seems to have come up again, although this time it’s an On Demand fleet. Which makes me think maybe it wasn’t something related to FleetIQ at all?

Fleet: arn:aws:gamelift:ap-southeast-2::fleet/fleet-3b702cb9-84c4-449d-a787-07b31aaf8ce8

Queue: us-east-1/game-session-queues/DEV-1-0-0-0

Would be great to get to the bottom of this, since if it’s impacting our dev fleets then worried it could happen in production as well.

Cheers,
Geordie

Hey @geordie

Were you able to confirm that you had available capacity in your fleet when you attempted to place the game session? In my previous response, I indicated that it looked like the reason you were not placing was because you did not have available capacity.

Hi @Matchmaker,

Sorry, think I must’ve missed your May 1st reply (don’t seem to get emails even though I’ve followed the question?). I didn’t know about the IdleHosts metric, but will check it out!

For the latest issue I posted about, I did try to check the metrics of the Queue, and didn’t notice any timeouts showing up. Although maybe that was just because I checked too early, before they’d actually had time to time out.

I definitely remember checking the fleet though in case we had game sessions that weren’t terminated properly or something. Maybe you could check the logs to make sure, but from memory that fleet had 11 “terminated” game sessions but none that were active, so I couldn’t think why the Queue wouldn’t want to place on it.

Cheers,
Geordie

Hey @Matchmaker,

Okay great, glad to know it was an issue on our end. Thanks so much for looking into it. (I think it was mainly the fact that there were no active game sessions on the fleet, yet it still wouldn’t place that confused me. But maybe server processes weren’t returning that they were healthy or something)

I haven’t used CloudWatch much, but seems very powerful. Have been wondering for a while if there was a way to see an overview of fleets across all regions, since we’ve been building out something similar as part of our deployment pipeline. Being able to assign fleets to metric groups is handy, although probably doesn’t allow for enough fine-grained info about which regions have fleets and the status of those fleets etc. for our purposes.

I couldn’t find a metric named IdleHosts, though assume IdleInstances is the same thing? I’ll be sure to check on that next time a queue won’t place on a fleet.

Thanks again,
Geordie

Hey @geordie

Sorry that previous post was unclear – we did confirm in the logs that you did not have available capacity in your fleet to place a game session during that time period where the backup region was selected.