Timeout trying to connect to RealTime server with certificate generation enabled

What else must be done to get WSS working? I’ve enabled certificate generation on the fleet, but I can’t seem to open a connection. Any insight into what I might be missing?

Here are the server logs as it starts up:

[INFO] (rt-logger.js) 59: Game session (PID: 13531) configured with logger: {"logDirPath":"/local/game/logs/13531","logLevelFilter":"*.info","filename":"server.log"}
[INFO] (index.js) 123: Game server was constructed with {"test_dist":false,"script":"../../../local/game/etag-005d5904bf82113cb0e612377793a8ab/src/server.js","port":1900,"end":2000,"udp_port":33400,"udp_port_end":33500,"devargs":"dev","enable_security":true}
[INFO] (gamelift.js) 195: Starting GameLift Realtime server process. PID: 13531...
[INFO] (gamelift.js) 200: Calling GameLiftServerAPI.SdkVersion...
[INFO] (gamelift.js) 205: GameLiftServerAPI.SdkVersion succeeded with result: 3.3.0
[INFO] (gamelift.js) 208: Calling GameLiftServerAPI.InitSDK...
[INFO] (gamelift.js) 213: GameLiftServerAPI.InitSDK succeeded
[INFO] (gamelift.js) 216: Waiting for Realtime server to start...
[INFO] (index.js) 149: Loading game server script at path: /local/game/etag-005d5904bf82113cb0e612377793a8ab/src/server.js
[INFO] (index.js) 177: Fetching credentials to establish secure connections...
[INFO] (gamelift.js) 170: Calling GameLiftServerAPI.GetInstanceCertificate
[INFO] (gamelift.js) 175: GameLiftServerAPI.DescribePlayerSessions succeeded with optional result: [object Object]
[INFO] (index.js) 185: Credential paths fetched: {"CertificatePath":"/local/gamemetadata/certificates/certificate.pem","PrivateKeyPath":"/local/gamemetadata/certificates/privateKey.pem","CertificateChainPath":"/local/gamemetadata/certificates/certificateChain.pem","HostName":"u2fp3zjxdovmooutoxufgldo641gkzfay986yrvaejfoyq5fsmv3w04sl24cfqd.whji3yx30fjwq55416k67pyobo9oi02c.us-west-2.amazongamelift.com","RootCertificatePath":"/local/gamemetadata/certificates/rootCertificate.pem"}
[INFO] (index.js) 202: Parsed credentials signed for hostname: u2fp3zjxdovmooutoxufgldo641gkzfay986yrvaejfoyq5fsmv3w04sl24cfqd.whji3yx30fjwq55416k67pyobo9oi02c.us-west-2.amazongamelift.com
[INFO] (index.js) 209: Initializing Realtime server event handlers...
[INFO] (dtls.js) 32: Creating DTLS server...
[INFO] (ws.js) 47: Creating WebSocket server over HTTPS...
[INFO] (ws.js) 64: Available cipher suite on host: aes128-gcm-sha256,aes128-sha,aes128-sha256,aes256-gcm-sha384,aes256-sha,aes256-sha256,dhe-psk-aes128-cbc-sha,dhe-psk-aes128-cbc-sha256,dhe-psk-aes128-gcm-sha256,dhe-psk-aes256-cbc-sha,dhe-psk-aes256-cbc-sha384,dhe-psk-aes256-gcm-sha384,dhe-psk-chacha20-poly1305,dhe-rsa-aes128-gcm-sha256,dhe-rsa-aes128-sha,dhe-rsa-aes128-sha256,dhe-rsa-aes256-gcm-sha384,dhe-rsa-aes256-sha,dhe-rsa-aes256-sha256,dhe-rsa-chacha20-poly1305,ecdhe-ecdsa-aes128-gcm-sha256,ecdhe-ecdsa-aes128-sha,ecdhe-ecdsa-aes128-sha256,ecdhe-ecdsa-aes256-gcm-sha384,ecdhe-ecdsa-aes256-sha,ecdhe-ecdsa-aes256-sha384,ecdhe-ecdsa-chacha20-poly1305,ecdhe-psk-aes128-cbc-sha,ecdhe-psk-aes128-cbc-sha256,ecdhe-psk-aes256-cbc-sha,ecdhe-psk-aes256-cbc-sha384,ecdhe-psk-chacha20-poly1305,ecdhe-rsa-aes128-gcm-sha256,ecdhe-rsa-aes128-sha,ecdhe-rsa-aes128-sha256,ecdhe-rsa-aes256-gcm-sha384,ecdhe-rsa-aes256-sha,ecdhe-rsa-aes256-sha384,ecdhe-rsa-chacha20-poly1305,psk-aes128-cbc-sha,psk-aes128-cbc-sha256,psk-aes128-gcm-sha256,psk-aes256-cbc-sha,psk-aes256-cbc-sha384,psk-aes256-gcm-sha384,psk-chacha20-poly1305,rsa-psk-aes128-cbc-sha,rsa-psk-aes128-cbc-sha256,rsa-psk-aes128-gcm-sha256,rsa-psk-aes256-cbc-sha,rsa-psk-aes256-cbc-sha384,rsa-psk-aes256-gcm-sha384,rsa-psk-chacha20-poly1305,srp-aes-128-cbc-sha,srp-aes-256-cbc-sha,srp-rsa-aes-128-cbc-sha,srp-rsa-aes-256-cbc-sha
[INFO] (ws.js) 89: Using secure options for websocket: {"ciphers":"TLS_AES_256_GCM_SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384","minVersion":"TLSv1.2","dhparam":"2048"}
[INFO] (index.js) 223: Initializing UDP connector...
[INFO] (index.js) 129: Attempting to open port 33400...
[INFO] (dtls.js) 56: DTLS server listening on 0.0.0.0:33400
[INFO] (index.js) 229: Initializing TCP connector...
[INFO] (index.js) 129: Attempting to open port 1900...
[INFO] (ws.js) 163: HTTP/S server listening on port: 1900
[INFO] (server.js) 605: Ready to host games...
[INFO] (index.js) 241: onProcessStarted success. Process ready for games.
[INFO] (index.js) 261: Game session initialized with port: 1900
[INFO] (gamelift.js) 223: Realtime server started! Calling GameLiftServerAPI.ProcessReady with processParameters: {"Port":1900,"LogParameters":{"LogPaths":["/local/game/logs/13531"]}}
[INFO] (gamelift.js) 229: Process advertised to AuxProxy! GameLiftServerAPI.ProcessReady succeeded
[INFO] (gamelift.js) 231: GameLift Realtime server process started successfully.

Edit:

The GameLift service is also continuously complaining about this fleet:

SERVER_PROCESS_SDK_INITIALIZATION_TIMEOUT
Server process started correctly but did not call InitSDK() within 5 minutes, launchPath(/local/NodeJS/bin/node), arguments(-- /local/game/src/gamelift.js --enable-security --script ../../../local/game/etag-005d5904bf82113cb0e612377793a8ab/src/server.js --devargs "dev"), instanceId(i-09f6e5e76f757c572)

Edit:

Connecting to an instance without certificate generation enabled:

$ telnet 35.166.244.249 1901
Trying 35.166.244.249...
Connected to ec2-35-166-244-249.us-west-2.compute.amazonaws.com.
Escape character is '^]'.

Trying to connect to an instance with certificate generation enabled:

$ telnet 34.223.223.14 1900
Trying 34.223.223.14...

I can, however, connect to the secure fleet from another fleet:

[gl-user-remote@ip-10-172-206-82 ~]$ telnet 34.223.223.14 1900
Trying 34.223.223.14...
Connected to 34.223.223.14.
Escape character is '^]'.

Do I need to configure some security groups or something with the secure fleet?

I am sorry you are having problems with this.

I’m going to let the GameLift team know, to see if they have any further insight.

I would though:

  • Make sure your Client side SDK is up-to-date to support secure fleets as I believe there is some handshake code required

The bigger worry is this warning:

Server process started correctly but did not call InitSDK() within 5 minutes, launchPath(/local/NodeJS/bin/node), arguments(-- /local/game/src/gamelift.js --enable-security --script ../../../local/game/etag-005d5904bf82113cb0e612377793a8ab/src/server.js --devargs "dev"), instanceId(i-09f6e5e76f757c572)

This means that GameLift will probably terminate your instances, this looks like you have a bug in your server script causing it fail. This may be the reason you can’t connect connect to your secure fleet.

I would:

  • Double check the documentation
  • Lint/validate your scripts
  • Simplify your script until you have a working fleet and then add in your logic in small sections.

Any new information on this? We have a functional fleet without certificate generation enabled.

I tried launching a secure fleet with a minimal, functional script. I’ve even attempted it with this example script from the Realtime docs – just to see:

/*
* All or portions of this file Copyright (c) Amazon.com, Inc. or its affiliates or
* its licensors.
*
* All use of this software is governed by the terms and conditions governing AWS
* Content in the AWS Customer Agreement at aws.amazon.com/agreement. Do not
* remove or modify any license notices. This file is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
*
*/

// Example minimal server file with no overridden callbacks or configuration

var gameSession;

// Called when game server is initialized, is passed server object of current session
function init(session) {
    gameSession = session;
}

exports.ssExports = {
    init: init
};

Unfortunately, the server does not respond and any connection attempt times out.

As I said previously, I am able to establish a connection from one fleet instance to another fleet instance. So, something is working in that regard. That’s why I asked about security groups.

We may need to set up a proxy server as a workaround for this issue, but I was hoping all the GameLift certificate generation / SSL bits would just work.

Looking forward to any insight on this issue, thanks!

Apologies, I forgot to follow up with you.

IMHO The minimal script isn’t that useful, as it will get you an active fleet but does very little. You ideally want something that logs around major lifecycle events including player connections.

The two obvious issues are that:

  • Server is listening on a port that you didn’t open in the gamelift fleet, but that seems unlikely because you’ve gotten non tls version working.
  • Your use of telnet to test connections is interesting (I had assumed that Telnet didn’t support TLS but as it sort of works for you it could be a root ca issue when the TLS handshake is timing out and failing.
    • BTW Have you tried other tools such as telnet-ssh and ```openssl s_client -connect :<993> to connect?
    • EC2 instances may have different telnet client than yours/version so that could be why they work. They also have all the right rootCAs installed from the beginning for ACM TLS certs.

If you’re still having problems could you provide fleet ids + region of the realtime fleets you couldn’t connect to? Can ask the GameLift service team to investigate this particular failure.

Hi Pip,

Region: us-west-2
FleetId: fleet-78d9529b-c2c8-4283-bd0c-711b2dd81e08

Yes, I’m just using telnet to test connectivity to the host. We’re unable to establish a connection, so we haven’t yet reached the point where the client can attempt a TLS handshake.

It just seems that external connections to the secure GameLift instances are not going through. For example, from a machine outside of EC2:

$ openssl s_client -host 52.38.139.32 -port 1900
connect: Operation timed out
connect:errno=60

From an ec2 instance, so I guess any machine inside the Amazon VPC/Cloud. This particular instance was in us-east-1, but I tried it from us-west-2 etc. still worked:

$ openssl s_client -host 52.38.139.32 -port 1900
CONNECTED(00000003)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = *.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com
verify return:1
---
Certificate chain
 0 s:CN = *.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
   i:C = US, O = Amazon, CN = Amazon Root CA 1
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFyDCCBLCgAwIBAgIQBaI9RCbX4Sn2O4Rzc1ypdzANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMDAyMjQwMDAwMDBaFw0yMTAzMjQx
MjAwMDBaMEoxSDBGBgNVBAMMPyouYmJ1NnB0amtiOGhiYnhjMzF0cW5xeTFpZ2xs
c2V5b3MudXMtd2VzdC0yLmFtYXpvbmdhbWVsaWZ0LmNvbTCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAK0G149xjEZWp4vgGWIyxIDOSg2Vbi3ctPzbBuGi
U4u28nAowTU0gpJ70NdMoaHkAA+GA1dm13vyk9pN+9Fjk2i+oQpo/ytergE3+ijA
KWwVzbFC8PgYlFnRENJykJyxAqkADtYgQwm6aPIA5TnaNQT9H8HvVrFGdoNccBAQ
EuZm6bxFbQ3BRxeg78LTc5wkYQW+A9yXiaR2cmudqdMhf8sFvCGwxvHsu14u+pJA
zkqtyJyXzq42fTt7qsNjTOqUZ44G/9qswGwby3IVI+fTuJquDyD9tB5GU8sYTLZA
XQZI2trqi6A37NDQ71Lu0xbycqcOHD3N91mA0AhNIPa20OMCAwEAAaOCAqwwggKo
MB8GA1UdIwQYMBaAFFmkZgZSoHuVkjyjlAcnlnRb+T3QMB0GA1UdDgQWBBRYxv+s
cZ5uG72Ig6FMA+DFfX2AiTBKBgNVHREEQzBBgj8qLmJidTZwdGprYjhoYmJ4YzMx
dHFucXkxaWdsbHNleW9zLnVzLXdlc3QtMi5hbWF6b25nYW1lbGlmdC5jb20wDgYD
VR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjA7BgNV
HR8ENDAyMDCgLqAshipodHRwOi8vY3JsLnNjYTFiLmFtYXpvbnRydXN0LmNvbS9z
Y2ExYi5jcmwwIAYDVR0gBBkwFzALBglghkgBhv1sAQIwCAYGZ4EMAQIBMHUGCCsG
AQUFBwEBBGkwZzAtBggrBgEFBQcwAYYhaHR0cDovL29jc3Auc2NhMWIuYW1hem9u
dHJ1c3QuY29tMDYGCCsGAQUFBzAChipodHRwOi8vY3J0LnNjYTFiLmFtYXpvbnRy
dXN0LmNvbS9zY2ExYi5jcnQwDAYDVR0TAQH/BAIwADCCAQUGCisGAQQB1nkCBAIE
gfYEgfMA8QB3ALvZ37wfinG1k5Qjl6qSe0c4V5UKq1LoGpCWZDaOHtGFAAABcHiV
aeEAAAQDAEgwRgIhAMLRvCwqjKF2e+DFT2Xd5czSdj4QLbidpFFbxC0VvwN0AiEA
mkahndLU0vNqhcNCHB7WUpBZKvJN2AgtQiXcYB8FB6sAdgCHdb/nWXz4jEOZX73z
bv9WjUdWNv9KtWDBtOr/XqCDDwAAAXB4lWoxAAAEAwBHMEUCIFUMh3fm96aw/IEp
HXXHpLben5kLtDJ2/3RNwffLJCiTAiEAiaealPByhEeaat476TN2QbJG4mzpQr44
QFEfqWXiLoswDQYJKoZIhvcNAQELBQADggEBABmCAQjHv62ary+PI/ZsefN+yxeD
CguWz7CBVaO0fBmJ9CQp8n68of+Wb9qkxH5eu47Wc8aTpnkINVTRdp1GGgd1/rzs
EljbAff/38IwR0mKoFdDqZzeVgaOlW8hgYjJcfnwuPdkUVmJg+eQsgJjADvWXghv
S/ZAiNHlsD8lVxoPK6LMSN2xtidUNU2BU69zRxm09JqyjwF4n9yIqRyljN2UrxoA
OZHKNJn59gkwKEOZ8mFkiYKrnVAjNF5wShrNFPdzx40EmOVO8IT7Dxt3IPbS2L3c
HwjFSdBlC7eIKqG1kT7s7bHvP3EosHUTXOHsYtC900ZcI79+VKzHyd5Qc3U=
-----END CERTIFICATE-----
subject=CN = *.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com

issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5586 bytes and written 407 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: F5C3E9533F34156C4AE0F60C82E7818FD58E40473B07EFE9708749B6EA3C97B6
    Session-ID-ctx:
    Master-Key: F3937EA600085B5AD015BCB16EECA90A3F4637B71CA90177E3B315FB3F241A3E0B5341B17185B608D4A805F829BA0277
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - a0 74 c4 cd 4e 5d d5 27-0f 28 02 5a c7 04 16 49   .t..N].'.(.Z...I
    0010 - 7b 4c 74 0c c3 53 e0 2d-0f 86 67 6c 04 be 72 e7   {Lt..S.-..gl..r.
    0020 - 27 a4 28 d8 05 5b f6 52-d9 8d 11 2a d7 c0 70 6b   '.(..[.R...*..pk
    0030 - d5 4e a3 94 a8 02 06 f1-dc 1e 72 54 7c 44 66 89   .N........rT|Df.
    0040 - 04 2a c7 79 cc 44 11 df-b7 e5 7f 01 a9 33 7c 43   .*.y.D.......3|C
    0050 - 9f 94 55 83 72 40 f3 6b-24 7d 68 14 3f bf fd b7   ..U.r@.k$}h.?...
    0060 - a3 f2 44 6a 97 95 0b 81-07 79 92 65 9a 42 b2 0d   ..Dj.....y.e.B..
    0070 - 31 ed db 44 51 07 94 26-a5 d1 2b 65 92 46 3a 4e   1..DQ..&..+e.F:N
    0080 - 90 da 00 b1 70 65 57 36-f1 08 91 9f 3b f6 ab e3   ....peW6....;...
    0090 - 4c 1a ef f0 39 55 49 8a-0b 6e 20 64 47 b3 c5 ee   L...9UI..n dG...
    00a0 - 04 9e f6 e7 69 56 6d b8-19 05 e9 25 4a f6 51 a4   ....iVm....%J.Q.
    00b0 - 58 c1 d9 c0 e2 9e 8a 15-31 0b 41 fe b0 53 66 2d   X.......1.A..Sf-
    00c0 - 60 68 28 a4 64 a3 a7 47-9d e5 b6 ea 0f 64 be e2   `h(.d..G.....d..

    Start Time: 1582574840
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

@Kip are you using Realtime with unity? We had a similar issue, it seems that for unity realtime sdk to work with TLS you need to build another SDK and include it in your project, unfortunately you can only find this information inside the GameliftRealtime SDK README, you need to build GameLiftRealtimeNative SDK and and add it into your project, then it should work.

Here’s the excerpt for building Native TLS library from the README:

Building the Native TLS library.

In order to leverage the TLS feature of this SDK, you will need to build the GameLiftRealtimeNative >library.
From the root directory:
On Windows:

cd Native
cmake ..
msbuild ALL_BUILD.vcxproj /p:Configuration=Release

On MacOS:

cd Native
cmake ..
make

Hi Kip,

I was able to get telnet and openssl to work with the DNS Name instead.

$ host eym3d569ljig2cuuzjs6zwe33fmo97zohaou3h7r7ztxu25wx6kok6opo6celn0.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com
eym3d569ljig2cuuzjs6zwe33fmo97zohaou3h7r7ztxu25wx6kok6opo6celn0.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com has address 52.38.139.32
$ telnet eym3d569ljig2cuuzjs6zwe33fmo97zohaou3h7r7ztxu25wx6kok6opo6celn0.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com 1900
Trying 52.38.139.32...
Connected to eym3d569ljig2cuuzjs6zwe33fmo97zohaou3h7r7ztxu25wx6kok6opo6celn0.bbu6ptjkb8hbbxc31tqnqy1igllseyos.us-west-2.amazongamelift.com.
Escape character is '^]'.
^CConnection closed by foreign host.

The DNS Name of the secured server can be retrieved via DescribeInstances: https://docs.aws.amazon.com/cli/latest/reference/gamelift/describe-instances.html, or CreateGameSession: https://docs.aws.amazon.com/cli/latest/reference/gamelift/create-game-session.html

Could you try and see if this works for you?

I’ve also tried to connect a realtime client with your server as the endpoint, and I was able to establish a TLS connection. You should be able to verify this in your game log – the connection was made at around 02/05 12:30pm PST.

Hi @James-AWS,

Thanks for taking a look at this! Since you managed to establish a connection, I decided to try connecting from a different network.

I tethered my machine to my mobile phone (so it’s the same machine, just a different network) and success:

$ telnet 52.38.139.32 1900
Trying 52.38.139.32...
Connected to ec2-52-38-139-32.us-west-2.compute.amazonaws.com.
Escape character is '^]'.

So, it appears the issue is on our end / our ISP’s end. I’ve gotten in contact with our ISP and they’re taking a look at it.

With that said, I find it strange that I have 0 issues connecting to the regular GameLift instances from this same network – but enabling certificate generation causes problems… So I’m not sure where / what in between AWS and our ISP the problem is. If you have any information that might be useful to pass on to the ISP, that would great.

@Pip @James-AWS

We’re still having a problem with the secure servers, unfortunately. In my first post I mentioned these errors:

SERVER_PROCESS_SDK_INITIALIZATION_TIMEOUT
Server process started correctly but did not call InitSDK() within 5 minutes, launchPath(/local/NodeJS/bin/node), arguments(-- /local/game/src/gamelift.js --enable-security --script ../../../local/game/etag-dafaf03254888cfe5ca81cd24932e321/src/server.js --devargs "dev"), instanceId(i-0bc14850c546e8889)

This error pops up if I set ConcurrentExecutions > 1. We’re currently trying to run the fleet with ConcurrentExecutions=16.

Taking a look at some of the logs in /local/game/logs, almost all of the logs look like this:

27 Feb 2020 19:42:07,272 [INFO] (rt-logger.js) 59: Game session (PID: 15334) configured with logger: {"logDirPath":"/local/game/logs/15334","logLevelFilter":"*.info","filename":"server.log"}
27 Feb 2020 19:42:09,796 [INFO] (index.js) 123: Game server was constructed with {"test_dist":false,"script":"../../../local/game/etag-dafaf03254888cfe5ca81cd24932e321/src/server.js","port":1900,"end":2000,"udp_port":33400,"udp_port_end":33500,"devargs":"dev","enable_security":true}
27 Feb 2020 19:42:09,797 [INFO] (gamelift.js) 195: Starting GameLift Realtime server process. PID: 15334...
27 Feb 2020 19:42:09,797 [INFO] (gamelift.js) 200: Calling GameLiftServerAPI.SdkVersion...
27 Feb 2020 19:42:09,797 [INFO] (gamelift.js) 205: GameLiftServerAPI.SdkVersion succeeded with result: 3.3.0
27 Feb 2020 19:42:09,798 [INFO] (gamelift.js) 208: Calling GameLiftServerAPI.InitSDK...
27 Feb 2020 19:42:10,67 [INFO] (gamelift.js) 213: GameLiftServerAPI.InitSDK succeeded
27 Feb 2020 19:42:10,67 [INFO] (gamelift.js) 216: Waiting for Realtime server to start...
27 Feb 2020 19:42:10,100 [INFO] (index.js) 149: Loading game server script at path: /local/game/etag-dafaf03254888cfe5ca81cd24932e321/src/server.js
27 Feb 2020 19:42:10,403 [INFO] (index.js) 177: Fetching credentials to establish secure connections...
27 Feb 2020 19:42:10,421 [INFO] (gamelift.js) 170: Calling GameLiftServerAPI.GetInstanceCertificate
27 Feb 2020 19:42:10,483 [INFO] (gamelift.js) 175: GameLiftServerAPI.DescribePlayerSessions succeeded with optional result: [object Object]
27 Feb 2020 19:42:10,483 [INFO] (index.js) 185: Credential paths fetched: {"CertificatePath":"/local/gamemetadata/certificates/certificate.pem","PrivateKeyPath":"/local/gamemetadata/certificates/privateKey.pem","CertificateChainPath":"/local/gamemetadata/certificates/certificateChain.pem","HostName":"7zdfw0j28fci4hn54kb2f73i20szskgfvxmx38zh2gxaw0zfayia7yzocay7hzc.d88fv375nd76vpbobccapyfxo4wn4bek.us-west-2.amazongamelift.com","RootCertificatePath":"/local/gamemetadata/certificates/rootCertificate.pem"}
27 Feb 2020 19:42:10,483 [INFO] (index.js) 202: Parsed credentials signed for hostname: 7zdfw0j28fci4hn54kb2f73i20szskgfvxmx38zh2gxaw0zfayia7yzocay7hzc.d88fv375nd76vpbobccapyfxo4wn4bek.us-west-2.amazongamelift.com
27 Feb 2020 19:42:10,484 [INFO] (index.js) 209: Initializing Realtime server event handlers...
27 Feb 2020 19:42:10,484 [INFO] (dtls.js) 32: Creating DTLS server...
27 Feb 2020 19:42:10,532 [INFO] (ws.js) 47: Creating WebSocket server over HTTPS...
27 Feb 2020 19:42:10,534 [INFO] (ws.js) 64: Available cipher suite on host: aes128-gcm-sha256,aes128-sha,aes128-sha256,aes256-gcm-sha384,aes256-sha,aes256-sha256,dhe-psk-aes128-cbc-sha,dhe-psk-aes128-cbc-sha256,dhe-psk-aes128-gcm-sha256,dhe-psk-aes256-cbc-sha,dhe-psk-aes256-cbc-sha384,dhe-psk-aes256-gcm-sha384,dhe-psk-chacha20-poly1305,dhe-rsa-aes128-gcm-sha256,dhe-rsa-aes128-sha,dhe-rsa-aes128-sha256,dhe-rsa-aes256-gcm-sha384,dhe-rsa-aes256-sha,dhe-rsa-aes256-sha256,dhe-rsa-chacha20-poly1305,ecdhe-ecdsa-aes128-gcm-sha256,ecdhe-ecdsa-aes128-sha,ecdhe-ecdsa-aes128-sha256,ecdhe-ecdsa-aes256-gcm-sha384,ecdhe-ecdsa-aes256-sha,ecdhe-ecdsa-aes256-sha384,ecdhe-ecdsa-chacha20-poly1305,ecdhe-psk-aes128-cbc-sha,ecdhe-psk-aes128-cbc-sha256,ecdhe-psk-aes256-cbc-sha,ecdhe-psk-aes256-cbc-sha384,ecdhe-psk-chacha20-poly1305,ecdhe-rsa-aes128-gcm-sha256,ecdhe-rsa-aes128-sha,ecdhe-rsa-aes128-sha256,ecdhe-rsa-aes256-gcm-sha384,ecdhe-rsa-aes256-sha,ecdhe-rsa-aes256-sha384,ecdhe-rsa-chacha20-poly1305,psk-aes128-cbc-sha,psk-aes128-cbc-sha256,psk-aes128-gcm-sha256,psk-aes256-cbc-sha,psk-aes256-cbc-sha384,psk-aes256-gcm-sha384,psk-chacha20-poly1305,rsa-psk-aes128-cbc-sha,rsa-psk-aes128-cbc-sha256,rsa-psk-aes128-gcm-sha256,rsa-psk-aes256-cbc-sha,rsa-psk-aes256-cbc-sha384,rsa-psk-aes256-gcm-sha384,rsa-psk-chacha20-poly1305,srp-aes-128-cbc-sha,srp-aes-256-cbc-sha,srp-rsa-aes-128-cbc-sha,srp-rsa-aes-256-cbc-sha
27 Feb 2020 19:42:10,534 [INFO] (ws.js) 89: Using secure options for websocket: {"ciphers":"TLS_AES_256_GCM_SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384","minVersion":"TLSv1.2","dhparam":"2048"}
27 Feb 2020 19:42:10,607 [INFO] (index.js) 223: Initializing UDP connector...
27 Feb 2020 19:42:10,640 [INFO] (index.js) 129: Attempting to open port 33400...

A few things to note:

  • GameLiftServerAPI.InitSDK succeeded – so why do the logs in the console say Server process started correctly but did not call InitSDK() within 5 minutes?

  • The logs end at Attempting to open port 33400...


On a regular fleet, there’s 16 node processes on 19xx ports:

$ netstat -plnt | grep node
tcp6       0      0 :::1900                 :::*                    LISTEN      20440/node
tcp6       0      0 :::1901                 :::*                    LISTEN      20631/node
tcp6       0      0 :::1902                 :::*                    LISTEN      20208/node
tcp6       0      0 :::1903                 :::*                    LISTEN      20536/node
tcp6       0      0 :::1904                 :::*                    LISTEN      20510/node
tcp6       0      0 :::1905                 :::*                    LISTEN      20701/node
tcp6       0      0 :::1906                 :::*                    LISTEN      20486/node
tcp6       0      0 :::1907                 :::*                    LISTEN      20463/node
tcp6       0      0 :::1908                 :::*                    LISTEN      20559/node
tcp6       0      0 :::1909                 :::*                    LISTEN      20674/node
tcp6       0      0 :::1910                 :::*                    LISTEN      20608/node
tcp6       0      0 :::1911                 :::*                    LISTEN      20769/node
tcp6       0      0 :::1912                 :::*                    LISTEN      20659/node
tcp6       0      0 :::1913                 :::*                    LISTEN      20585/node
tcp6       0      0 :::1914                 :::*                    LISTEN      20746/node
tcp6       0      0 :::1915                 :::*                    LISTEN      20723/node

On a secure fleet, there’s 1 node process on port 1900:

$ netstat -plnt | grep node
tcp6       0      0 :::1900                 :::*                    LISTEN      15388/node

On the secure fleet, there are a ton of writes happening in /local/whitewater/Logs/auxproxy.log. We’re not seeing the following message on a regular fleet, but it’s happening frequently on the secure fleet:

27 Feb 2020 20:01:02,448 [WARN]  (nioEventLoopGroup-3-8) io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377)
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1147)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

These are the fleets we’re working with:

Secure fleet

Region: us-west-2
FleetId: fleet-7618ec04-03c1-4538-9824-0c2ff97f200f

Regular fleet

Region: us-west-2
FleetId: fleet-1429ba8b-4a7e-466d-b579-062c97b6412a

I have the same issue in that I cannot have a certificate realtime server with more than 1 concurrent process. When the current process is > 1, then I get the Server process started correctly but did not call InitSDK() within 5 minutes error message.