Protobuf is throwing an AssertionError from the RealTime server app

I have a seemingly random problem with RealTime server processes crashing due to an exception being thrown by Protobuff from the base Node server application. It does not seem to be impacted by how “busy” the server is. I have seen this happen with no PlayerSessions connected, and I have seen it happen when our game is busy executing code based off of messages sent in from clients.

This does not happen often, but it happens enough to be a problem.

Below is an example of it happening when our game was quiet. We were not currently dealing with any optional messages sent in by clients. That is so say if the base C# SDK does send messages in the background we may have been processing those. The logs seem to show us only getting internal messages fro peer -1. These errors look more or less like this every time.

16 Sep 2021 19:59:26,396 [INFO] (index.js) 332: Received custom game message from sender: -1 with opCode: 1, targetPeerId: 0, targetGroupId: 1
16 Sep 2021 19:59:27,516 [FATAL] (gamelift.js) 190: Realtime server crashed due to: process.uncaughtException -- See stacktrace for more details:
AssertionError: Assertion failed
    at new goog.asserts.AssertionError (/local/game/node_modules/google-protobuf/google-protobuf.js:98:603)
    at Object.goog.asserts.doAssertFailure_ (/local/game/node_modules/google-protobuf/google-protobuf.js:99:126)
    at Object.goog.asserts.assert (/local/game/node_modules/google-protobuf/google-protobuf.js:99:385)
    at jspb.BinaryDecoder.readUnsignedVarint32 (/local/game/node_modules/google-protobuf/google-protobuf.js:322:304)
    at jspb.BinaryReader.readInt32 (/local/game/node_modules/google-protobuf/google-protobuf.js:349:457)
    at Function.proto.com.gamelift.rt.proto.Packet.deserializeBinaryFromReader (/local/game/src/proto/Packet_pb.js:171:49)
    at Function.proto.com.gamelift.rt.proto.Packet.deserializeBinary (/local/game/src/proto/Packet_pb.js:140:45)
    at exports.Server.wireMessageToPacket (/local/game/src/server/index.js:228:51)
    at Socket.<anonymous> (/local/game/src/server/connectors/udp.js:37:33)
    at Socket.emit (events.js:189:13)
1 Like

Hi @ElNahal ,

We are looking into this now, thanks for reaching out.

James
P52656679

1 Like

Hi @ElNahal

We have made an update to make RealTime server more resilient to incorrectly formatted UDP packets. In addition, we have also added logging to record the sender address and content of incorrectly formatted packets, so you can hopefully find out what/where these bad packets are coming from.

Here is the format of the error log:

Caught exception when receiving UDP message <packet> from <ip_address>:<port>.

Please create a new fleet with your existing build in order to apply this change.

Thanks,
James

Thank you. We will make a new fleet tonight and wait for a new error.