I have a seemingly random problem with RealTime server processes crashing due to an exception being thrown by Protobuff from the base Node server application. It does not seem to be impacted by how “busy” the server is. I have seen this happen with no PlayerSessions connected, and I have seen it happen when our game is busy executing code based off of messages sent in from clients.
This does not happen often, but it happens enough to be a problem.
Below is an example of it happening when our game was quiet. We were not currently dealing with any optional messages sent in by clients. That is so say if the base C# SDK does send messages in the background we may have been processing those. The logs seem to show us only getting internal messages fro peer -1. These errors look more or less like this every time.
16 Sep 2021 19:59:26,396 [INFO] (index.js) 332: Received custom game message from sender: -1 with opCode: 1, targetPeerId: 0, targetGroupId: 1 16 Sep 2021 19:59:27,516 [FATAL] (gamelift.js) 190: Realtime server crashed due to: process.uncaughtException -- See stacktrace for more details: AssertionError: Assertion failed at new goog.asserts.AssertionError (/local/game/node_modules/google-protobuf/google-protobuf.js:98:603) at Object.goog.asserts.doAssertFailure_ (/local/game/node_modules/google-protobuf/google-protobuf.js:99:126) at Object.goog.asserts.assert (/local/game/node_modules/google-protobuf/google-protobuf.js:99:385) at jspb.BinaryDecoder.readUnsignedVarint32 (/local/game/node_modules/google-protobuf/google-protobuf.js:322:304) at jspb.BinaryReader.readInt32 (/local/game/node_modules/google-protobuf/google-protobuf.js:349:457) at Function.proto.com.gamelift.rt.proto.Packet.deserializeBinaryFromReader (/local/game/src/proto/Packet_pb.js:171:49) at Function.proto.com.gamelift.rt.proto.Packet.deserializeBinary (/local/game/src/proto/Packet_pb.js:140:45) at exports.Server.wireMessageToPacket (/local/game/src/server/index.js:228:51) at Socket.<anonymous> (/local/game/src/server/connectors/udp.js:37:33) at Socket.emit (events.js:189:13)