A21.1 Server "Could Not Retrieve Server Information"

[KaldaraFox]

Alpha 21.1 Stable

 

I run four servers on a dedicated Linux box that does nothing but this.

 

They've been running well.

 

Today I took down and deleted two of the servers to replace them with custom maps and fresh start 7DTD games.

When the first of those two servers reported that it was up and running, I attempted to log in to the game to verify that all went well and got a "Could Not Retrieve Server Information" message. Checking the log file (https://pastebin.com/2iWZsMng (lines with my server address have been removed - otherwise this is unchanged) I found that the server crashed during login and restarted.

I tried a number of different times including deleting the instance, doing a hardware restart, reconstructing the instance, and logging back in.

 

No joy. This log file is the result of that last attempt.

 

The Discord channel offered some help (yours, not AMP's - they're insisting this is a Game Dev issue, nothing to do with them and they're pretty . . . let's be polite and say "firm" about that) from SylenThunder, but he may have gone dark for the evening as he stopped responding when I reported that the last suggestion failed to solve the problem. I don't know what time zone he's in and it's early evening for me (Midwest US).

He did say that some sockets are still being held open.

I deleted the instance, did a hardware restart, and rebuilt the instance and started it and it still failed.
 

I'm unclear how "sockets" (like I know what those are <grin>) are still open after that. Who or what is holding them (assuming that's the issue)? How do I get it/them to release them?

I've got about 20 people who play on my servers and right now half of them are down and there doesn't seem to be anyone who can fix this. If this isn't the right place to ask, please point me to where the right place is.

ETA: Literally nothing changed other than the map I was using between deletion of the old instance and the creation of the new one. I even tried deleting that and making a pure vanilla one - no addons other than the ones that come with the server that allow interaction with Twitch and the Navesgane map. No joy.

I haven't changed my port assignments. Heck, I haven't even updated Linux in a week.

Two nights ago this worked just fine. Now it's so broken that I've lost half my servers and I've got a dozen people wondering when they'll get back in.

Edited November 1, 2023 by KaldaraFox
Added some stuff. (see edit history)

[Beelzybub]

Your game port is on 27015 and telnet is connecting on 25017, which is a port that the game needs. So

move one or the other.

[Ando583]

I'm having the same issue as of a few days ago. Seems to be affecting Linux systems.

[KaldaraFox]

12 hours ago, Beelzybub said:

Your game port is on 27015 and telnet is connecting on 25017, which is a port that the game needs. So

move one or the other.

That sounds really authoritative and correct-ish, but how does it explain the two servers that are still running with the same basic pattern of ports (27015, 27018, 27021, 27024 with appropriate bumps in the other ports/port ranges)?

How does it explain that two days ago this server was running with the same configuration (just a different map and an older save) prior to deleting it and recreating it?

I'm not being argumentative. I just want to understand what is actually going on.

 

27015 was working for months. I deleted the server yesterday to make a new one and it stopped working.

Something obviously changed. I'm trying to find out what and who is responsible (the server management software support folks say it's TFP - I find that hard to believe - but I need facts that I can prove to push back with them).

 

Did Steam change something on their end that only affects newly defined servers (the other two servers are older - one on the order of three weeks, one around six weeks.

 

These are my current port assignments.

If I need to make changes to Server 1 or 2 I can do that risk-free.

Servers 3 and 4 are currently running and I'm loathe to make any changes there without phone support to fix them if they break.

https://imgur.com/a/ohCSDrt

 

Do I just need to change Server 1's ports (put them up above the Server 4 port numbers?s

If so, is the process....

1) Add port forwarding for 27027, 27043, 27044, 27045 to my router
2) On the Ubuntu Linux LTS box, issue

sudo ufw allow 27027

(followed by a similar command for each of the other ports).

I'm kinda new to Linux command line stuff. I've been a GUI user for a bit, but messing with command line stuff makes me nervous.

Edited November 2, 2023 by KaldaraFox (see edit history)

[KaldaraFox]

Okay, I've added ports 27027 and 27043-27045 to my router table.

 

I've run sudo uwf deny on ports 27015 and 27031-27033 on the Linux box running my servers.

 

I've run sudo uwf allow on ports 27027 and 27043-27045 on the Linux box running my servers.

 

I deleted and redefined the server I'm trying to get up and running (again - it was running just fine (now) three days ago)) and started it.

 

The log file (https://pastebin.com/n3Ry1yw7) shows that the Server Port is now 27027.

 

Yet I'm still getting that same error on first login.

 

I need help, guys. If this isn't where I go to get it, I need to know where I can go. No response for half a day on the Discord, the server software folks (AMP) say it's a TFP issue (I think they mean "it's not us so it must be someone else" but that might be Steam/Valve).

 

I'm up against it here.

 

I can't do this alone and there doesn't seem to be anyone who will respond with anything useful.

 

It's certainly not the "don't use 27015" issue because I'm no longer using that one and the error has not changed.

 

I kind of knew that was going to be the result because I've been using 27015 for months with no problem until day before yesterday.

 

I've been at this non-stop for about 18 hours now and I'm at the end of my rope.

 

Where is support? If not here, where do I go for it?

What is happening here? Why is it just happening now (I've made no changes to my system or my game parameters other than trying to add a new map and this last failure was using a generic, pregen map (Navezgane)). Something was changed somewhere in the chain of execution between my desktop server and where the game gets started and sets up communication and I can't even find out who is responsible for let alone ask them how to fix it.

 

The actual error text is here (it's in context in the log file, but I thought it might be simpler to give just that part here).

13:23:28

 

=================================================================
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (3514) : Trying to close low level socket support, but we still have sockets open!
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x7ff4a3d15062 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/294420/7DaysToDieServer_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so :
0x7ff4a3cbdd3d - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/294420/7DaysToDieServer_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so :
0x7ff4a3c43250 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/294420/7DaysToDieServer_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so :
0x7ff4aa51c520 - /lib/x86_64-linux-gnu/libc.so.6 :
0x7ff257dd97bb - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff2578ff103 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257906190 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257b26214 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257b22b89 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256d71303 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256d7221a - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256d73103 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256d73b61 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256c2cbb0 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256e2acbd - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256f9a3c2 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256f97128 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256f99928 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff256f99f58 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257a8046a - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257a7e541 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257a7ec3b - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff257a82a05 - /home/amp/.ampdata/instances/PhodgePodge7DTDZombieWarMap01/seven-days-to-die/linux64/steamclient.so :
0x7ff4aa56eac3 - /lib/x86_64-linux-gnu/libc.so.6 :
0x7ff4aa600a40 - /lib/x86_64-linux-gnu/libc.so.6 :
=================================================================
Telemetry Dumper:
=================================================================
Thread 0x7ff28cffd640 may have been prematurely finalized* Assertion at mono-threads.c:702, condition `info' not met, function:mono_thread_info_current,
src/steamnetworkingsockets/clientlib/steamnetworkingsockets_lowlevel.cpp (3514) : Trying to close low level socket support, but we still have sockets open!

Shutdown handler: cleanup.

System info is displayed in the log file, but I'll repeat it here as well:

 

2023-11-02T18:23:32 0.486 INF System information:
2023-11-02T18:23:32 0.486 INF OS: Linux 5.15 Ubuntu 22.04 64bit
2023-11-02T18:23:32 0.504 INF CPU: AMD Ryzen 5 1600 Six-Core Processor (cores: 12)
2023-11-02T18:23:32 0.505 INF RAM: 32034 MB
2023-11-02T18:23:32 0.505 INF GPU: Null Device (128 MB)

2023-11-02T18:23:32 0.509 INF Graphics API: NULL 1.0 [1.0] (shader level 5.0)

If there's anything else I need to provide, please let me know.

 

I did redact my server's actual IP address every time it showed in the log. I can't imagine that's necessary and it seemed a bad idea to publish that openly.

 

Edited November 2, 2023 by KaldaraFox (see edit history)

[meganoth]

Your first log seems to show that you downloaded an update to the server before starting the game. Were you running an older version before (which one?) and changed to latest stable with these that don't work? Or did you simply delete the server files and redownloaded again?

 

 

[KaldaraFox]

48 minutes ago, meganoth said:

Your first log seems to show that you downloaded an update to the server before starting the game. Were you running an older version before (which one?) and changed to latest stable with these that don't work? Or did you simply delete the server files and redownloaded again?

 

 

"Install" runs an update to a very basic "footprint" - that's normal for at least the AMP managed servers. Looks identical to the logs of the two servers still working.

I can even do an "Update" and draw down fresh copies of the software (kinda pointless because it's not changed).

 

 

Edited November 2, 2023 by KaldaraFox (see edit history)

[Jugginator]

I suppose the biggest/quickest troubleshooting would be to just run the server without the AMP (which tbh I'm unfamiliar with) on a new save after verifying your files through steam. If it works like that, then we've narrowed it down to AMP.

[KaldaraFox]

https://discourse.cubecoders.com/t/7-days-to-die-crashing-on-connection/6669

 

Appears to be a Steam/Valve issue. Telling here is that it works on a local connection, but not through Steam.

RIP remote access to my servers until/if this gets fixed.

 

[KaldaraFox]

2 minutes ago, Jugginator said:

I suppose the biggest/quickest troubleshooting would be to just run the server without the AMP (which tbh I'm unfamiliar with) on a new save after verifying your files through steam. If it works like that, then we've narrowed it down to AMP.

I have the AMP software because I have utterly no idea how to start a server without it.

 

I posted a link, oddly from same people whose representative bluntly told me it was a TFP issue, that pretty clearly indicates that it's a Steam/Valve issue.

 

Note the post that it works just fine with a local connection (indicating that there's nothing at all wrong with the install itself), but fails with a remote connection.

 

I'll mark this complete for now and look for an answer from Steam/Valve.

[Beelzybub]

2 hours ago, KaldaraFox said:

Okay, I've added ports 27027 and 27043-27045 to my router table.

Eh? The game uses the base TCP port and the corresponding UDP port +2 (or +3 to be safe). IE: if your base port is TCP 27027, you also need UDP 27027, 27028, & 27029 forwarded to your server. If using the defaults, you'd need TCP 26900, UDP 26900, 26901, & 26902.

 

edit: Never mind. I see from your Discord posts you already know this info.

Edited November 2, 2023 by Beelzybub (see edit history)

[Jugginator]

On 11/2/2023 at 3:55 PM, KaldaraFox said:

I have the AMP software because I have utterly no idea how to start a server without it.

 

It's just a file within the game directory. The config file is in the same directory too; once you run it it does everything else.

[KaldaraFox]

SylenThunder walked me through several attempts at a fix and found one that works.

Root Cause (for me) seems to be that the AMP default configuration for 7DTD is creating instance entry points in the 27xxx range of ports - which are known to be used by Steam.

 

While it works some of the time, I might argue most of the time, Steam still has those reserved and in my case two of the servers I was setting back up collided with Steam's use of the ports they wanted to use.

 

The solution was to move the server entry points and port ranges up from the 27xxx port range to the 31xxx port range.

 

The technical details of that are more than I can accurately describe in a way that would cover every Linux Distro, but the basic outline of what we did was this:

1) Deleted the failing instances.
2) Perform a shutdown on the server hardware.

3) Restart the server hardware.
4) Opened 10-port ranges in the router for each instance starting with 31030-31039 for the first one and building on that for each one.

5) Activated port listening for the newly opened port ranges.

6) Configure and start the new instances.

 

If you're using AMP's software, you'll need to be sure that your configuration includes manually overriding the port assignments for the server. It'll try to start at 27015.

 

If you're using someone else's software or doing it raw, obviously that doesn't apply.

 

Thank you for everyone who contributed to the problem definition, research, and especially to SylenThunder for the solution.

14 hours ago, Jugginator said:

 

It's just a file within the game directory. The config file is in the same directory too; once you run it it does everything else.

I get how to edit the file. I don't know how to cause the server to actually start. That was what I meant. Problem solved as far as this thread goes, though.