Unable to Connect to Server

[SylenThunder]

This issue is completely got me stumped.

 

I've wiped the server client and clean installed. (Even wiped the OS) I've completely nuked the client on my system.  Other people can connect to the server without a problem. Heck, I used to be able to connect to the server without a problem. I've tried completely nuking all of my player data from the server as well, yet I'm still unable to connect.

 

Client log

https://pastebin.com/raw/a1fea5Xh

 

Server log

https://pastebin.com/raw/WzfrmDKn

You can see where I am initiating a connection to the server starting at line 651.

 

2020-10-15T14:33:43 1754.790 INF NET: LiteNetLib: Connect from: 192.168.1.214:53466
2020-10-15T14:33:43 1754.800 INF Started thread NCS_Reader_2_0
2020-10-15T14:33:43 1754.800 INF Started thread NCS_Writer_2_0
2020-10-15T14:33:43 1754.810 INF Started thread NCS_Reader_2_1
2020-10-15T14:33:43 1754.810 INF Started thread NCS_Writer_2_1
2020-10-15T14:33:43 1754.810 INF [NET] PlayerConnected EntityID=-1, PlayerID='', OwnerID='', PlayerName=''
2020-10-15T14:33:43 1754.813 INF NPCID.Write
2020-10-15T14:33:43 1754.852 INF NET: LiteNetLib: Client disconnect from: 192.168.1.214:53466 (RemoteConnectionClose)
2020-10-15T14:33:43 1754.852 INF [NET] PlayerDisconnected EntityID=-1, PlayerID='', OwnerID='', PlayerName=''
2020-10-15T14:33:43 1754.852 INF Disconnected player not found in client list...
2020-10-15T14:33:43 1754.879 INF Player disconnected: EntityID=-1, PlayerID='', OwnerID='', PlayerName=''
2020-10-15T14:33:43 1754.880 INF [EAC] FreeUser: EntityID=-1, PlayerID='', OwnerID='', PlayerName=''

It just repeats this anywhere from five to twelve times and then I get a "Connection Failed" error in my client.

 

If I use a different user account, I can log in just fine. Which leads me to believe it's something on the server side with my character that is messed up, however deleting my character data has had zero effect.

[Beelzybub]

41 minutes ago, SylenThunder said:

2020-10-15T14:33:43 1754.852 INF Disconnected player not found in client list...

Is there a whitelist maybe that you're not on?

[SylenThunder]

29 minutes ago, Beelzybub said:

Is there a whitelist maybe that you're not on?

No whitelist on the servers. I've double-checked my blacklist as well.

 

[Beelzybub]

If you try to log in with the bad user account from a different computer, does it still fail? If so, that would be confirmation

of your suspicion that it's a server side issue.

[SylenThunder]

On 10/16/2020 at 1:10 AM, Beelzybub said:

If you try to log in with the bad user account from a different computer, does it still fail? If so, that would be confirmation

of your suspicion that it's a server side issue.

I can't believe I didn't try this. I'm sitting here literally surrounded by systems. Tested from my laptop and the min-spec testing machine and was able to connect without an issue.  Problem then lies in my PC, so I'll have some fun with that. 

I'm way past due for re-installing Windows.

 

 

UPDATE: Scratch that.  Apparently I can connect to one server, but not the other three. After further testing I was able to connect to that same server on my primary PC. On the others when I attempt a connection using the Steam Connect link, it shows that the server is not responding, and when I try to direct connect using local IP it works. Not sure what is causing this one Steam account to be blocked now. It just gets stranger the more I get into it.

[Jugginator]

Boy you're like me when it comes to issues: either don't have issues or have the weirdest issue imaginable lol. I'm still contemplating; this doesn't make any sense, but since this is one of those issues, perhaps your router just doesn't like the address it assigned your system? Perhaps clear all the cache/table/etc. out and maybe reinstall Steam/your network adapter. Just simple and quick things before reinstalling Windows.

[SylenThunder]

I'm starting to think it's some oddity on the Linux box causing the issue.  

 

We have four 7 Days servers, and one Minecraft server on this machine. Server has four network adapters.

 

Setup is as follows.

Server	Adapter	Server Ports	Forwarded range
7 Days 1	eno1	26910-26918	26900-26924
7 Days 2	eno2	26930-26938	26925-26949
7 Days 3	eno3	26950-26958	26950-26974
7 Days 4	eno4	26980-26988	26975-26999
Minecraft	eno4	26990-26995	26975-26999

Now I used to be able to connect to everything.  Now I can only connect to 7 Days 1, and Minecraft. This has been happening for quite some time now. At one point in the past all of the servers were on eno1, but I still had the issue then. I had hoped that separating things would help, but it's had no effect on it.

 

Now I'm starting to have problems with sites like BattleMetrics, GameTracker, or 7daystodieservers.com reaching the ports, yet the servers are available to play on, can be interacted with on all ports, and have no issues with the web maps or CSMM connections. And it's only the 7 Days servers having that issue. Minecraft is fine. An even further oddity is that as of about half an hour ago, suddenly BattleMetrics can see Servers 1 and 4, but 2 and 3 are not accessible by it. The other sites still cannot see any of the 7 Days servers, but have no issues with Minecraft.

 

This is a recently clean install of Linux too, so I'm pretty baffled on what the issue is at this point.

 

[Beelzybub]

Can you trace activity with something like Wireshark (on both client and server) to see where things are going off the tracks?

Is each game in it's own vm?

6 hours ago, SylenThunder said:

7 Days 2 eno2 26930-26928 26925-26949

I assume you mean 26938?

[SylenThunder]

8 hours ago, Beelzybub said:

I assume you mean 26938?

Yep 

[Gamida]

I also noticed the 26930 - 26928 that Beelzybub mentioned and was also wondering about the forwarding range of 7 Days To Die 4 and Minecraft server being the same. Would that make any difference?

Also even though you have the port forwarding of your four 7 Days servers set, isn't port 26900 you have for minecraft the default port for 7 Days? Would that screw anything up? Server settings are mysterious to me so I am asking them as if it was something I needed to know for myself

[SylenThunder]

Oh sheesh, even more port errors in that table. I must have been crosseyed or something.  Fixed it now. Minecraft was on 26990-26995.

 

[SylenThunder]

Was tinkering some more.  Noticed this in the log...

 

2020-10-19T22:11:07 17.906 INF Connect by IP
2020-10-19T22:11:07 17.910 INF Connecting to server 192.168.1.143:26950...
2020-10-19T22:11:07 17.911 INF [NET] Disabling protocol: SteamNetworking
2020-10-19T22:11:07 17.912 INF Exited thread ServerBrowserListUpdater
2020-10-19T22:11:07 17.913 INF NET: LiteNetLib trying to connect to: 192.168.1.143:26950
Curl error 7: Failed to connect to config.uca.cloud.unity3d.com port 443: Timed out

2020-10-19T22:15:15 29.237 INF Connect by IP
2020-10-19T22:15:15 29.241 INF Connecting to server 68.41.178.221:26950...
2020-10-19T22:15:15 29.243 INF [NET] Disabling protocol: SteamNetworking
2020-10-19T22:15:15 29.244 INF NET: LiteNetLib trying to connect to: 68.41.178.221:26950
2020-10-19T22:15:15 29.246 INF Exited thread ServerBrowserListUpdater
2020-10-19T22:15:21 34.884 INF NET: LiteNetLib: Connection failed: ConnectionFailed
2020-10-19T22:15:21 34.884 ERR [NET] Connection to server failed: Connection Failed
2020-10-19T22:15:24 37.879 INF Started thread ServerBrowserListUpdater
Curl error 7: Failed to connect to cdp.cloud.unity3d.com port 443: Timed out

The first one is using the internal IP, and the second is using the external IP. 

 

Was talking with Yuri in the official Discord because they were seeing a similar issue. They disabled LiteNetLib, and then the person was able to connect. Unfortunately, it made no difference in my case.

[meganoth]

What happens if you try to connect to internal and external address port 443 per browser or curl on the command line? You should be able to use traceroute as well to find out if some connection is not possible and whether it stops at the router or your local server

 

 

[Liesel Weppen]

11 hours ago, SylenThunder said:

The first one is using the internal IP, and the second is using the external IP.

They can't use the same IP, because they are running on the same port. So maybe that one using the local address is a fallback?

Or what did you mean by "the first one" and "the second one".

 

The question is, how does the server determine what his public IP is?

Maybe there is a limitation in that "retrive public IP" backend, not allowing multiple servers from the same IP?

 

Also i'm not sure how these steam urls work. I guess if you are using a steam link, the connection is established over internet, even if both computers are in the same LAN. If you input the LAN-IP manually, the connection is of course local only. So both approaches are probably not comparable.

 

Why are you using 4 NICs? How are they configured? Does each server use the correct nic? Does every NIC obtain their correct IP address?

Portforwardings in the router are correct, not only the ports for each running 7d2d server but also for the ip of every nic AND port of whatever server runs on what interface?

 

Your table above also shows wierd organization of ports. So you do use port ranges of 20 starting of with 26910 for every server but configured forwardings in ranges of 25 starting from 26900? Maybe you should do this equally to prevent mistakes...

And why even different ports if they are on different network interfaces (not just interfaces but also even real adapters) anyway?

 

Do the servers perhaps use UPNP, and that may cause troubles if there is more than one server in one network?

[SylenThunder]

Ok meant to reply to this a couple of days ago, but I keep getting sidetracked.

 

On 10/20/2020 at 9:19 AM, meganoth said:

What happens if you try to connect to internal and external address port 443 per browser or curl on the command line? You should be able to use traceroute as well to find out if some connection is not possible and whether it stops at the router or your local server

These tests ran fine without an issue. Webmin and Netdata respond to their requests as well. Can even drop IP:Port into Firefox and get the server listing information. Yet that same interface/port isn't responding to the sites like BattleMetrics.

 

On 10/20/2020 at 9:44 AM, Liesel Weppen said:

They can't use the same IP, because they are running on the same port. So maybe that one using the local address is a fallback?

I attempt to connect to the internal port.  Then I use VPN with internal LAN access blocked to connect to the external port. This fully simulates connecting from outside of my network. Have been using this method for load testing the dedi for years so I can run tests with a variety in the connection quality without having to go hit 20 different Starbucks/McDonalds locations.

 

On 10/20/2020 at 9:44 AM, Liesel Weppen said:

The question is, how does the server determine what his public IP is?

Maybe there is a limitation in that "retrive public IP" backend, not allowing multiple servers from the same IP?

It pulls the external IP from the gateway. Gateway network for the different game servers is segregated on a managed switch, and then tied together into the main router with a LAG connection. Haven't had this issue previously, but it did eventually provide a solution. Though I am still not certain of the cause.

 

On 10/20/2020 at 9:44 AM, Liesel Weppen said:

Why are you using 4 NICs? How are they configured? Does each server use the correct nic? Does every NIC obtain their correct IP address?

Portforwardings in the router are correct, not only the ports for each running 7d2d server but also for the ip of every nic AND port of whatever server runs on what interface?

The server has four built-in Gigabit ports. (Dell PowerEdge R710) We configured the servers to use separate NIC's when we were seeing an issue with data traffic having 10 servers going through the one port. Each server is bound to a specific adapter when it is started. The port range of 26900-26999 is segregated into 25-port groups, and each adapter has it's own group. There is specific forwarding configured for each NIC's IP, and each IP is assigned in a static configuration. When the servers are started, they are bound to the IP address. Now that we have reduced our number of 7 Days servers there is only one on each port range.  Previously we had mapped them every 5 ports with the CP/Telnet ports in the 8080+ range.

 

On 10/20/2020 at 9:44 AM, Liesel Weppen said:

And why even different ports if they are on different network interfaces (not just interfaces but also even real adapters) anyway?

As a whole, we cannot have duplicate ports unless we have multiple internet connections. This separation has worked out quite well for hosting upwards of 22 servers at a time for several years. (Not all 22 on the same machine mind you. We have more than one physical server.)

 

 

Today (well technically yesterday now), the issue expanded from just me to 98% of our player base. We suddenly went from only me having an issue to all but two people having the issue. I went through everything. Even started a clean server on port 26900 and left everything at defaults. No dice.

 

After losing most of the hair I have left, I finally fixed it. Basically just unplugged three of the NIC's and routed everything through one NIC again.  Will have to investigate later why the data transfer slowly degraded over time. It was working just peachy when we originally set it up.  Maybe I missed something in the configuration when I wiped and did the Ubuntu 20.04 install.

[Beelzybub]

Maybe there's some unwanted/leftover port aggregation/trunk/mirroring thing going on in the server nics or in the switch's port setups?

Are all the Dell nics in the same subnet?

I have an old Dell that has bios level settings for the nics. You might find something checking in there as well.

[meganoth]

I would try to give the ports different IP addresses. One IP on different NICs is unusual enough that hidden bugs might sit in the linux kernel, in steam, the router, EDIT: the NICs firmware, .... comfortably undetected.

 

[SylenThunder]

11 hours ago, meganoth said:

I would try to give the ports different IP addresses. One IP on different NICs is unusual enough that hidden bugs might sit in the linux kernel, in steam, the router, EDIT: the NICs firmware, .... comfortably undetected.

 

Yeah, each NIC had it's own IP address. And each client was bound to the respective address when started.  Somewhere in-between is the issue.  I don't recall having any troubles in 18.04, so it's likely some change in netdata with 20.04 that requires an extra configuration I've overlooked.

[new7daylive]

Amazing...  Sylen having the same issue I'm having  : )

 

I set up a second instance of 7dtd on my server.  The first on is vanilla and is on 26905 and the second is a romero server with other addons and its on 26908.  My 2 friends can access both servers with no issue.  I, on the other hand, cannot access the second (26908 romero) server and Im getting the same logs as you are Sylen.

 

Not sure what to do.  All ports are open (others can access).

 

If you have anymore insight, I'd appreciate hearing it.

 

Thanks

[new7daylive]

My original server has steamnetworking disabled.  On the new server, I disabled LiteNetLib....  i can connect now.  I can connect to both servers. 

 

Thanks for mentioning this potential workaround above in your post - I hope there is a real answer to this issue soon

[SylenThunder]

2 hours ago, new7daylive said:

The first on is vanilla and is on 26905 and the second is a romero server with other addons and its on 26908. 

This is part of your issue.  The servers need to be 5 apart.  Even though the server is only slated for using 26900-26902, if it doesn't have 26903 it won't function. Leave the extra port open for a gap, and then next usable port is 26905.  In your case you're starting at 26905, so your Romero server needs to be at 206910. I mean you could stick it on 26909, but then you get odd port numberings, and it's harder to keep track of things.  

 

Disabling LiteNetLib will have a negative impact on your servers network performance. Clients will get out of sync easily.  SteamNetworking is basically used to trigger UPnP for opening ports. If you have proper forwarding it should be disabled. If you are unable to manually forward, or are behind a double-NAT, then you want to enable it.