Extreme lag and rubberbanding on server since updating to alpha 20.4 (b42)

[kog]

Hello,

 

My server became unplayable since updating to the latest version. There were absolutely no issues in v20.3.

Setup:

Dell PowerEdge 420 with:

two quad core xeon E5-2407v2 processors

70GB DDR3 RAM

Gigabit connection with 2ms PING

OS: Ubuntu 20.04

 

Players:

2 external

1 internal on the local network (me)

 

Server config xml settings:

I set all the data exactly as in the old xml(which was completely stable). In the new xml the only thing that I had to set differently were the two lines regarding the region.

No mods are used.

Networking settings:

<!-- Networking -->

    <property name="ServerPort"                     value="26900"/>

    <property name="ServerVisibility"               value="2"/>    

    <property name="ServerDisabledNetworkProtocols" value=""/>  

    <property name="ServerMaxWorldTransferSpeedKiBs" value="1024"/>

 

Behaviour:

Extreme lag and rubber banding happens so often that the server is mostly unplayable. During horde night the two external players were kicked by the EAC because of the lag. As soon as that happened, and I was the only one playing, the lag and rubber banding disappeared.

 

We got the same thing happening while in a test server on one of the external players local server.

 

Console:

I am using Telnet to monitor the server remotely in real-time (I keep the game on one monitor, and the server console on the other one). No errors appear on the server console, except for skipped ticks caused by player lag. During horde night while all the players experienced extreme lag,( so much that we did not even get to see zombies)but the server was behaving normally, spawning zombies and getting them killed by the turrets and various traps. Processor usage was normal. In the end the players were kicked by EAC for lagging. At one point we were 5-7 seconds behind what the server console was showing.

 

What is going on here? It is clear that it is not a performance issue on the server part, or else I would have seen high processor/network usage. I have a hunch that there is something fishy going on with the steam connection/game network.

 

Any suggestions?

[Star69]

Can you post the server log and possibly one of the players logs experiencing the lag? Go to pastebin.com, paste a copy of the entire log there and post the link created here instead of pasting the entire log here. Maybe there will be a hint in the logs.

[Reytag]

See if the following helps in any way:

<property name="ServerDisabledNetworkProtocols"    value="SteamNetworking"/>

 

Worse case roll back to 20.3 for now so you can play, we are on 20.3 and enjoying it, 20.4 is a mess.

[kog]

Here is the server output log https://pastebin.com/i6LRJiLn The last part after the players were kicked out is me in god mode.

Here is the log from my client https://pastebin.com/MRG9S70R

I will try tonight to see what happens with steam networking disabled. Last time the other steam clients could not connect to the server with steamnet disabled.

[Beelzybub]

11 hours ago, kog said:

I will try tonight to see what happens with steam networking disabled. Last time the other steam clients could not connect to the server with steamnet disabled.

The LiteNetLib connection is failing, which points to a port problem.  You should have 26900 TDP and 26900-26903 UDP natted in your router to your server ip. And those ports need to be open on your linux firewall as well.

[SylenThunder]

Might also have your ISP test your lines. We had a similar issue a couple of years ago, and it turned out to be a bad ground with the cable coming to the house.  Speeds and everything else tested just fine, but we were noticing a lot of dropped/corrupt packets.

[kog]

All the necessary ports are forwarded in my router and I have them opened in the firewall as well. The game has been working flawlessly for about three months. When I updated it to 20.4 the trouble began.

My connection is working correctly. The optic fiber has no losses, and everything else I play/download online has absolutely no issues.

 

Last night I discovered that as long as only one(1) player is online he has absolutely no issues, even during horde night. One of the external players was testing the server and it worked fine until the second player joined. Then it started lagging for both of them. It does not seem to matter who is online, if the number is >1 the lagfest begins.  

[Beelzybub]

2 hours ago, kog said:

All the necessary ports are forwarded in my router and I have them opened in the firewall as well.

Is LiteNetLIb failing for all your clents, or just the one whos log you posted?

[kog]

It fails on all clients.

Here is the log from an external player:

https://pastebin.com/jZpvHx9D

[Beelzybub]

3 minutes ago, kog said:

It fails on all clients.

It appears to be enabled on the server, so the fails are port related.

This can be from your linux firewall or your router not having the right nat.

You already said they are ok, but, just for science, check your ports with an

external port scanner, like this one, or, there are many others.

 

And by way of explanation, LiteNetLib allows a direct connection between

the client and the server, which greatly reduces lag compared to SteamNetworking

which routes all traffic trough steam servers. This increases lag

and can vary from hour to hour and bad to worse depending on load and other

conditions.

Once you get LIteNetLib working, you should disable SteamNetworking.

[kog]

Today we ran tests with the firewall completely disabled on the server machine. No change in behavior. I will check the router settings to see if anything happened there.

[kog]

No success. The game fails to connect with steam networking disabled. This is probably the reason for the lag.

So why does litenetlib fail? All the ports are opened in both firewall and router. I have to mention that my IP is not traceable. My ISP gives me an internal ip different from what port scanners show. But this was not a problem before 20.4.

 

Hmmm

 

Edit: I found an older log from 20.3. Here the Litenetlib failed too in every session, but the game ran fine on steamnetworking. Maybe in 20.4 the game sends/receives a lot more data? And this cripples it?

Edited May 8, 2022 by kog (see edit history)

[SylenThunder]

1 minute ago, kog said:

I have to mention that my IP is not traceable. My ISP gives me an internal ip different from what port scanners show. But this was not a problem before 20.4.

 

Hmmm

This sounds like CG-NAT, which is extremely bad for hosting.

[kog]

Yup. See the edit too.

[SylenThunder]

The LiteNetLib messages are an effect. It's not an error. It is merely reporting a failure in the data connection, which is external to the client. The cause is the data pathing being lost.

Edited May 8, 2022 by SylenThunder (see edit history)

[ClarkParker]

I had the same Problem - in my case EAC was the troublemaker - switched it off and now every thing is running fine.

Unfortunately didn't find a solution to run the game with EAC at the moment so can't use Anti Cheat on my Server till I found a fix...

[kog]

UPDATE:

 

I downloaded the 20.5 experimental build and all the issues cleared:

As a local network player connecting directly to the LAN local server IP, the LiteNetLib service  connected successfully as it should.

The external players still get the LiteNetLib error, but during the horde night tests there was no more lag that the usual. The game is playable again.👍

[kog]

The issues are back. The server shows in the search list but nobody can connect. Server version appears to be "unknown". The server does not register our connection attempts.

I can connect directly by using the LAN IP without issues, but not from the lobby list.

Yesterday it worked fine. There were no changes made to clients or the server. I only turn on the server machine to play the game.

 

At this point I do not know what else to try.

 

 

[Beelzybub]

23 minutes ago, kog said:

At this point I do not know what else to try.

You could post your router config, maybe someone will see the issue.

Have you confirmed or excluded a double nat problem?

Does your server have a static internal ip? If not, has it changed? Does it match your forwarding entries?

Also posting new logs may show a new clue.

[kog]

@Beelzybub

The router is configured correctly, the required ports are forwarded. What do you mean by a double nat?

The server has a static IP(150) All the forwarded ports refer to the same IP.

 

Yesterday we could connect to the server, but hordenight was unplayable.

I will try to get some logs but they are probably useless. The worst is during horde night. I counted up to 15000ms ping.

 

I am also trying to setup a zerotier vpn network. Without success so far.

[Beelzybub]

24 minutes ago, kog said:

What do you mean by a double nat?

 

On 5/8/2022 at 10:55 AM, SylenThunder said:

This sounds like CG-NAT, which is extremely bad for hosting.

 

On 5/8/2022 at 11:03 AM, kog said:

Yup. See the edit too.

 

[kog]

Hello people of the forum!

 

I have not started a new thread since the problem is still here.

 

After a break of a few months I am back to troubleshooting. Some progress has been made, but I still can not connect to the server. Please read below:

 

So being in a CG-NAT network I tried to find various solutions to access my network from outside.

The last one that worked to some extent was to use a free DDNS offered by the provider. Basically it is an address xxxxxx.go.ro. From outside the network you write the adress and port number and you are set to go. I opened a port to test the accesibility of my ups network card and it worked. I could turn power to the server on or off.

 

Next step was to test it with the server. In the serverconfig I disabled steam networking, and tethered the main pc through my cellphone so i would be outside the provider network. When I write the adress xxxxx.go.ro and the gameport the game recognises the address and accesses the right IP (session address), but there is no connection going through to the game server. Am I doing something wrong? Or is it just not possible to work it this way?

 

Thank you.

[Star69]

I wasn't familiar with CG-NAT so I looked up information on it's wiki and came across this gem about disadvantages of this system:  "Carrier-grade NAT usually prevents ISP customers from using port-forwarding, because the network address translation(NAT) is usually implemented by mapping ports of the NAT devices in the network to other ports in the network interface." There is more info after this sentence on the wiki. So since your ISP uses CG-NAT, it sounds like any hosting software requiring port-forwarding is not going to work for you. I guess you could try contacting the ISP directly and see if they have some kind of answer for you but it sounds like you're in a bad position. The only other option is if one of your playing partners has a different ISP and is willing to host your server.

[kog]

I somehow fixed it, and I have no idea why it works. LiteNetLib works correctly and does not throw errors in my friend's console anymore. Also in the server console, it appears to be used for the connections without issues. When spawning 100 zombies, although my friend's pc lags badly it is because of his pc, the connection does not skip ticks anymore, and there are no sync issues.

 

What did I do? Nothing except forwarding two ports for my server's IP: 443 and 80. These have nothing to do with online gaming from what I know, but if it works...