Constant De-Sync of one out of 3 of us / Dedicated Server

[aprodigy]

Hi there,

 

getting out of ideas here and I'm wondering if any of you has any ideas of how to debug my situation any further:

 

Server side: AMD EPYC 7502P 32-Core / 64-Threads Processor, 128GB RAM, NVME RAID-1, Debian SID, 7DtD Engine version 2020.3.14f1 (d0d1bb862f9d). WAN uplink is 1Gbit/s fast.

World: RWG, GenSize 6144

 

2022-11-14T05:00:32 0.085 INF Version: Alpha 20.6 (b9) Compatibility Version: Alpha 20.6, Build: LinuxPlayer 64 Bit
2022-11-14T05:00:32 0.085 INF System information:
2022-11-14T05:00:32 0.085 INF    OS: Linux 5.10 Debian GNU/Linux  64bit
2022-11-14T05:00:32 0.096 INF    CPU: AMD EPYC 7502P 32-Core Processor (cores: 64)
2022-11-14T05:00:32 0.096 INF    RAM: 128750 MB
2022-11-14T05:00:32 0.096 INF    GPU: Null Device (128 MB)
2022-11-14T05:00:32 0.101 INF    Graphics API: NULL 1.0 [1.0] (shader level 5.0)
2022-11-14T05:00:32 0.102 INF Last played version: Alpha 20.6
2022-11-14T05:00:32 0.103 INF Local UTC offset: 1 hours

 

Client side: Core i9 9900k, 32GB RAM, AMD RX580 / NVidia RTX 2070, Windows 11 / Manajaro Linux / macOS 10.15.7, 7DTD Alpha 20.6b9. A Ping roundtrip to that dedicated server is somewhere between 10 and 15ms for all of us. None of us have noticed any connection hickups so far. Drivers are all up to their latest releases.

 

Current situation: Out of us three playing on that server, I'm the absolute only one running into constant de-syncs. They sometimes happen just seconds after me joining the server, sometimes after a couple of minutes or even after 15 minutes but I'm getting one sooner or later and after that I have to leave the game and rejoin the server to get back into action again. However, I have been able to play on that server alone for almost three hours without any de-sync.

 

As stated earlier and to rule out possible issues on my end, I now have tried it with three different operating systems out of which macOS and Linux were tried with a AMD as well as a NVidia graphics card (see specs above). I used a cable connection as well as a WiFi-connection and my internet speed is at 250Mbit/s without any VPN in use to connect to that server. We play a lot of multiplayer games together (Rust, ARK, Battlefield, Rocket League, MSFS et al.) and I have never experienced any issue like this.

 

The logs (client and server side) are mostly unobtrusive when it happens. This is the client log (just shows a couple of the same warning) and this is what the server shows when it happens:

 

2022-11-14T18:37:54 49041.421 INF VehicleManager saving 0 (0 + 0)
2022-11-14T18:37:54 49041.421 INF DroneManager saving 0 (0 + 0)
2022-11-14T18:37:54 49041.421 INF VehicleManager saved 9 bytes
2022-11-14T18:37:54 49041.421 INF DroneManager saved 9 bytes
2022-11-14T18:38:05 49052.602 INF Time: 817.33m FPS: 22.53 Heap: 2334.8MB Max: 2456.9MB Chunks: 387 CGO: 9 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3534.5MB
2022-11-14T18:38:35 49082.626 INF Time: 817.83m FPS: 20.92 Heap: 2337.7MB Max: 2456.9MB Chunks: 387 CGO: 12 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3534.5MB
2022-11-14T18:39:05 49112.635 INF Time: 818.33m FPS: 22.54 Heap: 2343.0MB Max: 2456.9MB Chunks: 387 CGO: 12 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3536.9MB
Calling Animator.GotoState on Synchronize layer

2022-11-14T18:39:35 49142.675 INF Time: 818.83m FPS: 21.60 Heap: 2349.2MB Max: 2456.9MB Chunks: 387 CGO: 12 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3539.7MB
2022-11-14T18:39:54 49161.434 INF VehicleManager saving 0 (0 + 0)
2022-11-14T18:39:54 49161.434 INF DroneManager saving 0 (0 + 0)
2022-11-14T18:39:54 49161.435 INF VehicleManager saved 9 bytes
2022-11-14T18:39:54 49161.435 INF DroneManager saved 9 bytes
2022-11-14T18:40:05 49172.700 INF Time: 819.33m FPS: 20.91 Heap: 2337.1MB Max: 2456.9MB Chunks: 387 CGO: 9 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3540.9MB
2022-11-14T18:40:35 49202.723 INF Time: 819.83m FPS: 20.93 Heap: 2338.4MB Max: 2456.9MB Chunks: 387 CGO: 12 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3541.4MB
2022-11-14T18:41:05 49232.767 INF Time: 820.33m FPS: 21.12 Heap: 2340.9MB Max: 2456.9MB Chunks: 387 CGO: 12 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3543.4MB
2022-11-14T18:41:35 49262.804 INF Time: 820.83m FPS: 21.09 Heap: 2344.7MB Max: 2456.9MB Chunks: 387 CGO: 9 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3543.4MB
2022-11-14T18:41:54 49281.458 INF VehicleManager saving 0 (0 + 0)
2022-11-14T18:41:54 49281.459 INF DroneManager saving 0 (0 + 0)
2022-11-14T18:41:54 49281.459 INF DroneManager saved 9 bytes
2022-11-14T18:41:54 49281.459 INF VehicleManager saved 9 bytes
2022-11-14T18:42:05 49292.807 INF Time: 821.33m FPS: 21.10 Heap: 2314.9MB Max: 2456.9MB Chunks: 387 CGO: 9 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3543.3MB
2022-11-14T18:42:35 49322.808 INF Time: 821.83m FPS: 21.30 Heap: 2316.4MB Max: 2456.9MB Chunks: 387 CGO: 9 Ply: 2 Zom: 0 Ent: 2 (52) Items: 0 CO: 2 RSS: 3543.3MB
2022-11-14T18:42:58 49345.172 INF Player aprodigy disconnected after 6.1 minutes
2022-11-14T18:42:58 49345.436 INF DynamicMusic: 804 successfully removed from Bloodmoon state cache
2022-11-14T18:42:58 49345.436 INF Player disconnected: EntityID=804, PltfmId='Steam_7656119802XxX', CrossId='EOS_0002649906XxX545806c4cXxX', OwnerID='Steam_76561198024876946', PlayerName='aprodigy'
2022-11-14T18:42:58 49345.436 INF GMSG: Player 'aprodigy' left the game
2022-11-14T18:42:58 49345.436 INF Exited thread NCS_Reader_2_0
2022-11-14T18:42:58 49345.436 INF Exited thread NCS_Reader_2_1
2022-11-14T18:42:58 49345.436 INF Exited thread NCS_Writer_2_0
2022-11-14T18:42:58 49345.436 INF Exited thread NCS_Writer_2_1
2022-11-14T18:42:58 49345.436 INF NET: LiteNetLib: Client disconnect from: xXx.134.XxX.247:9575 (DisconnectPeerCalled)

 

My client doesn't time out and I don't get disconnected by the game (at 18h42m58s i disconnected by myself and the server even noticed that). The world is still being rendered on my end and I can move around but every player or non-playable entity runs on the spot and I'm not able to interact with any object anymore until and as stated before, I leave the game on my own and rejoin the server. My buddies however, are still able to play when the de-sync happen to me and they have the server continuing to run perfectly fine for them.

 

Since my buddies don't experience those de-syncs we suspect that the server is actually running fine and we thought, that it must be something on my end then. However, having tried playing on that server with three totally different operating systems, two different graphics cards, via WiFI and via LAN, I kinda rule out a platform problem on my end, too. And it can't be about any anti virus solution or firewall since I don't use those two on Linux and macOS.

 

I also tried to use a new server-side player profile by removing the EOS_*.map and EOS_*.ttp files that correspond to my player id but that didn't help either.

 

The only thing we're able to reproduce is that when I'm playing alone, I don't get de-syncs. As soon as one of my buddies connects, me and just me gets de-syncs.

 

Playing around with server-side options (e.g. EAC enabled, disabled) didn't help either.

 

Let me know if any of you  needs additional information from me to get down to this.

 

Thanks.

 

-aprodigy

Edited November 17, 2022 by aprodigy
fixes part 3 (see edit history)

[aprodigy]

A full client log with multiple de-syncs from my end can be found here. Even though I still think they're pretty unobtrusive and most likely won't really help

[aprodigy]

An here the corresponding server log as an addendum.

[Star69]

On both the server and your computer, create exclusions in your antivirus software for the 7DaystoDie folder and see if that helps. Also, if you are running any software that says it optimizes your internet for gaming, including router settings, be sure to disable.

[aprodigy]

thanks for trying to help @Star69 but i'm pretty sure that on my server there no iptables / nftables rules [and|or] profiles active. that system routes all of its ipv4 and ipv6 traffic directly to its default gateway that is the providers default gateway - without any exception. and if it would be a server side issue, it would affect my buddies, too.

 

and, yep, i even tried running this game with windows, linux and macos on the client side. so there were different and even no security features active even on the client running at that time (espcially when running the game via manjaro; even tried ubuntu latetly).

 

it's something else that seems not to come to light via the logs.

Edited November 18, 2022 by aprodigy (see edit history)

[FinkPloyd]

At 18:44 you try to connect and the server IP is a default / placeholder of 0.0.0.0 and the other user is getting 1.1.1.1:

"2022-11-15T18:44:20 40.150 INF NET: LiteNetLib trying to connect to: 0.0.0.0:26900"

 

The server is saying TELNET is from 127.0.0.1 (the localhost / loopback IP)

 

Are you selecting the server from the list or connecting by IP? (or did you edit the IP adresses in the files?)

 

Not 100% sure but it looks like the server has issues resolving IPs / TCP IP adresses.

 

Edited November 18, 2022 by FinkPloyd
Posted by mistake (see edit history)

[aprodigy]

5 hours ago, FinkPloyd said:

At 18:44 you try to connect and the server IP is a default / placeholder of 0.0.0.0 and the other user is getting 1.1.1.1:

"2022-11-15T18:44:20 40.150 INF NET: LiteNetLib trying to connect to: 0.0.0.0:26900"

 

 

thank you but that's fine - i just replaced the source and destination ip's (as well es the id's) before posting the logs to pastebin.

and yes, i actually telnet to the service from the machine itself. it's my own server, so i'm used to ssh to machines.

Edited November 18, 2022 by aprodigy (see edit history)

[aprodigy]

we'll try to create a new map based off navezgane this evening to see if it's got anything to do with the rwg-seed we've been using. i will capture a new set of clean logs, too and post them here afterwards if needed even though i didn't find a way to raise debug levels, yet.

 

but if that won't help either i guess i'm sadly out of luck playing this game.

[aprodigy]

just as a heads up...

 

resetting the map from RWG to navezgane didn't help either... i started to fall through the map and finally got a de-sync again - for my buddies everything was working well again.

 

so today, i set up a new server and used version 20.5 (and removed all traces of the old map and version we've been playing, so generated a new navezgane map) - we downgraded all of our clients and tried again... was working fine for about 30 minutes or so but the i started to fall though the map again and again.

 

i'm not gonna post the logs since they're unobtrusive (and i know how to read them), so i guess for some still unknown reason i'm not able to play this game. but cyberpunk 2077 was working pretty good for me, so i guess i'm gonna continue on this one.

 

if any of the devs would like ssh-access to our server to debug things further, feel free to contact me.

Edited November 19, 2022 by aprodigy (see edit history)

[Star69]

35 minutes ago, aprodigy said:

just as a heads up...

 

resetting the map from RWG to navezgane didn't help either... i started to fall through the map and finally got a de-sync again - for my buddies everything was working well again.

 

so today, i set up a new server and used version 20.5 (and removed all traces of the old map and version we've been playing, so generated a new navezgane map) - we downgraded all of our clients and tried again... was working fine for about 30 minutes or so but the i started to fall though the map again and again.

 

i'm not gonna post the logs since they're unobtrusive (and i know how to read them), so i guess for some still unknown reason i'm not able to play this game. but cyberpunk 2077 was working pretty good for me, so i guess i'm gonna continue on this one.

 

if any of the devs would like ssh-access to our server to debug things further, feel free to contact me.

The only other thing that I see in your logs is that you have both litenetlib and steamnetworking as active protocols. It appears you are joining via steamnetworking, that has been known to create significant lag which could lead to de-sync. As a last resort, why not try to add steamnetworking to the serverdisablednetworkprotocols setting? At least worth a try since there don’t seem to be any other ideas out there.

[aprodigy]

Just now, Star69 said:

The only other thing that I see in your logs is that you have both litenetlib and steamnetworking as active protocols. It appears you are joining via steamnetworking, that has been known to create significant lag which could lead to de-sync. As a last resort, why not try to add steamnetworking to the serverdisablednetworkprotocols setting? At least worth a try since there don’t seem to be any other ideas out there.

 

ya, thank you - we tried it all so far... comment out that ServerDisabledNetworkProtocols completely, set it to "SteamNetworking", set it to "LiteNetLib" - didn't help on my end but hinder playing my buddies either

the question here stays...

 

* the server is running perfectly fine for my buddies

* i tend to get de-syncs and fall through the map

* i tried windows, macos and linux as a client which macos was running with AMD RX580 instead of an Nvidia 2070 RTX

[aprodigy]

the question here stays, though...

 

* the server is running perfectly fine for my buddies (v25 as well as v26), no matter if the map gets randomly generated or Navezgane

* i tend to get de-syncs and fall through the map, my buddies don't

* i tried windows, macos and linux as a client of which macos was running with AMD RX580 instead of an Nvidia 2070 RTX

* used WLAN and LAN
* and every other game runs perfectly fine on my system, and one of is MSFS, which is prone to fail because of the "sun's shining"

* none of the logs, client or server side, is showing anything unusual, unwanted disconnects or anything relevant

 

all of this seems to be like an unhandled exception that is not being captured. we're still willing to help but a dev would be in need of.

Edited November 19, 2022 by aprodigy (see edit history)

[aprodigy]

news: i've set up a wireguard vpn tunnel to the server my two buddies play on for me and welp, i've been able to play online without any hickups for the first time ever. so, it might be something provider related even though i'm still not sure why it occurs sometimes right after joing a server and sometimes 15 minutes later but i didn't get a chance yet to look at my wireshark protocols.

 

it's still something, as far as i'm concerned, that the devs should take a look on.

[meganoth]

Is your server in your local network with the client or at a hoster?

 

I have a friend who is often (once or twice per evening) getting disconnected and we still have no idea where the problem is. One likely culprit is the network, i.e either his router, the provider or the rest of the network in some mysterious ways. Probably then some idiosyncracy or even bug in the network layer of the game is part of the problem.

 

The other possibility is his PC which is in fact a laptop with (one could imagine) possible heat issues(?). But so far nothing really points to this being the issue.

 

One suggestion we never got to try was for him to transport his PC for a session to one of our homes so he could try his PC with a different internet connection. If you have the ability to do this without too much inconvenience please do. I think this would help immensely with pinpointing the issue.

 

Another aspect is that network code is currently being worked on by a newly hired developer inside TFP and the next alpha of the game will likely act differently in some respects. The issue might or might not be gone. But mostly it means that for TFP A20 is not anymore the version they look at to find bugs. So I would also suggest to try out A21 when it is out. And if the problem persists to make a bug report instead of using General Support.

 

 

[aprodigy]

Just now, meganoth said:

Is your server in your local network with the client or at a hoster?

 

 One likely culprit is the network, i.e either his router, the provider or the rest of the network in some mysterious ways. Probably then some idiosyncracy or even bug in the network layer of the game is part of the problem.

 

Another aspect is that network code is currently being worked on by a newly hired developer inside TFP and the next alpha of the game will likely act differently in some respects. The issue might or might not be gone. But mostly it means that for TFP A20 is not anymore the version they look at to find bugs. So I would also suggest to try out A21 when it is out. And if the problem persists to make a bug report instead of using General Support.

 

 

 

thank you, magnoth and ya, since the game is finally running fine for me (windows via openvpn, macos via wireguard) using a vpn but not without any, we kinda suspect the same, i.e. it seems to be a network-related issue. my server is running at hetzner, a hoster located in southern germany but since my other two buddies can play on that server without a vpn all fine, it shouldn't be the server or its connection.

 

but good to know they are working hard to get a21 out. i'm looking forward to it and in the meantime i just play using a vpn to connect to our server.

 

if the problem persists with a21 i will provide you guys with a proper bug report then.

[aprodigy]

another heads up (for the devs and maybe others) after having played for another 6 to 8 hours now... game's running absolutely fine for me via a VPN to the dedicated server... without it, i keep continue falling through the world and get disconnects.

 

thread can be closed.

Edited November 21, 2022 by aprodigy (see edit history)