Multiplayer Server Laggy When Under Stress

[Bigun]

I have a dedicated private server running in a Linux container, with 16GB of RAM and 16 Cores, and is running on local SSD storage configured in a hardware RAID-5, all running on Proxmox.  I have more resources I can throw at it, but I only assign resources as it's needed.

 

Mind you, I'm sitting right next to the server and connect via local IP, so I have sincere doubts that the issue is network connectivity.  Plus outside players have 1 Gbps fiber to the server, so again, I have doubts it's a network issue.

 

The scenario is as follows:

 

Seven players (including myself) are connected.  When all of us are running on Motorcycles, the servers lags and kicks players around the map (including me).  Sometimes when players get off the bikes, the bike disappears and reappears where home base is.  While this is happening, I'm watching the server performance via htop and through Proxmox, and the CPU averages about 40% to 60% use, RAM hovers around 6 to 8 GB of use (about 50% used), swap space ends up being full (makes me believe the server isn't using all the RAM for some reason), and according to Proxmox there is no unreasonable I/O delay from storage.

Most other scenarios, the server performs just fine.  Running on foot or on bicycles doesn't seem to cause any issues.

Not sure what I can change to help with this, but any advice is welcomed.

Edited May 24, 2022 by Bigun (see edit history)

[Bigun]

A new discovery, I thought maybe a lack of swap may have been the cause, so it now has 8GB of swap.  Retested with just 5 people, and the same thing happened.  Not even half the RAM used and the cores sitting around 30%.

What is going on?!

[meganoth]

Considering you are seeing this while driving around data path between RAM and SSD would be the first guess. This doesn't depend on SSD and CPU alone, what generation is your CPU/motherboard?

 

My best guess for the CPU never showing 100% is that the game loop is too big for internal CPU caches.

 

Since you have it running in a virtual machine, maybe the VM has some overhead for anything coming from SSD (i.e. an additional copy operation for all data) ? Normally I would say current VM's should have almost no overhead for this but I don't know enough about that to be sure one way or the other.

 

[Bigun]

On 5/30/2022 at 6:01 AM, meganoth said:

Considering you are seeing this while driving around data path between RAM and SSD would be the first guess. This doesn't depend on SSD and CPU alone, what generation is your CPU/motherboard?

 

My best guess for the CPU never showing 100% is that the game loop is too big for internal CPU caches.

 

Since you have it running in a virtual machine, maybe the VM has some overhead for anything coming from SSD (i.e. an additional copy operation for all data) ? Normally I would say current VM's should have almost no overhead for this but I don't know enough about that to be sure one way or the other.

 

16 x Intel(R) Xeon(R) CPU E5-2440 v2 @ 1.90GHz

Still at a loss, anything over 6 players and the starts to derp out.

[joshwa0816]

Here's my experience since starting my own server on Alpha 15, 

 

I run windows exclusively since Linux is a headache for me.

 

I would suggest a CPU upgrade. E5 2440 v2 is from 2014 with a max turbo of just 2.4ghz and ddr3 ram. it may do fine for other things but obviously is not enough for 7dtd DS based on your own words. The server minimum recommended is 2.4ghz, 3ghz is the recommended. So you're at the minimum there.

 

I run a DS on a nvme 1tb 970 evo plus, i7 10700 65w with 64gb ddr4(3000mhz) ram and with 8/8 we were doing fine while 3 were looting a tier 5. Horde night with 12 zombies per player, 3 grouped and the other 2 grouped 4km away (5 total)was not a slide show. 

 

Currently waiting on a 13700k to arrive to see how that changes things. 8/8 did drop the server fps from 70s to mid 40s sometimes dropping into the 30s but didn't affect anyone not questing the tier 5 but there was hiccups when they'd wake the whole floor. 

 

My experience was coming from a i5 8400 24gb ddr4(2400mhz) 250gb 960 evo nvme we had delays opening anything, lag and stutters with more than 5, all in different areas.  The CPU upgrade made a difference going from 8th to 10th gen, i5 to i7. Now waiting to see if 3 more generations will make a similar difference as well as 65w to 125w

 

 

I also limit the ping! anyone spending a considerable amount of time above 150 ping is booted. Had a few connected with over 270 ping that would jump to 400 and as soon as they were gone everyone recovered from what felt like lag when more than 6 were connected not including myself. This was in a18 haven't since allowed anyone connecting from across the oceans. 

 

I also play from the same computer. When over 4 are connected, I usually stay home and build. I limit myself to 40fps when doing both, as well as give the server high priority and remove myself from 2 of the 8 real cores and 2 of the hyper threaded cores 

 

I temporarily let the server (a19.1) run alone on a Xeon x5660 with 20gb ddr3 and a cheap 240gb sata SSD, while I built a newer PC over several weeks.  Horde night was awful with 6 people separated and I was playing from a separate PC this time. It could have been 1 or all 3 of the old hardware causing trouble. This setup did fine for Minecraft, left 4 dead 2 and killing floor 2 dedicated servers (not at once) this is just fyi on how I got to where I was over several alphas 

 

The CPU upgrades felt smoother but I did keep myself on an NVMe when possible.

 

Hope this helps! 

 

 

[meganoth]

12 hours ago, Bigun said:

16 x Intel(R) Xeon(R) CPU E5-2440 v2 @ 1.90GHz

Still at a loss, anything over 6 players and the starts to derp out.

 

As joswha says, that is quite an old cpu. The 16 cores don't help much as unity is still single-core and the game uses 3 more cores max.

 

You should check out the servers logfiles. There is a line with "FPS" in it occasionally. This is actually "server ticks per second" (not frames) and should generally stay above  20.

 

The game is very resource-heavy due to using voxels and each further player adds a lot of stress, especially in A20. And each further alpha added to the load. TFP hired a programmer to improve network performance so partly this may improve somewhat in the next alphas, especially the additional load from more players, but who knows how much that brings.

 

I'd say you have two choices if you want to play with more than 6: Improve the hardware or go back to an earlier alpha, the farther back the better for performance. And you can hope for alpha21 or 22 to improve the situation.

 

 

[Cr0wst0rm]

Must be somehow related to chunks, but i have no frikin clue why it happens even when you sit next to it...

 

Either the sender (server) takes too long to transfer data or receiver (player) takes too long to receive. How large is your world? How modified are all these chunks? Do you have Dynamic Mesh enabled for the server? Can you disable that? How large is your claim block size? How large is your bed size? Either many unoptimized data is being send/less but bad HW specs for either receiver/sender

[Bigun]

22 hours ago, Cr0wst0rm said:

Must be somehow related to chunks, but i have no frikin clue why it happens even when you sit next to it...

 

Either the sender (server) takes too long to transfer data or receiver (player) takes too long to receive. How large is your world? How modified are all these chunks? Do you have Dynamic Mesh enabled for the server? Can you disable that? How large is your claim block size? How large is your bed size? Either many unoptimized data is being send/less but bad HW specs for either receiver/sender

The Pregen8K map.

[Cr0wst0rm]

45 minutes ago, Bigun said:

The Pregen8K map.

Tried on different maps?

[Bigun]

I started on the Pregen10K, the Navezgane is way too small.

Checked the logs, FPS wonders between 40+ to the lowest I saw as 9'ish, but seems to have an average above 20.

 

The game runs fine until more than 4 people join. 

Also wondering about the ping.  I know a few players who exclusively use wifi and I'm wondering if that is hindering everyone else.

 

Pending that check, the lag may just have to be dealt with until I can get some funds.

[Cr0wst0rm]

1 minute ago, Bigun said:

I started on the Pregen10K, the Navezgane is way too small.

Checked the logs, FPS wonders between 40+ to the lowest I saw as 9'ish, but seems to have an average above 20.

 

The game runs fine until more than 4 people join. 

Also wondering about the ping.  I know a few players who exclusively use wifi and I'm wondering if that is hindering everyone else.

 

Pending that check, the lag may just have to be dealt with until I can get some funds.

Yes, that might be problem. See, when a player does something to world, it has to send that info to server. This server then sends this updated info to all other players.

Depending on what they do, the game can optimize the data flow (not send entire chunk all over again, but only that modified block)

Now when your friends change something, send that to wifi -> server -> other players, that creates a LOT of delay, especially if they are not wired but wireless. I used antenna on my pc because i generally try to avoid wires, but after i tried it i dont really want to go back. It will really ease the things for you and your friends i believe. 

Best way you can fix things in this scenario is therefore HW side, not SW (unless you are downloading on background or anything else (you can check task manager)). Try wires.

ALSO THIS IS IMPORTANT. If you have 4 Close players, that should be HARDER, than having 20 players spread equally around entire world, because certain players DONT have to update (they dont have shared chunks) Shared chunks are EVIL and cause lag. Again scheme (sender -> server -> all other players 4* in your case)

[meganoth]

As Cr0wst0rm  said. WiFi has been the source of problems multiple times already. 

 

 

16 hours ago, Bigun said:

Checked the logs, FPS wonders between 40+ to the lowest I saw as 9'ish, but seems to have an average above 20.

 

Was it above 20 even with more than 6 players? If yes, your servers speed is should be ok and it points to the network as the likely culprit.