Server performance issues Alpha 20 (b238)

[Wycofane]

Dear community.

 

First, sorry for my English.

I am currently hosting a public server (wycoserver.com) and I have a problem.

The server is running on 10 AMD Epyc 7452 cores and 48G RAM, the server version is Alpha 20 (latest patch).

It's running good, but I noticed some significant issues. Like that the server is running on max. 30-40% CPU usage, and the server is laggy (ingame) and I don't know what I can do to change that.

RAM usage is most of the time at 20%.

Some values, maybe they are important:

 

Time: 432.67m FPS: 35.53 Heap: 5130.5MB Max: 5767.4MB Chunks: 2978 CGO: 171 Ply: 12 Zom: 10 Ent: (288) Items: 1 CO: 12 RSS: 7394.9MB - 20 - 30% CPU usage

Time: 465.19m FPS: 22.17 Heap: 4951.4MB Max: 5767.4MB Chunks: 2908 CGO: 206 Ply: 12 Zom: 19 Ent: 52 (339) Items: 11 CO: 12 RSS: 7328.3MB - 35 - 40% CPU usage

 

Sometimes it drops to 25 FPS.

 

Please help me if you have any idea!

 

Here is my config:

GamePref.AirDropFrequency = 72

GamePref.AirDropMarker = True

GamePref.AutopilotMode = 0

GamePref.BedrollDeadZoneSize = 15

GamePref.BedrollExpiryTime = 45

GamePref.BlockDamageAI = 100

GamePref.BlockDamageAIBM = 50

GamePref.BlockDamagePlayer = 100

GamePref.BloodMoonEnemyCount = 6

GamePref.BloodMoonFrequency = 7

GamePref.BloodMoonRange = 0

GamePref.BloodMoonWarning = 1

GamePref.BuildCreate = False

GamePref.ConnectToServerIP = 127.0.0.1

GamePref.ConnectToServerPort = 26900

GamePref.CraftTimer =

GamePref.CreateLevelDim = 8

GamePref.CreateLevelName = My Level

GamePref.CreativeMenuEnabled = False

GamePref.DayCount = 3

GamePref.DayLightLength = 18

GamePref.DayNightLength = 120

GamePref.DebugMenuEnabled = False

GamePref.DebugMenuShowTasks = False

GamePref.DebugStopEnemiesMoving = False

GamePref.DropOnDeath = 1

GamePref.DropOnQuit = 0

GamePref.DynamicMeshDistance = 1000

GamePref.DynamicMeshEnabled = False

GamePref.DynamicMeshLandClaimBuffer = 2

GamePref.DynamicMeshLandClaimOnly = True

GamePref.DynamicMeshMaxItemCache = 2

GamePref.DynamicMeshMaxRegionCache = 1

GamePref.DynamicMeshUseImposters = False

GamePref.DynamicSpawner =

GamePref.EACEnabled = True

GamePref.EnemyDifficulty = 0

GamePref.EnemySpawnMode = True

GamePref.FavoriteServersList =

GamePref.FragLimit = 20

GamePref.GameDifficulty = 3

GamePref.GameGuidClient =

GamePref.GameMode = GameModeSurvival

GamePref.GameName = My Game

GamePref.GameNameClient = My Game

GamePref.GameVersion = Alpha 20

GamePref.GameWorld = Wycoserver

GamePref.HideCommandExecutionLog = 0

GamePref.LandClaimCount = 3

GamePref.LandClaimDeadZone = 30

GamePref.LandClaimDecayMode = 2

GamePref.LandClaimExpiryTime = 14

GamePref.LandClaimOfflineDelay = 0

GamePref.LandClaimOfflineDurabilityModifier = 64

GamePref.LandClaimOnlineDurabilityModifier = 8

GamePref.LandClaimSize = 101

GamePref.LootAbundance = 100

GamePref.LootRespawnDays = 2

GamePref.LootTimer =

GamePref.MatchLength = 10

GamePref.MaxSpawnedAnimals = 30

GamePref.MaxSpawnedZombies = 64

GamePref.MaxUncoveredMapChunksPerPlayer = 131072

GamePref.NoGraphicsMode = True

GamePref.PartySharedKillRange = 10000

GamePref.PersistentPlayerProfiles = False

GamePref.PlayerAutologin = False

GamePref.PlayerKillingMode = 2

GamePref.PlayerName = Player

GamePref.PlayerSafeZoneHours = 5

GamePref.PlayerSafeZoneLevel = 5

GamePref.PlayerToken =

GamePref.PlaytestBiome = 3

GamePref.RebuildMap = False

GamePref.SelectionContextMode = 0

GamePref.SelectionOperationMode = 0

GamePref.ServerAdminSlots = 3

GamePref.ServerAdminSlotsPermission = 0

GamePref.ServerDescription = [0F52BA]Die Regeln befinden sich auf der Website[-]

[0F52BA]The rules are on the website[-]

GamePref.ServerDisabledNetworkProtocols = SteamNetworking

GamePref.ServerIP = 85.214.65.203

GamePref.ServerIsPublic = True

GamePref.ServerLoginConfirmationText =

GamePref.ServerMaxAllowedViewDistance = 7

GamePref.ServerMaxPlayerCount = 32

GamePref.ServerMaxWorldTransferSpeedKiBs = 1300

GamePref.ServerName = Alpha 20 pvp - Wycoserver langzeit/longterm server

GamePref.ServerPort = 26900

GamePref.ServerReservedSlots = 0

GamePref.ServerReservedSlotsPermission = 100

GamePref.ServerVisibility = 2

GamePref.ServerWebsiteURL = https://wycoserver.com/

GamePref.ShowFriendPlayerOnMap = True

GamePref.TerminalWindowEnabled = True

GamePref.TwitchBloodMoonAllowed = False

GamePref.TwitchServerPermission = 90

GamePref.UNUSED_PlayerId =

GamePref.UserDataFolder = /root/.local/share/7DaysToDie

GamePref.WorldGenSeed = asdf

GamePref.WorldGenSize = 6144

GamePref.XPMultiplier = 100

GamePref.ZombieBMMove = 3

GamePref.ZombieFeralMove = 3

GamePref.ZombieFeralSense = 2

GamePref.ZombieMove = 0

GamePref.ZombieMoveNight = 3

GamePref.ZombiePlayers = True

 

 

Greetings Wycofane

[meganoth]

If the 30-40% CPU means the usual, then 3-4 of your 10 CPUs run at top speed which would be quite normal. The game can use only 4 CPUs as far as I know and it has a main thread which is the one that determines how fast the game can go because the tasks in there can't be easily delegated to other CPUs. I think it is the thread which is handling Unity, so it would need an engine rewrite to move some of the stuff out.

 

How many players do you have on your server when it has performance problems? Do you use mods? Are all those mods updated versions for A20 ?

 

If you want more help, please post your complete logfile with the help of pastebin.com (there is a pinned thread in here telling you how to in detail).

Edited December 23, 2021 by meganoth (see edit history)

[Wycofane]

Firstly, thank you very much for your answer.

 

So I am monitoring all of my cores at the same time, and all the 10 cores are utilized with 7 days to die threads (workload balanced between those cores). 

21 Player were on my server today running the exact same configuration.

Sometimes it has 40 FPS and then dips to 7 FPS on the server, log can be seen here https://pastebin.com/1qvcfqzf

 

I am thinking about upgrading the resources to 16 Cores and 60 GB test wise.

 

I have to cut the log for data security reasons (would be too much work to cut all IP's out)

 

The Server is currently running on Linux Ubuntu 20.04 latest patch. Alpha 20 latest patch (b238). I am using some mods, everything is for Alpha 20.

 

- Craftable Acid
- A20 HUD AGF
- AGF Backpack 72 Slots
- Ana vehicle respawner
- RELXIX Rnable Target HP Bar
- Syco A19 Farming for Alpha 20
- My own (changing Airdrop UI to 1x1 square) for vote reward
- z30K Itemstack

 

 

With 8 Players (around 35-40 server fps):

→ When 20 Players or more online, CPU usage per core about 30-40% and 7 fps.

 

 

 

Currently 9 players:

[Wycofane]

I wrote a script for better visualization of the data.

 

The script takes every INF post of the Server containing the FPS and Player count, zombie count etc.

And puts the CPU usage per core at the end of it.

 

https://pastebin.com/CBCxKe34

[meganoth]

5 minutes ago, Wycofane said:

I wrote a script for better visualization of the data.

 

The script takes every INF post of the Server containing the FPS and Player count, zombie count etc.

And puts the CPU usage per core at the end of it.

 

https://pastebin.com/CBCxKe34

 

Nice. The pastebin seems not to be from the previous logfile as I don't see any <20 FPS lines in there.

 

Server may be a bit different than client because the server doesn't have to do any unity stuff and therefore there is not this single thread dominating all the others.

But the server needs to do a lot of data shuffling. My suspicion is that data paths and especially caches are at the limit and the cpus are waiting while all the cache-lines produce misses and have to be constantly reloaded.

 

In another currently active thread in the forum a server admin says that A20 can handle a lot less players than previous alphas. I assume you know that TFP only guarantees up to 8 players per server. Unofficially you always could get away with a lot more, but that seems to have taken a hit with A20 for the time being. Possibly it gets better with more bug fixes in this experimental

 

 

 

 

[Wycofane]

Yes that is true, the script weren’t running with 20 players on the server. But it was just a small example of the utilization of all cores (or to be more precise workload balanced between the cores) 

 

I am expecting 21+ players this evening and I’m gonna log all the actions.

 

Yes I read those threads about the server performance with 8+ players.

 

Let’s hope that my problem get solved in later updates. I will still answer this thread with my result after 20+ players for others to check them (maybe other interested server admins).

 

I will do more research maybe I will find the issue and can improve the server quality.

 

Have a nice day

[Wycofane]

6 hours ago, meganoth said:

 

Nice. The pastebin seems not to be from the previous logfile as I don't see any <20 FPS lines in there.

 

Server may be a bit different than client because the server doesn't have to do any unity stuff and therefore there is not this single thread dominating all the others.

But the server needs to do a lot of data shuffling. My suspicion is that data paths and especially caches are at the limit and the cpus are waiting while all the cache-lines produce misses and have to be constantly reloaded.

 

In another currently active thread in the forum a server admin says that A20 can handle a lot less players than previous alphas. I assume you know that TFP only guarantees up to 8 players per server. Unofficially you always could get away with a lot more, but that seems to have taken a hit with A20 for the time being. Possibly it gets better with more bug fixes in this experimental

 

 

 

 

 

Here is the log with fps < 20:

https://pastebin.com/wtzmSHr4

 

Now it is running on 16 cores instead of 10 and 60 GB Ram instead of 48 GB.

 

Any suggestions? - just waiting for updates?

 

 

[meganoth]

4 hours ago, Wycofane said:

 

Here is the log with fps < 20:

https://pastebin.com/wtzmSHr4

 

Now it is running on 16 cores instead of 10 and 60 GB Ram instead of 48 GB.

 

Any suggestions? - just waiting for updates?

 

 

 

Well, you can't change the memory throughput or cache size of your server. 

 

[Wycofane]

Just now, meganoth said:

 

Well, you can't change the memory throughput or cache size of your server. 

 

 

Thats true

[Beelzybub]

12 hours ago, Wycofane said:

Any suggestions?

Try fewer cores instead of more.

[Wycofane]

16 minutes ago, Beelzybub said:

Try fewer cores instead of more.

But the server is running smoother now with 16 cores.