Stopping lag.. I could use some help.

[nafeasonto]

So I run a server with a healthy population.  After a few hours the server starts to lag. I use all the tools available, such as RAT.

Now, here is my stats, I run my server on a 2016 VM on a Dell host. THe stats are:

1.  PowerEdge R720 with Duel  E5-2697 v2 @ 2.70GHz

2.  256GB of ram, of which 16GB is dedicated to the VM.

3.  8 CPU's are dedicated to the VM.

4.  The server is running on a 10 RAID of 10K drives.

 

The server lagging after awhile makes no sense, this is with about 25 people.

 

It gets annoying to the players that I have to restart every few hours to fix this.  What is causing this lag after awhile.  Everyone's ping is below 100, some as long as 10.

 

What is causing this, the rubber banding gets worse and worse etc.  Zombies don't move.

 

Can I get some help on this please.

[SylenThunder]

21 minutes ago, nafeasonto said:

So I run a server with a healthy population.  After a few hours the server starts to lag. I use all the tools available, such as RAT.

Now, here is my stats, I run my server on a 2016 VM on a Dell host. THe stats are:

1.  PowerEdge R720 with Duel  E5-2697 v2 @ 2.70GHz

2.  256GB of ram, of which 16GB is dedicated to the VM.

3.  8 CPU's are dedicated to the VM.

4.  The server is running on a 10 RAID of 10K drives.

 

The server lagging after awhile makes no sense, this is with about 25 people.

 

It gets annoying to the players that I have to restart every few hours to fix this.  What is causing this lag after awhile.  Everyone's ping is below 100, some as long as 10.

 

What is causing this, the rubber banding gets worse and worse etc.  Zombies don't move.

 

Can I get some help on this please.

Ok for starters you shouldn't exceed 16-20 players with that hardware, and that's assuming you're running bare metal with adequate I/O.

16GB RAM is a bit low. I would recommend 32GB if you're going to be exceeding what the game is designed for. (8 players, 8k map)

Curious what kind of speeds you get with those SAS drives. That is going to be another bottleneck. Run CrystalDiskMark on "Real world performance" and share the results. I would bet that even with RAID-10 you aren't going to hit SSD speeds, and SSD's in RAID would be recommended for exceeding the supported values that much.

"I use all the tools available" is not only pretty vague, but could also be a large part of your problem. Much more information is needed.

And then there's your settings. What size map? What mods? What are the values in your config?  There are a large number of other things that could tank your performance that are left out of your post.

 

Read the Pinned thread on how to report an issue, and follow the instructions please.

[nafeasonto]

My VM is not even exceeding 5GB of that RAM, nor is the CPU utiliziation exceeding 20%, How is my hardware not good enough? That makes no sense. 

 I failed to mention my upload and download speed which is a gig, sorry about that.

 

However, my I/O on a RAID 10 is damn good. I can try pushing the server to my ISCSI unit and see if that improves, but what could be causing this. 

 

Also I wasn't spamming, I don't know who is on what forums and where.  How am I supposd to know you are on three places?

Edited July 18, 2021 by nafeasonto (see edit history)

[SylenThunder]

Modern games almost require SSD's. When you're hosing a server that's accessing literal gigabytes of data on a regular basis with constant read/write on the drive it's extremely important.  

Regarding how good your hardware is, it's not about % utilization. It's about processing speed and bandwidth.

 

As for what is causing it, I still need more data.

 

As for spamming, all three locations are official contacts. Using common sense, you would assume that support contacts are in all of those locations. (I will admit though that I do not regularly check Twitter or Facebook.) 

[nafeasonto]

18 minutes ago, SylenThunder said:

Modern games almost require SSD's. When you're hosing a server that's accessing literal gigabytes of data on a regular basis with constant read/write on the drive it's extremely important.  

Regarding how good your hardware is, it's not about % utilization. It's about processing speed and bandwidth.

 

As for what is causing it, I still need more data.

 

As for spamming, all three locations are official contacts. Using common sense, you would assume that support contacts are in all of those locations. (I will admit though that I do not regularly check Twitter or Facebook.) 

 

This was your message from 2020:

 

 

"I'm currently running a Dell R710 with two Xeon X5675's at 3.07GHz with 80GB RAM and SSD's in RAID0 for the client. I can host 8 servers with 20 players, and it maxes out at about 45% system load. The setup cost me about $550.

I have run a single server on an i7 3.2GHz with 16GB RAM and a single SSD before. It was fine up to about 10 players with default horde size. If I increased it to 16 per wave, we'd start maxing out the capabilities until I reduced players to about 6."

 

If you can handle 20 people on that old CPU, i can easily handle more.  So I am confused how you say my hardware is too old to handle more then 10 players, while you handled multiple servers.

 

What is the EXACT information you want to know?

Edited July 18, 2021 by nafeasonto (see edit history)

[SylenThunder]

OK, the CPU's in comparison, yours is hardly faster when it comes to single-core processing speed. Which is what counts for the main client thread.

For drive I/O we've for four Evo 860's in RAID-0. We're exceeding 45k IOPS read/write in real-world values. (Not raw performance) 

And it doesn't matter the number of servers very much because each is running on it's own set of cores. The only real limitation there is hard disk performance and RAM speeds.

Speaking of RAM, how is yours configured? With these servers, that is going to make a huge impact on performance. With our RAM in triple-channel we achieve maximum throughput. (quad is actually slower) We're also clocking these CPU's a bit higher under load.
I'll also note that we are running bare metal headless Linux. That alone is going to add a performance improvement of about 10-15% over a bare metal Windows machine, and you're adding an extra later to the mix on top of that with the VM.

Once you add all of this together, our old CPU's are likely going to out-perform your slightly less old CPU by a good bit when you put all the parts together.

 

I never said your hardware was too old to handle more than 10 players. You made that up yourself.
I said "you shouldn't exceed 16-20 players with that hardware", and that statement is based on months of testing with what the client is capable of on a wide variety of hardware. And lets be crystal clear here. The biggest limitation is the client and the fact that this is a Voxel game. I cover that in some detail in the Pinned FAQ thread. You've apparently done some research, but somehow missed all the really important data.

 

Exact information? I cover this in the Pinned thread I already directed you at, and provided you tips on the details I need. Configuration, full setup, ect. The thread even gives you instructions on how to get it, and how to provide it properly so you don't make a mess out of the forum.

[Beelzybub]

1 hour ago, nafeasonto said:

What is the EXACT information you want to know?

A full log.

What Hypervisor?

What else is the server working on?

Antivirus?

Are you monitoring the network/router?

What size world?

Mods?

Server managers?

Edited July 18, 2021 by Beelzybub (see edit history)

[The_Great_Sephiroth]

I am curious also. I have an old i5-2400 with 16GB of RAM that I ran 7 Days on for ages without a problem, but we had eight players max. It was an old desktop system, an Optiplex 960 maybe? I ran it through A18.

 

My question would be this. Is it hardware-induced lag (CPU spiking, RAM exhausted, etc) or network-induced? Have you watched the CPU, RAM, and disk on the VM from within the VM while the lag (assuming latency spikes) occur?

[nafeasonto]

16 hours ago, SylenThunder said:

OK, the CPU's in comparison, yours is hardly faster when it comes to single-core processing speed. Which is what counts for the main client thread.

For drive I/O we've for four Evo 860's in RAID-0. We're exceeding 45k IOPS read/write in real-world values. (Not raw performance) 

And it doesn't matter the number of servers very much because each is running on it's own set of cores. The only real limitation there is hard disk performance and RAM speeds.

Speaking of RAM, how is yours configured? With these servers, that is going to make a huge impact on performance. With our RAM in triple-channel we achieve maximum throughput. (quad is actually slower) We're also clocking these CPU's a bit higher under load.
I'll also note that we are running bare metal headless Linux. That alone is going to add a performance improvement of about 10-15% over a bare metal Windows machine, and you're adding an extra later to the mix on top of that with the VM.

Once you add all of this together, our old CPU's are likely going to out-perform your slightly less old CPU by a good bit when you put all the parts together.

 

I never said your hardware was too old to handle more than 10 players. You made that up yourself.
I said "you shouldn't exceed 16-20 players with that hardware", and that statement is based on months of testing with what the client is capable of on a wide variety of hardware. And lets be crystal clear here. The biggest limitation is the client and the fact that this is a Voxel game. I cover that in some detail in the Pinned FAQ thread. You've apparently done some research, but somehow missed all the really important data.

 

Exact information? I cover this in the Pinned thread I already directed you at, and provided you tips on the details I need. Configuration, full setup, ect. The thread even gives you instructions on how to get it, and how to provide it properly so you don't make a mess out of the forum.

So here is some details:

 

1.  PowerEdge R720 Running Dual 2690V2s.  10 CPUs are dedicated to the VM

2.  16GM of the Ram is dedicated the server.

3.  Server is now running on my ISCSI 10Gb Raid.  No change at all in performance.

4.  Esxi Ver 6.7

5.  Windows 2019 VM (made a mistake wasn't 2016).

 

Mods running:

 

https://7daystodiemods.com/snufkins-community-pack-server-side-vehicles/ (Snufkin’s Community Pack Server Side Vehicles)

Snufkin_CommunityPack_ServerSideVehicles_A19_2021June12

Snufkin_CustomZombies_A19-Stable_2020Oct22

Alloc's WEb and Map Rendering/Command Extensions/Command Func

BCManager

 

Logs:

 

For some reason the log files are not populating.   The latest one i have is 6/27.  So that needs to be fixed.

 

6.  max people of the server is 25, this lag starts just with 10. 

 

7.  CPu/RAM ulitlization are not even close to max on the Guest, or the Host system indictation.

8.  FPS on RAT shows 6.  I can't believe it's that low.

 

Please ask for more info. 

 

Does leaving Vehicles on the ground, do anything bad?

 

Picture of Server SETTINGS:

 

https://ibb.co/hcXNXZQ

https://ibb.co/HHjPHLJ

 

Edited July 19, 2021 by nafeasonto (see edit history)

[Jugginator]

4 minutes ago, nafeasonto said:

Does leaving Vehicles on the ground, do anything bad?

 

If your vehicles.dat file has been corrupted I have seen servers be brought to their knees due to invisible dupes. Known issue but deleting it will cure it. Just have everyone pick their vehicles up first, and yeah ones left out does allow the issue to happen.

[SylenThunder]

Logs are kind of critical.  The FPS in that screenshot with 15 players is absolutely abysmal. 

 

Ideal FPS is 35-40. Anything below 25 is an indication of a bottleneck, and anything below 20 will start having a serious impact on performance. Yours isn't even at 5.

 

Leaving the vehicles on the ground isn't inherently bad. However many of the servers exceeding known stable values have had issues with them. A few have a job that wipes the vehicles.dat at every restart and just tell their players not to leave them laying about.

[nafeasonto]

4 minutes ago, SylenThunder said:

Logs are kind of critical.  The FPS in that screenshot with 15 players is absolutely abysmal. 

 

Ideal FPS is 35-40. Anything below 25 is an indication of a bottleneck, and anything below 20 will start having a serious impact on performance. Yours isn't even at 5.

 

Leaving the vehicles on the ground isn't inherently bad. However many of the servers exceeding known stable values have had issues with them. A few have a job that wipes the vehicles.dat at every restart and just tell their players not to leave them laying about.

Okay so far, erasing the vehicles got my FPS back to 20.  But there has to be another bottleneck somewhere, where do I enable logs, because I don't ever remember disabling those. 

7 minutes ago, SylenThunder said:

Logs are kind of critical.  The FPS in that screenshot with 15 players is absolutely abysmal. 

 

Ideal FPS is 35-40. Anything below 25 is an indication of a bottleneck, and anything below 20 will start having a serious impact on performance. Yours isn't even at 5.

 

Leaving the vehicles on the ground isn't inherently bad. However many of the servers exceeding known stable values have had issues with them. A few have a job that wipes the vehicles.dat at every restart and just tell their players not to leave them laying about.

 

i got the log, let me pastebin it:

 

https://pastebin.com/EabpX6ys

[SylenThunder]

3 minutes ago, nafeasonto said:

Okay so far, erasing the vehicles got my FPS back to 20.  But there has to be another bottleneck somewhere, where do I enable logs, because I don't ever remember disabling those. 

It's part of the batch file that starts the server, and is written so that each log is dated. Argument is -logfile=

[nafeasonto]

2 minutes ago, SylenThunder said:

It's part of the batch file that starts the server, and is written so that each log is dated. Argument is -logfile=

See above, log file in a pastebin. I moved the directory, was looking in the default accidently. 

[SylenThunder]

The log doesn't give me much to work with. The server is shutdown before it even finishes initializing.

 

Curious question though, how many players are in Players.xml and the Players folder?

[nafeasonto]

455 Files in Players folder, divide by 2 for BAKup, so 220

 

Players.xml is 477 lines. 

 

Of which, 134 player id= lines are

Edited July 19, 2021 by nafeasonto (see edit history)

[nafeasonto]

12 minutes ago, SylenThunder said:

The log doesn't give me much to work with. The server is shutdown before it even finishes initializing.

 

Curious question though, how many players are in Players.xml and the Players folder?

 

I have lots of Output logs, which specifically you want.  There are a lot of timestamps.  Do you want multiple?

 

Also answered your question above. 

[SylenThunder]

Ok, so the issue probably isn't a bloated player list.

 

The best log would be one where you start experiencing the issue, so we can try to determine what's going on in the server when the issue starts to occur.

[nafeasonto]

4 minutes ago, SylenThunder said:

Ok, so the issue probably isn't a bloated player list.

 

The best log would be one where you start experiencing the issue, so we can try to determine what's going on in the server when the issue starts to occur.

 

It gets 10x worse on horde night.  Okay thed issue started happening a week ago, the earliest log i have is 7-11. I am going to paste it here in a minute. 

9 minutes ago, SylenThunder said:

Ok, so the issue probably isn't a bloated player list.

 

The best log would be one where you start experiencing the issue, so we can try to determine what's going on in the server when the issue starts to occur.

 

Here is the log file:

 

https://drive.google.com/file/d/1lRl_859Npby3ju-qLIuADgzDHPAK5O-e/view?usp=sharing

 

Too big for pastebin. 

Edited July 19, 2021 by nafeasonto (see edit history)

[nafeasonto]

26 minutes ago, SylenThunder said:

Ok, so the issue probably isn't a bloated player list.

 

The best log would be one where you start experiencing the issue, so we can try to determine what's going on in the server when the issue starts to occur.

 

I could just send you an entire zip file of all the logs, if you can't find anything in the one i just linked to you. The big one. 

[Jugginator]

2 hours ago, SylenThunder said:

Leaving the vehicles on the ground isn't inherently bad. However many of the servers exceeding known stable values have had issues with them. A few have a job that wipes the vehicles.dat at every restart and just tell their players not to leave them laying about.

 

Not inherently bad, no, but the corrupted vehicle.dat file causing the vehicle issues isn't related to big servers, can repro it with one person in a SP game too. Just conjecture, gotta catch up with the rest.

[SylenThunder]

45 minutes ago, Jugginator said:

 

Not inherently bad, no, but the corrupted vehicle.dat file causing the vehicle issues isn't related to big servers, can repro it with one person in a SP game too. Just conjecture, gotta catch up with the rest.

I see.  

 

2 hours ago, nafeasonto said:

Here is the log file:

 

https://drive.google.com/file/d/1lRl_859Npby3ju-qLIuADgzDHPAK5O-e/view?usp=sharing

 

Too big for pastebin. 

File is locked. You can use justpaste.it or open it up.  

Work is especially busy tonight, so may take me a bit to get through it.

[Pichii]

23 hours ago, nafeasonto said:

So I run a server with a healthy population.  After a few hours the server starts to lag. I use all the tools available, such as RAT.

Now, here is my stats, I run my server on a 2016 VM on a Dell host. THe stats are:

1.  PowerEdge R720 with Duel  E5-2697 v2 @ 2.70GHz

2.  256GB of ram, of which 16GB is dedicated to the VM.

3.  8 CPU's are dedicated to the VM.

4.  The server is running on a 10 RAID of 10K drives.

 

The server lagging after awhile makes no sense, this is with about 25 people.

 

It gets annoying to the players that I have to restart every few hours to fix this.  What is causing this lag after awhile.  Everyone's ping is below 100, some as long as 10.

 

What is causing this, the rubber banding gets worse and worse etc.  Zombies don't move.

 

Can I get some help on this please.

I had/have alot of issues with EAC and lag sometimes, depending on the 7d2d patch, steam patch and EAC updates and windows .netframework probz.
My suggestion would be to make sure windows has allllll its updates, disable EAC (At least to test and see if it reduces lag, if it does, probably EAC bottlekneck, nothing you can do cept disable EAC) and double check steams bad self for updates.
If youve already done that, ignore me and best of luck :x (Im assuming windows? Again, ignore me if youve already checked this stuff.)

Edited July 19, 2021 by Pichii (see edit history)

[The_Great_Sephiroth]

OK, I always leave my vehicles laying about and have never had issues to my knowledge. What triggers the corruption?

[nafeasonto]

1 hour ago, SylenThunder said:

I see.  

 

File is locked. You can use justpaste.it or open it up.  

Work is especially busy tonight, so may take me a bit to get through it.

 

Unlocked

 

https://drive.google.com/file/d/1lRl_859Npby3ju-qLIuADgzDHPAK5O-e/view?usp=sharing

 Yeah take your time. Thanks for the help./