Stopping lag.. I could use some help.

[BFT2020]

5 hours ago, Jugginator said:

 

If your vehicles.dat file has been corrupted I have seen servers be brought to their knees due to invisible dupes. Known issue but deleting it will cure it. Just have everyone pick their vehicles up first, and yeah ones left out does allow the issue to happen.

Does the vehicles have to be on the player or can they be on storage?  I got a marker for a minibike that disappeared on me and couldn’t find it with god mode enable.

 

I been having issues with my bow firing as soon as I go to pull back on the string snd wondering if this is tied to it.

[Jugginator]

54 minutes ago, The_Great_Sephiroth said:

OK, I always leave my vehicles laying about and have never had issues to my knowledge. What triggers the corruption?

 

That is a gigantic, recursive can of worms, to put it lightly. Every cause is sporadic; I had to put a BOLO for a game save with a vehicle dupe in it and I spent many hours (besides the many hours investigating outside of it) debugging/breaking/investigating/reproing various issues in that save. It's nested deep in the backend, although enough information should be there to get it fixed. Pretty much the way the vehicle manager is handled sometimes causes it to mix up the stored data, sometimes you'll see a vehicle reappear where you left it before, a parked one reappearing in a different nearby spot, etc. Related to unload/reloading chunks/regions. Once it breaks, it turns into a waterfall; you'll spot many vehicles saved in the manager when you only have one or two or whatever and have invisible ones everywhere.

 

I will look at this log later on tomorrow while working myself. 

1 minute ago, BFT2020 said:

Does the vehicles have to be on the player or can they be on storage?  I got a marker for a minibike that disappeared on me and couldn’t find it with god mode enable.

 

I been having issues with my bow firing as soon as I go to pull back on the string snd wondering if this is tied to it.

 

No only vehicles handled in the world. Maybe. I do believe they are temporarily stored as idle (the x/z saved is x active z inactive) for a moment when you put them away but I'm pretty confident they aren't active as far as the vehicle manager is concerned. But, you can test this by going to where your minibike vanished and do a "killall all" in the console; if it's there invisible, it will blow up. Deleting the vehicle.dat file will only wipe vehicles placed in the world (or, invisible duped ones), so pick up the ones you want and delete it. 

 

There is also an inventory glitch, that's another issue with the manager. Both have been worked out for reproduction, though.

[SylenThunder]

Ok right at the start, the server FPS is 20. Then it gets up to the 30's.

Then after you've had a couple of people connect, you get this error...

Spoiler

2021-07-11T08:29:43 625.636 ERR Could not save file 'E:\steamcmd\7_days_to_die_server\Saves/MerlinsBeard/MerlinsCape\Player/76561199149499235.map': Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
2021-07-11T08:29:43 625.637 EXC Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
IOException: Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <567df3e0919241ba98db88bec4c6696f>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode) [0x00000] in <567df3e0919241ba98db88bec4c6696f>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode)
  at DatabaseWithFixedDS`2[KEY,DATA].Save (System.String _dir, System.String _filename) [0x00027] in <858b342e2ec84f598a70e52b667e9627>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
Logger:masterLogException(Exception)
Logger:Exception(Exception)
Log:Exception(Exception)
DatabaseWithFixedDS`2:Save(String, String)
MapChunkDatabase:SaveAsync(TaskInfo)
ThreadManager:myQueuedTaskInvoke(Object)
System.Threading.QueueUserWorkItemCallback:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
System.Threading.ThreadPoolWorkQueue:Dispatch()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()

 

The server FPS for a while varies from 29-32. Serviceable, but not a lot of room for overhead. It does occasionally pick up to 35 or 36.

 

I'm seeing a number of errors for corrupted map tiles, and the above error appears a couple more times.

 

Sharing violation is most commonly a result of not excluding the client from security software. It could also be a result of some other process accessing the files, the drive not performing well enough to handle requests in a timely fashion, or a simple permission issue.

[nafeasonto]

22 minutes ago, SylenThunder said:

Ok right at the start, the server FPS is 20. Then it gets up to the 30's.

Then after you've had a couple of people connect, you get this error...

  Hide contents

2021-07-11T08:29:43 625.636 ERR Could not save file 'E:\steamcmd\7_days_to_die_server\Saves/MerlinsBeard/MerlinsCape\Player/76561199149499235.map': Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
2021-07-11T08:29:43 625.637 EXC Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
IOException: Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <567df3e0919241ba98db88bec4c6696f>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode) [0x00000] in <567df3e0919241ba98db88bec4c6696f>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode)
  at DatabaseWithFixedDS`2[KEY,DATA].Save (System.String _dir, System.String _filename) [0x00027] in <858b342e2ec84f598a70e52b667e9627>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
Logger:masterLogException(Exception)
Logger:Exception(Exception)
Log:Exception(Exception)
DatabaseWithFixedDS`2:Save(String, String)
MapChunkDatabase:SaveAsync(TaskInfo)
ThreadManager:myQueuedTaskInvoke(Object)
System.Threading.QueueUserWorkItemCallback:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
System.Threading.ThreadPoolWorkQueue:Dispatch()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()

 

The server FPS for a while varies from 29-32. Serviceable, but not a lot of room for overhead. It does occasionally pick up to 35 or 36.

 

I'm seeing a number of errors for corrupted map tiles, and the above error appears a couple more times.

 

Sharing violation is most commonly a result of not excluding the client from security software. It could also be a result of some other process accessing the files, the drive not performing well enough to handle requests in a timely fashion, or a simple permission issue.

 

Okay so I am going to give the EVERYONE permission to the 7 Days Server directory. Start that way.

I have no security software, except  for Windows Defender, and the Firewall is off as well.

 

What else should I look at?

 

 

[SylenThunder]

31 minutes ago, nafeasonto said:

I have no security software, except  for Windows Defender

You have Defender. Ensure it is fully excluded. Even if Defender is disabled.

[nafeasonto]

4 hours ago, SylenThunder said:

You have Defender. Ensure it is fully excluded. Even if Defender is disabled.

 

Indeed, will do. 

 

Edit: Excluded the entire AppData 7 days folder, and Entire 7 days server data folder.

Edited July 19, 2021 by nafeasonto (see edit history)

[nafeasonto]

9 hours ago, SylenThunder said:

You have Defender. Ensure it is fully excluded. Even if Defender is disabled.

 

If I were to move this to an SSD RAID 0, or RAID 1, what SSD would you recommend?

Edited July 19, 2021 by nafeasonto (see edit history)

[SylenThunder]

2 hours ago, nafeasonto said:

 

If I were to move this to an SSD RAID 0, or RAID 1, what SSD would you recommend?

Honestly, don't use anything but Samsung Evo's.

1. They are supported by your RAID controller.

2. They will last for literal years.

3. They are faster than most others.

 

We have multiple R710's with all the drive slots filled. Here is what we've run on them.

• Our Evo 850's lasted almost ten years. One of which is still in use.
• I tried Kingston drives. After about a year they were at 80% wear, and speed was reduced by 40%.
• I tried Crucial drives. They didn't last 6 months. Critical drive failures across the board. Got them RMA'ed, next set did the exact same thing.
• Tried Inland Professional. Same end result as the Kingston's, except we started seeing the drops in performance at about 6-9 months. I don't recall what the health was on these when I pulled them offhand. I think it was around 90%. I do know that when we pulled them, they were almost as slow as a freaking platter drive.

So now our array is almost fully Evo 860's with long term media storage being on Seagate Ironwolf drives. (Note that WD NAS drives are not worth the metal they are printed on.)

 

Our current drive layout.

On Miata the Linux OS is on a pair of 15k SAS drives in RAID-0 (650GB). The servers run on a pair of 860's in RAID-0 (500GB). The game save data is on 4 860's in RAID-0 (1TB). And the backups are stored on a pair of Seagate Firecuda SSHD's in RAID-0 (2TB), and mirrored to Challenger.

On Challenger the Windows OS is on an Evo 850 (120GB, health is at 93%, and estimated life remaining is about two years. It's only ever been a Windows drive). The servers and save data run on three 860's in RAID-0 (750GB). Media storage and all backups are on a Seagate IronWolF (8TB). One slot is currently reserved for another IronWolf drive that we are saving up for, but prices have been pretty steep for a year now.

The next system is planned to have NVMe's, and I am considering testing Sabrient or another brand I cannot for the life of me recall the name of now.

[The_Great_Sephiroth]

I know the EVo's are good, but I got better performance on a SiliconPower NVME compared tot he same size Evo at half the price. Been using it for a few years now, no issues. My SP chip has the same NAND that the Samsung did, so...

[nafeasonto]

22 minutes ago, The_Great_Sephiroth said:

I know the EVo's are good, but I got better performance on a SiliconPower NVME compared tot he same size Evo at half the price. Been using it for a few years now, no issues. My SP chip has the same NAND that the Samsung did, so...Do 

 

 

Do you do it in a raid 0 for increased i/o?

[The_Great_Sephiroth]

Sadly my board only has a single NVME slot, so no RAID. I do run a BTRFS RAID10 array of WD Black, single-platter disks for normal games (ie: Not Ark or Cyberpunk), but those are SATA disks.

 

You do realize that NVME is right on the PCI-E bus, right? NVME is faster than RAID on SATA even with a single disk. Mine averages around 3,500~3,600MB/s reads and around 3,300~3,500MB/s writes. If I was on Linux and could use F2FS instead of NTFS it would be even higher.

Edited July 19, 2021 by The_Great_Sephiroth (see edit history)

[SylenThunder]

43 minutes ago, The_Great_Sephiroth said:

You do realize that NVME is right on the PCI-E bus, right? NVME is faster than RAID on SATA even with a single disk. Mine averages around 3,500~3,600MB/s reads and around 3,300~3,500MB/s writes. If I was on Linux and could use F2FS instead of NTFS it would be even higher.

Absolutely correct. Huge difference between NVMe and the SSD's on our servers. And additionally between NTFS and F2FS. I can hit 5802 MB/sec with our fastest array, but that's pretty much the upper limit. (It can hit 6659 MB/sec in performance tests, but the limit for the RAID controller is 6.5GB/s)  You could do that with just a pair of NVMe's in RAID-0.

A standard NVMe array could easily hit 26GB/s with a full array of 6-8 drives. 

 

The AI server I used to get the 30-player limit on was running a full deck of Intel Optane drives at 45GB/s.

 

 

[The_Great_Sephiroth]

If you setup F2FS correctly, it can get far better speeds. The key is the correct block size (-w parameter) and using a good compression (mount option compress_algorithm). Between those two it DRASTICALLY decreases writes, prolonging life of your SSD/NVME and increasing throughput. It also tends to cache a lot of stuff in RAM so it isn't reading and writing as much. It's by far the best flash-based file-system I have used. Gentoo is the only OS that seems to do it right. I tried it on PCLOS and it cannot boot from F2FS but the root partition CAN live on F2FS. I love BTRFS, but for single-SSD setups or where you run RAID and only a logical volume is exposed, F2FS is the heat.

[nafeasonto]

So if im reading this right, The server backend is very drive intensive. For now, should i run this off my ISCSi, RADZFS2 array with 10 gbps throughput or keep it on the 10k RAID10 raid until i get my ssds?

 

Because I honestly can't tell which has better I/O.

Edited July 20, 2021 by nafeasonto (see edit history)

[SylenThunder]

Just now, nafeasonto said:

So if im reading this right, The server backend is very drive intensive. For noe, should i run this off my ISCSi array with 10 gbps throughput or keep it on the 10k 10 raid until i get my ssds.

Yes. Especially if you have players spread about. It is actively loading and writing to region files that could literally be gigabytes in size. It's doing it live as changes happen, and as data needs to be loaded when people move around.  It's a large reason for the current limitations on vehicle speeds.

And you need not just fast storage, but a CPU and RAM that can keep up, and the bandwidth on the motherboard to keep up.

Our main Linux array is a little bit overkill, but I set this system up to run several 7 Days servers and a full Ark cluster.  For your particular case, I would advise at least 3-4GB/s throughput on the drives. 

[nafeasonto]

5 minutes ago, SylenThunder said:

Yes. Especially if you have players spread about. It is actively loading and writing to region files that could literally be gigabytes in size. It's doing it live as changes happen, and as data needs to be loaded when people move around.  It's a large reason for the current limitations on vehicle speeds.

And you need not just fast storage, but a CPU and RAM that can keep up, and the bandwidth on the motherboard to keep up.

Our main Linux array is a little bit overkill, but I set this system up to run several 7 Days servers and a full Ark cluster.  For your particular case, I would advise at least 3-4GB/s throughput on the drives. 

 

Yes to what? 10K drives, or ISCSI?

[SylenThunder]

7 minutes ago, nafeasonto said:

 

Yes to what? 10K drives, or ISCSI?

I still don't know what the actual bandwidth of your 10k drives are. You never said. My 15k drives get about 890MB/s, but to be fair, there are only two of them.  You've never provided the full details on your configuration there. I did ask about it in the very first post.

[nafeasonto]

18 minutes ago, SylenThunder said:

I still don't know what the actual bandwidth of your 10k drives are. You never said. My 15k drives get about 890MB/s, but to be fair, there are only two of them.  You've never provided the full details on your configuration there. I did ask about it in the very first post.

 

What specifics do you want to know though. They are 12 - Dell 600GB 10K drives in a RAID 10, off the RAID card in the PowerEdge R720.  What's a good program to test the i/o, because I am running it through a VM/datastore, so won't really get an accurate reading. 

Edited July 20, 2021 by nafeasonto (see edit history)

[SylenThunder]

On 7/18/2021 at 1:15 AM, SylenThunder said:

Run CrystalDiskMark on "Real world performance" and share the results.

Run it from the 2019 instance that your server runs from.

[nafeasonto]

12 hours ago, SylenThunder said:

Run it from the 2019 instance that your server runs from.

 

It's basically almost equal on performance, the 10K RAID beats but like a few megs.  I got SSD's on the way.  I increased the CPU's to 12 on the server, that brought up the performance, but the CPU utilization is at like 2%.  I have a feeling it's the drives that are the bottleneck at this point because I can see how much I/O is going through it from the 7 days server through the performance monitor.

 

So we'll see when they arrive. 

[nafeasonto]

17 hours ago, SylenThunder said:

Run it from the 2019 instance that your server runs from.

 

Also what are other hardware issues you have had, because you had good performance with your older chipset cpus.  So most likely this is all Hard drive issues, and if so, why isn't this game caching the items it uses from the hard drive to memory.   It's acting like a SQL database. 

[SylenThunder]

1 hour ago, nafeasonto said:

 

Also what are other hardware issues you have had, because you had good performance with your older chipset cpus.  So most likely this is all Hard drive issues, and if so, why isn't this game caching the items it uses from the hard drive to memory.   It's acting like a SQL database. 

If it cached the gigabytes of data instead of writing it live you run into two issues. 1. Massive amounts of RAM being used. 2. Every time it saves the cache, it's going to tank performance.

Take Valheim as an example here. Even with as little data as it's processing, and it being written to a very small flat database, having it cache the data causes the game to almost pause every time it does a save. And the server isn't even hosting most the work!  You can see this in action on the PS4 and Xbox versions of the game, because they perform a similar operation. Game will freeze once every three minutes because it's performing a save.

[nafeasonto]

13 minutes ago, SylenThunder said:

If it cached the gigabytes of data instead of writing it live you run into two issues. 1. Massive amounts of RAM being used. 2. Every time it saves the cache, it's going to tank performance.

Take Valheim as an example here. Even with as little data as it's processing, and it being written to a very small flat database, having it cache the data causes the game to almost pause every time it does a save. And the server isn't even hosting most the work!  You can see this in action on the PS4 and Xbox versions of the game, because they perform a similar operation. Game will freeze once every three minutes because it's performing a save.

 

Hmm interesting, why is there such a huge deal writing to RAM, compared to a hd/ssd though?

[SylenThunder]

8 minutes ago, nafeasonto said:

 

Hmm interesting, why is there such a huge deal writing to RAM, compared to a hd/ssd though?

The issue isn't writing to RAM, it's that when you're storing a lot of data in RAM, and then have to write it to the disk, it's going to be an extremely large operation.  Make a copy of the Regions folder, and see how long that takes. Now imagine it has to do that every few minutes instead of just writing to the files live.  Also look at the total size of the files. At least that much RAM would be in use by the client.

 

This is a pretty basic representation, and it could be optimized a little bit. However the current method is the most optimal.

[nafeasonto]

16 minutes ago, SylenThunder said:

The issue isn't writing to RAM, it's that when you're storing a lot of data in RAM, and then have to write it to the disk, it's going to be an extremely large operation.  Make a copy of the Regions folder, and see how long that takes. Now imagine it has to do that every few minutes instead of just writing to the files live.  Also look at the total size of the files. At least that much RAM would be in use by the client.

 

This is a pretty basic representation, and it could be optimized a little bit. However the current method is the most optimal.

I see makes sense.