A18 and A19 Server Issues

[Arcueid_Brunestud]

So in both A18 and A19 I've been having some troubles with a dedicated server. I'm running it on the same machine that i play the game on.

And generally i don't have issues, unless i get a map with fairly densely populated POIs. Base map and average maps seem fine.

 

The issue comes in the form of the server being in a state of lockup/desync. And the general symptoms are:

• Zombies will stop really moving, they may stand up if shot/interacted with but then will only do the jerky standing animation.
• Sound effects stops occuring, such as reloading and footsteps.
• Lootable containers stop being able to be opened.
• All other players look to be walking in place.
• If you disconnect you generally can't re-connect. (more info below on this one).

 

Its not really a full server crash at this point, and people are still "connected" but its as if the server is hung up on something.

One of 3 things can then occur.

1. The server fixes itself, doing a bit of a "catch-up".
2. The server doesn't fix itself, but still accepts console commands allowing a "kickall", if its still taking console commands and all players disconnect, players can then reconnect.
3. The server doesn't fix itself, and doesn't accept console commands. This requires a forcekill of the task in the task manager because it won't exit out no matter what you do.
   1. As a note when the forcekill of the task occurs all connected players are disconnected, forcing them to the title screen. Which suggests some level of connection was being maintained to the clients; just the server itself was hung up on something.

 

Now the console still shows certain things, but generally only players connecting/disconnect. I've attached a screenshot of someone attempt to connect during this time also, where it doesn't even fill in any of the info, and puts in a seemingly default IP/port into the chat.

 

The output logs don't show anything as there's never any unifying thing between the incidents, and its just usually a full halt of activities in the log, outside of player disconnections.

I've monitored the resources, but memory and cpu usage seem to be fine; did a memtest just to be certain as well.

Overall nothing else is crashing/behaving this way which is why its so odd.

 

I have a Ryzen 9 3900X 3.79 GHz 12-Core; 32 GB of DDR4 ram (running at 2133), effectively triple that in a paging file.
Both the game and the server are running on an SSD.

I've hosted other games/game servers in the past, so this hasn't been an issue before, but obviously different game could have different conflicts.

 

I've been all around trying to figure this out but these are the only real factors that we've noticed.

• It tends to happen most on super densely POI populated maps.
  • the base map, and others seemed fine, its only these rare maps that it seems to occur on.
• Happens a bit more right around the bloodmoon
  • Though cutting the max zombies on the map down didn't help; and even with no zombies around its occured.
• Happens if theres 2 or 8 players. (average is about 4 players on).
• Happens more if i'm specifically playing 7 days to die; (if i'm playing something else, and even if its something thats also memory/CPU-intensive it doens't seem to happen as much)
  • Its still happened while i'm not playing, just less frequently.
• The world does a small rollback after this, as it seems around the exact moment the server locks up thats where it last saved. Reloading the map causes a small rollback then; as this is desynced from player saves/inventory it can result in players getting duplicated or lost items depending on their last actions before this.
  • I thought this meant it was an issue with it saving, but changing the save directory, double-ensuring the directory isn't read-only, and ensuring theres exclusion paths on the game and save folder didn't seem to help.

 

Overall haven't been able to track this one down, and its been infuriating trying to troubleshoot this one; as i can't seem to find a specific cause, rhyme, or reason. The only thing I can really say is its not a network issue as as the host, doing a localhost connection to my own game doesn't work.

I even once tried moving it from the dedicated server to the main game client and running it from there. This was considerably worse, because when the freeze occured it would take the game client with it.

[SylenThunder]

Logs?

Pagefile should be set to Windows managed. You should absolutely not be manually configuring it. Doing so will cause issues not only with your OS, but with Unity.
Also, curious as to why your RAM is at 2133MHz. 

[Arcueid_Brunestud]

I'll try the paging file thing; i had gotten into just manually calculating the size because windows was handling it poorly for quite a few versions. And i never had any stability issues otherwise from it.

The ram is 2133 MHz because that's what the bios has it auto configured at; and i haven't adjusted it because if i set it any higher i'd be overclocking it.

 

I've attached the various log files between my restarts for the past couple days. I wasn't able to find anything that really stood out as generally it would just stop writing to the log when it crashed and the last thing it did was never seemingly the same.

The only thing it would write was player login/logouts as players left/attempted to rejoin during this.

output_log_dedi__2020-07-15__22-34-17.txt output_log_dedi__2020-07-15__22-39-41.txt output_log_dedi__2020-07-15__22-46-56.txt output_log_dedi__2020-07-15__22-51-25.txt output_log_dedi__2020-07-15__23-31-43.txt output_log_dedi__2020-07-16__08-16-21.txt output_log_dedi__2020-07-16__14-17-42.txt output_log_dedi__2020-07-17__01-52-52.txt output_log_dedi__2020-07-17__01-55-29.txt output_log_dedi__2020-07-17__02-26-03.txt output_log_dedi__2020-07-17__02-59-51.txt output_log_dedi__2020-07-17__03-45-00.txt

[Arcueid_Brunestud]

Here's another 2 logfiles from when it just happened, after readjusting the paging file.

output_log_dedi__2020-07-17__01-51-23.txt output_log_dedi__2020-07-17__18-29-00.txt

output_log_dedi__2020-07-17__23-02-38.txt

 

Edit; added another where it locked up as soon as i started it back up. I loaded in; someone else was still in the "creating player" step, and then you can see the blank playerID/ect.. where i disconnect and attempt to reconnnect.

output_log_dedi__2020-07-17__23-16-36.txt

[Arcueid_Brunestud]

This one is new, middle of the bloodmoon i decided to check the client side. Youc an see the "failed to load audio source" the moment the server freezes.

But right before it is "paritcle system is trying to spawn on a mesh with zero surface area" which is new.

 

As a note all players get this audio error/bug; not just my client.

output_log_dedi__2020-07-17__23-23-16.txt

[SylenThunder]

2 minutes ago, Arcueid_Brunestud said:

This one is new, middle of the bloodmoon i decided to check the client side. Youc an see the "failed to load audio source" the moment the server freezes.

But right before it is "paritcle system is trying to spawn on a mesh with zero surface area" which is new.

output_log_dedi__2020-07-17__23-23-16.txt 102.14 kB · 0 downloads

I haven't had a chance to go through the logs yet, but this issue specifically was fixed in b169.
"Particle System is trying to spawn on player mesh with zero surface area, aka console spam when player is on fire"
 

[Haelikor]

Interestingly I'm running into a very similar issue too with A19, no mods, smaller map, similar dedicated server settings.

If I'm on by myself, exiting and re-entering the game client basically does a soft reset, but it's as though things desynced. If I'm on with multiple players, we have to hard kill the server to address this desync.

Will be watching this thread.

[Arcueid_Brunestud]

I assume that is because the map was made before the latest update. That said, this is the version the server says.

[SylenThunder]

Looking at a few of your logs, I'm not seeing any issues server-side outside of probably out-dated mods. (He hasn't updated those since he released them, and both are likely broken now)

 

If your client updated to 169?

[Arcueid_Brunestud]

1 minute ago, SylenThunder said:

Looking at a few of your logs, I'm not seeing any issues server-side outside of probably out-dated mods. (He hasn't updated those since he released them, and both are likely broken now)

 

If your client updated to 169?

Client is same version, just updated today.
It was doing this yesterday before the patch was pushed to steam; and was doing this in A18; both client and server same versions. I checked with the other users on the server, and they're the same version.

 

As for mods, we can try disabling this, but we actually didn't have them the first time this happened during A18.

The two mods don't do a ton; one just modifies the range on some of the zombie attacks; and another modifies bag drop-rate and despawn duration.
The only problem with the reach one is it doesn't modify some of the newer zombies, and we had to comment out a removed zombie type as the mod gave an error when we first loaded it, commenting out that entry fixed it.

[Arcueid_Brunestud]

Tried it without the mods.  We even tried it with a single player (not me, just to rule out my game running at the same time being the problem).
They'd progress a a minute or so into the bloodmoon and things would become bad again. The telltale sign is the zombie AI just begins to degrade. They start walking into walls alot more, or they'll attack, but you'll kill one, they'll still be standing without a head, or they'll just keep walking forward without a head into a wall.
Until eventually they all just stop moving and just do their idle standing animation in place and the full desync/freeze of the server occurs.

 

Once again, we were able to get this to work fine on 3 of the 5 maps we've run. But anytime we generate a map with a very high density of POIs it seems to go south like this in both A18 and A19. I thought it may have even been number of zombies, but even reducing the max zombies below the default didn't help.

 

The best way i can describe it is its almost as if the server just chokes on things being within such close range. Especially because it doesn't seem to try and allocate any more or less ram, it usually at 8-9 gigs when it first starts/at peak activity, and when idle or low activity can be about 1.3-2 gigs. CPU usage remains fairly small.

I even during one of the attempts tried messing with priority and CPU affinity, but it was the same as all the other freezes.

 

It happens a from time to time, but it seems to be the absolute worst during the bloodmoon right now. Me being in the game or not seemed to make no difference; i sat back while 2 other players connected to the server, would play for a little bit and it would crash; which is what lead to our eventual trying it with a single person (who wasn't me).

[Arcueid_Brunestud]

Continued lockups of the server; we have just started throwing anything at it and seeing what sticks.

 

• Tried reducing the zombie spawns.
  • Because it gets the absolute worst during the bloodmoons , though it still happens outside of it.
• Tried changing the system time.
  • Because it seemed to happen more at night, but its might be just luck that our bloodmoons happen at this time; so tried changing the system time.
• tried having specific players not connect for a bit just to see if its down to who is connecting.

None of these worked. All and all i'm absolutely stumped.

[SylenThunder]

I'm a bit stumped on it too, because I've been testing on my PC that has similar specs, and I'm not having any issues at all. My R710 with two X5670's is holding up just fine running multiple servers. Tested both Windows and Linux sides as well.

 

I have seen a few odd reports that are similar to yours, but I have been completely unable to replicate it on any of the systems I have here. I've tested both dirty upgrades, and clean installs. Even mis-matched the versions, and only saw the expected issues with those.

[Arcueid_Brunestud]

I guess a random additive, our height is 193.1 trying to find similar factors. We have/made a fairly tall tower for bloodmoon defense.

 

I can't confirm/deny this; but its possible its had issues when we're at that height? Its hard to say, we're consistently there during the bloodmoon of course; and someone could be up there during the other times? We don't always have this issue in the tower, but its the only other constant i can think of right now. I'll keep monitoring for other factors.
Let me know if you need me to upload/share any other details that might help, at this point though i'm not sure if some sort of setting/factor could be there.

 

I'm really starting to think being near something that high might be a factor? Just because the server did a lockup as i approached our tower (3 screamers were at the bottom, and a few summoned zombies were there), killed 2 screamers, and then all the zombies began standing in place and it crashed. No proof, but 2 of us were on, away gathering/crafting/ect.. got back and suddenly problems.

[Arcueid_Brunestud]

Thought we might have had a breakthrough and to a small degree we did.

 

We found if we waited long enough the server would catch back up. The issue is that it would either then lock up again, or it would continue for a bit without issue. Generally staying locked up if the bloodmoon was happening.

 

Now we had some luck with players leaving one at a time, but the delay as we did that might not have been related.

 

I'm starting to think between the screamers and the bloodmoon its zombie spawn related though; and due to the map density its having some sort of problems.

 

For some reason images won't attach so i had to use links:

 

1. i saw this for one of the players which was new.

 

2.As a note when its stuck the CPU usage never really goes up at all,the memory stays about here, the disk usage is effectively 0 at this point.

[Lockjaw]

I downloaded A19 7D2D as soon as it arrived and was enjoying the online game until this most recent update. Now I'm unable to join any A19 servers without the "Exception Error" - Red-Scrolling-Errors below. I can play “single-player” but my character freezes-up and I have to return to my base in order to get moving again.

After reading the 7D2D Forums regarding this issue, I uninstalled (as suggested) but after reinstalling it, I verified the integrity of the files but still had the same issues... I'm perplexed!!! Any suggestions?

[SylenThunder]

21 minutes ago, Lockjaw said:

I downloaded A19 7D2D as soon as it arrived and was enjoying the online game until this most recent update. Now I'm unable to join any A19 servers without the "Exception Error" - Red-Scrolling-Errors below. I can play “single-player” but my character freezes-up and I have to return to my base in order to get moving again.

After reading the 7D2D Forums regarding this issue, I uninstalled (as suggested) but after reinstalling it, I verified the integrity of the files but still had the same issues... I'm perplexed!!! Any suggestions?

 

 

 

Did you wipe all of the cached data, previous profile data, and all previous generated worlds?

[Arcueid_Brunestud]

Well, the only real recreate-able thing is that non-bloodmoon is fine. And to a certain degree up until the bloodmoon is fine. The literal moment it turned 22:00 and the spooky music played things went to crap.

I've been timing out the freezes now attempting to see how long it takes the server to respond. It seems to be about 5-10 minutes  At least by myself on this map; will need to test further with more people, the server unfroze after its 3rd time at 22:06, was then good until 00:35 of the next day, got better, then froze again at about 3:40.
I'm going to try not forcekilling the task and just having people wait and see if it behaves itself better. Mostly because i think the game has to respawn all the zombies on a new load-in even if its mid bloodmoon. Meaning it could be just repeating the issue over and over anytime we don't let it sort itself out.

 

Been able to just over and over recreate this exact timing by just doing settime commands to the day of the bloodmoon (either back or forward doesn't seem to matter).

 

It usually finally clears up near the end of the night, 2:00-ish and if everyone online died (usually due to being frozen for 99% of the bloodmoon) zombies just kinda stop spawning then.

The server does seem to be fully stuck as i started paying closer attention to the clock and noticed it would rollback; even though while the server is stuck the clock ticks forward for the client-side.

Once again performance-wise nothing else on the PC is having issues, lagging,ect.. and what makes it amazing is we tend to not even see many/any zombies for the first ingame hour of the bloodmoon just  because they need to climb so many stairs, so noone even has any FPS drops or anything.

 

I keep wanting to say this is spawn related, yet I'll watch the server handle wandering hordes and big groups on non-bloodmoon days just fine. I even tested that apparently had no issues right up until the bloodmoon for players while i was asleep, which further seems to rule out it being related to me playing at the same time or not.

 

My running theory between a bloodmoon and normal, is normally the number of zombies isn't a factor, but the bloodmoon adds in more aggressive spawns and needs to automatically path them towards you. And i feel that combination between it trying to figure out where to place them in such a densely populated area of POIs and having to path around it is somehow just making the server choke (and not even attempt to allocate more resources).

[Arcueid_Brunestud]

So, we figured it out.

 

This tower i had described was the problem. We had gotten a little artsy with it, had some odd shaped blocks on theoutside; had alot of walkways on the outside, then honeycombed it to try and make some blast layers against the demolition guys. So it had alot of empty space inside, and was absurdly tall.

 

Apparently we think the game was choking on where to place zombies during the bloodmoon, and was churing through so many possible spawn locations that it just seized up.

 

We tested this by doing a bloodmoon in the tower, then a bloodmoon down the street from the tower in a random building. We then proceeded to cut off all the bells and whistles off the tower and cut as may possible spawn points that were outside of the land claim at the top. Suddenly a perfectly smooth bloodmoon.

[SylenThunder]

Good to hear you figured it out. I was really confused as to how you were having such issues on this hardware.  I even tried a slow platter drive and couldn't replicate it.

[Arcueid_Brunestud]

6 hours ago, SylenThunder said:

Good to hear you figured it out. I was really confused as to how you were having such issues on this hardware.  I even tried a slow platter drive and couldn't replicate it.

As a quick final note. This might still be worth looking into as a potential bug/improvement.

 

Mostly because the part that still strikes me as odd is the game never seemed to attempt to use more CPU or allocate more resources to get the job done. Even when it had plenty of room to do so. Which somewhat suggests some sort of bottleneck, artificial or otherwise, to limit how quickly it can do such things.

 

I mostly mention this because on the 3 other maps we did that didn't have this issue, we had bloodmoons so crazy i was getting significant FPS drops and so on just due to the sheer volume of enemies; and the server never skipped a beat; and now here due to the funky layout of a tower it was too much for even a early gamestage bloodmoon for a solo player; then after correcting things we had probably the most people we've had online in a bit do a fairly busy bloodmoon in the newly revised tower.

[SylenThunder]

9 hours ago, Arcueid_Brunestud said:

As a quick final note. This might still be worth looking into as a potential bug/improvement.

 

Mostly because the part that still strikes me as odd is the game never seemed to attempt to use more CPU or allocate more resources to get the job done. Even when it had plenty of room to do so. Which somewhat suggests some sort of bottleneck, artificial or otherwise, to limit how quickly it can do such things.

 

I mostly mention this because on the 3 other maps we did that didn't have this issue, we had bloodmoons so crazy i was getting significant FPS drops and so on just due to the sheer volume of enemies; and the server never skipped a beat; and now here due to the funky layout of a tower it was too much for even a early gamestage bloodmoon for a solo player; then after correcting things we had probably the most people we've had online in a bit do a fairly busy bloodmoon in the newly revised tower.

IMHO, I think this might be a Unity/AMD things, and is something I've been looking into for a couple of years now. As well as the issue where Unity appears to have 1080 and 2080 series GPU's for some stupid reason. (I literally get better performance on a 1060 than people are getting on 2080's.) Sadly I didn't have AMD hardware until a couple of months ago, and cannot afford a new GPU to test.