Jump to content
madmole

Alpha 19 Dev Diary

Recommended Posts

1 hour ago, Blake_ said:

I forgot to show you this beauty on my performance post. Take a look at those gorgeous bumps of "606 ms saving the chunks" ((EDIT: last line of code in the console)) while that amount (135) should take on average no more than 80ms. It's a common freeze when walking through the world and quite intense.

 

20200923184019_1.thumb.jpg.9921da4c5eb025e6961680b9a2ec921b.jpg

Except that save message only happens on a full save, like when you Exit the world, do a save in the prefab editor or generate a world.

I am flying around the world and blowing the ground up with dynamite and that code is never called.

 

Are you using any mods?

  • Like 2

Share this post


Link to post
Share on other sites

On the topic of performance, i posted some benchmark results in #general support if anyone is interested. The tests where done on 2 systems, amd and intel both tests with 2 graphics option configs. One system was also tested in SLI and many cores vs fewer cores, smt on/off. Some interesting results, but best viewed on desktop the tables get a bit wide XD

  • Thanks 1

Share this post


Link to post
Share on other sites
18 hours ago, faatal said:

Except that save message only happens on a full save, like when you Exit the world, do a save in the prefab editor or generate a world.

I am flying around the world and blowing the ground up with dynamite and that code is never called.

 

Are you using any mods?

EDIT: This post is cool, but read my next one which has "lotz" of screenshots and hopefully me and all the potato guys can get help (performance help). Also, whenever I said chunk saving I really meant (and expressed myself poorly) a "PAUSED game" telltale of an inmediate previous bump directly proportional to DECO handling/writing, which does happen without pausing.

 

No mods, Never mods. Always Vanilla. Full wipe. The game is paused AFTER the fact, so that 606 is a saving and the bump did come just before (DECO handling). I have a lot of screenshots with 100-120 ms console entries of chunks saved in pauses after the bumps so  that one was one of a kind. Edit:I did not leave the world nor did I just got in the game. Three things are for sure:

 

1. Anything more than a paused chunk saving time of 80ms is a tip for the previously handled loading/unloading of assets causing big momentary freezes in real time.

2. The biggest freezes usually happen when the yellow "no child for position" warnings are involved PLUS the DECO writing notification (always together),resulting in said paused chunk notification surpassing the 70-80ms mark. When there's no warnings, it can still happen, but beefy writes are rare when there's little to handle.

3. When digging looking down, there are no bumps and after pausing the writing times are just around 60ms from 40-60 chunks. 

 

Other stuff

Spoiler

Textures are too high res in lower sizes. They should be sluggish BUT not the readable stuff. Readable stuff shouldn't downscale more than half. After that it should stay the same to not affect experience and, ya know, be readable.

 

When I say we need washed out low textures I am talking about the main texture map with the details and colours.

 

The issue affects Zd texture scaling too. Lowest is lowest and lower end texture streaming too . Cardboard washed until in my face.

I hope this helps. I have many more screenshots, but all similar to the above except 100-120 paused notifications  just after the bumps instead of that one entry of 606 (more time without pausing) . I need to point out though, I paused AFTER the fact, so each chunk saving is a pause. The saving time directly tells how big of a bump the DECO assets caused. Neat trick, if not very helpful.

Edited by Blake_ (see edit history)

Share this post


Link to post
Share on other sites

The new door physics prototype look awesome. Any chance we'll be able to shoot through the holes that are created when the door is damaged?

  • Like 3

Share this post


Link to post
Share on other sites
15 hours ago, madmole said:

As long as I can get Fallout 5, Elderscrolls 6 on PC and mods I'll be happy.

How about Starfield? :)

Share this post


Link to post
Share on other sites
2 hours ago, MechanicalLens said:

The new door physics prototype look awesome. Any chance we'll be able to shoot through the holes that are created when the door is damaged?

WHERE? WHAT? WHERE DID YOU SEE THAT? SPEAK!!! :yell: :eek2:

Share this post


Link to post
Share on other sites
4 minutes ago, Jost Amman said:

WHERE? WHAT? WHERE DID YOU SEE THAT? SPEAK!!! :yell: :eek2:

Most likely MechanicalLens is a tester for TFP :)

  • Thanks 1

Share this post


Link to post
Share on other sites
2 hours ago, MechanicalLens said:

The new door physics prototype look awesome. Any chance we'll be able to shoot through the holes that are created when the door is damaged?

Any physics-based asset maintains the collisions of each door piece both intact and in shambles, resulting in YES, you can shoot across any hole in the door. That's one of the benefits of the technology actually. Also, It's lots of fun.  In other words, no more swapping door models, but rather "simulating" a broken one by doing a "smarter localized swapping/breaking" also called " "physics simulation" as is very close to the real thing. Yes, it affects performance and no, not much.

Edited by Blake_ (see edit history)
  • Like 1

Share this post


Link to post
Share on other sites
33 minutes ago, Annihilatorza said:

Most likely MechanicalLens is a tester for TFP :)

Nope, we watch these from here 

 

  • Like 4
  • Thanks 2

Share this post


Link to post
Share on other sites
1 hour ago, Annihilatorza said:

Most likely MechanicalLens is a tester for TFP :)

Yes, I am indeed a tester for TFP. I'm currently testing a new mechanic where one in every five shots, your gun jams and explodes in your hands, dealing of -50 HP of damage. It's a really fun new mechanic and one that I'm certain all of you are going to enjoy. ;)

 

 

 

(jk on both of these, nope, just a regular dude, and the Twitter link is above, thanks beerfly)

  • Haha 2

Share this post


Link to post
Share on other sites

Hi again @faatal .First of all Thx for reading and interacting on this subject. Your help is very welcome. I'm decided to keep on the feedback while I try to painfully test and, after going through the screenshots I noticed something that might help. The time between DECO writes seems to be a minute. If you bring it down to 10 or 15 seconds it should have a bigger overall overhead but WAAAY more stable framerate as there's  four to six times less to write. As the problem is worse when the load is higher (when there are chunks with buildings) the way to go might be to  just increase the writing frequency in smaller batches.

 

In theory, what I'm suggesting should work and reduce almost any writing-related bumps except the theoretical high time of the worse case scenario (606ms worth of stuff in real time when paused).

 

In my later test I encountered several 189ms bumbs, so I could say that the 606ms one is a 1% case (it did happen, is just that I paused after, more of that below) I also defragmented as you suggested and it did not do much, but yes, a little bit better, I would say a 5ms improvement on average tops (It's only been 2 weeks since my last one).

 

As you can see, the 606ms saving one was a big bumb, and it really makes reference to all the DECO handling and stuff happening before the fact,  but because I paused it after due to that pesky Zombie in order to take the screenshot then the feedback might be contaminated. I have more screenshots, no threats and in real time. Each chunk saving is from pausing I guess, but there's a relation between the size, the bumps and the DECO handling.

Spoiler

 20200923184019_1.thumb.jpg.e5137f6ae43695a54274b39cbea6b3d4.jpg20200923183704_1.thumb.jpg.c394bb577c39b3f27750bb56fae5834e.jpg20200923172932_1.thumb.jpg.425d059cdf2d14e7e9e2b67024b42925.jpg20200923170309_1.thumb.jpg.3e7b03ebce973019927bd006676098db.jpg

 

As you can see in the 3rd screenshot, there's no chunk writing because I didn't pause, but the Warnings hint at the problem and in every screenshot I did get a bumb.

 

The lightest one being 120ms (last screenshot)  but anything over 70ms after the paused fact is a direct tell of an incremental bump of proportional intensity just before  (lots of DECO being handled at once).

 

CONCLUSION: Plz, it sounds counterintuitive, but increase the DECO writing (saving?) frequency while spreading the load/unload. There's got to be a point in which if the frequency is too high the performance takes too big of a hit, but if you save DECO, say, each 10-15 seconds, the average fps could get just a small drop in exchange for a smoother experience. That,  plus your recent 19.2 fixes, should do it for a19 performance and beyond.

 

If it works, the huge 606ms rare drop "tip paused result" should still give away the  big freeze moment, but at 10 seconds DECO handling there would be 6 times less stuff to write so the DECO impact could theoretically be reduced. From there, you can try 5 second DECO saving/ writing and see what happens. 5-second writing could be taxing OR sustainably bearable. I would take the last one any day.

 

Again, thx and I hope this helps. Edit: Gamesparks and EAC is off in all the tests. The issue happens in ALL game modes.

 

Edit: I suspect that the DECO handling is heavily microhandled and I do still tend to mistake console notifications for the real thing. The core is right though. You need to micromanage it so it loads/unloads more often and in THE LEAST cuantities possible, resulting in a a more stable experience with not so big of a fps average hit.

 

Edited by Blake_ (see edit history)

Share this post


Link to post
Share on other sites
2 hours ago, beerfly said:

Nope, we watch these from here 

 

Wow! That looks friggin' A-Mazing!! :eek2:

  • Like 1

Share this post


Link to post
Share on other sites

Does this new door tech include one way doors and keyed locking with lock picking or is it just nicer looking/breaking doors?

Share this post


Link to post
Share on other sites

i am kinda enjoying that new door breaking animation, still looks a bit clunky but im not much for graphics over fun gameplay, would we soon have this same kinda breaking effect for walls and other types on blocks down the road or strictly doors?  I imagine it would work well with windows too

Share this post


Link to post
Share on other sites
On 9/23/2020 at 7:00 PM, faatal said:

I actually enjoy optimizing code. One of my favorite things to do, because sometimes you do find that big win and multiple small gains together do lead to moderate gains.

You often don't know how it will turn out until you do it. That is life.

I see you.

 

In my profession there are also things that awaken my "investigative curiosity" why some things broke apart, or why it held together for quite some time despite being installed in a "not-so-inspired" way, while other things that were concerted perfectly and should have lasted forever, have worn out pretty fast without no obvious reason. 

Most of the time it amounts to some boring/stupid reasons.

But sometimes you discover the hidden real reason that has the potential to break down a seemingly perfect machinery, and revealing that (and so being able to prevent future failures) is such a satifying thing, hard to find words for!

 

So no offense meant in my previous post, your work is appeciated much!

I am happy you see it as a satisfying challenge, and not  - as i have assumed/represented - hard work for little benefit

 

Good hunting! 

 

 

10 hours ago, beerfly said:

Nope, we watch these from here 

 

Oh YEAH really BIG THX to TFP this is exactly the thing I was hoping for!!!

 

*dancing/jumping naked on the couch while eating a pizza*

 

Edited by meilodasreh (see edit history)
  • Like 1

Share this post


Link to post
Share on other sites
15 hours ago, Blake_ said:

Hi again @faatal .First of all Thx for reading and interacting on this subject. Your help is very welcome. I'm decided to keep on the feedback while I try to painfully test and, after going through the screenshots I noticed something that might help. The time between DECO writes seems to be a minute. If you bring it down to 10 or 15 seconds it should have a bigger overall overhead but WAAAY more stable framerate as there's  four to six times less to write. As the problem is worse when the load is higher (when there are chunks with buildings) the way to go might be to  just increase the writing frequency in smaller batches.

 

In theory, what I'm suggesting should work and reduce almost any writing-related bumps except the theoretical high time of the worse case scenario (606ms worth of stuff in real time when paused).

 

In my later test I encountered several 189ms bumbs, so I could say that the 606ms one is a 1% case (it did happen, is just that I paused after, more of that below) I also defragmented as you suggested and it did not do much, but yes, a little bit better, I would say a 5ms improvement on average tops (It's only been 2 weeks since my last one).

 

As you can see, the 606ms saving one was a big bumb, and it really makes reference to all the DECO handling and stuff happening before the fact,  but because I paused it after due to that pesky Zombie in order to take the screenshot then the feedback might be contaminated. I have more screenshots, no threats and in real time. Each chunk saving is from pausing I guess, but there's a relation between the size, the bumps and the DECO handling.

  Reveal hidden contents

 20200923184019_1.thumb.jpg.e5137f6ae43695a54274b39cbea6b3d4.jpg20200923183704_1.thumb.jpg.c394bb577c39b3f27750bb56fae5834e.jpg20200923172932_1.thumb.jpg.425d059cdf2d14e7e9e2b67024b42925.jpg20200923170309_1.thumb.jpg.3e7b03ebce973019927bd006676098db.jpg

 

As you can see in the 3rd screenshot, there's no chunk writing because I didn't pause, but the Warnings hint at the problem and in every screenshot I did get a bumb.

 

The lightest one being 120ms (last screenshot)  but anything over 70ms after the paused fact is a direct tell of an incremental bump of proportional intensity just before  (lots of DECO being handled at once).

 

CONCLUSION: Plz, it sounds counterintuitive, but increase the DECO writing (saving?) frequency while spreading the load/unload. There's got to be a point in which if the frequency is too high the performance takes too big of a hit, but if you save DECO, say, each 10-15 seconds, the average fps could get just a small drop in exchange for a smoother experience. That,  plus your recent 19.2 fixes, should do it for a19 performance and beyond.

 

If it works, the huge 606ms rare drop "tip paused result" should still give away the  big freeze moment, but at 10 seconds DECO handling there would be 6 times less stuff to write so the DECO impact could theoretically be reduced. From there, you can try 5 second DECO saving/ writing and see what happens. 5-second writing could be taxing OR sustainably bearable. I would take the last one any day.

 

Again, thx and I hope this helps. Edit: Gamesparks and EAC is off in all the tests. The issue happens in ALL game modes.

 

Edit: I suspect that the DECO handling is heavily microhandled and I do still tend to mistake console notifications for the real thing. The core is right though. You need to micromanage it so it loads/unloads more often and in THE LEAST cuantities possible, resulting in a a more stable experience with not so big of a fps average hit.

 

I thought the deco writes were 0 as the log says, so it was confusing what was going on, but it turns out the list is cleared before the log happens, so it is writing out, in a 8k world, about 71000 entries, totally over 1 MB. Tried on my HD, but still not horribly slow, then on a flash drive, which gave a 7 second write time! The save timer is every minute, but there is a dirty check, so it does not always do the save.

 

The deco save code was changed a few months ago by another programmer to fix a problem with old saves, so it might have been only doing partial saves before, but since it does the whole thing now in any event, and on the main thread, it does lead to a large frame spike.

 

The simple solution was to write it into a memory stream, which is faster than a file stream anyway, then another thread does the actual file save, so minimal frame spike. This probably won't make 19.2, but hopefully 19.3.

 

Thanks

  • Like 13
  • Thanks 3

Share this post


Link to post
Share on other sites
2 hours ago, faatal said:

The simple solution was to write it into a memory stream, which is faster than a file stream anyway, then another thread does the actual file save, so minimal frame spike. This probably won't make 19.2, but hopefully 19.3.

Beautiful!

Share this post


Link to post
Share on other sites
5 hours ago, faatal said:

I thought the deco writes were 0 as the log says, so it was confusing what was going on, but it turns out the list is cleared before the log happens, so it is writing out, in a 8k world, about 71000 entries, totally over 1 MB. Tried on my HD, but still not horribly slow, then on a flash drive, which gave a 7 second write time! The save timer is every minute, but there is a dirty check, so it does not always do the save.

 

The deco save code was changed a few months ago by another programmer to fix a problem with old saves, so it might have been only doing partial saves before, but since it does the whole thing now in any event, and on the main thread, it does lead to a large frame spike.

 

The simple solution was to write it into a memory stream, which is faster than a file stream anyway, then another thread does the actual file save, so minimal frame spike. This probably won't make 19.2, but hopefully 19.3.

 

Thanks

Yeah, DECO was saying 0 but I could "feel" otherwise. Output log is quite revealing too because it registers the actual writen entries.

 

Despite my crazy explanation and boring journalistic feedback, you managed to find a lead, so I'm glad that I could help in the end. I will be testing 19.2 and 19.3 in detail too, hopefully posting feedback with less words this time around. 

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites

Will hatches be given the same treatment as doors will in the future? New physics, one stage instead of multiple stages, etc.

 

Also, will vault hatches and/or fully upgraded iron hatches be nerfed at some point? Using them as a retractable wall has become the new pole wall into terms of a meta defense.

One option could be increasing the amount of damage that zombies deal against hatches by 6x. Just spitballing here, in fact this entire point I brought up is nothing more than a giant spitball.

  • Like 1

Share this post


Link to post
Share on other sites
On 9/23/2020 at 4:55 AM, faatal said:

Do you know that this is a team with many different skill sets, like pretty much every game team? Most people on the team can't fix FPS issues, so they do spent their time working on their "stupid" stuff.

 

FPS issues don't have an "I Win" button you can press to make it all better. It takes lots of time to analyze performance and find bottlenecks, which then often have no easy fix, but to rewrite and retest something, which often results in minimal gains. Features have maximum limits at which they can be optimized to run.

 

I've actually been spending half my time the last few weeks doing that and cleaning up the project of old/unused files. For example, I found an old unneeded water depth camera, that was rendering nothing each frame. After time spend researching it, removing and testing, I saw a 3% FPS gain. Not much. Path grid updates have also been optimized, which is not something increasing overall FPS, but reduces frame spikes. Terrain shader has been updated for good gains. Tree batching/instancing is being analyzed and tweaked. To get large gains, you have to do that over and over on many systems. Today I started moving many of those improvements to 19.2 for testing, so the next exp will have them.

Thank you very much Faatal, now that you're working optimizing terrain shader please take a look at this thread, there is a massive 40% fps drop from low to med terrain quality setting:

 

 

 

Edited by HeLLKnight (see edit history)

Share this post


Link to post
Share on other sites
On 9/23/2020 at 5:55 AM, faatal said:

Do you know that this is a team with many different skill sets, like pretty much every game team? Most people on the team can't fix FPS issues, so they do spent their time working on their "stupid" stuff.

 

FPS issues don't have an "I Win" button you can press to make it all better. It takes lots of time to analyze performance and find bottlenecks, which then often have no easy fix, but to rewrite and retest something, which often results in minimal gains. Features have maximum limits at which they can be optimized to run.

 

I've actually been spending half my time the last few weeks doing that and cleaning up the project of old/unused files. For example, I found an old unneeded water depth camera, that was rendering nothing each frame. After time spend researching it, removing and testing, I saw a 3% FPS gain. Not much. Path grid updates have also been optimized, which is not something increasing overall FPS, but reduces frame spikes. Terrain shader has been updated for good gains. Tree batching/instancing is being analyzed and tweaked. To get large gains, you have to do that over and over on many systems. Today I started moving many of those improvements to 19.2 for testing, so the next exp will have them.

@faatal

Thank You for telling us all this. I think it really calms some people (including me :)) more than just generalized promises.

This is how i always imagined real communication with game developers.

Edited by n2n1 (see edit history)

Share this post


Link to post
Share on other sites

Hello again @faatal . Another a19.1 b8 performance test :

 

This time around I detected a problem with the AIDirector . AIDirector aka the "wandering horde troll" knows that spawning entities all at once results in a freeze of 5-15 seconds to an already taxed cpu (that hurt). 

The freeze was the hardest I've ever seen. 4 entities at the same second. My 2.7ghz 4C/8T said nope for quite a while there.

 

Here's the screenshot, check that 19:52:26 timestamp :

Spoiler

20200925195241_1.thumb.jpg.07f5e1cad8ca0d67cd7b208896087a2f.jpg

 

I don't know if it's an easy fix. The solution would be to force the AIDirector to spawn each entity every 2 seconds. Ideally each one on readjusted coordinates from each other (to account for entities moving) provided that they still aren't noticed by the player. But the main problem is not the conga line, but the performance hit of simultaneous spawning.

 

Happens with different POI simultaneous sleeper volumes too. That one you already know  and I'll be testing it further in 19.2 // 19.3

 

 

Edited by Blake_ (see edit history)
  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...