Fallout 3 [BLUS30541]: Frequent vertex explosions

[Shoegzer]

Quick summary

Vertex explosions occur frequently while ingame:

Details

• Vertex explosions can usually be triggered by drawing a weapon, firing at an enemy or entering an area such as Megaton at the beginning of the game, but can also cocur on their own.
• Frequency of occurrence is typically between 1-10 minutes of play.
• These can usually be cleared up by entering a different area or going to pause menu, but they eventually come back.
• Many options were tried in attempting to resolve this issue, including OpenGL and debug settings, but to no avail.

Attach a log file

RPCS3 Log: RPCS3.log.zip

Attach capture files for visual issues

RSX Capture: BLUS30451_20231201180021_capture.rrc.gz

System configuration

• OS: Linux Mint 21.2 / kernel 6.2
• CPU: Intel i7-10870H
• GPU: Nvidia GTX 3070 / driver 535.129
• Build tested: c245637

Other details

Save file from an area of the game that frequently (always?) triggers the issue: BLUS30185-SAVE-PLAYER-66.zip

[Darkhost1999]

If I remember correctly this is antagonized by write color buffers.

[Shoegzer]

Thanks, I asume by antagnozied you mean WCB causes this. I've been testing with it on since it prevents save game thumbnail corruption but I'll and try without it for a while.

[Shoegzer]

Update: Vertex explosions occur even without WCB, as I've just discovered in testing.

[Darkhost1999]

Yes, but does it happen less often or have the same triggers?

[Shoegzer]

That's why I assumed that by "antagonized" you meant that WCB causes the problem, not to be pendantic. I can check again to see if I encounter the issue less frequently, though it did occur soon after going ingame with WCB turned off, so I'm not sure if there's a conclusive way to measure its impact. As far as triggers, it was caused by going into a new area as can happen with WCB on. Have your tests been different?

[Shoegzer]

Vertex explosions seem to be occurring just as frequently, and triggered the same way, so I don't think we can conclude that WCB has any real effect on them, positive or negative.

[kd-11]

I don't see any RSX heap overflows so it's more likely one of two things:

1. Vertex shader corruption.
2. Verts coming in already messed up from CELL,

Has this error always been present? It would be easier to figure out which scenario is the culprit if it is a regression of some kind.

[Shoegzer]

I'm not exactly sure if it has always been present, but at least the writeup on the title's Wiki page which references the issue was there before I started testing it a few weeks ago. I know that's much help but perhaps there's a way to determine when that was posted.

That said, I'm happy to gather any other information for you that may help in tracking this down.

[Darkhost1999]

Does Shader Precision Ultra help at all?
My tests were when I got the game and not very long after that epic gave the game away for free. So I have 0 progress in RPCS3. Just remember getting to that 1st town with a bomb in the middle of it and there being vertex stuff that made it impossible to see anything.
Then on this approximately 0.0.22 or 0.0.23 build I disabled wcb and never saw the issue again.
Nowadays things could've changed yet the game doesn't appeal to me very much.

[Shoegzer]

Hey thanks, I turned shader quality to Ultra and will let you know if it helps. That's one of the few settings I didn't test.

And yeah, that first town with the bomb in it is "Megaton" as I had mentioned in my original post, and that's the first time I encountered it too. FWIW assuming you followed a typical game path as I did, that's the first area of the game with such a large amount of scene geometry, before that point you're primarily in the vault which is nowhere near as complex graphically. Perhaps not by coincidence then, usually when I enter Megaton the vertex explosions occur.

[Shoegzer]

Actually, I got the vertex explosions immediately on loading the game with Ultra shader quality, so that doesn't help unfortunately.

[kd-11]

This isn't a shader bug. The memory for some vertex stream likely got overwritten. I wonder how to catch the culprit.

[Shoegzer]

Thanks @kd-11. FWIW I've just dicovered an area of the game that seems to always trigger this issue, if you think that might help I can provide some RSX captures from there or anything else you may need.

[kd-11]

Thanks @kd-11. FWIW I've just dicovered an area of the game that seems to always trigger this issue, if you think that might help I can provide some RSX captures from there or anything else you may need.

A save file would be better. I need to run the game on a modified version of the emulator to catch the issue.

[Shoegzer]

Done - posted above. It's a serialized save so if you need the entire lot just let me know.

[Shoegzer]

Also, in case it helps I've just come across this post which shows that the issue must have been present over four years ago (though whether it was resolved and then later regressed again I don't know):

The game suffers from...major graphical glitches like stretching polygons covering the screen or when using any scoped weapon (it doesn't always happen, though).

Incidentally the "scoped weapon" comment only holds true in the most recent builds if the screen is already showing vertex explosions.

[jeremybris]

I got this bug more frequently and consistantly in Oblivion and Fallout 3, when enabling high frame rate patches and increasing the vblank rate. Dunno if that helps.

Edit: Specifically 120 F.P.S., 120 vblank in Oblivion.

Edit 2: I rarely get this bug outside of the above settings.

[Shoegzer]

Interesting. I've tried almost everything but can't seem to change its frequency positively or negatively, but then again I always run at ps3-native resolutions and framerates. I'll tinker with those a bit to see if there's any observable difference on my end.

Also interesting to know that this happens on Oblivion too, though I suppose not too surprising given the same developer and engine. I'm curious if this is observed in Fallout New Vegas or Skyrim.

[Shoegzer]

Update: I discovered that setting the PPU Thread Count to 1 in Debug seems to eliminate this issue, at least in testing so far. At least, it hasn't occurred after entering areas such as the Rivet City Marketplace multiple times, whereas the problem is always observed with the default Thread Count of 2.

I'll report further if I get the issue again though.

[kd-11]

Test with #14907
If you enable WCB, also enable RCB. Same for RDB/WDB, try to pair them together.

[Shoegzer]

Thanks. Unfortunately the vertex explosions are still occurring with the new PR merged and WCB + RCB enabled.

Log: RPCS3.log
RSX Capture: here (too large to upload here)

Also: Is it possible your new PR could resolve the issue with this title freezing occasionally? I'm not sure if that's related, and it's still early, but it hasn't frozen in testing so far.

[Darkhost1999]

Also: Is it possible your new PR could resolve the issue with this title freezing occasionally? I'm not sure if that's related, and it's still early, but it hasn't frozen in testing so far.

#9133 (comment)
That's what this was describing so I hope so.
Along with other games than just Fallout 3.

[Shoegzer]

Actually, after further testing it did freeze on me once. At that frequency I'm not convinced it's a RPCS3 issue though - the Bethesda games were known for freezing on PS3 and even PC. This is the GOTY edition which I assume incorporates all previous patches, but I'm pretty sure even that edition had issues like this.

[Shoegzer]

Incidentally, turning on RCB frequently produces issues of its own, and only when the game is drawing other objects in the foreground, as shown with the in-game menu here:

Such issues are not present when only WCB is on.

Log: BLUS30451_20231229172639_capture.rrc.gz

As this is likely to be unrelated, I can move this to a separate issue if needed.

[Triticum0]

Does the issue get worse or better when using the Spu interpreter?

[Shoegzer]

@Triticum0 are you asking about the main issue or the one I just raised? And also, which interpreter - static, dynamic or ASMJIT?

[Triticum0]

@Shoegzer Interpreter Static or dynamic it doesn't matter. I just had issue with vertex explosions on an issue which got worse using the Interpreter and just wanted to know if it related to this issue or not.#15021

[Triticum0]

Incidentally, turning on RCB frequently produces issues of its own, and only when the game is drawing other objects in the foreground, as shown with the in-game menu here:

Also Ideally RCB and WCB are meant to used together as that how ps3 does it. So having graphical issue with one turn off is to be expected.

Kd-11 Quote about data leaking causing graphical issues

As I noted in #14907, there is no way to avoid the leaks without both RCB+WCB (or RDB+WDB in some games) enabled at the same time.

[Shoegzer]

The vertex explosion issue is no better or worse with the interpreters (I just tested all three). The only way to avoid this problem (90% of the time at least), is to set the PPU Thread Count to 1. Try that and see if it helps your case.

I realize RCB and WCB are meant to be used together. My point is that the issue only occurs with both of them turned on.

[Triticum0]

Thanks, just wanted to know if the issue was related looks like it is not.

With the RCB issue, you should probably comment on #9133

[Shoegzer]

Done.

Also, I discovered this video that demonstrates the vertex explosion issue at least with Fallout: New Vegas (though it's the same with Fallout 3).

[IAmBatby]

Just thought I'd mention that Bloodborne via ShadPS4 seems to be having the same issue (Seemingly random vertex explosions after the game tries to load a region that can be resolved by reloading)

shadps4-emu/shadPS4#1232

At least in Bloodborne's case almost every instance of a vertex explosion is tied to the files used in the tool FACEGEN, This is a tool that is also predominantly used in the fallout series.

This is linked in my Bloodborne issue but I found a mod for the PC mod "Fallout: London" that also fixes similar corruption by modifying FACEGEN related files

https://www.nexusmods.com/fallout4london/mods/490

Furthermore, While this may be unrelated I also found that Xenoblade Chronicles X via Cemu has an active vertex explosion problem that is noted to be due to some sort of scheduling issue, with resolution and framerate tweaking being observed to increase the likely hood of this occurring. I mention this because both Fallout 3 and Bloodborne are known to have a pretty inconsistent framerate as well as being heavily tied to the target framerate they shipped with

https://wiki.cemu.info/wiki/Xenoblade_Chronicles_X

From the examples we have it seemed like the niche facegen file setup is victim to some sort of data corruption issue that happens due to possibly inconsistent framerates and/or scheduling at the time of load/reload?