Biffing discussion, [Topic split by Baronius] Options

[Marupal]

@The Bigg
Basically, I had progressingly worse stutter the more animations I had (E.G. having all six party members walk across the screen), from as soon as I started a new game. Sounds would play normally, but the screen would have a very low frame per second; if I paused the game (or didn't have characters walking) the screen would update without noticeable lag.

Same here. From the minute that the game was loaded. There's no stutter on the menus, ever. i.e. save, character creation, backpack, journal, etc. Framerate is probably close to 0.2 fps. Hitting pause takes around 2 seconds to take effect.

@Baronius
BG2 doesn't seem to have more than 200-250 open handles at the same time, and only a small (< 10) percent of these are file handles (most of them are for the constantly used bif files and music). It didn't keep open the handles of WAV files it played from the Override, they were immediately closed after use, just like .itm, .spl etc. files. Of course, the fact it correctly closes handles doesn't mean it also frees certain objects (this is why some memory profiling would have been handy). On the other hand, as far as I've seen, it keeps open the handles of music files (even if the music isn't played anymore).

That's interesting.
Here's another strange thing about it: configuration->game autopause on mini map->no. No stutter while watching the character cross the mini map.

Furthermore, what kind of WAV files are these (i.e. the most of your 2.2GB), e.g. dialogue sounds of mods?

It appears to be two thirds related to dialogue, but there's quite a bit that I can't identify what mod installed it or what it's purpose is (except to play. )

Quick question: did you guys check the memory usage of BG2 during the stutters? (Sorry if it's a stupid question and you would have already mentioned it if this was the case.) For example, you can use the Task Manager's process list for it.

I'll have to check into this later today.
It doesn't appear to be memory leak related though. I've not had the game crash on me yet just because it had been running too long and I've left it running for days. I wish Internet Explorer behaved so well.

It didn't keep open the handles of WAV files it played from the Override, they were immediately closed after use, just like .itm, .spl etc. files.

Maybe this is the source of the problem. Maybe the .itm, .spl, etc files don't have an impact on speed because they are (usually) so small and don't require much in memory allocation but wavs do require more in the loading and unloading and logic to determine when that ambient (or whatever) needs to be unloaded. Of course you dropped all the wavs in your override and didn't have any problem.

I also forgot to ask it from you and TheBigg: is the stutter due to WAV files eventually caused by intensive I/O operations (i.e. your hard disk LED is active)?

Not much more activity than would be expected there for me.

[Baronius]

I'll proceed with making even more unverified assumptions.

It's interesting where we are: the game apparently stutters as the number of animations increases, but the phenomenon somehow correlates with the presence of a lot of WAV files in Override.

Of course you dropped all the wavs in your override and didn't have any problem.

As I've written, I played only for minutes, the game didn't access more than a few tens of WAV files at most. It was a quick, incomplete test.

Same here. From the minute that the game was loaded. There's no stutter on the menus, ever. i.e. save, character creation, backpack, journal, etc.

This is because BG2 has a GUI thread, which runs concurrently with the scheduler/engine thread (examining BG2's scheduler is another topic I would be interested in, if I had time). The stutter (unsurprisingly) happens in the engine thread.

The stutter may be due to an 'expensive' operation performed a lot of times. What can execute things a lot of times in BG2? For example, the scheduler -- which also deals with the script blocks. But this is the point where there are too many assumptions and too few certain things. Considering neither of you confirmed that you experienced I/O overload (i.e. hard disk LED blinking intensively) and the game doesn't access thousands of WAV files in the period you were playing (no matter if they are in a bif or in the Override), the explanation might be totally elsewhere than where we're searching for it.

The best would be if I could get a copy of your installation (with all the necessary files to build the same installation, i.e. chitin.key and the mod files, including the WAV files; but excluding original copyrighted game files, of course). I know it's gigabytes, so I don't suppose you would want to upload it e.g. to BWL. It's just an idea.

However, one point:

Here's what I am seeing though. Going through and biffing the BWP install-> 44,000+ files, 2.7 GB override folder, 2.2GB used by wavs. Biffing the 32,000 (non-wav) files had no noticable impact on speed. I haven't touched the bmps, bams, spls, pros, or 2das (cumulative 6000+ files there.) I'm currently half way through biffing the wavs and immediately see a major change in game speed.

The 32,000 non-wav files only used up ~500 MB for you, while even the half of the WAV files is 1.1 GB! This means that the whole thing might not necessary have to do with WAV files, but rather the size of Override (earlier TheBigg had a similar note somewhere, i.e. that the problem arises if the Override size is comparable to the RAM -- however, I don't remember if there has been an explanation for it too).

By the way, does this WAV bif/non-bif problem arise for others too (possibly without BWP), or did both of you experience it only with BWP? It might be a BWP-specific issue, perhaps an error in BWP somewhere.

I have another assumption here: as far as I can remember, the game queries the Override folder sometimes (lists the files in it). Perhaps it does something additional with the WAV files it finds (and doesn't do it if they're in BIFs). But these are just assumptions again, this is why it would be better if I had the same BWP installation as you guys (no, I don't have time to build it from pieces, from scratch), because I could reproduce the phenomenon.

[The Bigg]

The 32,000 non-wav files only used up ~500 MB for you, while even the half of the WAV files is 1.1 GB! This means that the whole thing might not necessary have to do with WAV files, but rather the size of Override (earlier TheBigg had a similar note somewhere, i.e. that the problem arises if the Override size is comparable to the RAM -- however, I don't remember if there has been an explanation for it too).

After that (early) post, I tried to reproduce the problem on my machine (by using /maxmem=512), but failed without lots of WAV space.

By the way, does this WAV bif/non-bif problem arise for others too (possibly without BWP), or did both of you experience it only with BWP? It might be a BWP-specific issue, perhaps an error in BWP somewhere.

When I experienced it (two years ago I believe) there wasn't even a BWP - we still had the (non-WeiDU) BP-BGT-NeJ compilation from plaguezone (and I didn't have that either).

This post has been edited by The Bigg: Sep 10 2008, 11:50 PM

[Marupal]

@Baronius
The best would be if I could get a copy of your installation (with all the necessary files to build the same installation, i.e. chitin.key and the mod files, including the WAV files; but excluding original copyrighted game files, of course). I know it's gigabytes, so I don't suppose you would want to upload it e.g. to BWL. It's just an idea.

OK. I'll zip it up today. It really is going to be big though. 6 GB in the data folder and 2.7 gb override folder. You'll need the dialog.tlk and the chitin.key. I think that should be everything you need to drop it onto a normal BG2 install to get it to work. Can you think of anything else?
Where would you like me to upload it to. This is a BGT installation and it includes all the BG1 files so someplace private would be appropriate to keep from violating the copyright. I already know you have both BG 1&2. Et tu The Bigg?

By the way, does this WAV bif/non-bif problem arise for others too (possibly without BWP), or did both of you experience it only with BWP? It might be a BWP-specific issue, perhaps an error in BWP somewhere.

I've seen a few other posts about stutters. The Bigg also wrote a util to test script blocks to see if they are culprits. I've never experienced stutter without mods installed.

When I experienced it (two years ago I believe) there wasn't even a BWP - we still had the (non-WeiDU) BP-BGT-NeJ compilation from plaguezone (and I didn't have that either).

I also had the same problem then but the BWP install is much worse. Still very much worth playing though. Had to add that.
Let me know where to upload this (it'll probably be the end of the day before I can upload at the earliest) and maybe I'll break it up into a couple of files.

[The Bigg]

Et tu The Bigg?

The last time I had problems it was two years ago. I don't have the stuff available any more.

I've seen a few other posts about stutters. The Bigg also wrote a util to test script blocks to see if they are culprits. I've never experienced stutter without mods installed.

It's two different kinds of stutter: here we're discussing low FPS, whereas that script is about characters who stop their action for no particular reason.

[Marupal]

@The Bigg
It's two different kinds of stutter: here we're discussing low FPS, whereas that script is about characters who stop their action for no particular reason.

It's good to know that the scripts aren't responsible for FPS stutter. Thanks.

[The Bigg]

It's good to know that the scripts aren't responsible for FPS stutter. Thanks.

Well, it's possible (it's known that Improved Anvil has FPS stutter if you have Bottomless Bags) - however, my script isn't made to deal with those, it's made to deal with a specific kind of character stuttering (as opposed to game stuttering as we're discussing here).

[Marupal]

Just biffed the tis files using NI and got the same corrupted graphics. I know I've read about DLTCEP being a GUI front-end for Weidu but NI doesn't use Weidu to accomplish things, does it?

[The Bigg]

NI doesn't use Weidu to accomplish things

No, it doesn't.

[Baronius]

I made some partial tests a week ago (many thanks to Marupal for providing me his BWP installation), the results can be read below. Note that I didn't test any (wav) biffing (and how it affects performance), all the files were copied from Marupal's files to my existing BG2:ToB installation (e.g. my Override has 51563 files). The computer I used for the test is an old one, P4 3.2 Ghz (HyperThreading), 1 GB RAM, 2 MB cache, etc.

The initial startup time wasn't so bad for me, the game loaded for cca. 90 seconds. It loaded for you much longer, Marupal, didn't it? Reason: BG2 lists all files in Override when it starts.

I loaded an earlier save game (in Chateau Irenicus), and I experienced stutter too. When I command the party to move to a point, the game stutters, and after seconds, the party usually appears at the destination point. Reason: unknown from these tests. Nothing seems to access the WAV files in Override meanwhile, by the way.

It's interesting though (at least to me, I didn't know it earlier) that when the party is moving, the game accesses 11 SPCLxxx.SPL (= innate abilities, HLA abilities) files, very many times, i.e. with a high frequency (I didn't measure whether they correlated with the AI updates, but as far as I remember, there were hundreds of accesses in a few seconds, always the same sequence of certain SPCL files). I doubt this causes the stutter, it's just an interesting thing I noticed.

The game shows some additional activity (that can be monitored) during the movement of the party, but I didn't examine it (if I remember correctly, they were some memory sections or temporary files created -- perhaps some DirectX stuff).

By the way, the BWP main screen/menus are very nice. Who made it? Really nice work.

[The Bigg]

I loaded an earlier save game (in Chateau Irenicus), and I experienced stutter too. When I command the party to move to a point, the game stutters, and after seconds, the party usually appears at the destination point. Reason: unknown from these tests. Nothing seems to access the WAV files in Override meanwhile, by the way.

Did you check RAM usage and/or open handles? My wild theory is that, when the game opens the dialog.tlk, it loads all WAV files referenced in the tlk in RAM (or keeps the file open, either for fast referencing or because of a coding error).

It's interesting though (at least to me, I didn't know it earlier) that when the party is moving, the game accesses 11 SPCLxxx.SPL (= innate abilities, HLA abilities) files, very many times, i.e. with a high frequency (I didn't measure whether they correlated with the AI updates, but as far as I remember, there were hundreds of accesses in a few seconds, always the same sequence of certain SPCL files). I doubt this causes the stutter, it's just an interesting thing I noticed.

Does the weidu.log contain Refinements? I know Refs does lots of spell tricks under the hood (I don't think this causes issues by itself, since I played the mod on a P4 1.6Ghz with 256MB of RAM).

The game shows some additional activity (that can be monitored) during the movement of the party, but I didn't examine it (if I remember correctly, they were some memory sections or temporary files created -- perhaps some DirectX stuff).

MMIO mapping of animation files? The engine used for The Broken Hourglass does this to reduce RAM usage, for instance.

By the way, the BWP main screen/menus are very nice. Who made it? Really nice work.

Unless BWP has GUIs of his own, each of the 'large mods' adds its own main screen to a setup-gui.tp2 mod (which is written on the fly each time you install a new mod with a gui). Thus, looking at the comment string in weidu.log for that file, you can figure out which mod is it from (and hopefully, from the readme, the author of said GUI).

This post has been edited by The Bigg: Sep 27 2008, 08:19 PM

[Marupal]

@The Bigg
Does the weidu.log contain Refinements? I know Refs does lots of spell tricks under the hood (I don't think this causes issues by itself, since I played the mod on a P4 1.6Ghz with 256MB of RAM).

It does but just the revised HLA's and Universal Lesser Mage Robes.

@Baronius
By the way, the BWP main screen/menus are very nice. Who made it? Really nice work.

That one is from BGT. Couldn't find who the particular author of "Elminster holding the staff" was.

The initial startup time wasn't so bad for me, the game loaded for cca. 90 seconds. It loaded for you much longer, Marupal, didn't it? Reason: BG2 lists all files in Override when it starts

The computer I used was even older->AMD 2.6 768 MB RAM. I'm surprised it loaded that fast for you when it takes quite a few minutes for me. Will check next time how long it takes unbiffed.

@The Bigg
Did you check RAM usage and/or open handles? My wild theory is that, when the game opens the dialog.tlk, it loads all WAV files referenced in the tlk in RAM (or keeps the file open, either for fast referencing or because of a coding error).

That would make sense. It's pretty strange that having 30,000 files =500MB biffed doesn't make much of an impact on performance (game initialization does speed up though) but 6,000 files=2.2GB biffed takes away from the "teleportation" stutter. Maybe someday they'll release the source code and everything will make sense then

[The Bigg]

It does but just the revised HLA's and Universal Lesser Mage Robes.

It's the Revised HLAs which access spcl* lots of times.

The computer I used was even older->AMD 2.6 768 MB RAM. I'm surprised it loaded that fast for you when it takes quite a few minutes for me. Will check next time how long it takes unbiffed.

Perhaps it's related to disk speed.

Maybe someday they'll release the source code and everything will make sense then

Doubtful, Bioware is now a property of EA.

[Baronius]

Did you check RAM usage and/or open handles? My wild theory is that, when the game opens the dialog.tlk, it loads all WAV files referenced in the tlk in RAM (or keeps the file open, either for fast referencing or because of a coding error).

RAM usage is approximately constant, cca. 110-115 MB. Before loading the savegame, it's about 50 MB.

The open handles are just the usual ones (GUI biffs, SOS & ROT biffs, log file etc.)

As far as your wild theory is concerned, during my test, no Override WAV files were accessed from game startup to the end of the savegame load. After that, only 4-5 area-related WAVs and some character sounds (female1i.wav etc.) are accessed, continuously (i.e. with time, as I click on characters more times and move in the area). Even these WAV handles aren't kept open, but their content are stored in the game's address space (unlike with CRE, ITM, GUI BAM etc. files which are read from the file each time they are referenced by the game). Based on this, your assumption that it keeps WAV file content in memory sounds to be reasonable, but the lots of WAV files in Override just doesn't seem to be accessed, as I've written.

The game shows some additional activity (that can be monitored) during the movement of the party, but I didn't examine it (if I remember correctly, they were some memory sections or temporary files created -- perhaps some DirectX stuff).

MMIO mapping of animation files? The engine used for The Broken Hourglass does this to reduce RAM usage, for instance.

I checked them again and as far as I can see, they just seem to be temporary files related to Sound Card drivers or something like that. When the characters move, 2-3 such file handles are open and then immediately closed.

It does but just the revised HLA's and Universal Lesser Mage Robes.

It's the Revised HLAs which access spcl* lots of times.

It does it from some continously running script I suppose, considering the very high frequency of accesses to the same set of files. It's interesting though that about a week ago, I removed the SPCL files in question from the Override, and as far as I can remember, SOS-BIF.bif (or something of similar name) accesses appeared instead, so the chitin.key entries for those eleven spcl spl files point to the SOS-bif (Shadows over Soubar?) -- as far as I can remember (I haven't checked chitin.key).

[Jarno Mikkola]

I checked them again and as far as I can see, they just seem to be temporary files related to Sound Card drivers or something like that. When the characters move, 2-3 such file handles are open and then immediately closed.

Were the ambient sounds on(not turned to zero), as they do real hazards for fps counts if there is an error in their assignments in one or more of the mods, as the behavior would suggest it.

By the way, it would be good, if the particular BWP's version and every other essential info(WeiDU.log) would be documented, but then again it would turn this thread to something completely different, which wouldn't be The Biggs problem.

This post has been edited by Jarno Mikkola: Sep 29 2008, 09:41 AM