Biffing discussion, [Topic split by Baronius] Options

[Jarno Mikkola]

The biff files don't contain the file name, it's only listed in the chitin.key. Also, if a duplicate entry is found (E.G. data/ambsound.bif and data/rtwsnd.bif both contain ambsound01.wav), then the game will crash before even showing the main gui (SoA/ToB selection). WeiDU (including WesDU) does remove chitin.key duplicates when biffing.

You do realize that that kinda defeats the purpose off biffing when it's done multiple times... Now, is there anyway to prevent the a WeiDU mod from biffing itself and the files while it's installing, other than editing the tp2 file? As it would reduce the multiple .bif entries in the data folder, when installing...(with auto installed) the BWP. In which case the Biffing could actually done last, and in many cases is done either way still with the End_Biff, for the other files.

[The Bigg]

Arguably, the best way is to convince King Diamond, Ascension and all the others that no, you don't need to biff non-bulky files, because an override with many files but small in megabytes doesn't hurt performance at all.

Failing that, comfort yourself in the fact that at least it works correctly, as long as you don't use the Stack Uninstall when dealing with big mods (I.E. you manually uninstall up to them, uninstall them one-by-one, and reinstall the temp uninstalled mods manually). WeiDU has the facilities to biff files Stack-friendly, but unfortunately the various Big Mods authors want to biff itm files, thus not being able to use the stack-friendly facility.

[Galactygon]

..., because an override with many files but small in megabytes doesn't hurt performance at all.

I thought it was the other way around; more files mean having to load more indexes.

-Galactygon

[Jarno Mikkola]

How ever it is-

Arguably, the best way is to convince King Diamond, Ascension and all the others that no, you don't need to biff non-bulky files, because an override with many files but small in megabytes doesn't hurt performance at all.

Pretty sure that ain't gonna happen... as even the BGT checks for the .bif files, not a file(index, what ever) in on somewhere else.

Now that I think, the game uses biffs to read from the old data on the temps, so don't think Galactygon is right, but then again I know nothing about the comps, so...

Failing that, comfort yourself in the fact that at least it works correctly... when dealing with big mods

Yeah, and when dealing with big mods, isn't it always best to start from the fresh, backup? It takes less time.
Now, if only we would have them all.

[The Bigg]

I thought it was the other way around; more files mean having to load more indexes.

Tutu has an override of ~10000 files and it works pretty well on low-end machines. If you install a bunch of NPC mods, you might end up with 2/3000 files and half a gigabyte, which is too much for low-end machines. YMMV of course.

[Baronius]

I thought it was the other way around; more files mean having to load more indexes.

Tutu has an override of ~10000 files and it works pretty well on low-end machines. If you install a bunch of NPC mods, you might end up with 2/3000 files and half a gigabyte, which is too much for low-end machines. YMMV of course.

This is only relevant if you've also tested whether those 2000-3000 files using ~0.5 GB don't cause the same performance problem when biffed. In other words, it might have been simply a matter of operative memory on the low-end machine in question (the Override of EasyTutu is only about 120 MB).

Generally, using bifs is good and encouraged (provided it's done reasonably, e.g. the same stuff isn't biffed several times ), because the Infinity Engine games were optimized for reading 90% of their files from bif collections. They weren't prepared to deal with thousands of distinct files.

There can be a number of reasons how lots of (possibly small) files can decrease BG2 performance. For example, each -- even the smallest -- file uses up at least one file system block size (so your 314 byte-long .itm file will also use up e.g. 4KB), and for thousands of such files, the degree of the resulting internal fragmentation might be great (for example, if the files of BG2:ToB weren't in bif collections, BG2 would need +90 MB disk space). Beyond the impact it has on disk read I/O, this fragmentation might influence the caching mechanisms of the system (e.g. the file system cache) as well. Of course, further research and tests would be required, if we want(ed) more accurate information on this.

[The Bigg]

This is only relevant if you've also tested whether those 2000-3000 files using ~0.5 GB don't cause the same performance problem when biffed. In other words, it might have been simply a matter of operative memory on the low-end machine in question (the Override of EasyTutu is only about 120 MB).

It did for me when I biffed the WAV files

provided it's done reasonably, e.g. the same stuff isn't biffed several times

This wouldn't really change anything at all, since current tools are smart enough to delete duplicated entries. The only real risk is that, if you use `weidu --remove-biff' you risk losing access to files. But since all mods make a backup of chitin.key and restore that, this is not a risk for people installing/uninstalling mods 'normally'.

There can be a number of reasons how lots of small files can decrease BG2 performance. For example, each -- even the smallest -- file uses up at least one file system block size (so your 314 byte-long .itm file will also use up e.g. 4KB), and for thousands of such files, the degree of the resulting internal fragmentation might be great (for example, if the files of BG2:ToB weren't in bif collections, BG2 would need +90 MB disk space). Beyond the impact it has on disk read I/O, this fragmentation might influence the caching mechanisms of the system (e.g. the file system cache) as well. Of course, further research and tests would be required, if we want(ed) more accurate information on this.

On FAT32, the block size is 32 KB, on NTFS 4 KB. Even in the worst scenario (say, 20000 files on FAT32), the disk occupation difference would be completely negligible on a modern machine (less than half a gigabyte). Regarding the caching / fragmentation issue, it appears to not affect performance (I had overrides of 15000 files multiple times on my old laptop with both FAT32 and NTFS without issues).
That being said, it can happen that systems other than the game might have serious performance issues when dealing with some thousands of files in the same folder (say, an antivirus, a backup system, etc.)

EDIT: it might be interesting to mention that, after rebooting with /maxmem=512, the game lagged only minorly (20000 files, 600 MB).
EDIT II: my override is made up as follows:

CODE

2da 177 998K
are 1035 8.4M
bam 683 15M
bcs 2596 357M
bmp 265 2.7M
chu 3 24K
cre 7649 31M
dlg 2092 19M
eff 782 782K
gam 1 12K
ids 23 158K
itm 3487 5.3M
mos 51 29M
pro 13 43K
spl 1227 2.9M
sto 264 726K
tis 34 65M
txt 20 20K
vvc 88 88K
wav 210 54M
wed 25 208K
wmp 2 576K

that is, mostly made up of BCS files. I don't have the statistics for my previously mentioned case where I needed biffing, but IIRC most of the override was made up of WAV files. Perhaps WAV files in the override are all kept in memory, whereas BCS files aren't? Somebody more bored than me should do some experiments

This post has been edited by The Bigg: Sep 6 2008, 04:43 PM

[Baronius]

Regarding the caching / fragmentation issue, it appears to not affect performance (I had overrides of 15000 files multiple times on my old laptop with both FAT32 and NTFS without issues).

It also depends on the configuration and state of the concrete system where the player runs the game. For example, as far as I can remember, defragmentation helped some BG2 players with a big Override, the "stutter" they experienced ceased. I don't know how many (and how comprehensive) tests you've made.

Perhaps WAV files in the override are all kept in memory

Generally, you may have a point there (for example, it would also mean that certain mod types -- i.e. with lots of files of a particular type -- are capable of causing more performance issues than other mod types), however, what's the difference then between bif storage and Override? If the game stores their contents in memory, it doesn't matter where the files were loaded from, they will use up the virtual memory and the system will eventually start writing out the stuff of other processes to the pagefile (=> game slowdown). Of course, as I've written, we don't know about the exact optimizations the game might contain for biffed files (that is, it might handle non-biffed resources in a different, perhaps less efficient way, though I can't think of a good reason for that at the moment).

[The Bigg]

Generally, you may have a point there (for example, it would also mean that certain mod types -- i.e. with lots of files of a particular type -- are capable of causing more performance issues than other mod types), however, what's the difference then between bif storage and Override? If the game stores their contents in memory, it doesn't matter where the files were loaded from, they will use up the virtual memory and the system will eventually start writing out the stuff of other processes to the pagefile (=> game slowdown). Of course, as I've written, we don't know about the exact optimizations the game might contain for biffed files (that is, it might handle non-biffed resources in a different, perhaps less efficient way, though I can't think of a good reason for that at the moment).

Whatever the reason, that is what I know (= WAV files in the override cause more performance problems than biffed WAV files or unbiffed BCS files).

[Galactygon]

This is good news if I would be safe to biff all .wav and .bam files while ignoring the rest.

Skewing this slightly off topic, I have found that multiple (3-4) .wav files played simultaneously ingame have as much a severe impact on performance, as when the entire screen filled with several large .bams. This means that if you wish to play some sound at a louder volume, the only good way to do it is edit the sound itself by amplifying the waves and saving it as a different .wav rather than playing one sound many times.

-Galactygon

[Baronius]

provided it's done reasonably, e.g. the same stuff isn't biffed several times

This wouldn't really change anything at all, since current tools are smart enough to delete duplicated entries. The only real risk is that, if you use `weidu --remove-biff' you risk losing access to files. But since all mods make a backup of chitin.key and restore that, this is not a risk for people installing/uninstalling mods 'normally'.

I suppose that you don't consider the installation of The Bigg World Project 'normal' then, do you? Because if I've interpreted Marupal's posts correctly, the BWP biffed the same resources multiple times, because some of the member mods also biff themselves, and then comes the BWP biffing. I'm not familiar with BWP, so I might be missing something.

[The Bigg]

I suppose that you don't consider the installation of The Bigg World Project 'normal' then, do you? Because if I've interpreted Marupal's posts correctly, the BWP biffed the same resources multiple times, because some of the member mods also biff themselves, and then comes the BWP biffing. I'm not familiar with BWP, so I might be missing something.

No, I consider BWP biffing 'working', but 'stupid'. Even if you biff any file multiple times, it'll be available exactly once in chitin.key, so the game will work. WeiDU '[U]ninstall' (as coded in the various big mods) works flawlessly as well as long as you don't use stack uninstall. The only thing I find stupid is that it biffs itm, spl, etc. files, not because of performance reasons, but because "it looks messy otherwise", and by doing so it makes stack uninstall unsafe.

[Jarno Mikkola]

...The Bigg World Project...

No, I consider BWP biffing 'working', but 'stupid'. Even if you biff any file multiple times, it'll be available exactly once in chitin.key, so the game will work. WeiDU '[U]ninstall' (as coded in the various big mods) works flawlessly as well as long as you don't use stack uninstall. The only thing I find stupid is that it biffs itm, spl, etc. files, not because of performance reasons, but because "it looks messy otherwise", and by doing so it makes stack uninstall unsafe.

And not only is it a bit stupid, but it wastes huge spaces if the same resource is biffed many times in different places, because it has been changed... but that's a very minor thing as long as it works, and it does.

@Baronius, It's the "Big World Project", it's has nothing to do with the honorable The Bigg.

[Baronius]

Whoops. I knew it wasn't his project, but it seems I've seen too many Biggs recently

The only thing I find stupid is that it biffs itm, spl, etc. files, not because of performance reasons, but because "it looks messy otherwise", and by doing so it makes stack uninstall unsafe.

Why is it unsafe exactly? (An example is also enough, if you prefer that.) By the way, it's good to gather these facts in one place here, they might be useful for IEMT too. Thanks for your time and answers, TheBigg.

[The Bigg]

Well, the way it's done in the Big Mods*, at uninstall the old chitin.key is restored but not reloaded. Thus, if (fantasy example) baldur.bcs was in data\scripts.bcs and data\tddbcs.bif but not in the override, during a stack uninstall of tdd, later mods trying to append baldur.bcs will load the copy in data\tddbcs.bif rather than the copy in data\scripts.bcs, possibly failing if data\tddbcs.bif was removed.
Later WeiDUs have a command to create biffs in a completely stack-safe manner (MAKE_BIFF), but unfortunately it hasn't caught on (since I refuse to add a DELETE or MOVE command, making it impossible to biff the various itm,cre,spl... files in the override).

Big Mods = TDD, BGT, CtB and the likes.
Big World Project: a pdf file maintained by Leonardo Watson containing install instructions for a large selection of mods (over 600).

[Marupal]

@Baronius
Generally, you may have a point there (for example, it would also mean that certain mod types -- i.e. with lots of files of a particular type -- are capable of causing more performance issues than other mod types), however, what's the difference then between bif storage and Override?

I don't remember where I heard this but somebody claimed that the BG engine(s) didn't use normal file i/o. Instead they used file-mapping (memory mapped files) to pull off the ability to have so many large biffs in memory and still be able to run on a small system (the first computer I installed BG on only had 128 meg.) Don't know if they knew what they were talking about but this makes some interesting reading:
http://msdn.microsoft.com/en-us/library/ms810613.aspx

[Baronius]

Instead they used file-mapping (memory mapped files) to pull off the ability to have so many large biffs in memory [..]

Large (or any) biffs aren't stored in memory even with memory mapping (it wouldn't make any sense to waste so much memory). It doesn't require memory mapping to read just a certain part of a large file to memory, so I'm sure it's used for other reasons in BG (e.g. more efficient I/O due to buffering, transparent handling of files etc. -- they probably didn't want to implement these on application level).

Bif files simply contain game files. When the game needs an item, spell, etc. resource and it can't be found in the Override, it finds the resource data in the corresponding bif file (based on the information in the KEY hashtable in memory), and reads all resource data (and not just a part of it: e.g. it can't just partially deserialize an item, because all data are required for an in-game item object). So usually, in terms of memory usage, it probably doesn't matter whether a file is in Override or in a bif -- it needs to be completely deserialized.

So it's more likely that the game handles WAV files read from bif in a different way than WAV files from the Override (this is based on what TheBigg has written about WAV performance, I haven't tested it). For example, it has a cache of a limited size for WAV files read from bifs (and if it's full, it drops old elements), but there is no similar limit for files read from Override, or something like this. This is a primitive and highly unlikely example, just to illustrate what sort of things I can think of in this case. Nonetheless, I really haven't done any research on this (so the case might be totally different and there is no difference between how bif and Override WAVs are handled, and thus we're missing something else), so I really don't want to make further assumptions. As TheBigg has said, some tests might help to find out more information on this.

[Marupal]

It doesn't require memory mapping to read just a certain part of a large file to memory, so I'm sure it's used for other reasons in BG

I really can't confirm whether BG does file-mapping or not. Somebody may have been making it up. It was just something interesting that came up. I had never heard of it before then.

e.g. it can't just partially deserialize an item

"Deserialize"? Deserialize:verb->Something I had to look up. Tis and wav files don't need to be deserialized though. Even the compressed versions are done with zlib so will do streaming decompression. Nothing but opinion here: having all the wavs in one file (or very few files) and only needing handles to those very few files would be easier to maintain than having to account for the possibility of hundreds (or thousands maybe) of handles being open and in use (ambients, cre sounds, npc sounds, mus, etc.)
Here's what I am seeing though. Going through and biffing the BWP install-> 44,000+ files, 2.7 GB override folder, 2.2GB used by wavs. Biffing the 32,000 (non-wav) files had no noticable impact on speed. I haven't touched the bmps, bams, spls, pros, or 2das (cumulative 6000+ files there.) I'm currently half way through biffing the wavs and immediately see a major change in game speed. Tried all kinds of different settings in configure options including pathfinding. Nothing had any noticable impact except biffing the wavs. It was so bad that you could click on a destination and the game would look like it locked up and wouldn't do anything again until it showed you at the destination. Really bad stutter.
Maybe next time I'll start by biffing the wavs first and see if get that same increase immediately.

[Baronius]

Here's what I am seeing though. Going through and biffing the BWP install-> 44,000+ files, 2.7 GB override folder, 2.2GB used by wavs. Biffing the 32,000 (non-wav) files had no noticable impact on speed. I haven't touched the bmps, bams, spls, pros, or 2das (cumulative 6000+ files there.) I'm currently half way through biffing the wavs and immediately see a major change in game speed.

What does this exactly mean? How exactly the performance problems arise (this is also directed to TheBigg)? After playing through a few areas, perhaps after playing hours, or immediately? Does it happen only at certain points of the game? Or is the stuttering periodic (with a high frequency)? Does reload or game restart+reload help (the latter one certainly eliminates memory leak caused by unfreed objects accumulated during a longer time of playing)? Furthermore, what kind of WAV files are these (i.e. the most of your 2.2GB), e.g. dialogue sounds of mods?

If the stutter is due to some memory leak (or inefficient memory utilization), it should only happen when or after a lot of WAV files have been accessed. The game doesn't access thousands of WAV files at the same time, so I suppose the stutter doesn't happen quickly for you (and reload should also eliminate it). I also forgot to ask it from you and TheBigg: is the stutter due to WAV files eventually caused by intensive I/O operations (i.e. your hard disk LED is active)? I suppose it does (and this obviously doesn't answer why it's happening, because more things can cause Windows to write the pages of processes to disk), but I ask it just to make sure.

Nothing but opinion here: having all the wavs in one file (or very few files) and only needing handles to those very few files would be easier to maintain than having to account for the possibility of hundreds (or thousands maybe) of handles being open and in use (ambients, cre sounds, npc sounds, mus, etc.)

I don't have a proper testing environment set up at the moment (for example, I have no 2.2 GB of WAV files of mods), just an installed BG2:ToB. I extracted all 9800+ WAV files from the bif collections to the Override folder, and quickly visited 2-3 areas (one of them included a battle with Draconis) in the game. There was no stutter. Of course, this doesn't prove or confirm anything (e.g. I should've played for a much longer time, and should have at least made some memory profiling), but at least I can react to your comment about handles:

(1) I doubt that a high number of handles can cause such performance issues that you & TheBigg experienced. The system can cope with it. (This has nothing to do with my primitive test mentioned above -- it's a general statement.)
(2) BG2 doesn't seem to have more than 200-250 open handles at the same time, and only a small (< 10) percent of these are file handles (most of them are for the constantly used bif files and music). It didn't keep open the handles of WAV files it played from the Override, they were immediately closed after use, just like .itm, .spl etc. files. Of course, the fact it correctly closes handles doesn't mean it also frees certain objects (this is why some memory profiling would have been handy). On the other hand, as far as I've seen, it keeps open the handles of music files (even if the music isn't played anymore).

On a side note, BG2 (unsurprisingly) uses DirectSound to play audio files. Perhaps the code that passes data to the DirectSound component is buggy for WAV files from the Override. But this is just another of the many assumptions I've made. Some day when I have time, I should really do some more useful tests in a proper environment.

As far as file-memory mapping is concerned, some components used by BG2 (perhaps some DirectSound audio stuff IIRC) seemed to use it, so your information source might be partially correct with that (but I doubt that BG2 itself accesses files in that way). But as I've written earlier, I think this doesn't matter in terms of the performance problem we're discussing.

Quick question: did you guys check the memory usage of BG2 during the stutters? (Sorry if it's a stupid question and you would have already mentioned it if this was the case.) For example, you can use the Task Manager's process list for it.

This post has been edited by Baronius: Sep 9 2008, 07:55 PM

[The Bigg]

Basically, I had progressingly worse stutter the more animations I had (E.G. having all six party members walk across the screen), from as soon as I started a new game. Sounds would play normally, but the screen would have a very low frame per second; if I paused the game (or didn't have characters walking) the screen would update without noticeable lag.

Can't be sure about memory use, but IIRC it was in the same range before and after biffing.