You do realize that that kinda defeats the purpose off biffing when it's done multiple times... Now, is there anyway to prevent the a WeiDU mod from biffing itself and the files while it's installing, other than editing the tp2 file? As it would reduce the multiple .bif entries in the data folder, when installing...(with auto installed) the BWP. In which case the Biffing could actually done last, and in many cases is done either way still with the End_Biff, for the other files.

Arguably, the best way is to convince King Diamond, Ascension and all the others that no, you don't need to biff non-bulky files, because an override with many files but small in megabytes doesn't hurt performance at all.

Failing that, comfort yourself in the fact that at least it works correctly, as long as you don't use the Stack Uninstall when dealing with big mods (I.E. you manually uninstall up to them, uninstall them one-by-one, and reinstall the temp uninstalled mods manually). WeiDU has the facilities to biff files Stack-friendly, but unfortunately the various Big Mods authors want to biff itm files, thus not being able to use the stack-friendly facility.

I thought it was the other way around; more files mean having to load more indexes.

-Galactygon

How ever it is- Pretty sure that ain't gonna happen... as even the BGT checks for the .bif files, not a file(index, what ever) in on somewhere else.

Now that I think, the game uses biffs to read from the old data on the temps, so don't think Galactygon is right, but then again I know nothing about the comps, so...

Yeah, and when dealing with big mods, isn't it always best to start from the fresh, backup? It takes less time.
Now, if only we would have them all.

Tutu has an override of ~10000 files and it works pretty well on low-end machines. If you install a bunch of NPC mods, you might end up with 2/3000 files and half a gigabyte, which is too much for low-end machines. YMMV of course.

This is only relevant if you've also tested whether those 2000-3000 files using ~0.5 GB don't cause the same performance problem when biffed. In other words, it might have been simply a matter of operative memory on the low-end machine in question (the Override of EasyTutu is only about 120 MB).

Generally, using bifs is good and encouraged (provided it's done reasonably, e.g. the same stuff isn't biffed several times ), because the Infinity Engine games were optimized for reading 90% of their files from bif collections. They weren't prepared to deal with thousands of distinct files.

There can be a number of reasons how lots of (possibly small) files can decrease BG2 performance. For example, each -- even the smallest -- file uses up at least one file system block size (so your 314 byte-long .itm file will also use up e.g. 4KB), and for thousands of such files, the degree of the resulting internal fragmentation might be great (for example, if the files of BG2:ToB weren't in bif collections, BG2 would need +90 MB disk space). Beyond the impact it has on disk read I/O, this fragmentation might influence the caching mechanisms of the system (e.g. the file system cache) as well. Of course, further research and tests would be required, if we want(ed) more accurate information on this.

It did for me when I biffed the WAV files

This wouldn't really change anything at all, since current tools are smart enough to delete duplicated entries. The only real risk is that, if you use `weidu --remove-biff' you risk losing access to files. But since all mods make a backup of chitin.key and restore that, this is not a risk for people installing/uninstalling mods 'normally'.


On FAT32, the block size is 32 KB, on NTFS 4 KB. Even in the worst scenario (say, 20000 files on FAT32), the disk occupation difference would be completely negligible on a modern machine (less than half a gigabyte). Regarding the caching / fragmentation issue, it appears to not affect performance (I had overrides of 15000 files multiple times on my old laptop with both FAT32 and NTFS without issues).
That being said, it can happen that systems other than the game might have serious performance issues when dealing with some thousands of files in the same folder (say, an antivirus, a backup system, etc.)

EDIT: it might be interesting to mention that, after rebooting with /maxmem=512, the game lagged only minorly (20000 files, 600 MB).
EDIT II: my override is made up as follows:

that is, mostly made up of BCS files. I don't have the statistics for my previously mentioned case where I needed biffing, but IIRC most of the override was made up of WAV files. Perhaps WAV files in the override are all kept in memory, whereas BCS files aren't? Somebody more bored than me should do some experiments

It also depends on the configuration and state of the concrete system where the player runs the game. For example, as far as I can remember, defragmentation helped some BG2 players with a big Override, the "stutter" they experienced ceased. I don't know how many (and how comprehensive) tests you've made.


Generally, you may have a point there (for example, it would also mean that certain mod types -- i.e. with lots of files of a particular type -- are capable of causing more performance issues than other mod types), however, what's the difference then between bif storage and Override? If the game stores their contents in memory, it doesn't matter where the files were loaded from, they will use up the virtual memory and the system will eventually start writing out the stuff of other processes to the pagefile (=> game slowdown). Of course, as I've written, we don't know about the exact optimizations the game might contain for biffed files (that is, it might handle non-biffed resources in a different, perhaps less efficient way, though I can't think of a good reason for that at the moment).

Whatever the reason, that is what I know (= WAV files in the override cause more performance problems than biffed WAV files or unbiffed BCS files).

This is good news if I would be safe to biff all .wav and .bam files while ignoring the rest.

Skewing this slightly off topic, I have found that multiple (3-4) .wav files played simultaneously ingame have as much a severe impact on performance, as when the entire screen filled with several large .bams. This means that if you wish to play some sound at a louder volume, the only good way to do it is edit the sound itself by amplifying the waves and saving it as a different .wav rather than playing one sound many times.

-Galactygon

I suppose that you don't consider the installation of The Bigg World Project 'normal' then, do you? Because if I've interpreted Marupal's posts correctly, the BWP biffed the same resources multiple times, because some of the member mods also biff themselves, and then comes the BWP biffing. I'm not familiar with BWP, so I might be missing something.

No, I consider BWP biffing 'working', but 'stupid'. Even if you biff any file multiple times, it'll be available exactly once in chitin.key, so the game will work. WeiDU '[U]ninstall' (as coded in the various big mods) works flawlessly as well as long as you don't use stack uninstall. The only thing I find stupid is that it biffs itm, spl, etc. files, not because of performance reasons, but because "it looks messy otherwise", and by doing so it makes stack uninstall unsafe.

And not only is it a bit stupid, but it wastes huge spaces if the same resource is biffed many times in different places, because it has been changed... but that's a very minor thing as long as it works, and it does.

@Baronius, It's the "Big World Project", it's has nothing to do with the honorable The Bigg.

Whoops. I knew it wasn't his project, but it seems I've seen too many Biggs recently


Why is it unsafe exactly? (An example is also enough, if you prefer that.) By the way, it's good to gather these facts in one place here, they might be useful for IEMT too. Thanks for your time and answers, TheBigg.

Well, the way it's done in the Big Mods*, at uninstall the old chitin.key is restored but not reloaded. Thus, if (fantasy example) baldur.bcs was in data\scripts.bcs and data\tddbcs.bif but not in the override, during a stack uninstall of tdd, later mods trying to append baldur.bcs will load the copy in data\tddbcs.bif rather than the copy in data\scripts.bcs, possibly failing if data\tddbcs.bif was removed.
Later WeiDUs have a command to create biffs in a completely stack-safe manner (MAKE_BIFF), but unfortunately it hasn't caught on (since I refuse to add a DELETE or MOVE command, making it impossible to biff the various itm,cre,spl... files in the override).

Big Mods = TDD, BGT, CtB and the likes.
Big World Project: a pdf file maintained by Leonardo Watson containing install instructions for a large selection of mods (over 600).

I don't remember where I heard this but somebody claimed that the BG engine(s) didn't use normal file i/o. Instead they used file-mapping (memory mapped files) to pull off the ability to have so many large biffs in memory and still be able to run on a small system (the first computer I installed BG on only had 128 meg.) Don't know if they knew what they were talking about but this makes some interesting reading:
http://msdn.microsoft.com/en-us/library/ms810613.aspx

Large (or any) biffs aren't stored in memory even with memory mapping (it wouldn't make any sense to waste so much memory). It doesn't require memory mapping to read just a certain part of a large file to memory, so I'm sure it's used for other reasons in BG (e.g. more efficient I/O due to buffering, transparent handling of files etc. -- they probably didn't want to implement these on application level).

Bif files simply contain game files. When the game needs an item, spell, etc. resource and it can't be found in the Override, it finds the resource data in the corresponding bif file (based on the information in the KEY hashtable in memory), and reads all resource data (and not just a part of it: e.g. it can't just partially deserialize an item, because all data are required for an in-game item object). So usually, in terms of memory usage, it probably doesn't matter whether a file is in Override or in a bif -- it needs to be completely deserialized.

So it's more likely that the game handles WAV files read from bif in a different way than WAV files from the Override (this is based on what TheBigg has written about WAV performance, I haven't tested it). For example, it has a cache of a limited size for WAV files read from bifs (and if it's full, it drops old elements), but there is no similar limit for files read from Override, or something like this. This is a primitive and highly unlikely example, just to illustrate what sort of things I can think of in this case. Nonetheless, I really haven't done any research on this (so the case might be totally different and there is no difference between how bif and Override WAVs are handled, and thus we're missing something else), so I really don't want to make further assumptions. As TheBigg has said, some tests might help to find out more information on this.

I really can't confirm whether BG does file-mapping or not. Somebody may have been making it up. It was just something interesting that came up. I had never heard of it before then.

"Deserialize"? Deserialize:verb->Something I had to look up. Tis and wav files don't need to be deserialized though. Even the compressed versions are done with zlib so will do streaming decompression. Nothing but opinion here: having all the wavs in one file (or very few files) and only needing handles to those very few files would be easier to maintain than having to account for the possibility of hundreds (or thousands maybe) of handles being open and in use (ambients, cre sounds, npc sounds, mus, etc.)
Here's what I am seeing though. Going through and biffing the BWP install-> 44,000+ files, 2.7 GB override folder, 2.2GB used by wavs. Biffing the 32,000 (non-wav) files had no noticable impact on speed. I haven't touched the bmps, bams, spls, pros, or 2das (cumulative 6000+ files there.) I'm currently half way through biffing the wavs and immediately see a major change in game speed. Tried all kinds of different settings in configure options including pathfinding. Nothing had any noticable impact except biffing the wavs. It was so bad that you could click on a destination and the game would look like it locked up and wouldn't do anything again until it showed you at the destination. Really bad stutter.
Maybe next time I'll start by biffing the wavs first and see if get that same increase immediately.

What does this exactly mean? How exactly the performance problems arise (this is also directed to TheBigg)? After playing through a few areas, perhaps after playing hours, or immediately? Does it happen only at certain points of the game? Or is the stuttering periodic (with a high frequency)? Does reload or game restart+reload help (the latter one certainly eliminates memory leak caused by unfreed objects accumulated during a longer time of playing)? Furthermore, what kind of WAV files are these (i.e. the most of your 2.2GB), e.g. dialogue sounds of mods?

If the stutter is due to some memory leak (or inefficient memory utilization), it should only happen when or after a lot of WAV files have been accessed. The game doesn't access thousands of WAV files at the same time, so I suppose the stutter doesn't happen quickly for you (and reload should also eliminate it). I also forgot to ask it from you and TheBigg: is the stutter due to WAV files eventually caused by intensive I/O operations (i.e. your hard disk LED is active)? I suppose it does (and this obviously doesn't answer why it's happening, because more things can cause Windows to write the pages of processes to disk), but I ask it just to make sure.


I don't have a proper testing environment set up at the moment (for example, I have no 2.2 GB of WAV files of mods), just an installed BG2:ToB. I extracted all 9800+ WAV files from the bif collections to the Override folder, and quickly visited 2-3 areas (one of them included a battle with Draconis) in the game. There was no stutter. Of course, this doesn't prove or confirm anything (e.g. I should've played for a much longer time, and should have at least made some memory profiling), but at least I can react to your comment about handles:

(1) I doubt that a high number of handles can cause such performance issues that you & TheBigg experienced. The system can cope with it. (This has nothing to do with my primitive test mentioned above -- it's a general statement.)
(2) BG2 doesn't seem to have more than 200-250 open handles at the same time, and only a small (< 10) percent of these are file handles (most of them are for the constantly used bif files and music). It didn't keep open the handles of WAV files it played from the Override, they were immediately closed after use, just like .itm, .spl etc. files. Of course, the fact it correctly closes handles doesn't mean it also frees certain objects (this is why some memory profiling would have been handy). On the other hand, as far as I've seen, it keeps open the handles of music files (even if the music isn't played anymore).

On a side note, BG2 (unsurprisingly) uses DirectSound to play audio files. Perhaps the code that passes data to the DirectSound component is buggy for WAV files from the Override. But this is just another of the many assumptions I've made. Some day when I have time, I should really do some more useful tests in a proper environment.

As far as file-memory mapping is concerned, some components used by BG2 (perhaps some DirectSound audio stuff IIRC) seemed to use it, so your information source might be partially correct with that (but I doubt that BG2 itself accesses files in that way). But as I've written earlier, I think this doesn't matter in terms of the performance problem we're discussing.

Quick question: did you guys check the memory usage of BG2 during the stutters? (Sorry if it's a stupid question and you would have already mentioned it if this was the case.) For example, you can use the Task Manager's process list for it.

Basically, I had progressingly worse stutter the more animations I had (E.G. having all six party members walk across the screen), from as soon as I started a new game. Sounds would play normally, but the screen would have a very low frame per second; if I paused the game (or didn't have characters walking) the screen would update without noticeable lag.

Can't be sure about memory use, but IIRC it was in the same range before and after biffing.

Same here. From the minute that the game was loaded. There's no stutter on the menus, ever. i.e. save, character creation, backpack, journal, etc. Framerate is probably close to 0.2 fps. Hitting pause takes around 2 seconds to take effect.

That's interesting.
Here's another strange thing about it: configuration->game autopause on mini map->no. No stutter while watching the character cross the mini map.
It appears to be two thirds related to dialogue, but there's quite a bit that I can't identify what mod installed it or what it's purpose is (except to play. )

I'll have to check into this later today.
It doesn't appear to be memory leak related though. I've not had the game crash on me yet just because it had been running too long and I've left it running for days. I wish Internet Explorer behaved so well.
Maybe this is the source of the problem. Maybe the .itm, .spl, etc files don't have an impact on speed because they are (usually) so small and don't require much in memory allocation but wavs do require more in the loading and unloading and logic to determine when that ambient (or whatever) needs to be unloaded. Of course you dropped all the wavs in your override and didn't have any problem.

Not much more activity than would be expected there for me.

I'll proceed with making even more unverified assumptions.

It's interesting where we are: the game apparently stutters as the number of animations increases, but the phenomenon somehow correlates with the presence of a lot of WAV files in Override.


As I've written, I played only for minutes, the game didn't access more than a few tens of WAV files at most. It was a quick, incomplete test.


This is because BG2 has a GUI thread, which runs concurrently with the scheduler/engine thread (examining BG2's scheduler is another topic I would be interested in, if I had time). The stutter (unsurprisingly) happens in the engine thread.

The stutter may be due to an 'expensive' operation performed a lot of times. What can execute things a lot of times in BG2? For example, the scheduler -- which also deals with the script blocks. But this is the point where there are too many assumptions and too few certain things. Considering neither of you confirmed that you experienced I/O overload (i.e. hard disk LED blinking intensively) and the game doesn't access thousands of WAV files in the period you were playing (no matter if they are in a bif or in the Override), the explanation might be totally elsewhere than where we're searching for it.

The best would be if I could get a copy of your installation (with all the necessary files to build the same installation, i.e. chitin.key and the mod files, including the WAV files; but excluding original copyrighted game files, of course). I know it's gigabytes, so I don't suppose you would want to upload it e.g. to BWL. It's just an idea.

However, one point:

The 32,000 non-wav files only used up ~500 MB for you, while even the half of the WAV files is 1.1 GB! This means that the whole thing might not necessary have to do with WAV files, but rather the size of Override (earlier TheBigg had a similar note somewhere, i.e. that the problem arises if the Override size is comparable to the RAM -- however, I don't remember if there has been an explanation for it too).

By the way, does this WAV bif/non-bif problem arise for others too (possibly without BWP), or did both of you experience it only with BWP? It might be a BWP-specific issue, perhaps an error in BWP somewhere.

I have another assumption here: as far as I can remember, the game queries the Override folder sometimes (lists the files in it). Perhaps it does something additional with the WAV files it finds (and doesn't do it if they're in BIFs). But these are just assumptions again, this is why it would be better if I had the same BWP installation as you guys (no, I don't have time to build it from pieces, from scratch), because I could reproduce the phenomenon.

After that (early) post, I tried to reproduce the problem on my machine (by using /maxmem=512), but failed without lots of WAV space.


When I experienced it (two years ago I believe) there wasn't even a BWP - we still had the (non-WeiDU) BP-BGT-NeJ compilation from plaguezone (and I didn't have that either).

OK. I'll zip it up today. It really is going to be big though. 6 GB in the data folder and 2.7 gb override folder. You'll need the dialog.tlk and the chitin.key. I think that should be everything you need to drop it onto a normal BG2 install to get it to work. Can you think of anything else?
Where would you like me to upload it to. This is a BGT installation and it includes all the BG1 files so someplace private would be appropriate to keep from violating the copyright. I already know you have both BG 1&2. Et tu The Bigg?

I've seen a few other posts about stutters. The Bigg also wrote a util to test script blocks to see if they are culprits. I've never experienced stutter without mods installed.

I also had the same problem then but the BWP install is much worse. Still very much worth playing though. Had to add that.
Let me know where to upload this (it'll probably be the end of the day before I can upload at the earliest) and maybe I'll break it up into a couple of files.

The last time I had problems it was two years ago. I don't have the stuff available any more.


It's two different kinds of stutter: here we're discussing low FPS, whereas that script is about characters who stop their action for no particular reason.

It's good to know that the scripts aren't responsible for FPS stutter. Thanks.

Well, it's possible (it's known that Improved Anvil has FPS stutter if you have Bottomless Bags) - however, my script isn't made to deal with those, it's made to deal with a specific kind of character stuttering (as opposed to game stuttering as we're discussing here).

Just biffed the tis files using NI and got the same corrupted graphics. I know I've read about DLTCEP being a GUI front-end for Weidu but NI doesn't use Weidu to accomplish things, does it?

No, it doesn't.

I made some partial tests a week ago (many thanks to Marupal for providing me his BWP installation), the results can be read below. Note that I didn't test any (wav) biffing (and how it affects performance), all the files were copied from Marupal's files to my existing BG2:ToB installation (e.g. my Override has 51563 files). The computer I used for the test is an old one, P4 3.2 Ghz (HyperThreading), 1 GB RAM, 2 MB cache, etc.

The initial startup time wasn't so bad for me, the game loaded for cca. 90 seconds. It loaded for you much longer, Marupal, didn't it? Reason: BG2 lists all files in Override when it starts.

I loaded an earlier save game (in Chateau Irenicus), and I experienced stutter too. When I command the party to move to a point, the game stutters, and after seconds, the party usually appears at the destination point. Reason: unknown from these tests. Nothing seems to access the WAV files in Override meanwhile, by the way.

It's interesting though (at least to me, I didn't know it earlier) that when the party is moving, the game accesses 11 SPCLxxx.SPL (= innate abilities, HLA abilities) files, very many times, i.e. with a high frequency (I didn't measure whether they correlated with the AI updates, but as far as I remember, there were hundreds of accesses in a few seconds, always the same sequence of certain SPCL files). I doubt this causes the stutter, it's just an interesting thing I noticed.

The game shows some additional activity (that can be monitored) during the movement of the party, but I didn't examine it (if I remember correctly, they were some memory sections or temporary files created -- perhaps some DirectX stuff).

By the way, the BWP main screen/menus are very nice. Who made it? Really nice work.

Did you check RAM usage and/or open handles? My wild theory is that, when the game opens the dialog.tlk, it loads all WAV files referenced in the tlk in RAM (or keeps the file open, either for fast referencing or because of a coding error).

Does the weidu.log contain Refinements? I know Refs does lots of spell tricks under the hood (I don't think this causes issues by itself, since I played the mod on a P4 1.6Ghz with 256MB of RAM).


MMIO mapping of animation files? The engine used for The Broken Hourglass does this to reduce RAM usage, for instance.


Unless BWP has GUIs of his own, each of the 'large mods' adds its own main screen to a setup-gui.tp2 mod (which is written on the fly each time you install a new mod with a gui). Thus, looking at the comment string in weidu.log for that file, you can figure out which mod is it from (and hopefully, from the readme, the author of said GUI).

It does but just the revised HLA's and Universal Lesser Mage Robes.


That one is from BGT. Couldn't find who the particular author of "Elminster holding the staff" was.

The computer I used was even older->AMD 2.6 768 MB RAM. I'm surprised it loaded that fast for you when it takes quite a few minutes for me. Will check next time how long it takes unbiffed.


That would make sense. It's pretty strange that having 30,000 files =500MB biffed doesn't make much of an impact on performance (game initialization does speed up though) but 6,000 files=2.2GB biffed takes away from the "teleportation" stutter. Maybe someday they'll release the source code and everything will make sense then

It's the Revised HLAs which access spcl* lots of times.

Perhaps it's related to disk speed.


Doubtful, Bioware is now a property of EA.

RAM usage is approximately constant, cca. 110-115 MB. Before loading the savegame, it's about 50 MB.

The open handles are just the usual ones (GUI biffs, SOS & ROT biffs, log file etc.)

As far as your wild theory is concerned, during my test, no Override WAV files were accessed from game startup to the end of the savegame load. After that, only 4-5 area-related WAVs and some character sounds (female1i.wav etc.) are accessed, continuously (i.e. with time, as I click on characters more times and move in the area). Even these WAV handles aren't kept open, but their content are stored in the game's address space (unlike with CRE, ITM, GUI BAM etc. files which are read from the file each time they are referenced by the game). Based on this, your assumption that it keeps WAV file content in memory sounds to be reasonable, but the lots of WAV files in Override just doesn't seem to be accessed, as I've written.


I checked them again and as far as I can see, they just seem to be temporary files related to Sound Card drivers or something like that. When the characters move, 2-3 such file handles are open and then immediately closed.


It does it from some continously running script I suppose, considering the very high frequency of accesses to the same set of files. It's interesting though that about a week ago, I removed the SPCL files in question from the Override, and as far as I can remember, SOS-BIF.bif (or something of similar name) accesses appeared instead, so the chitin.key entries for those eleven spcl spl files point to the SOS-bif (Shadows over Soubar?) -- as far as I can remember (I haven't checked chitin.key).

Were the ambient sounds on(not turned to zero), as they do real hazards for fps counts if there is an error in their assignments in one or more of the mods, as the behavior would suggest it.

By the way, it would be good, if the particular BWP's version and every other essential info(WeiDU.log) would be documented, but then again it would turn this thread to something completely different, which wouldn't be The Biggs problem.