I am in the middle of installing Big World and specifcally the Lost Crossroads Spell Pack for Baldur's Gate 2 = SpellPackB5 and am encountering an error.

When trying to install the Core Files and Graphics it fails with the following error. "DEITM049.ITM: read out of bounds".

Can anyone help resolve this.

Thanks.

You are not the first to report an out of bounds error.

I do not have DEITM049.ITM on my install, so I am assuming it comes from one of your mods. Recalling past experience, I recommend you:
1.) open DEITM049.ITM using Near Infinity or DLTCEP.
2.) Just save the file.
3.) Rerun the setup.

Before you attempt the fix, could you do me the favour and attatch the file? It will be useful for my testing purposes.

With more and more people running out of bounds, I will do everything to fix it by B6.

-Galactygon

DEITM049.ITM is a broken file from FR_ROV (has 6 effects, but the header says it has 10). The problem's been around since forever, but nobody cared to fix it.

I usually workaround problems with that file (and other similarly broken files) by using ELSE in all READ statements (READ_* offset variablename ELSE 0). If the read is out of bound, 0 is returned instead. Use a fallback number so that no action will be taken (for ex., if you're looking for readVal = 0, use ELSE 1 instead).

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

Additionally, I've never understood why this "ELSE 0" useful. It hides the presence of bugs or corrupt files instead of causing the installation to fail. In other words, it hides bugs instead of revealing them. Do we want players to install mods in any way (even if they contain buggy files) or do we want them to install bug-free mods, possibly at the cost of some initial, temporary installation failures?

Beats me.


The game usually plays fine if you use a file whose structure is broken like that. In this event, simply fixing and/or ignoring the problem is generally better than having 1,000,000 threads asking "how do I fix that"? See also: ToD vs. UB.

ToD vs. UB is a wrong example. UB applies a typical antipattern which introduces very serious incompatibility risks. The only case when such errors should be fixed in other mods (rather than in the mod which introduces them) is when the mod which introduces them isn't supported any more.

If FR_ROV isn't supported any more, then you're right that fixing the problem in other mods is okay. In fact, even if it was supported but a quick reaction is required, it's OK to fix bugs like this (as they don't introduce any further risks, the file is corrupt and that's all).

However, ignoring is a completely different matter:

This is exactly the fundamental difference between your and e.g. BWL's Improved Anvil's philosophy. (That is, we find every detail, feature of the mod important, because seamless details are what make a mod really excellent and powerful. Of course, this depends on the mod's type as well -- I'll explain it below -- but to most mods, this is true.)

So, assuming "ELSE 0" was used, even if the file works correctly in-game, there are still two major problems:
(1) If the problem hasn't been revealed yet, such "ELSE 0" instructions just keep in hidden further. This prevents/hinders the detection of corrupt, buggy files in (possibly widely used) mods.
(2) The mod which would have failed without "ELSE 0" tried to do something with the corrupt file, which means it wanted to add/extend/modify content. In other words, modifying that file is supposed to be a feature of the mod, something which belongs to its content.

You rather prefer happy players, who don't complain or possibly put apart the mod due to the negative experience with the installation failure; happy players, who play the mod, even if with missing features of the mod. If the author of the mod is happy with this fact, then OK, it's his or her decision. Of course, it's not a disaster if a mod like SpellPack can't apply its changes to 1-2 files from the thousands of files, but as far as I've noticed, this "ELSE 0" approach has been generally encouraged to be used in TP2 of all WeiDU-based mods. There are many cases where it's totally incorrect, e.g. for mods which apply changes primarily to specific elements (rather than applying general changes to all elements of a given type).

On a side note, there is no problem with a lot of threads, because that is why patches can be released. Once it's done, it can just be linked in every thread (and obviously there won't be hundreds of threads, just a few tens at most). The patch can be added as a pinned topic.

This "ELSE 0" approach (provided it's encouraged for all TP2s, and it is) also fits to the attitude I've noticed in case of certain mod developers/sites:

"Let's suppress, disguise problems rather than undertaking the inconvenience they cause and the effort required to solve them"

It's present at different levels and in different forms (that is, not all modders follow it to the same degree).

In certain special cases, "ELSE 0" can be justified, but not generally (for example, for mods which e.g. read all BAMs in the game, it can be useful). But usually, it's not required at all. Any WeiDU tutorial content (or anything like that, such as sample code) that generally encourages the use of "ELSE 0" should be corrected, or it will be misleading, especially for new modders who search for help material and base source code for their mods.

To sum up: except in special, justified cases, I believe that preventing bugs for the sake of "successful installation" instead of recognizing and fixing them is a strongly incorrect attitude. Of course, in this concrete case (SpellPack), it might be okay (usually, for instructions such as COPY_EXISTING_REGEXP with a predictably high number of matches, it should be OK).

I wouldn't call neither UB nor FR_ROV maintained, since their author and maintainer are (almost) retired. True, JC and/or SC could fix them by authority (I'd have done so if I were an admin). The reason why this doesn't happen beats me, but I suppose it's related to a misguided sense of copyright. Incidentally, in the case of FR_ROV, the mod's forum contains the fix as an attachment in a post (not even a pinned one), but it hasn't been applied since forever.

I find your argument misguided. Between threads, bumps, and unrelated bumps, there are at least 300 people who have failed to read the pinned topic before asking about ar0100.bif.


I believe that FR_ROV was made in 2002 with some older teambg tool (not IEEP). I doubt that this type of file corruption is something you must watch out for in newer mods. Also, "Be liberal in what you accept, and conservative in what you send." (Jon Postel, RFC 760).


I don't understand this from a grammar POV. Do you mean that it's morally wrong for Spellpack to alter FR_ROV, or that Spellpack will have broken features?
In meaning 1, well, it's not really the fault of ELSE 0 for a morally wrong action. In meaning 2, I believe you're misunderstanding what ELSE 0 does. Generally speaking, ITM patching code looks like

If an item has 6 effects but states it has 10 (deitm049's situation), then using ELSE 0 will mean that the condition in the 'if' will be automatically false for effects 6-9, whereas it'd be applied normally for effects 0-5. If your code doesn't have the 'if', then the mod will generally fail at the doSomething() step.

It's meaning 2, but with the code you've provided, it's OK (Galactygon can use that in his code). It's a code to manipulate many files of the same type, and it's reasonable to make such a process fault-tolerant (especially because it won't really affect the mod's important features if a few items won't be updated, as opposed to mods where each change is very important).

When I state that it shouldn't be used everywhere, I refer to cases when it's used to omit a feature just to suppress an installation error regardless of the importance of the omitted change. The installer's logic is "if I get a valid offset, apply the changes for this feature; if I get an invalid one (e.g. due to ELSE 0), skip this feature". For example, this is the case when an offset in a header is incorrect; the feature of the mod that needed that file will missing from the installed mod, and no one might notice it (while in case of an installation error, the problem is recognized and can be fixed).

If the probability of this type of corruptions is so low, then why to ignore them? Since they don't happen often at all -- but when they do, they can possibly prevent a feature of a mod from installing if 'ELSE 0' is used -- why is it a problem to deal with those few cases (even if they result in many player reports)? In this respect, they're like disk errors: they don't happen often at all, but when they do, their effect is really undesired (= a mod's feature isn't installed and no one might notice this). That's why it's not a good practise for new modders to use "ELSE 0" everywhere to suppress errors.


I meant that you make a pinned topic about the patch, and link to that topic in every new thread about the problem. Honestly, I don't think it's such a horrible burden for a mod author to reply with a link e.g. once per day (and it's probably an exaggeration that such reports are posted in every day by players). It especially shouldn't be considered a burden, because it's about fixing a bug, a problem. Should we suppress bugs because authors are lazy? (Again, I'm not referring to the -- really narrow -- group of mods which use COPY_EXISTING_REGEXP or GLOB or whatever, with quite wide masks) It should be up to the author to decide, yet most WeiDU-based mods (and sample codes) are full of "ELSE 0" statements, clearly suggesting that it's a generally good practice and should be used everywhere. Since new modders will check the code of existing mods, their mods will also contain the same code, which newer modders will use as a basis, and so on... The point is, many new modders who use sample code / existing code don't know what "ELSE 0" actually does, they just use it because they believe it to be some necessary thing or good practice.

About a year ago, I've seen some G3 WeiDU tutorials and mod codes, and they were full of "ELSE 0" statements, but without explanations. Since new modders consider such codes ("TP2 ninja" codes) as the most credible WeiDU TP2 sources, there should definitely be more documentation and explanation. It is similar to the problem that codes obviously use absolute offsets, but no one warns the new modders who use the code as a basis that those offsets can be different for their own files (inside variable-length sections).

To cut a long story short, modders (who are less familiar with this type of stuff, unlike you or me) shouldn't be kept in the misbelief that "ELSE 0" is something necessary; unfortunately, that's exactly what this "use existing code / use sample code" approach does. Tutorials should explicitly explain what the purpose of "ELSE x" is, and some modders should surely update their mods (I can imagine that there are many mods which use "ELSE 0" when they shouldn't). So each modder could decide on his/her own whether he/she wants to detect or suppress errors. Generally, the first case is much more beneficial.

About sample codes (as I'm sure most modders learn "ELSE 0" from such codes):

I tend to criticize certain modders/sites for influencing new modders and players too much (occassionally with unacceptable methods), convincing new modders and players that their methods are the best and superior, and others methods (such as Improved Anvil's solutions) are inferior. Actually, 'sample codes' are also a very effective way of influencing modders, and of course a totally acceptable way (most modders share their code also to help others, no doubt). However, sample codes have the disadvantage that they don't help the modder understand the interrelations and actual solutions, and on top of it all, they implicitly influence the modder's beliefs about what correct coding is. This is why tutorials are much better than such codes (but I understand that not all modders have time to write tutorials while sharing code is instant). Tutorials give general and widely usable information. So WeiDU code sharing is like an element of the child-parent relationship: the parent shows an example, the child (implicitly) learns it and starts to use it. For example, Gibberlings3 WeiDU coding standards and the associated "compatibility" definition was a typical example to this phenomenon: certain modders believed (or believe) that Improved Anvil isn't compatible with other mods because it's not designed as smartly as e.g. Gibberlings3 mods that are advertised as compatible with tons of other mods. (Needless to say that it's false; Improved Anvil's quality and robustness can be thanked to its great design and careful interoperability policy with other mods. There would be no way to make it more compatible with mods without giving up its quality or robustness.) They had/has this misbelief because they aren't skilled enough; they just implicitly learnt the coding standards and samples of many Gibberlings3 mods (and they didn't even notice how much they're influenced -- the child also doesn't know he/she is biased due to the example shown by the parents). If there were comprehensive tutorials and guides available, these modders would be provided with real information and knowledge, and thus they would get the chance to make unbiased decisions and judgements on their own. Actually, in this respect, sample codes (and the use of existing mod codes as a basis without understanding the deeper interrelations) can be considered as a -- sometimes probably unintentional -- way of making modders biased; if modders were forced to learn and understand the basics due to the lack of tutorials and sample codes, they would be able to make much more conscious judgements and decisions. This paragraph has become quite long, so let me close it with a message to all readers who are interested in modding with WeiDU: learn the interrelations, try to understand everything! Learn, if you want to make a good, above average mod! The statement "modding/coding is surprisingly easy" is a humbug; you will experience it on your own if you don't believe me

Here is the file before I did as you suggested. I will let you know what happens.

Opened the file in Near Infinity and then saved it then reran the install. Install failed with the same error message.

Is there a way I can edit this file to get it to work? Please keep in mind that I am pretty much of a Noobie in this area so will need pretty simple and specific instructions.

Thanks.

In my case, ar0100.bif isn't a bug, and the only critical bug I'm aware of was never reported
While I don't get them one per day, I still get them with an high frequency (I might get ten in a day if I was mentioned on a particularly well read magazine).


I'm not really interested in bickering back and forth about how The IA Way is good vs. how The IA Way is bad, so I'll leave you to your sense of victory.

Beach: remove the file from the override directory, install Spellpack, and put it back there.

BeachBum: as far as I can see, the file you attached is still invalid (IEMF detected an error). I quickly corrected it with a hex editor, the feature block offsets seemed to be invalid in the extended headers. So you can also use the file I attached (right-click the attachment, and choose "Save As...", save it to your Override and rerun the setup). In this way, you will have a valid file in your Override (not just for SpellPack but e.g. for further mods as well that might access it).


There is nothing that can be discussed there, because I merely mentioned it as an example of how sample code can influence modders. (It applies to the use of "ELSE x" as well.)

Your reaction reflects the problem very well, by the way -- there is no such thing as generally "Good way" or "Bad way". Each mod and mod type is unique to a certain extent, and something which is a good way for a certain mod might not be as good for another one. This is why it is very narrow-minded (especially without development skills) to write harsh criticism about a mod just because its system doesn't match the mods favoured by the critic.

So the point wasn't that "modders should realize how Good the 'IA way' is"; it was "modders should try to understand the deeper details as well, so they can understand the technical decisions behind the system of all mods (possibly including the 'IA way' too)". It would allow their opinion on 'IA way' to be a bit more scientific than just "It doesn't work in the way Gibberlings3 modders make their mods". Such a knowledge of interrelations would also allow them to make reasonable decisions on their own (e.g. whether "ELSE x" is needed or not for a particular installation block).

Baronius -

Thanks for the help. This got me by the problem. It did appear that there were somethings that did not install properly but the majority did. I think???

Since no one has replied yet:

I suppose that if it has said "successfully installed ..." at the end, then it should be fine, but only Galactygon knows this for sure (SpellPack is his mod), so hopefully you will get a reply from him soon. I've only restored your file's consistency.

Unfortunately, I cannot return to SpellPack until December (with a release earliest in January) due to the following reasons:

1.) I have Lost Crossroads installed, and I am in the middle of writing some code and managing stuff, so I cannot uninstall and reinstall SpellPack.
2.) BscArchitecture is harder than I thought and is taking it's toll. Basically you take 30 credits worth of classes spread out over 13 weeks. I can check stuff almost every day, and reply to posts, but not much more than that.

With due patience, I will run both files through my macros and report the results.

-Galactygon

I think BeachBum was merely wondering whether or not his latest installation has been successful. If the installation has ended with the message "SUCCESFULLY INSTALLED...", it should be fine, shouldn't it? It's just a general assumption from me, as I don't know whether installation warnings have any importance in case of SpellPack. Of course, you might have meant that that you also want to make sure whether the files he installed are okay (regardless if his installation ended with a "success" or with a failure message).


Do you refer to the files posted by BeachBum and myself? The file I attached has consistent structure for 100% (of course, I don't know if it meets your criteria content-wise as well; I merely fixed the file attached by BeachBum), so no need to sacrifice your free time for its verification (but of course, if you verify it, it will be at least a good test file to see how your macros work).

If someone else verifies this for me, that would be good. I still wonder how WeiDU would bail out but the game or any of the other editors I regularly use (DLTCEP and NI) would not.

-Galactygon

Just to make sure I understand your question: are you asking why WeiDU has problems with that file ("read out of bounds") while the usual editors you use don't?

Yes, or more like how is the game able to run smoothly and how can I replicate it in WeiDU?

-Galactygon

If you want to suppress such installation errors in WeiDU, use "ELSE x" in the proper context, as TheBigg has said.

For example, here is a "patching" function call with two arguments that reads a 32-bit integer from file position 1234h to the variable "some_header_field":

READ_LONG 0x1234 "some_header_field"

If offset 0x1234 is out of range (e.g. the file size is smaller or equal to 0x1238 bytes), the operation will fail, causing an installation error. On the other hand, the following form works differently:

READ_LONG 0x1234 "some_header_field" ELSE 666
PATCH_IF ("%some_header_field%" != 666) ...

In this case, the value of some_header field will be 666, if the offset is beyond the end of stream. There will be no installation error. You can detect that there was an error by checking the returned value (obviously, to indicate the error you should use a value that can't occur in normal circumstances).

You said you learnt some Java earlier. You can think of the difference between the two solutions above as an Exception and a null check. That is, a run-time exception interrupts the normal flow of the program, while a null check can be used to detect a problem "silently" when you surely know that the returned value is a valid reference in normal circumstances (and you don't want to use exception handling, e.g. because the failure is expected to happen relatively often).

So you should add "ELSE 0" statements (with additional code if needed, I don't know your concrete code) to every function call where you expect failed reads and want to suppress (or specially process) the error. As I've emphasized, suppressing errors in mods is not always desired; if you specifically want to set a field/entry of a particular file and the file is corrupt, the installation won't fail, and the missing field (corrupt file) might break one or more features of the mod (which have the field or entry as their dependency). On the other hand, suppressing errors for features which absence can be tolerated is another question (it's a question of taste/style and/or of the mod's goals) .

Of course, there are other ways as well in WeiDU to detect that a file is corrupt (e.g. SOURCE_SIZE -- or something like that -- returns the buffer/file size, so you can check whether it has the required minimal size).


It ignores the problem if an invalid value is returned (exactly what you do with "ELSE 0" ). (I would've guessed that the game crashes when a referenced feature block doesn't exist, but if this inconsistent deitm049.itm works for people, then my guess was wrong and the game just ignores such problems). I didn't test how exactly the game handles item files that are invalid in this way, but it's a general fact that the file structure specifications are stricter than the implementation that deals with these files. This allows the game to be more fault-tolerant when it's possible (for example, a missing dialogue file doesn't cause any crash, but an area without a tileset cannot be tolerated ).

As far as I've noticed, DLTCEP tries to read most file formats in a robust way too. How can it be accomplished? I haven't checked the implementations in DLTCEP and NI, but I guess that e.g. DLTCEP often simply considers the file as a sequence of contiguous sections (e.g. header, extended headers, feature blocks, equipping feature blocks). Where possible, it reads all sections (when it finishes the reading of a Section A, it immediately starts to read Section B that follows Section A, regardless what the file header says about the position of Section B ). Of course, this is a bit simplified, but you get the idea. This allows DLTCEP to read files with inconsistent or sometimes even corrupt structure, and when you save it again, it provides valid values for the header fields that were incorrect in the originally loaded file -- which results in a consistent, valid output file. This is how "broken files" can be fixed.

However, this robust approach has its own drawbacks. First of all, since these editors (such as DLTCEP) aren't built for data recovery from corrupt IE mod/game files, the algorithms aren't too complicated, there aren't too advanced heuristics. Notice that data recovery and file editing have different goals; data recovery has its own uncertainty (the returned data might be corrupt) and it usually requires human interaction, while file editing should always guarantee a consistent structure, and when this isn't possible, it should indicate an error. It's typical example of the question whether we want to correct the error or to indicate it. The latter one is usually easier and -- more importantly -- safe, because correction may fail and return corrupt data without indicating that it's corrupt (but discussing this would exceed the scope of the present topic). This drawback is present in DLTCEP as well: when it reads dialog.tlk (probably with the aforementioned procedure of continous reading that ignores certain offsets) and there is an inconsistency, it offers you to make an attempt to correct the problem. If you click 'Yes', it applies its "corrections" and saves the file. Depending on the type of the faulty field in dialog.tlk, this can corrupt dialog.tlk! For example, the unofficial Russian dialog.tlk works perfectly in the game, but it contains a very tricky offset assignment (I wonder who had that idea and what he/she wanted to accomplish with it) which DLTCEP detects as a consistency error and offers to make an attempt to fix it. Its fix will corrupt the file (the game will not crash, but many dialog.tlk strings will become totally senseless, corrupt), because the assumptions made by its heuristics prove to be wrong. Needless to say that DLTCEP doesn't detect that it partially corrupted the file (this is a common drawback of correcting errors, as I've said), because structurally, the file will be consistent, but semantically, it will contain several corrupt, unreadable texts.

Considering BeachBum could open and save deitm049 in Near Infinity, this software probably uses the same or a similar algorithm to read items as DLTCEP (i.e. trying to be fault-tolerant when possible), but when you save the file, it does not correct the invalid fields. It probably writes back the same values as it read out (this has an advantage as well: it won't accidently corrupt an inconsistent file that possibly works in the game itself).

I hope I managed to answer your question Galc, and hopefully the readers of this topic also got an insight to the question of when "ELSE 0" should be used and when it shouldn't (and why).

I'd guess that the game (being coded in C++) simply blits the file in memory and then uses pointer arithmetic to read the various effect blocks, thus reading past the end of the actual file if fed wrong positions (causing random behavior, generally without crashing visibly). WeiDU had a similar unchecked memory access in SET_2DA_ENTRY, which used to randomly work fine and crash before I enabled safe arrays.

Possible. One more reason why I generally don't recommend to base important decisions/statements on observations such as "it works in the game", "it doesn't cause any crash in the game". Such a decision can be whether to follow the IESDP descriptions when implementing a file format handler (or when editing a game file for a mod). Generally, it's a good idea to keep conformance with IESDP, because -- apart from a few cases -- IESDP is relatively accurate. (On a side note, however, sometimes it's a matter of interpretation as well: for example, the BIF entry searching/matching behaviour implied by IESDP's file description is not what actually the Infinity Engine games do. As far as I remember, gemRB does resource locator comparison between the KEY BIF and the BIF BIF entries, which is a completely incompatible behaviour with the Infinity Engine games, and as such, certain files which worked perfectly e.g. in BG2 will behave differently in gemRB. In such cases, it might be a better idea to follow the original game's behaviour, considering that IESDP merely specifies file formats and not the associated behaviour.)

Oh dear.

I wish Real Life wouldn't prevent me from reading the forums more often. You two are worse than Kulyok and me.

FR_ROV_updates_for_v5.rar fixes (DEITM049.ITM: read out of bounds). I'm sure it'll be rolled into v6.

I actually meant that the third-party code (e.g. in this case, SpellPack) could fix it instead of suppressing possible errors due to it (with "ELSE 0"). It's acceptable in cases when the buggy mod isn't supported (and that is what I suspected that time).

As far as kulyok's sayings are concerned, it's no surprise for me (she has been spreading lies about Vlad's BWL mods as well, I guess she wants everyone to use Pocketplane WeiDU mods). I wonder if she is paid for it, or if not then why exactly she needs to criticize everything and make attempts to manipulate players... As TheBigg would say: beats me.

Yes, I am thinking on ways to do it. I can already guess that in some cases the last effect in the last header isn't fully defined (with the opcode being a BYTE instead of SHORT), and I have to insert 0 bytes until it is.



While there might be people who tolerate this, I would like to make myself clear: I don't. Regardless of who is right or who isn't. Let's not bash people and stay on topic if you please. This goes for everyone just as much.

Thank you,
Galactygon

Glad to hear that. I hope you will practise that intolerance the other way around as well, i.e. when you see real bashing against BWL members and works.




Regarding the solution of the concrete problem discussed in the present topic, I made a reference to it in this paragraph:



As TheBigg has said, the culprit is DEITM049, and if you want to fix it, the base of the solution can be read above.

If you're speaking generally (more precisely, in the context of other out-of-bounds error with SpellPack), this 0-byte insertion doesn't sound like an elegant solution (I even don't remember why such a thing is needed in WeiDU). If you show a code example where you believe there is such a problem, I can say more if you're interested.

When anything like that happens in my subforums, I will. I am not the type who goes into stuff like this in other threads in other forums, but when it spills over to here, I have to draw a line.



This is what I meant, really. We were cross-talking a bunch. A macro that fixes the number of effects if there is a gaping disagreement between that and SOURCE_SIZE.

I do not have my macros up at the moment and I have never done anything like this, so I am posting my thoughts rather than testing them through trial and error.

1.) The macro looks at the last header, counts the number of effects and compares it to SOURCE_SIZE.

2.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE but not by more than 48 bytes, then we assume the last feature block didn't get filled in. So we expand the SOURCE_SIZE gets with zeroes as needed.

3.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE by at least 48 bytes, then we assume there is something wrong with the number of effects. We then start counting the number of effects manually until we run out of bounds, counting feature blocks that might be less than 48 bytes. In that case, we go to 2.).

This sounds somewhat right, would I have to account for headers that go before the last header, or is it always the last header that causes all the problems?

-Galactygon

Where do such corrupt files occur? The algorithm you outlined is a general one, but still only solves one type of problem (with certain assumptions). There are many
ways a file can be corrupt, is this "broken effect" problem typical to certain files encountered by SpellPack (and this is why you find it reasonable to add generic fix code for it)? Where do these files originate from?

Yes, this would solve only one type of problem, but it happens frequently to warrant my attention. I can still have my macros bail out and skip that .itm/.spl file if the file is hopelessly corrupted and all else fails.

-Galactygon

I see, but I asked what mods (or perhaps the original game?) add these corrupt files. Normally, such horribly broken files shouldn't occur often at all. If they do, it means something is corrupting or has corrupted game/mod files.


Filled in by what? A WeiDU installer of a mod? If so, we're talking about mods which produce broken files (I don't refer to the question of mod quality now, I am just wondering).

On a side note, in case of DEITM049, nothing was broken in such a cruel way (i.e. end-of-file in the middle of a block). It just had incorrect feature block offsets (indexes) assigned (if I remember correctly, their value was greater -- or smaller? -- by one than the actual position).

A .spl or .itm file consists of a single 114 byte header, multiple (56 bit for .itm and 40 bit for .spl) extended headers, and multiple 48 byte feature blocks or "effects".

I once ran a test on one of the corrupted files, and found out there was nothing wrong with any part of the file except the last effect in the last header. When I READ_SHORT the opcode, the installer ran out of bounds and bailed out. When I opened the .spl in DLTCEP and looked at that effect, I noticed that all fields except the opcode number were filled in their default values - zeroes.

Therefore, I am making a guess that the last effect in the last header wasn't filled out to 48 bytes and that DLTCEP filled it out correctly. What I am talking about if I could replicate this correctly in WeiDU, and correct corrupted files on the fly rather than worry about them individually or going up to each modder or skipping them altogether.

But then again, I could be completely wrong, since I did not do -- what they call -- rigorous testing.

-Galactygon

So you don't want to reveal where do these "corrupted files" come from (i.e. original game, some mod etc.) -- as you wish, though I don't see the reason. (As I said, it would help identify the problem, because as far as I can see, even the exact nature of the problem isn't known.)



Why do you make guesses when you could check it for sure? Opening the file in a hex editor to check the last header (or even the other headers) isn't really rigorous testing, it takes 1-2 minutes at most.

I don't get it.



I will have to figure out the correct offsets that lead to the last effect of the last header and compare it with whatever that file says.

Due to my studies, that will happen late-December the earliest.

-Galactygon

Perhaps I'm misinterpreting something, so let me summarize what I believed the problem is:
- when SpellPack is installed, it modifies (with WeiDU parlance: "patches") many game files, often based on regular expression matching
- some files SpellPack tries to modify are corrupt (e.g. bad offsets/indexes in extended headers)
- SpellPack's installer refers to an invalid offset (due to the invalid data it read from the file) which obviously results in an out-of-bounds error

Is this correct? If yes, my question was simple: your words gave the impression that we're talking about more than just 1-2 corrupt files (you said that the typical problem of read-out-bounds occured with more files), so I ask where do the corrupt files come from? What mod installed them before SpellPack? I am wondering why they're corrupt. For example, they are a few corrupt BAM files (even in the original game), but I haven't really seen items corrupted in the way we've discussed.

Correct.



So far, I only know of NeJ. But people have hinted of other mods, and chances are they are right. In that case, the macro is worth the effort.



I really don't know.

-Galactygon

First, just a quick note: SOURCE_SIZE in WeiDU returns the size of the complete "patched" resource, doesn't it? In this case, you need to take into account the item header size and the extended headers as well in your calculation when making a comparison with SOURCE_SIZE. In your current algorythm, you're comparing the total size of feature blocks with the file size, instead of comparing it with the expected size of the feature block section. You write "plus whatever other offsets there might be", if this is meant to cover the item header and the extended headers, then ignore what I've written above.

It will be an interesting discovery if your suspicion about partial effect blocks proves to be true. To tell the truth, I don't give much chance to it, but we will see. Instead, I assume that the case is similar to the present problem of DEITM049.ITM: the feature block indices of extended headers are incorrect, they point beyond the end of file (but the feature blocks themselves are okay, i.e. their total size is a whole multiple of 48 bytes). I don't find your suspicion too likely, because the original game doesn't contain such files, which means they must have been generated by an editor. I doubt any item editors of the past generated partial blocks (such severely damaged data structures in this case can only be the result of some extraordinary problem such as damage on files on the media or during their transfer). On the other hand, faulty calculation of feature block indices is much more likely for an editor software, so perhaps one of the old editors has this bug. The fact that the base of FR_ROV (DEITM049.ITM ) and NEJ (mentioned by you) is relatively old, it's possible that some older files in these mods were made with an early item editor which might have generated faulty indices to feature blocks in extended headers. Of course, I would like to emphasize that this is also just an assumption.

SOURCE_SIZE returns the initial size of the file (and is not correct if you use INNER_ACTION or similar). BUFFER_SIZE returns the current size.


I recall that CTB had (or still has) such a broken file as well, so that one falls under "old tools" as well (don't these mods predate even IEEP?). I guess somebody at SHS knows which files are broken in both CTB and NEJ.

I also meant SOURCE_SIZE in that way (my wording was unfortunate), I should have written 'the size of the complete resource to be "patched"' instead. Either way, I think he needs to include the main and extended headers in the algorithm he outlined.

A nice treasure is hidden in this thread.

So I'll let you know, that we have included the deitm049.itm fix in the BiG World Fixpack.

Greetings Leomar

This was fixed some time ago with Nythrun's macro and I think also the "Resource Fixer" which is buried in one of your old BWP threads, unless that is what you used.

Of course, it has not much to do with LC, except LC could use the same macro to avoid bombing out when modding erroneous items like that one.

Yes we use the "Resource Fixer", but only for BP like you've told us. So we install the fixer directly before BP and only with it together. Therefore the deitm049.itm fix is very useful for us.

Greetings Leomar

This issue has been fixed.

-Galactygon