DEITM049.ITM Error Options

[BeachBum]

I am in the middle of installing Big World and specifcally the Lost Crossroads Spell Pack for Baldur's Gate 2 = SpellPackB5 and am encountering an error.

When trying to install the Core Files and Graphics it fails with the following error. "DEITM049.ITM: read out of bounds".

Can anyone help resolve this.

Thanks.

[Galactygon]

You are not the first to report an out of bounds error.

I do not have DEITM049.ITM on my install, so I am assuming it comes from one of your mods. Recalling past experience, I recommend you:
1.) open DEITM049.ITM using Near Infinity or DLTCEP.
2.) Just save the file.
3.) Rerun the setup.

Before you attempt the fix, could you do me the favour and attatch the file? It will be useful for my testing purposes.

With more and more people running out of bounds, I will do everything to fix it by B6.

-Galactygon

[The Bigg]

DEITM049.ITM is a broken file from FR_ROV (has 6 effects, but the header says it has 10). The problem's been around since forever, but nobody cared to fix it.

I usually workaround problems with that file (and other similarly broken files) by using ELSE in all READ statements (READ_* offset variablename ELSE 0). If the read is out of bound, 0 is returned instead. Use a fallback number so that no action will be taken (for ex., if you're looking for readVal = 0, use ELSE 1 instead).

This post has been edited by The Bigg: Oct 7 2008, 02:41 PM

[Baronius]

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

Additionally, I've never understood why this "ELSE 0" useful. It hides the presence of bugs or corrupt files instead of causing the installation to fail. In other words, it hides bugs instead of revealing them. Do we want players to install mods in any way (even if they contain buggy files) or do we want them to install bug-free mods, possibly at the cost of some initial, temporary installation failures?

[The Bigg]

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

Beats me.

Additionally, I've never understood why this "ELSE 0" useful. It hides the presence of bugs or corrupt files instead of causing the installation to fail. In other words, it hides bugs instead of revealing them. Do we want players to install mods in any way (even if they contain buggy files) or do we want them to install bug-free mods, possibly at the cost of some initial, temporary installation failures?

The game usually plays fine if you use a file whose structure is broken like that. In this event, simply fixing and/or ignoring the problem is generally better than having 1,000,000 threads asking "how do I fix that"? See also: ToD vs. UB.

[Baronius]

ToD vs. UB is a wrong example. UB applies a typical antipattern which introduces very serious incompatibility risks. The only case when such errors should be fixed in other mods (rather than in the mod which introduces them) is when the mod which introduces them isn't supported any more.

If FR_ROV isn't supported any more, then you're right that fixing the problem in other mods is okay. In fact, even if it was supported but a quick reaction is required, it's OK to fix bugs like this (as they don't introduce any further risks, the file is corrupt and that's all).

However, ignoring is a completely different matter:

[..] or ignoring the problem is generally better than having 1,000,000 threads asking "how do I fix that"?

This is exactly the fundamental difference between your and e.g. BWL's Improved Anvil's philosophy. (That is, we find every detail, feature of the mod important, because seamless details are what make a mod really excellent and powerful. Of course, this depends on the mod's type as well -- I'll explain it below -- but to most mods, this is true.)

So, assuming "ELSE 0" was used, even if the file works correctly in-game, there are still two major problems:
(1) If the problem hasn't been revealed yet, such "ELSE 0" instructions just keep in hidden further. This prevents/hinders the detection of corrupt, buggy files in (possibly widely used) mods.
(2) The mod which would have failed without "ELSE 0" tried to do something with the corrupt file, which means it wanted to add/extend/modify content. In other words, modifying that file is supposed to be a feature of the mod, something which belongs to its content.

You rather prefer happy players, who don't complain or possibly put apart the mod due to the negative experience with the installation failure; happy players, who play the mod, even if with missing features of the mod. If the author of the mod is happy with this fact, then OK, it's his or her decision. Of course, it's not a disaster if a mod like SpellPack can't apply its changes to 1-2 files from the thousands of files, but as far as I've noticed, this "ELSE 0" approach has been generally encouraged to be used in TP2 of all WeiDU-based mods. There are many cases where it's totally incorrect, e.g. for mods which apply changes primarily to specific elements (rather than applying general changes to all elements of a given type).

On a side note, there is no problem with a lot of threads, because that is why patches can be released. Once it's done, it can just be linked in every thread (and obviously there won't be hundreds of threads, just a few tens at most). The patch can be added as a pinned topic.

This "ELSE 0" approach (provided it's encouraged for all TP2s, and it is) also fits to the attitude I've noticed in case of certain mod developers/sites:

"Let's suppress, disguise problems rather than undertaking the inconvenience they cause and the effort required to solve them"

It's present at different levels and in different forms (that is, not all modders follow it to the same degree).

In certain special cases, "ELSE 0" can be justified, but not generally (for example, for mods which e.g. read all BAMs in the game, it can be useful). But usually, it's not required at all. Any WeiDU tutorial content (or anything like that, such as sample code) that generally encourages the use of "ELSE 0" should be corrected, or it will be misleading, especially for new modders who search for help material and base source code for their mods.

To sum up: except in special, justified cases, I believe that preventing bugs for the sake of "successful installation" instead of recognizing and fixing them is a strongly incorrect attitude. Of course, in this concrete case (SpellPack), it might be okay (usually, for instructions such as COPY_EXISTING_REGEXP with a predictably high number of matches, it should be OK).

[The Bigg]

ToD vs. UB is a wrong example. UB applies a typical antipattern which introduces very serious incompatibility risks. The only case when such errors should be fixed in other mods (rather than in the mod which introduces them) is when the mod which introduces them isn't supported any more.

If FR_ROV isn't supported any more, then you're right that fixing the problem in other mods is okay. In fact, even if it was supported but a quick reaction is required, it's OK to fix bugs like this (as they don't introduce any further risks, the file is corrupt and that's all).

I wouldn't call neither UB nor FR_ROV maintained, since their author and maintainer are (almost) retired. True, JC and/or SC could fix them by authority (I'd have done so if I were an admin). The reason why this doesn't happen beats me, but I suppose it's related to a misguided sense of copyright. Incidentally, in the case of FR_ROV, the mod's forum contains the fix as an attachment in a post (not even a pinned one), but it hasn't been applied since forever.

On a side note, there is no problem with a lot of threads, because that is why patches can be released. Once it's done, it can just be linked in every thread (and obviously there won't be hundreds of threads, just a few tens at most). The patch can be added as a pinned topic.

I find your argument misguided. Between threads, bumps, and unrelated bumps, there are at least 300 people who have failed to read the pinned topic before asking about ar0100.bif.

This is exactly the fundamental difference between your and e.g. BWL's Improved Anvil's philosophy.

Even if the file works correctly in-game, there are still two major problems:
(1) If the problem hasn't been revealed yet, such "ELSE 0" instructions just keep in hidden further. This prevents/hinders the detection of corrupt, buggy files in (possibly widely used) mods.

I believe that FR_ROV was made in 2002 with some older teambg tool (not IEEP). I doubt that this type of file corruption is something you must watch out for in newer mods. Also, "Be liberal in what you accept, and conservative in what you send." (Jon Postel, RFC 760).

(2) The mod which would have failed without "ELSE 0" tried to do something with the corrupt file, which means it wanted to add/extend/modify content. In other words, modifying that file is supposed to be a feature of the mod, something which belongs to its content.

I don't understand this from a grammar POV. Do you mean that it's morally wrong for Spellpack to alter FR_ROV, or that Spellpack will have broken features?
In meaning 1, well, it's not really the fault of ELSE 0 for a morally wrong action. In meaning 2, I believe you're misunderstanding what ELSE 0 does. Generally speaking, ITM patching code looks like

CODE

for all effects
  read the opcode
  read another parameter
  if (opcode = X) and (parameter = Y) then
    doSomething()

If an item has 6 effects but states it has 10 (deitm049's situation), then using ELSE 0 will mean that the condition in the 'if' will be automatically false for effects 6-9, whereas it'd be applied normally for effects 0-5. If your code doesn't have the 'if', then the mod will generally fail at the doSomething() step.

[Baronius]

(2) The mod which would have failed without "ELSE 0" tried to do something with the corrupt file, which means it wanted to add/extend/modify content. In other words, modifying that file is supposed to be a feature of the mod, something which belongs to its content.

I don't understand this from a grammar POV. Do you mean that it's morally wrong for Spellpack to alter FR_ROV, or that Spellpack will have broken features?
In meaning 1, well, it's not really the fault of ELSE 0 for a morally wrong action. In meaning 2, I believe you're misunderstanding what ELSE 0 does. Generally speaking, ITM patching code looks like [..]

It's meaning 2, but with the code you've provided, it's OK (Galactygon can use that in his code). It's a code to manipulate many files of the same type, and it's reasonable to make such a process fault-tolerant (especially because it won't really affect the mod's important features if a few items won't be updated, as opposed to mods where each change is very important).

When I state that it shouldn't be used everywhere, I refer to cases when it's used to omit a feature just to suppress an installation error regardless of the importance of the omitted change. The installer's logic is "if I get a valid offset, apply the changes for this feature; if I get an invalid one (e.g. due to ELSE 0), skip this feature". For example, this is the case when an offset in a header is incorrect; the feature of the mod that needed that file will missing from the installed mod, and no one might notice it (while in case of an installation error, the problem is recognized and can be fixed).

I believe that FR_ROV was made in 2002 with some older teambg tool (not IEEP). I doubt that this type of file corruption is something you must watch out for in newer mods.

If the probability of this type of corruptions is so low, then why to ignore them? Since they don't happen often at all -- but when they do, they can possibly prevent a feature of a mod from installing if 'ELSE 0' is used -- why is it a problem to deal with those few cases (even if they result in many player reports)? In this respect, they're like disk errors: they don't happen often at all, but when they do, their effect is really undesired (= a mod's feature isn't installed and no one might notice this). That's why it's not a good practise for new modders to use "ELSE 0" everywhere to suppress errors.

On a side note, there is no problem with a lot of threads, because that is why patches can be released. Once it's done, it can just be linked in every thread (and obviously there won't be hundreds of threads, just a few tens at most). The patch can be added as a pinned topic.

I find your argument misguided. Between threads, bumps, and unrelated bumps, there are at least 300 people who have failed to read the pinned topic before asking about ar0100.bif.

I meant that you make a pinned topic about the patch, and link to that topic in every new thread about the problem. Honestly, I don't think it's such a horrible burden for a mod author to reply with a link e.g. once per day (and it's probably an exaggeration that such reports are posted in every day by players). It especially shouldn't be considered a burden, because it's about fixing a bug, a problem. Should we suppress bugs because authors are lazy? (Again, I'm not referring to the -- really narrow -- group of mods which use COPY_EXISTING_REGEXP or GLOB or whatever, with quite wide masks) It should be up to the author to decide, yet most WeiDU-based mods (and sample codes) are full of "ELSE 0" statements, clearly suggesting that it's a generally good practice and should be used everywhere. Since new modders will check the code of existing mods, their mods will also contain the same code, which newer modders will use as a basis, and so on... The point is, many new modders who use sample code / existing code don't know what "ELSE 0" actually does, they just use it because they believe it to be some necessary thing or good practice.

About a year ago, I've seen some G3 WeiDU tutorials and mod codes, and they were full of "ELSE 0" statements, but without explanations. Since new modders consider such codes ("TP2 ninja" codes) as the most credible WeiDU TP2 sources, there should definitely be more documentation and explanation. It is similar to the problem that codes obviously use absolute offsets, but no one warns the new modders who use the code as a basis that those offsets can be different for their own files (inside variable-length sections).

To cut a long story short, modders (who are less familiar with this type of stuff, unlike you or me) shouldn't be kept in the misbelief that "ELSE 0" is something necessary; unfortunately, that's exactly what this "use existing code / use sample code" approach does. Tutorials should explicitly explain what the purpose of "ELSE x" is, and some modders should surely update their mods (I can imagine that there are many mods which use "ELSE 0" when they shouldn't). So each modder could decide on his/her own whether he/she wants to detect or suppress errors. Generally, the first case is much more beneficial.

About sample codes (as I'm sure most modders learn "ELSE 0" from such codes):

I tend to criticize certain modders/sites for influencing new modders and players too much (occassionally with unacceptable methods), convincing new modders and players that their methods are the best and superior, and others methods (such as Improved Anvil's solutions) are inferior. Actually, 'sample codes' are also a very effective way of influencing modders, and of course a totally acceptable way (most modders share their code also to help others, no doubt). However, sample codes have the disadvantage that they don't help the modder understand the interrelations and actual solutions, and on top of it all, they implicitly influence the modder's beliefs about what correct coding is. This is why tutorials are much better than such codes (but I understand that not all modders have time to write tutorials while sharing code is instant). Tutorials give general and widely usable information. So WeiDU code sharing is like an element of the child-parent relationship: the parent shows an example, the child (implicitly) learns it and starts to use it. For example, Gibberlings3 WeiDU coding standards and the associated "compatibility" definition was a typical example to this phenomenon: certain modders believed (or believe) that Improved Anvil isn't compatible with other mods because it's not designed as smartly as e.g. Gibberlings3 mods that are advertised as compatible with tons of other mods. (Needless to say that it's false; Improved Anvil's quality and robustness can be thanked to its great design and careful interoperability policy with other mods. There would be no way to make it more compatible with mods without giving up its quality or robustness.) They had/has this misbelief because they aren't skilled enough; they just implicitly learnt the coding standards and samples of many Gibberlings3 mods (and they didn't even notice how much they're influenced -- the child also doesn't know he/she is biased due to the example shown by the parents). If there were comprehensive tutorials and guides available, these modders would be provided with real information and knowledge, and thus they would get the chance to make unbiased decisions and judgements on their own. Actually, in this respect, sample codes (and the use of existing mod codes as a basis without understanding the deeper interrelations) can be considered as a -- sometimes probably unintentional -- way of making modders biased; if modders were forced to learn and understand the basics due to the lack of tutorials and sample codes, they would be able to make much more conscious judgements and decisions. This paragraph has become quite long, so let me close it with a message to all readers who are interested in modding with WeiDU: learn the interrelations, try to understand everything! Learn, if you want to make a good, above average mod! The statement "modding/coding is surprisingly easy" is a humbug; you will experience it on your own if you don't believe me

[BeachBum]

Here is the file before I did as you suggested. I will let you know what happens.

Attached File(s)

 DeITM049.itm ( 466bytes ) Number of downloads: 6

 

[BeachBum]

Opened the file in Near Infinity and then saved it then reran the install. Install failed with the same error message.

Is there a way I can edit this file to get it to work? Please keep in mind that I am pretty much of a Noobie in this area so will need pretty simple and specific instructions.

Thanks.

[The Bigg]

I meant that you make a pinned topic about the patch, and link to that topic in every new thread about the problem. Honestly, I don't think it's such a horrible burden for a mod author to reply with a link e.g. once per day (and it's probably an exaggeration that such reports are posted in every day by players). It especially shouldn't be considered a burden, because it's about fixing a bug, a problem.

In my case, ar0100.bif isn't a bug, and the only critical bug I'm aware of was never reported
While I don't get them one per day, I still get them with an high frequency (I might get ten in a day if I was mentioned on a particularly well read magazine).

<...>

I'm not really interested in bickering back and forth about how The IA Way is good vs. how The IA Way is bad, so I'll leave you to your sense of victory.

Beach: remove the file from the override directory, install Spellpack, and put it back there.

[Baronius]

BeachBum: as far as I can see, the file you attached is still invalid (IEMF detected an error). I quickly corrected it with a hex editor, the feature block offsets seemed to be invalid in the extended headers. So you can also use the file I attached (right-click the attachment, and choose "Save As...", save it to your Override and rerun the setup). In this way, you will have a valid file in your Override (not just for SpellPack but e.g. for further mods as well that might access it).

I'm not really interested in bickering back and forth about how The IA Way is good vs. how The IA Way is bad, so I'll leave you to your sense of victory.

There is nothing that can be discussed there, because I merely mentioned it as an example of how sample code can influence modders. (It applies to the use of "ELSE x" as well.)

Your reaction reflects the problem very well, by the way -- there is no such thing as generally "Good way" or "Bad way". Each mod and mod type is unique to a certain extent, and something which is a good way for a certain mod might not be as good for another one. This is why it is very narrow-minded (especially without development skills) to write harsh criticism about a mod just because its system doesn't match the mods favoured by the critic.

So the point wasn't that "modders should realize how Good the 'IA way' is"; it was "modders should try to understand the deeper details as well, so they can understand the technical decisions behind the system of all mods (possibly including the 'IA way' too)". It would allow their opinion on 'IA way' to be a bit more scientific than just "It doesn't work in the way Gibberlings3 modders make their mods". Such a knowledge of interrelations would also allow them to make reasonable decisions on their own (e.g. whether "ELSE x" is needed or not for a particular installation block).

Attached File(s)

 deitm049.itm ( 466bytes ) Number of downloads: 18

 

[BeachBum]

Baronius -

Thanks for the help. This got me by the problem. It did appear that there were somethings that did not install properly but the majority did. I think???

[Baronius]

Since no one has replied yet:

I suppose that if it has said "successfully installed ..." at the end, then it should be fine, but only Galactygon knows this for sure (SpellPack is his mod), so hopefully you will get a reply from him soon. I've only restored your file's consistency.

[Galactygon]

Unfortunately, I cannot return to SpellPack until December (with a release earliest in January) due to the following reasons:

1.) I have Lost Crossroads installed, and I am in the middle of writing some code and managing stuff, so I cannot uninstall and reinstall SpellPack.
2.) BscArchitecture is harder than I thought and is taking it's toll. Basically you take 30 credits worth of classes spread out over 13 weeks. I can check stuff almost every day, and reply to posts, but not much more than that.

With due patience, I will run both files through my macros and report the results.

-Galactygon

[Baronius]

I think BeachBum was merely wondering whether or not his latest installation has been successful. If the installation has ended with the message "SUCCESFULLY INSTALLED...", it should be fine, shouldn't it? It's just a general assumption from me, as I don't know whether installation warnings have any importance in case of SpellPack. Of course, you might have meant that that you also want to make sure whether the files he installed are okay (regardless if his installation ended with a "success" or with a failure message).

I will run both files through my macros and report the results.

Do you refer to the files posted by BeachBum and myself? The file I attached has consistent structure for 100% (of course, I don't know if it meets your criteria content-wise as well; I merely fixed the file attached by BeachBum), so no need to sacrifice your free time for its verification (but of course, if you verify it, it will be at least a good test file to see how your macros work).

[Galactygon]

The file I attached has consistent structure for 100% (of course, I don't know if it meets your criteria content-wise as well; I merely fixed the file attached by BeachBum), so no need to sacrifice your free time for its verification (but of course, if you verify it, it will be at least a good test file to see how your macros work).

If someone else verifies this for me, that would be good. I still wonder how WeiDU would bail out but the game or any of the other editors I regularly use (DLTCEP and NI) would not.

-Galactygon

[Baronius]

Just to make sure I understand your question: are you asking why WeiDU has problems with that file ("read out of bounds") while the usual editors you use don't?

[Galactygon]

Yes, or more like how is the game able to run smoothly and how can I replicate it in WeiDU?

-Galactygon

[Baronius]

If you want to suppress such installation errors in WeiDU, use "ELSE x" in the proper context, as TheBigg has said.

For example, here is a "patching" function call with two arguments that reads a 32-bit integer from file position 1234h to the variable "some_header_field":

READ_LONG 0x1234 "some_header_field"

If offset 0x1234 is out of range (e.g. the file size is smaller or equal to 0x1238 bytes), the operation will fail, causing an installation error. On the other hand, the following form works differently:

READ_LONG 0x1234 "some_header_field" ELSE 666
PATCH_IF ("%some_header_field%" != 666) ...

In this case, the value of some_header field will be 666, if the offset is beyond the end of stream. There will be no installation error. You can detect that there was an error by checking the returned value (obviously, to indicate the error you should use a value that can't occur in normal circumstances).

You said you learnt some Java earlier. You can think of the difference between the two solutions above as an Exception and a null check. That is, a run-time exception interrupts the normal flow of the program, while a null check can be used to detect a problem "silently" when you surely know that the returned value is a valid reference in normal circumstances (and you don't want to use exception handling, e.g. because the failure is expected to happen relatively often).

So you should add "ELSE 0" statements (with additional code if needed, I don't know your concrete code) to every function call where you expect failed reads and want to suppress (or specially process) the error. As I've emphasized, suppressing errors in mods is not always desired; if you specifically want to set a field/entry of a particular file and the file is corrupt, the installation won't fail, and the missing field (corrupt file) might break one or more features of the mod (which have the field or entry as their dependency). On the other hand, suppressing errors for features which absence can be tolerated is another question (it's a question of taste/style and/or of the mod's goals) .

Of course, there are other ways as well in WeiDU to detect that a file is corrupt (e.g. SOURCE_SIZE -- or something like that -- returns the buffer/file size, so you can check whether it has the required minimal size).

[..] how is the game able to run smoothly [..]

It ignores the problem if an invalid value is returned (exactly what you do with "ELSE 0" ). (I would've guessed that the game crashes when a referenced feature block doesn't exist, but if this inconsistent deitm049.itm works for people, then my guess was wrong and the game just ignores such problems). I didn't test how exactly the game handles item files that are invalid in this way, but it's a general fact that the file structure specifications are stricter than the implementation that deals with these files. This allows the game to be more fault-tolerant when it's possible (for example, a missing dialogue file doesn't cause any crash, but an area without a tileset cannot be tolerated ).

As far as I've noticed, DLTCEP tries to read most file formats in a robust way too. How can it be accomplished? I haven't checked the implementations in DLTCEP and NI, but I guess that e.g. DLTCEP often simply considers the file as a sequence of contiguous sections (e.g. header, extended headers, feature blocks, equipping feature blocks). Where possible, it reads all sections (when it finishes the reading of a Section A, it immediately starts to read Section B that follows Section A, regardless what the file header says about the position of Section B ). Of course, this is a bit simplified, but you get the idea. This allows DLTCEP to read files with inconsistent or sometimes even corrupt structure, and when you save it again, it provides valid values for the header fields that were incorrect in the originally loaded file -- which results in a consistent, valid output file. This is how "broken files" can be fixed.

However, this robust approach has its own drawbacks. First of all, since these editors (such as DLTCEP) aren't built for data recovery from corrupt IE mod/game files, the algorithms aren't too complicated, there aren't too advanced heuristics. Notice that data recovery and file editing have different goals; data recovery has its own uncertainty (the returned data might be corrupt) and it usually requires human interaction, while file editing should always guarantee a consistent structure, and when this isn't possible, it should indicate an error. It's typical example of the question whether we want to correct the error or to indicate it. The latter one is usually easier and -- more importantly -- safe, because correction may fail and return corrupt data without indicating that it's corrupt (but discussing this would exceed the scope of the present topic). This drawback is present in DLTCEP as well: when it reads dialog.tlk (probably with the aforementioned procedure of continous reading that ignores certain offsets) and there is an inconsistency, it offers you to make an attempt to correct the problem. If you click 'Yes', it applies its "corrections" and saves the file. Depending on the type of the faulty field in dialog.tlk, this can corrupt dialog.tlk! For example, the unofficial Russian dialog.tlk works perfectly in the game, but it contains a very tricky offset assignment (I wonder who had that idea and what he/she wanted to accomplish with it) which DLTCEP detects as a consistency error and offers to make an attempt to fix it. Its fix will corrupt the file (the game will not crash, but many dialog.tlk strings will become totally senseless, corrupt), because the assumptions made by its heuristics prove to be wrong. Needless to say that DLTCEP doesn't detect that it partially corrupted the file (this is a common drawback of correcting errors, as I've said), because structurally, the file will be consistent, but semantically, it will contain several corrupt, unreadable texts.

Considering BeachBum could open and save deitm049 in Near Infinity, this software probably uses the same or a similar algorithm to read items as DLTCEP (i.e. trying to be fault-tolerant when possible), but when you save the file, it does not correct the invalid fields. It probably writes back the same values as it read out (this has an advantage as well: it won't accidently corrupt an inconsistent file that possibly works in the game itself).

I hope I managed to answer your question Galc, and hopefully the readers of this topic also got an insight to the question of when "ELSE 0" should be used and when it shouldn't (and why).