DEITM049.ITM Error Options

[The Bigg]

It ignores the problem if an invalid value is returned (exactly what you do with "ELSE 0" ). (I would've guessed that the game crashes when a referenced feature block doesn't exist, but if this inconsistent deitm049.itm works for people, then my guess was wrong and the game just ignores such problems). I didn't test how exactly the game handles item files that are invalid in this way, but it's a general fact that the file structure specifications are stricter than the implementation that deals with these files. This allows the game to be more fault-tolerant when it's possible (for example, a missing dialogue file doesn't cause any crash, but an area without a tileset cannot be tolerated ).

I'd guess that the game (being coded in C++) simply blits the file in memory and then uses pointer arithmetic to read the various effect blocks, thus reading past the end of the actual file if fed wrong positions (causing random behavior, generally without crashing visibly). WeiDU had a similar unchecked memory access in SET_2DA_ENTRY, which used to randomly work fine and crash before I enabled safe arrays.

[Baronius]

I'd guess that the game (being coded in C++) simply blits the file in memory and then uses pointer arithmetic to read the various effect blocks, thus reading past the end of the actual file if fed wrong positions (causing random behavior, generally without crashing visibly).

Possible. One more reason why I generally don't recommend to base important decisions/statements on observations such as "it works in the game", "it doesn't cause any crash in the game". Such a decision can be whether to follow the IESDP descriptions when implementing a file format handler (or when editing a game file for a mod). Generally, it's a good idea to keep conformance with IESDP, because -- apart from a few cases -- IESDP is relatively accurate. (On a side note, however, sometimes it's a matter of interpretation as well: for example, the BIF entry searching/matching behaviour implied by IESDP's file description is not what actually the Infinity Engine games do. As far as I remember, gemRB does resource locator comparison between the KEY BIF and the BIF BIF entries, which is a completely incompatible behaviour with the Infinity Engine games, and as such, certain files which worked perfectly e.g. in BG2 will behave differently in gemRB. In such cases, it might be a better idea to follow the original game's behaviour, considering that IESDP merely specifies file formats and not the associated behaviour.)

[Azazello]

DEITM049.ITM is a broken file from FR_ROV (has 6 effects, but the header says it has 10). The problem's been around since forever, but nobody cared to fix it.

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

Beats me.

Oh dear.

I wish Real Life wouldn't prevent me from reading the forums more often. You two are worse than Kulyok and me.

FR_ROV_updates_for_v5.rar fixes (DEITM049.ITM: read out of bounds). I'm sure it'll be rolled into v6.

[Baronius]

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

I actually meant that the third-party code (e.g. in this case, SpellPack) could fix it instead of suppressing possible errors due to it (with "ELSE 0"). It's acceptable in cases when the buggy mod isn't supported (and that is what I suspected that time).

As far as kulyok's sayings are concerned, it's no surprise for me (she has been spreading lies about Vlad's BWL mods as well, I guess she wants everyone to use Pocketplane WeiDU mods). I wonder if she is paid for it, or if not then why exactly she needs to criticize everything and make attempts to manipulate players... As TheBigg would say: beats me.

[Galactygon]

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

I actually meant that the third-party code (e.g. in this case, SpellPack) could fix it instead of suppressing possible errors due to it (with "ELSE 0"). It's acceptable in cases when the buggy mod isn't supported (and that is what I suspected that time).

Yes, I am thinking on ways to do it. I can already guess that in some cases the last effect in the last header isn't fully defined (with the opcode being a BYTE instead of SHORT), and I have to insert 0 bytes until it is.

As far as kulyok's sayings are concerned, it's no surprise for me (she has been spreading lies about Vlad's BWL mods as well, I guess she wants everyone to use Pocketplane WeiDU mods). I wonder if she is paid for it, or if not then why exactly she needs to criticize everything and make attempts to manipulate players... As TheBigg would say: beats me.

While there might be people who tolerate this, I would like to make myself clear: I don't. Regardless of who is right or who isn't. Let's not bash people and stay on topic if you please. This goes for everyone just as much.

Thank you,
Galactygon

[Baronius]

Glad to hear that. I hope you will practise that intolerance the other way around as well, i.e. when you see real bashing against BWL members and works.

Why isn't the file fixed in such cases when the problem is known? If it's just about a header field, it wouldn't result in any modification of the file content-wise.

I actually meant that the third-party code (e.g. in this case, SpellPack) could fix it instead of suppressing possible errors due to it (with "ELSE 0"). It's acceptable in cases when the buggy mod isn't supported (and that is what I suspected that time).

Yes, I am thinking on ways to do it. I can already guess that in some cases the last effect in the last header isn't fully defined (with the opcode being a BYTE instead of SHORT), and I have to insert 0 bytes until it is.

Regarding the solution of the concrete problem discussed in the present topic, I made a reference to it in this paragraph:

BeachBum: as far as I can see, the file you attached is still invalid (IEMF detected an error). I quickly corrected it with a hex editor, the feature block offsets seemed to be invalid in the extended headers.

As TheBigg has said, the culprit is DEITM049, and if you want to fix it, the base of the solution can be read above.

If you're speaking generally (more precisely, in the context of other out-of-bounds error with SpellPack), this 0-byte insertion doesn't sound like an elegant solution (I even don't remember why such a thing is needed in WeiDU). If you show a code example where you believe there is such a problem, I can say more if you're interested.

[Galactygon]

Glad to hear that. I hope you will practise that intolerance the other way around as well, i.e. when you see real bashing against BWL members and works.

When anything like that happens in my subforums, I will. I am not the type who goes into stuff like this in other threads in other forums, but when it spills over to here, I have to draw a line.

If you're speaking generally (more precisely, in the context of other out-of-bounds error with SpellPack), this 0-byte insertion doesn't sound like an elegant solution (I even don't remember why such a thing is needed in WeiDU). If you show a code example where you believe there is such a problem, I can say more if you're interested.

This is what I meant, really. We were cross-talking a bunch. A macro that fixes the number of effects if there is a gaping disagreement between that and SOURCE_SIZE.

I do not have my macros up at the moment and I have never done anything like this, so I am posting my thoughts rather than testing them through trial and error.

1.) The macro looks at the last header, counts the number of effects and compares it to SOURCE_SIZE.

2.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE but not by more than 48 bytes, then we assume the last feature block didn't get filled in. So we expand the SOURCE_SIZE gets with zeroes as needed.

3.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE by at least 48 bytes, then we assume there is something wrong with the number of effects. We then start counting the number of effects manually until we run out of bounds, counting feature blocks that might be less than 48 bytes. In that case, we go to 2.).

This sounds somewhat right, would I have to account for headers that go before the last header, or is it always the last header that causes all the problems?

-Galactygon

[Baronius]

Where do such corrupt files occur? The algorithm you outlined is a general one, but still only solves one type of problem (with certain assumptions). There are many
ways a file can be corrupt, is this "broken effect" problem typical to certain files encountered by SpellPack (and this is why you find it reasonable to add generic fix code for it)? Where do these files originate from?

[Galactygon]

Where do such corrupt files occur? The algorythm you outlined is a general one, but still only solves one type of problem (with certain assumptions). There are many
ways a file can be corrupt, is this "broken effect" problem typical to certain files encountered by SpellPack (and this is why you find it reasonable to add generic fix code for it)? Where do these files originate from?

Yes, this would solve only one type of problem, but it happens frequently to warrant my attention. I can still have my macros bail out and skip that .itm/.spl file if the file is hopelessly corrupted and all else fails.

-Galactygon

[Baronius]

I see, but I asked what mods (or perhaps the original game?) add these corrupt files. Normally, such horribly broken files shouldn't occur often at all. If they do, it means something is corrupting or has corrupted game/mod files.

[..]then we assume the last feature block didn't get filled in

Filled in by what? A WeiDU installer of a mod? If so, we're talking about mods which produce broken files (I don't refer to the question of mod quality now, I am just wondering).

On a side note, in case of DEITM049, nothing was broken in such a cruel way (i.e. end-of-file in the middle of a block). It just had incorrect feature block offsets (indexes) assigned (if I remember correctly, their value was greater -- or smaller? -- by one than the actual position).

[Galactygon]

Filled in by what? A WeiDU installer of a mod? If so, we're talking about mods which produce broken files (I don't refer to the question of mod quality now, I am just wondering).

A .spl or .itm file consists of a single 114 byte header, multiple (56 bit for .itm and 40 bit for .spl) extended headers, and multiple 48 byte feature blocks or "effects".

I once ran a test on one of the corrupted files, and found out there was nothing wrong with any part of the file except the last effect in the last header. When I READ_SHORT the opcode, the installer ran out of bounds and bailed out. When I opened the .spl in DLTCEP and looked at that effect, I noticed that all fields except the opcode number were filled in their default values - zeroes.

Therefore, I am making a guess that the last effect in the last header wasn't filled out to 48 bytes and that DLTCEP filled it out correctly. What I am talking about if I could replicate this correctly in WeiDU, and correct corrupted files on the fly rather than worry about them individually or going up to each modder or skipping them altogether.

But then again, I could be completely wrong, since I did not do -- what they call -- rigorous testing.

-Galactygon

[Baronius]

So you don't want to reveal where do these "corrupted files" come from (i.e. original game, some mod etc.) -- as you wish, though I don't see the reason. (As I said, it would help identify the problem, because as far as I can see, even the exact nature of the problem isn't known.)

I once ran a test on one of the corrupted files, and found out there was nothing wrong with any part of the file except the last effect in the last header. When I READ_SHORT the opcode, the installer ran out of bounds and bailed out. When I opened the .spl in DLTCEP and looked at that effect, I noticed that all fields except the opcode number were filled in their default values - zeroes.

Therefore, I am making a guess that the last effect in the last header wasn't filled out to 48 bytes and that DLTCEP filled it out correctly.[...]

Why do you make guesses when you could check it for sure? Opening the file in a hex editor to check the last header (or even the other headers) isn't really rigorous testing, it takes 1-2 minutes at most.

[Galactygon]

So you don't want to reveal where do these "corrupted files" come from (i.e. original game, some mod etc.) -- as you wish, though I don't see the reason. (As I said, it would help identify the problem, because as far as I can see, even the exact nature of the problem isn't known.)

I don't get it.

Why do you make guesses when you could check it for sure? Opening the file in a hex editor to check the last header (or even the other headers) isn't really rigorous testing, it takes 1-2 minutes at most.

I will have to figure out the correct offsets that lead to the last effect of the last header and compare it with whatever that file says.

Due to my studies, that will happen late-December the earliest.

-Galactygon

This post has been edited by Galactygon: Oct 23 2008, 05:28 PM

[Baronius]

I don't get it.

Perhaps I'm misinterpreting something, so let me summarize what I believed the problem is:
- when SpellPack is installed, it modifies (with WeiDU parlance: "patches") many game files, often based on regular expression matching
- some files SpellPack tries to modify are corrupt (e.g. bad offsets/indexes in extended headers)
- SpellPack's installer refers to an invalid offset (due to the invalid data it read from the file) which obviously results in an out-of-bounds error

Is this correct? If yes, my question was simple: your words gave the impression that we're talking about more than just 1-2 corrupt files (you said that the typical problem of read-out-bounds occured with more files), so I ask where do the corrupt files come from? What mod installed them before SpellPack? I am wondering why they're corrupt. For example, they are a few corrupt BAM files (even in the original game), but I haven't really seen items corrupted in the way we've discussed.

[Galactygon]

Is this correct? If yes, my question was simple: your words gave the impression that we're talking about more than just 1-2 corrupt files (you said that the typical problem of read-out-bounds occured with more files),...

Correct.

...so I ask where do the corrupt files come from? What mod installed them before SpellPack?

So far, I only know of NeJ. But people have hinted of other mods, and chances are they are right. In that case, the macro is worth the effort.

I am wondering why they're corrupt.

I really don't know.

-Galactygon

[Baronius]

1.) The macro looks at the last header, counts the number of effects and compares it to SOURCE_SIZE.

2.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE but not by more than 48 bytes, then we assume the last feature block didn't get filled in. So we expand the SOURCE_SIZE gets with zeroes as needed.

3.) If the (number of effects times the feature block size plus whatever other offsets there might be) is larger than the SOURCE_SIZE by at least 48 bytes, then we assume there is something wrong with the number of effects. We then start counting the number of effects manually until we run out of bounds, counting feature blocks that might be less than 48 bytes. In that case, we go to 2.).

First, just a quick note: SOURCE_SIZE in WeiDU returns the size of the complete "patched" resource, doesn't it? In this case, you need to take into account the item header size and the extended headers as well in your calculation when making a comparison with SOURCE_SIZE. In your current algorythm, you're comparing the total size of feature blocks with the file size, instead of comparing it with the expected size of the feature block section. You write "plus whatever other offsets there might be", if this is meant to cover the item header and the extended headers, then ignore what I've written above.

It will be an interesting discovery if your suspicion about partial effect blocks proves to be true. To tell the truth, I don't give much chance to it, but we will see. Instead, I assume that the case is similar to the present problem of DEITM049.ITM: the feature block indices of extended headers are incorrect, they point beyond the end of file (but the feature blocks themselves are okay, i.e. their total size is a whole multiple of 48 bytes). I don't find your suspicion too likely, because the original game doesn't contain such files, which means they must have been generated by an editor. I doubt any item editors of the past generated partial blocks (such severely damaged data structures in this case can only be the result of some extraordinary problem such as damage on files on the media or during their transfer). On the other hand, faulty calculation of feature block indices is much more likely for an editor software, so perhaps one of the old editors has this bug. The fact that the base of FR_ROV (DEITM049.ITM ) and NEJ (mentioned by you) is relatively old, it's possible that some older files in these mods were made with an early item editor which might have generated faulty indices to feature blocks in extended headers. Of course, I would like to emphasize that this is also just an assumption.

[The Bigg]

First, just a quick note: SOURCE_SIZE in WeiDU returns the size of the complete "patched" resource, doesn't it?

SOURCE_SIZE returns the initial size of the file (and is not correct if you use INNER_ACTION or similar). BUFFER_SIZE returns the current size.

The fact that the base of FR_ROV (DEITM049.ITM ) and NEJ (mentioned by you) is relatively old, it's possible that some older files in these mods were made with an early item editor which might have generated faulty indices to feature blocks in extended headers. Of course, I would like to emphasize that this is also just an assumption.

I recall that CTB had (or still has) such a broken file as well, so that one falls under "old tools" as well (don't these mods predate even IEEP?). I guess somebody at SHS knows which files are broken in both CTB and NEJ.

[Baronius]

SOURCE_SIZE returns the initial size of the file (and is not correct if you use INNER_ACTION or similar). BUFFER_SIZE returns the current size.

I also meant SOURCE_SIZE in that way (my wording was unfortunate), I should have written 'the size of the complete resource to be "patched"' instead. Either way, I think he needs to include the main and extended headers in the algorithm he outlined.

[Leomar]

A nice treasure is hidden in this thread.

So I'll let you know, that we have included the deitm049.itm fix in the BiG World Fixpack.

Greetings Leomar

[Tervadh]

So I'll let you know, that we have included the deitm049.itm fix in the BiG World Fixpack.

This was fixed some time ago with Nythrun's macro and I think also the "Resource Fixer" which is buried in one of your old BWP threads, unless that is what you used.

Of course, it has not much to do with LC, except LC could use the same macro to avoid bombing out when modding erroneous items like that one.