How do area maps work (i.e. how does QEdit visualize areas)?

[Wilhuf]

I want to play around with quest files, and visualizing the areas with monsters and other objects would be a fun exercise. But I just can't find any information about area maps. Where can I find the actual map data that e.g. QEdit uses to visualize areas and how is the data formatted? Well, I assume the data is included with the client, but I have no idea on how to parse it.

If anyone happens to have to QEdit source code and willing to share it, that would be great too.

Can anyone point me in the right direction?

 

[Aleron Ives]

Qedit reads the raw map files from BB. Schthack never released any documentation on how he did it, so your only option is to reinvent the wheel and reverse engineer the map files yourself.

 

[Andy]

Qedit reads the raw map files from BB. Schthack never released any documentation on how he did it, so your only option is to reinvent the wheel and reverse engineer the map files yourself.

ezpz lemon squeezy

 

[Wilhuf]

so your only option is to reinvent the wheel and reverse engineer the map files yourself.

I feared this would be the case, but I'm already on it. So far I've figured out that the files named map_*c.rel contain the triangle (I'm assuming triangles) meshes that make up the PSO world. These files are neither compressed nor encrypted.

I've added some pictures that plot the x and z components of coordinate triples of 32 bit floats starting from byte offsets 0, 4 and 8 (pictures in random order). A point's hue indicates it's location in the file, bright red means start of the file, blue means the end. Compare this with the Qedit visualization of forest 1, there are obvious parallels.

So far I've managed to tease out what looks like the monster roaming volume/mesh for the top-left room (the large blocky shape around the room). The vertex data are located between byte offsets 30232 inclusive and 30352 exclusive (0-indexed).

@Aleron Ives, you are omnipresent in the PSO community, you seem to be on good terms with a lot of people that worked on reverse-engineering PSO and development of all the tools and software we use. Is there anyone you could shoot an e-mail or PM to help out or can you give me names, forums, anything else? It seems like Schthack and Lee are the two people most knowledgeable about the game's workings, but they've both completely disappeared (are they okay?). Our own @Soly and @Sodaboy know a thing or two too. But neither knows a lot about this sort of stuff, or at least that's what I gather from examining the type of software they write. E.g., the Tethealla code doesn't seem to care about area maps, it just believes whatever the clients tells it as long as you don't "teleport" between e.g. forest 1 and cave 3.

I think it's very important for the health of the PSO community that all the technical information about the game is easily available. I'll keep working on figuring it out and I will document my findings and open source any useful code I write.

 

[Soly]

While working in the XJ file format I looked at the maps too but I couldn't get much out of it.
Besides not having enough time, I got bored after a while of not getting results (I still can't export correctly the XJ models... normals pretty much) so yeah... maybe I should dive into it again.

 

[Aleron Ives]

The c.rel files contain collision data, i.e. the boundaries of the map for where you can walk and where you can't (because the boundary is solid), as well as where the monster boundaries are (for the doorways they can't pass through). As you've already noticed, this is how Qedit draws the 2D overhead map of each area. The n.rel files contain map models and the instructions for how to animate them. I have no idea what the tam files do...

 

[Wilhuf]

@Aleron Ives, it makes sense that Qedit uses collision data since it has to make sure monsters etc. are in valid locations.

Things are starting to make sense... So far I've figured out that you read these files back to front. I've managed to extract all the vertices from map_forest01c.rel and also figured out where triangle indices are. So I should be able to visualize f1 in a way similar to Qedit soon.

Unsolved problems:

• Where to start reading exactly (I can't figure it out automatically yet)
• How to differentiate the different types of data (Qedit can do it)

I've also written a small JavaScript tool to help me figure all this stuff out. It's basically a hex viewer with 2d plotting of graphical data. The plots really help you figure out what you're looking at. This tool and the fact that I know what I was looking for was the only reason I was able to figure this stuff out. Looking at a huge pile of bytes with a regular hex editor would have driven me mad.

I'll probably make it a bit more generic and put it online somewhere. It's going to help me with future RE of graphical file formats. You might find it useful too, @Soly, for figuring out that XJ format.

And now some colorful pictures to keep things interesting.

This is the numerical and graphical representation of the monster bounding area for the south-east part of forest 1.



These are all the vertices from the file, each block of vertices is colored differently (they're split up in 68 blocks). If you look closely you can see the exact same set of vertices in Qedit.

 

[Aleron Ives]

it makes sense that Qedit uses collision data since it has to make sure monsters etc. are in valid locations.

Thankfully, Qedit doesn't check your monster positioning. There are so many gaps in our knowledge of PSO file formats that Qedit would surely start complaining at you about trying to put a monster in an invalid location, even if the location you were choosing was perfectly valid.

It's worth noting that REL is a generic format Sega created to store all kinds of data types. The filenames can come at the beginning or the end of the file, but all rel files have a series of pointers either at the start or the end to describe where the data chunks are within the rel. Beyond that, most rels in the game are undocumented. I keep hoping somebody will crack the n.rel format so that we can build new rel files to fix all of the missing and broken animations that the game has, but sadly nobody has done it yet.

 

[Soly]

I actually have most of the data required from the models but the problem I have is generating the triangles out of the vertices and indices, some models generate correctly (like the simplest item box) but others don't and have messed up normals.

Also as Ives said c is collision and n is models, I looked at the n files, which seems more complicated than c ones (at least are bigger).
The 3 files n, r and c are read from the bottom up, yeah.
@tofuman posted a tiny guide on how to read the itempmt.prs file which is the same way these files should be read (obviously the data is different).

 

[Soly]

This reply has mixed up hex and decimal
To give you a quick idea on how to read the files....

size - 16, read a pointer, this might or might not be needed
size - 32, read a pointer, this one is required ("short pointer table")
size - 28, this is the count of "pointers" in the short pointer table
size - 24, this is unknown to me atm, always 1, it seems.

Go to the location of the short pointer table and read like this
previous + short * 4;
offset + 2;

For example, the file map_aboss01r.rel would yield the following data
// map_aboss01r.rel

// First pointer (size - 16)
// 1A4

// Short pointer table (size - 32)
// 0 + 4c * 4 = 130
// 130 + 1 * 4 = 134
// 134 + 6 * 4 = 14C
// 14C + 15 * 4 = 1A0
// 1A0 + 1 * 4 = 1A4

As you can see, the last pointer in this table is the same as the "main pointer" read on size - 16... so its some sort of quick access or something like that

 

[Nikki]

I'm going to insert my dumb curiousity in here but if we understood how area maps work could we change their layouts or even add rooms to existing vanilla maps that aren't the ones we already have?

 

[Mylandra]

You could look at Schthack map viewer. It's the old source code and the fps sucks, but the file structure is working fine. Not sure if that's what you're looking for.

 

[Soly]

Here is the plot of the first pointer's data of map_aboss01c.rel
Doesn't really show much, but well the structure is obvious (also easy to read as there is not much data).

 

[Aleron Ives]

Schthack probably already reached the point where he could have added entirely new map configurations to the game and then created quests to use those new maps.

 

[Wilhuf]

You could look at Schthack map viewer. It's the old source code and the fps sucks, but the file structure is working fine. Not sure if that's what you're looking for.

Holy shit that's a goldmine of information. Thank you for that. That rar file contains a textual explanation of the *n.rel format and working Delphi code which loads *n.rel files and texture files. I guess Delphi will be the second language I'm learning during this project...

And a big thanks to @Soly, you gave me the missing pieces of the puzzle. I can now extract all vertices, triangle vertex indices and triangle normals.

I actually have most of the data required from the models but the problem I have is generating the triangles out of the vertices and indices, some models generate correctly (like the simplest item box) but others don't and have messed up normals.

If you're talking about the *c.rel files, I'll post some documentation and sample code tomorrow, otherwise take a look at Schthack's tool.

Next up I'll either try positioning quest entities on maps or I'll make things look nicer first with the *n.rel file data.

Results so far:

 

[Wilhuf]

I finished writing an 80-90% complete spec for the c.rel format. You can find it in this document under "REL Format":
https://docs.google.com/document/d/1B8bQsCJ9gU005INzePoK8fekj0O3Vootsq1FEpa8m6Y/edit#

@Soly, can you take a look and possibly fill in any holes? Other suggestions are always welcome too, of course.

I think I'll start looking at the n.rel format now, it would be cool to have a nice 3D quest viewer (QEdit's 3D window, if it works, barely does so for me). So far I've been visualizing things with WebGL, I'll probably make a web viewer so you don't even need to install anything. I have plenty of ideas for making a better viewer/editor (I may or may not be thinking of writing a better QEdit), hopefully I'll keep interest.

And of course a nice picture to finish off:

 

[ScoobyDew]

Wow...all this CSE stuff puts all my MAT stuff to shame.

Good stuff!

You're taking a look at .rel?? Great stuff!

 

[Andy]

 

[Soly]

I think I'll start looking at the n.rel format now, it would be cool to have a nice 3D quest viewer (QEdit's 3D window, if it works, barely does so for me). So far I've been visualizing things with WebGL, I'll probably make a web viewer so you don't even need to install anything. I have plenty of ideas for making a better viewer/editor (I may or may not be thinking of writing a better QEdit), hopefully I'll keep interest.

QEdit's 3D viewer works fine for me, but as QEdit in general, is not very user friendly.... I don't use the 3D viewer anyway.

I've been toying with the idea of making a "better" script editor adding a very basic intellisense-like suggestion system for instructions, editing as text and not as a "line list", etc... but as with everything else I have been putting it aside, the last thing I did was completing a very small portion of the instructions.

 

[Wilhuf]

Intellisense in QEdit would be pretty cool. I don't know how complex the scripting/instructions are, but maybe we could even make a nice scripting language? Possibly even a visual language, so anyone would be able to create and tweak quests.