Xj Conversion Notes
- Thread starter kion
- Start date
kion
Garbage Human
- Gender
- Male
On a level of one to dumb, this is pretty dumb. But I published my file-to-excel-based-hex-spreadsheet as a command line tool on npm. So if you install Nodejs, you can open the command line and enter
which will install my stupid utility to the path. You can then use my stupid utility from the command line inside any directory with
It will write a filename.xlsx file in the same directory as the source file.
Code:
> npm install hexcel -g
Code:
> hexcel <filename.ext>kion
Garbage Human
- Gender
- Male
Still a work in progress, but Glader's been putting in a lot work on forest and it's really starting to look amazing.
https://i.gyazo.com/4336934e4ee4e3d3d6e ... df6e8d.mp4
https://i.gyazo.com/08eb7bcfa1cad9eaeff ... 240593.mp4
Hopefully I can get some better exports of the other maps. I can so some basic rips, the translations and rotations are still off, so I have some more reverse engineering to take care of.
https://i.gyazo.com/4336934e4ee4e3d3d6e ... df6e8d.mp4
https://i.gyazo.com/08eb7bcfa1cad9eaeff ... 240593.mp4
Hopefully I can get some better exports of the other maps. I can so some basic rips, the translations and rotations are still off, so I have some more reverse engineering to take care of.
kion
Garbage Human
- Gender
- Male
Progress report. Tried a couple of different things over the last few days. Now that I can more or less parse XJ models, I took another look at PSOBB maps and wrote and exporter from scratch.
Not perfect. I need to fix the textures as there were a few I couldn't parse out and I have to trace back through Schthack's Map Viewer to try and find those. The difference is I think I managed to fix rotations. The way maps work is they have sections. Each section makes a small "scene" that is rendered at the origin. Each scene is then rotated around the Y-axis (left or right) and then translated by an x,y value to place that scene some where.
To give an example the forest is made of a 4x4 grid, where each scene makes up a section of the grid. For pioneer 2 the shops are scene, so is the hunter's guild, ect. I haven't gotten Y translations yet, so pioneer 2 ends up squished. But other maps like caves looks okay for now.
This is what I messing around with today. I can take a map from PSO version 2 for PC, read the buffers for the models, rearrange them and overwrite the pointers to create a "home grown" nj file. Right now I'm just doing it for one section at a time. Next I need to try adding an NJTL header on the top, so it actually has textures. I can try adding rotations and translations onto the model, but I don't think that will work out very well. I think translations and rotations are applied by scale, which would cause models to explode everywhere, which is why sega might have chosen to package them in a format like this.
I laughed when I saw this. I've been pretty vocal against animations for stage files, but I think I ran into them. 0300200 is the flags used at the top of .njm files and the format looks kind of looks like a ninja animation. The image from the hex above is from the lobby in Pso Version 2 for PC. I was tracing through the file for my map to nj conversion script and I noticed this.
Each map section contains two model definitions the first one seems to be static contents, and the second seems to have an entry for .njm. The steps for applying this to PSOBB maps would be something like: 1) write out the models, look at them, 2) try to write out the njm, 3) apply the njm to the models to see if they work. 4)Document that. 5) find the same model and then 6) figure out how to apply animations for xj.
And for 6, that's a pretty massive investment because for most of the models in PSOBB, they use .nj models with a slightly different format format for animations which is pretty easy to revert. But for xj, it's a different story. The xj models in PSOBB like fences and warps have a different format for animations. Probably not anything too difficult, but it would require a complete and accurate model for xj before than can be looked into any real detail.
Not to mention that from there you'd have to figure out how to find the location where the pointer to the animation is supposed to be, splice the home-grown animation into there, and then go back and fix every pointer in the file because the file size changed and they longer match up. So not exactly a practical undertaking, but I can at least try to document the file format for maps and for njm if anyone wants to try dicking around with them.
Also ShinAsura asked about adding reflections to objects in maps. For PSOBB, I haven't found anything in PSOBB directly for how they work. I can tell you where it's not. Maps are just packaged xj models, so I'd try with xj models to try and make them reflective first. XJ models are a model node definition, a model which contains pointers to a vertex list and two indices lists. The indice lists is where that information might be stored, but I'm not entirely sure. Everything else is pretty much vertex definitions.
For NJ models I have a better idea of where that would be at. NJ models are a node, model, vertexes and triangle list. The first couple of dwords in the NJ triangle define things like diffuse and ambiance. Though I really don't what what does what as far as textures go. My focus is still on just getting the model structure correct.
Anyways, sorry my goals don't match up with those of the community trying to mod the game. But I still want to try and write out what I learn as I learn it, so at least the information is available and out there on the interwebs.
Not perfect. I need to fix the textures as there were a few I couldn't parse out and I have to trace back through Schthack's Map Viewer to try and find those. The difference is I think I managed to fix rotations. The way maps work is they have sections. Each section makes a small "scene" that is rendered at the origin. Each scene is then rotated around the Y-axis (left or right) and then translated by an x,y value to place that scene some where.
To give an example the forest is made of a 4x4 grid, where each scene makes up a section of the grid. For pioneer 2 the shops are scene, so is the hunter's guild, ect. I haven't gotten Y translations yet, so pioneer 2 ends up squished. But other maps like caves looks okay for now.
This is what I messing around with today. I can take a map from PSO version 2 for PC, read the buffers for the models, rearrange them and overwrite the pointers to create a "home grown" nj file. Right now I'm just doing it for one section at a time. Next I need to try adding an NJTL header on the top, so it actually has textures. I can try adding rotations and translations onto the model, but I don't think that will work out very well. I think translations and rotations are applied by scale, which would cause models to explode everywhere, which is why sega might have chosen to package them in a format like this.
I laughed when I saw this. I've been pretty vocal against animations for stage files, but I think I ran into them. 0300200 is the flags used at the top of .njm files and the format looks kind of looks like a ninja animation. The image from the hex above is from the lobby in Pso Version 2 for PC. I was tracing through the file for my map to nj conversion script and I noticed this.
Each map section contains two model definitions the first one seems to be static contents, and the second seems to have an entry for .njm. The steps for applying this to PSOBB maps would be something like: 1) write out the models, look at them, 2) try to write out the njm, 3) apply the njm to the models to see if they work. 4)Document that. 5) find the same model and then 6) figure out how to apply animations for xj.
And for 6, that's a pretty massive investment because for most of the models in PSOBB, they use .nj models with a slightly different format format for animations which is pretty easy to revert. But for xj, it's a different story. The xj models in PSOBB like fences and warps have a different format for animations. Probably not anything too difficult, but it would require a complete and accurate model for xj before than can be looked into any real detail.
Not to mention that from there you'd have to figure out how to find the location where the pointer to the animation is supposed to be, splice the home-grown animation into there, and then go back and fix every pointer in the file because the file size changed and they longer match up. So not exactly a practical undertaking, but I can at least try to document the file format for maps and for njm if anyone wants to try dicking around with them.
Also ShinAsura asked about adding reflections to objects in maps. For PSOBB, I haven't found anything in PSOBB directly for how they work. I can tell you where it's not. Maps are just packaged xj models, so I'd try with xj models to try and make them reflective first. XJ models are a model node definition, a model which contains pointers to a vertex list and two indices lists. The indice lists is where that information might be stored, but I'm not entirely sure. Everything else is pretty much vertex definitions.
For NJ models I have a better idea of where that would be at. NJ models are a node, model, vertexes and triangle list. The first couple of dwords in the NJ triangle define things like diffuse and ambiance. Though I really don't what what does what as far as textures go. My focus is still on just getting the model structure correct.
Anyways, sorry my goals don't match up with those of the community trying to mod the game. But I still want to try and write out what I learn as I learn it, so at least the information is available and out there on the interwebs.
Mylandra
Member
To be completely honest with you, pso modding has gone downhill to the point it's almost completely dead since Schthack left years ago.
Most of the things that have been done/released after this point were mostly rehash of stuff that already existed and then got buried.
Is it possible to add new maps to the game? Yes, it is, it has already been done as well. Is it possible to add new monsters to the game? Again, this is something that could be done. But when you gauge the amount of people with the technical skills to do any of this, it's all very pointless. Heck, most people can't even write a proper quest using the quest editor.
I feel as if it's gonna be a long work with this pso rewriting project, but it's probably still more productive than staying around here doing mods, because more people will be able to join the project in the long run.
Most of the things that have been done/released after this point were mostly rehash of stuff that already existed and then got buried.
Is it possible to add new maps to the game? Yes, it is, it has already been done as well. Is it possible to add new monsters to the game? Again, this is something that could be done. But when you gauge the amount of people with the technical skills to do any of this, it's all very pointless. Heck, most people can't even write a proper quest using the quest editor.
I feel as if it's gonna be a long work with this pso rewriting project, but it's probably still more productive than staying around here doing mods, because more people will be able to join the project in the long run.
kion
Garbage Human
- Gender
- Male
Yeah, overall i think modding tools for this game have diminishing returns because you're having to create custom tools from the ground up for a closed game. If there are simple mods I could apply by working on exporting models I might try and go for it, but animations for maps just seems like way too massive of an under taking.
For Phantasy Star Online Version 2:
Nj models: these still give me a bit of trouble. Overall I have the structure and can parse most of the file, it's the triangle strips that give me trouble. Rather than a rigid structure, the strip list will have "triangle left (4)" or "triangle right (5)" as well as uv and material information. I found the code Kryslin uses to parse these in ExMLD.net if I need to go back and do trace through it that could be an option later.
Xmv, Pvr textures: It would be nice to have a png conversion for these instead of having to export .dds files from Schthack's texture converter. It's pretty easy to unpack the pvr textures from their XVM packages, and read the headers on the pvr files, I just haven't found enough information on the body to try writing out the images. It would be nice if each pixel was just a 16 bit color entry, but generally everything I've read about the format is how to parse the header and very little information on the body, so I'd have to look for patterns in the length.
n.rel maps: Right now I can export each section into its own .nj file. I need to add a header on to each one. I can try, but I don't think there's an easy way too apply translation and rotation to convert the whole map to a single .nj file (otherwise sega would have probably done it that way to begin with). I'd need to learn how to parse the nj triangle strips before I can try exporting these on my own. Though I think PSOGC uses this format, so this is something I'd like to attempt eventually for the episode III maps.
For Phantasy Star Online Blue Burst:
XJ: I'm getting pretty close to being able to parse out the whole file. I need to go back, debug and fix my translations. If i get that then I'll at least have these parsed out for static files in terms of exporting weapons from PSOBB.
Xvm: PSOBB uses a compressed version of the PVR format used in PSOv2. Luckily Schthack left the code for how to extract these in his released mapviewer source code, so there's something to go on there. Though since the textures technically can be exported and converted to png from there, this is more of a minor annoyance that would be nice to have, but not a high priority.
n.rel: I think I need to go back and fix my xj definition for translations before I can get clean rips of map files. The other thing I need to do is make sure I have the right texture number for each group. One thing that drives me nuts about these is Sega's seemingly lack of respect for keeping their x,y,z axis consistent.
Priorities:
Right now most of the models from Episode I can be extracted, character models, character animations, enemies, enemy animations, weapons and set pieces. The only major aspect remaining is maps, everything after that I think I can go back and address. So I think I'll focus on cleaning up xj a little more and try to clean up my map exports.
For Phantasy Star Online Version 2:
Nj models: these still give me a bit of trouble. Overall I have the structure and can parse most of the file, it's the triangle strips that give me trouble. Rather than a rigid structure, the strip list will have "triangle left (4)" or "triangle right (5)" as well as uv and material information. I found the code Kryslin uses to parse these in ExMLD.net if I need to go back and do trace through it that could be an option later.
Xmv, Pvr textures: It would be nice to have a png conversion for these instead of having to export .dds files from Schthack's texture converter. It's pretty easy to unpack the pvr textures from their XVM packages, and read the headers on the pvr files, I just haven't found enough information on the body to try writing out the images. It would be nice if each pixel was just a 16 bit color entry, but generally everything I've read about the format is how to parse the header and very little information on the body, so I'd have to look for patterns in the length.
n.rel maps: Right now I can export each section into its own .nj file. I need to add a header on to each one. I can try, but I don't think there's an easy way too apply translation and rotation to convert the whole map to a single .nj file (otherwise sega would have probably done it that way to begin with). I'd need to learn how to parse the nj triangle strips before I can try exporting these on my own. Though I think PSOGC uses this format, so this is something I'd like to attempt eventually for the episode III maps.
For Phantasy Star Online Blue Burst:
XJ: I'm getting pretty close to being able to parse out the whole file. I need to go back, debug and fix my translations. If i get that then I'll at least have these parsed out for static files in terms of exporting weapons from PSOBB.
Xvm: PSOBB uses a compressed version of the PVR format used in PSOv2. Luckily Schthack left the code for how to extract these in his released mapviewer source code, so there's something to go on there. Though since the textures technically can be exported and converted to png from there, this is more of a minor annoyance that would be nice to have, but not a high priority.
n.rel: I think I need to go back and fix my xj definition for translations before I can get clean rips of map files. The other thing I need to do is make sure I have the right texture number for each group. One thing that drives me nuts about these is Sega's seemingly lack of respect for keeping their x,y,z axis consistent.
Priorities:
Right now most of the models from Episode I can be extracted, character models, character animations, enemies, enemy animations, weapons and set pieces. The only major aspect remaining is maps, everything after that I think I can go back and address. So I think I'll focus on cleaning up xj a little more and try to clean up my map exports.
Oskitar1981
Member
- Gender
- Male
Hi all,
I really apreciate your work kion, i´m following your updates in GitHub. I´ll love to see a model importer/exporter and so a map importer/exporter for PSO.
I would like to help you in some way but i´m not a programmer so i doubt it.
Anyway i encourage you to continue, no hurry, take your time but please don´t give up.
Regards. Oscar
I really apreciate your work kion, i´m following your updates in GitHub. I´ll love to see a model importer/exporter and so a map importer/exporter for PSO.
I would like to help you in some way but i´m not a programmer so i doubt it.
Anyway i encourage you to continue, no hurry, take your time but please don´t give up.
Regards. Oscar
Aleron Ives
Member
If you want to manipulate PVR textures, the PhotoShop plugin from the Katana SDK is the only way to go. It's the only tool that supports every PVR format.kion said:
Uh, the XVR textures in BB aren't PVR files; they're DDS files with a different header (because the Xbox used an nVidia GPU, Sega used an nVidia texture format for the game, as opposed to DC, which had a Power VR GPU, hence the use of PVR textures).kion said:
I highly disagree. It doesn't matter how many people are able to do something, as long as at least one person is capable of doing it and decides to spend the time necessary to do so. The reason people don't want to do it is that there are so few players these days that they figure it isn't worth their time to create a project for the sake of all five remaining players.Mylandra said:
