Phantasmal Quest Editor

[Wilhuf]

Phantasmal Quest Editor will be a quest editor that is capable of doing all things QEdit can is capable of and more. This thread is meant for brainstorming ideas and having a place to point to the current prototype.

Everyone's encouraged to come up with ideas and help with development if you have the skills. If you're a frequent user of QEdit, tell us what you miss in it. If you want to create quests but can't figure out how to do it, please tell us what the biggest hurdles are. If you don't have technical skills but still want to help, you can write documentation and tutorials.

@ToasterMage is thinking of making a nice scripting language for quests (so you don't need to be software engineer to make interesting quests). Hopefully his work can be integrated into this editor in some way.

Prototype (try uploading a .qst file): http://www.phantasmal.world/
Code (for developers): https://github.com/DaanVandenBosch/phantasmal-quest
Resources for developers: http://sharnoth.com/psodevwiki/doku.php?id=start

Top Feature Requests

• Better way to place and move monsters/objects
• Better labelling of all kinds of parameters
• Easier script editing

Roadmap
This project will take quite a bit of time and it's out my comfort zone as a web developer so I'm not even sure whether it's doable, but this is my plan for next few weeks:

1. Basic quest viewer with a 3d wireframe view and basic quest information (name, description,...)
2. Minimal editing, e.g. moving monsters around
3. At this point we'll now how viable the project is and we'll see from there

My ideas:

For users:

1. Tutorials and user documentation (can be done by non-technical people)
2. Make a web application, this way it works everywhere and you don't need to install anything (optionally save quests to dropbox/cloud/...)
   1. You could share quests by simply copy-pasting a link to the forums a la Google Docs, people could immediately see it in front of them
3. See the world as it is in the game (3D, fully textured)
4. Drag and drop enemy/object placement
5. Step through the waves of enemies in a quest (is this possible?)
6. Quest statistics
   1. Enemy counts
   2. Box counts
   3. What else are people interested in?
7. I'd like to see a visual language for the scripting part (this is harder to do, but can we keep this in mind while working on the regular version of the scripting language or is this just a silly, almost impossible idea?)

For developers:

1. Open source (so this project doesn't die when the current developers lose interest)
2. Documentation for the technical parts (file formats, opcodes,...)

 

[GuardianGirth]

I like toys.



AND I WANT TO PLAY.


...they l00k de same...

Honestly, anything to do with scripting is out of my education and 99% of the things you talk about here I wont be able to....*actively engage* as if I know what i'm talking about. I know a good 95% of the actual design and items and what they do but the biggest thing I need is someone to label enemy/item parameters that are still just 1-6 or etc. Or even just for me, a checklist of what certain unknown items do(they're literally called unknown item). Small things like collision detection and such I havent fucked much with. Lordyata is my Script skank. If theres anyone directly to get a hold of, he would definitely be the guy. I actually mentioned though that you should come on our ts sometime so we can discuss shit. Throw me a PM whenever yo.

 

[Wilhuf]

Honestly, anything to do with scripting is out of my education and 99% of the things you talk about here I wont be able to....*actively engage* as if I know what i'm talking about. I know a good 95% of the actual design and items and what they do but the biggest thing I need is someone to label enemy/item parameters that are still just 1-6 or etc. Or even just for me, a checklist of what certain unknown items do(they're literally called unknown item). Small things like collision detection and such I havent fucked much with. Lordyata is my Script skank. If theres anyone directly to get a hold of, he would definitely be the guy. I actually mentioned though that you should come on our ts sometime so we can discuss shit. Throw me a PM whenever yo.

No worries, we want to make everything easier, not just scripting. Like moving things around and rotating them. It's very cumbersome at the moment in my opinion (as far as I understand you have to click a button and then somewhere on the map to move something). Is this a big one for you?

And labeling things should be pretty easy (unless we have absolutely no idea what something's for). Once we get something out there you can see if things have improved and suggest better names if necessary. I have no idea what the unknown items do, what happens when you add them to a quest?

And about TS, I have a very crappy microphone and my English speaking skills aren't nearly as good as my writing skills, but I'll give it a try. ToasterMage wanted to talk via Skype or Discord, but TS might be ok too, we'll see what he says.

Also, this Lordyata person doesn't seem to be on the forums here?

 

[GuardianGirth]

Yeah moving enemies is pretty slow unless you have a handle on just putting coordinates (which with some room setups is equal hell).
Labeling enemy and item parameters that currently have no labels would MASSIVELY expand my ability to do little shit here and there. I haven't fucked around enough with unknown items #'s 1- how ever the fuck many to really know what changes if I add them. And personally I dont care if its TS, discord or w/e, communication is just the biggest factor and you're more than welcome to hit me up for the address. If your mic is garby, then no worries as long as the communication is there. Also Lordyata has a name on the forums. Take a look at my Quest topic and you'll see his name. He's definitely interested in anything that makes this easier.

 

[Matt]

The main issue I have with Qedit is how labour intensive placing spawns is. There's no snap, and you have to go through menus/windows to start placing your spawns. It gets tiring very, very quickly. Furthermore, many of the variables and fields are unnamed and require a lot of guesswork/trial and error to figure out what they do.

When it comes to the scripting language itself, I personally have no issues with it and am not so sure what the huge dislike for it is for. It's very simple and doesn't need any technical know-how, just a quick runthrough of what things mean and do, but scripting for a quest in the program itself is tiring as you need to do one line at a time. While you can write in a text file, the spacing is awkward and messing it up even slightly will cause the program to hate you.

All I'd personally want in a new quest program is to make placing spawns more convenient, and allow people to write scripts in a text-editor that sorts out any spacing for you.

 

[Mylandra]

Placing enemies is convenient if you can easily do it from the 3d view.
For instance, SEGA had a debug console over the game with a debug menu allowing you to add enemies, objects and npcs without the need for an external tool. While the feature has been removed upon release, you could still implement something similar whether it's through the game client itself or a web-based editor.

A nice feature would be to be able to test whatever you changed without having to save the file, copy it to the server folder, restart the server, start the game, load the quest, move to location that needs testing.
That's partly why I made a quest debugger back then so that changes you make are applied directly to the game client and you can test the changes as soon as you type a line of code or edit any other assets.
You can still do that from a web interface, adding a quest vm for testing is just a nice addon and most people probably wouldn't even think of these possible upgrades either way. There are features that are handled just the same as you would from the game client, for instance select monster wave and it rotates to that wave in the 3d viewer. Or alternatively let the user type the command from the 3d viewer, through debug menu, specific hotkey or whatever you see fit.

As other people have mentioned, the major issue is the lack of quest project files, the quest files located in the project folder are mostly binaries (or rather raw packets embedding the quest binary itself). It's the same as converting a C file to binary, you can still retrieve the assembly code, but you can't retrieve the variable/function names since it's not part of the binary. You'd either have to create a new file format so that people can save their own variable/function names and make the user save the binary independently or make the server load this new file format and parse it correctly to the clients.

 

[Vmufan]

An easy way to import script would be nice, qedit doesn't allow to import a few lines into the current script.

Right now I'm doing research on the Opcode npc_text, and it's time consuming to add the lines one by one, or maybe I'm doing something wrong.

 

[Wilhuf]

Thanks for the suggestions, everybody. I added the top feature requests to my first post and added a little road map for the coming weeks. This doesn't mean other feature requests will be ignored. It's just that even reaching a point where this thing will be somewhat useful will take quite a while and I don't want to make it look like version 1 will be able to do everything immediately. Please brainstorm away and tell us what you want, even if it's been said before as knowing what many people find important helps decide priorities.

I also want to point out that the new scripting language would be optional. And I have to say I'm amazed that most people are not bothered by the ASM. Maybe people will change their minds if ToasterMage makes some examples.

A nice feature would be to be able to test whatever you changed without having to save the file, copy it to the server folder, restart the server, start the game, load the quest, move to location that needs testing.
That's partly why I made a quest debugger back then so that changes you make are applied directly to the game client and you can test the changes as soon as you type a line of code or edit any other assets.

This sounds really cool and possibly doable with a browser plugin. Would you be willing to share your debugger's source code? Or could we integrate it in some other way?

As other people have mentioned, the major issue is the lack of quest project files, the quest files located in the project folder are mostly binaries (or rather raw packets embedding the quest binary itself). It's the same as converting a C file to binary, you can still retrieve the assembly code, but you can't retrieve the variable/function names since it's not part of the binary. You'd either have to create a new file format so that people can save their own variable/function names and make the user save the binary independently or make the server load this new file format and parse it correctly to the clients.

I'm not sure what you mean here.

Also managed to write some quest parsing code today, so I should be able to work on showing the quest's monsters in a 3d view tomorrow. The first proof of concept will probably just be red boxes on a wire frame view though. Hopefully I'll have the first ultra crappy prototype done this weekend so there's something out there and maybe this will spark interest with other developers.

 

[Wilhuf]

The first prototype is finished: https://www.daanvandenbosch.com/pso/phantasmal-quest/

As I promised, it barely does anything. It only works well for episode I quests that take place in forest 1 and 2 since it can't yet determine which specific map version to use of e.g. cave 2. I'm mainly putting this out there in hopes of generating more interest from other developers.

Code: https://github.com/DaanVandenBosch/phantasmal-quest

 

[Andy]

I'll take a look at the code sometime Soon™ and see if I can add anything or make some suggestions.

 

[Mylandra]

This sounds really cool and possibly doable with a browser plugin. Would you be willing to share your debugger's source code? Or could we integrate it in some other way?

I'm not sure what you mean here.

Doubtful, the code is interacting with the game client which is unlikely to be of any use in this case since the web-based client isn't gonna run the client. Also, what I mean is that the quest binaries do not contain variable names. In a programming language, you would name your functions and your variables. This is not saved when you compile to an executable. Hence, the quest binary file format is not good enough to fully customize the language. While it's good for new quests, I wonder just how many of the existing ones would be converted manually by people.

 

[Wilhuf]

Doubtful, the code is interacting with the game client which is unlikely to be of any use in this case since the web-based client isn't gonna run the client.

Your debugger could provide an http server that the quest editor could talk to for example.

Also, what I mean is that the quest binaries do not contain variable names. In a programming language, you would name your functions and your variables. This is not saved when you compile to an executable. Hence, the quest binary file format is not good enough to fully customize the language. While it's good for new quests, I wonder just how many of the existing ones would be converted manually by people.

Oh you're talking about the scripting language. Yeah that's going to be a problem, could be solved with a new format or possibly a very clever decompiler.

But seriously, about this debugger, show us more please. It sounds awesome. Could you make a screencast or something if you don't want to share the source code? Or explain the basics of how you did it? I'm very curious.

 

[Aleron Ives]

It really isn't that difficult to come up with standards for which registers and function numbers you will always use for specific things. If you have trouble remembering, keep notes of which registers and functions do what in each quest you make.

A nice feature would be to be able to test whatever you changed without having to save the file, copy it to the server folder, restart the server, start the game, load the quest, move to location that needs testing.

Maybe you should fix the server software to not cache quest files and thus not need restarting to test changes. I would never be able to get anything done if I had to go through all that trouble to test each change.

 

[Mylandra]

Your debugger could provide an http server that the quest editor could talk to for example.

Oh you're talking about the scripting language. Yeah that's going to be a problem, could be solved with a new format or possibly a very clever decompiler.

But seriously, about this debugger, show us more please. It sounds awesome. Could you make a screencast or something if you don't want to share the source code? Or explain the basics of how you did it? I'm very curious.

The source code is using Delphi (Mostly for native UI), but most of the codebase is pure win32 asm which I'm sure you're not going to like. Obviously, the main part works only using native language and requires proper understanding of the game internals to work it out.

Even if I wanted to hand out the tool, I cannot distribute it over here since it'd be considered a massive cheating tool allowing you to alter game content on the fly. I tend to give vague answers around here since the more in-depth mods sometimes have the potential to break the game more than the cheats you can currently find on the web.

In short, just locate the useful quest data such as registers and such in memory, the debugging part is pretty straightforward, you just have to catch where the quest decides which function to execute based on the compiled bytecode and inject a breakpoint over there. Alternatively, you could set a read breakpoint on that same compiled bytecode and achieve the same result. The rest is pretty much debugging features that exist in other programming softwares. This allows you for instance to step through bytecode as it passes through the same code region over and over and you can also read the registers since you also snatched that info earlier (Giving you the equivalent of programming language « watches »). In my case, there are nice features allowing you to view the value history of registers and highlighting the current value whenever it changes. Other features allowing you to spawn anything anywhere live/Run quest function on the fly/etc. are just features from the client which you can reuse to your heart's content. Again, injecting code and such works with the game. You barely write any code at all since you're reusing the game code at least 80% of the time anyway making it quick and easy to code. Autocomplete, parameter hinting, colored syntax, etc. were just nice to have but not required features so the tool lets you do programming as you would in any other IDE.

I'm not too eager about having socket based debuggers, especially when not limited to local network. If you do, make sure it's fully protected so it doesn't allow just anyone to remote control the computer through this protocol. There's already enough security flaws in pso as it is.

Maybe you should fix the server software to not cache quest files and thus not need restarting to test changes. I would never be able to get anything done if I had to go through all that trouble to test each change.

I guess if you do that, work on the same computer and save the files directly, it drastically limits the amount of time required before testing changes.

 

[Soly]

Maybe you should fix the server software to not cache quest files and thus not need restarting to test changes. I would never be able to get anything done if I had to go through all that trouble to test each change.

Right now, the new ship I am testing can reload the quest (with an in game command), it kicks everyone inside a quest... but its something ... so you don't have to restart the ship and log back in... (just remake the room).

I was thinking of making it different, loading just the quest metadata for the quest menu you are currently looking at, and then loading the quest data (into the "party" class, not a global one) once you pick a quest... but this doesn't seem very efficient if the ship has to decompress all the quests to look at their metadata.
Probably I can cache the metadata instead... but who knows... I'll see when I get to that.

 

[Aleron Ives]

Don't bother trying to read the title/short/long from the quest files; many of them have garbage data that you wouldn't want to display, anyway. Just have a separate text file where you can write a static description for each quest. That way, you don't have to modify original Sega quest files to display proper quest descriptions. Apparently this is how Sega did it, since their quest list would have looked quite a bit different if they had been reading the metadata from the quests' bin files.

You should only have to restart the ship to alter the description file, e.g. to add a new quest to the list; just load the bin/dat or qst file every time somebody requests it and don't waste time caching it at all; it's not like loading a 20 KiB quest every five minutes is going to put massive disk I/O strain on the server. Of course being able to re-parse the quest text file, e.g. quest.xml, without restarting the server is ideal, but it's hardly mandatory.

 

[Wilhuf]

The source code is using Delphi (Mostly for native UI), but most of the codebase is pure win32 asm which I'm sure you're not going to like. Obviously, the main part works only using native language and requires proper understanding of the game internals to work it out.

I love Delphi and ASM, I keep wanting to delve into ASM (like write a simple compiler) but end up doing something else.

Even if I wanted to hand out the tool, I cannot distribute it over here since it'd be considered a massive cheating tool allowing you to alter game content on the fly.

Any technical info about the game can be used to improve cheats. And cheating is only discouraged, you will only be banned for deliberately harming the experience of others. We're trying to do the opposite here.

I tend to give vague answers around here since the more in-depth mods sometimes have the potential to break the game more than the cheats you can currently find on the web.

Even if you want to keep cheaters from getting even better, a little video wouldn't hurt, would it? Please satisfy my curiosity, man. You can't go around talking about this awesome tool you made and not even show us...

And also I just don't understand how cheating is even possible in this game anymore. Some simple sanity checks on the server would counter most cheating, no?

I know the server decides which monsters spawn and it decides which items drop. That's about the end of my knowledge when it comes to the protocol. Even if clients decide which monsters get hit (could a client say it hit every monster in the room 50 times in 1 millisecond?), it would be impossible to get more xp or items than is reasonable.

You could of course write a perfect bot that just runs around rooms and kills everything, picks up the items and feeds 10 mags at the same time. Make it's behavior indistinguishable from a player and you broke the game. The only way to counter this would be to ask players captcha-like questions every now and then or e.g. remove all monsters from a quest except for one rappy to see if they notice.

Am I correct in thinking that cheating outside of the perfect bot scenario is only possible in a very minimal way unless a server is very, very trusting of its players?

Don't bother trying to read the title/short/long from the quest files; many of them have garbage data that you wouldn't want to display, anyway. Just have a separate text file where you can write a static description for each quest. That way, you don't have to modify original Sega quest files to display proper quest descriptions. Apparently this is how Sega did it, since their quest list would have looked quite a bit different if they had been reading the metadata from the quests' bin files.

Why not just fix the files, so standard tools like qedit can see the nicer text too? Also Japanese is not garbage. Or maybe I haven't encountered the sort of files you're talking about.

You should only have to restart the ship to alter the description file, e.g. to add a new quest to the list; just load the bin/dat or qst file every time somebody requests it and don't waste time caching it at all; it's not like loading a 20 KiB quest every five minutes is going to put massive disk I/O strain on the server. Of course being able to re-parse the quest text file, e.g. quest.xml, without restarting the server is ideal, but it's hardly mandatory.

I would just listen for file system changes and reload a file when it changes or remake the list when a file gets added/removed. Takes a bit of work to do it this way though, don't know how much of a priority this is for Soly.

 

[Wilhuf]

I'll take a look at the code sometime Soon™ and see if I can add anything or make some suggestions.

That would be greatly appreciated. And if you decide to write some code yourself, please tell me what you're working on so we don't end up writing the same piece of code twice.

At the moment I'm working on a disassembler for the quest script that outputs a nice data structure that will initially be used to detect which episode the quest is for and which versions of the area maps should be used (e.g. caves 1 has 6 different versions, but the viewer always just shows the first version).

 

[Crimzanku]

I'm new to coding for games but I know how to code in c and c# and some miscellaneous languages. what language and programing environment do you use I wouldn't mind looking at it and learning new things, plus making quests for one of my all time favorite games would be awesome.

 

[Wilhuf]

I added the github repo link to the initial post, so you can take a look. It's entirely in JavaScript and so far everything happens on the client. I'll add some information on getting started with development to the readme (basically install yarn (npm would work too probably) and run "yarn start" from the project directory).