You’ve found one of the mistakes made at the very beginning of Vassal (when it was still VASL). The long-term solution is that we won’t be doing that in V4.
OK so in the mean time can we have the ability to delete and re-upload scenarios so we don’t have to wait until version 4?
If the scenario files cannot be used with versions of the module other than the one they were made with, why not package them as scenarios inside the module? That way, users cannot get the wrong file.
I hope you are not serious with that suggestion. Many 10’s of packaged scenarios only adds many MBs to the module - for little gain. Keeping the scenarios separate is intentional to avoid main module file size bloat.
Have a look at the vassal webpage for this game (world in flames), note the number of available scenarios and their file size.
You may not think that what is stored on the vassal server is important, thus who cares about what the actual module owner can or cannot do with the ‘stuff’ on THEIR vassal webpage. I do, however, care. Removal of waste, in any environment, is paramount. Anything else is wasteful. Of, perhaps, time, storage, efficiency, logic, to name a few.
Just do it and let us manage. We do not need, nor want, micromanagement overlord processes.
All my suggestions are serious.
I totaled up the size of the 2.1.4 scenario files and got 819MB. That’s over the current file size limit, but I could increase that to accommodate if someone asked.
Once Vassal 3.8 is released, you can expect the scenario files to be significantly smaller—likely half their current size—due to 15032 and 15060. Thus in the near future, you’re looking at more like 400MB instead of 800MB.
“Module file size bloat” would seem to imply that what’s being stored is superfluous or unnecessary—e.g., unused images. I take it that the scenarios are not bloat, because they’re wanted and used.
I spent 18 months on the module library conversion. If that doesn’t establish for you that I think the module library is important, I’d be interested to hear (a) why not and (b) what would.
Please read my post above regarding storage costs. There is nothing significant to be gained here. If the WiF scenario files were reduced to zero size, we would save around two cents per month on storage. It is not worth having an argument of this length over two cents per month.
That said, putting the scenario files into the module or leaving them as they are makes no difference for storage costs: They’re the same size regardless.
What will make a difference for the size of the scenario files is Vassal 3.8, which you could already be using if you wanted----3.8.0-beta2 was released two weeks ago.
Thanks to Del for that innovation. Not to mention that Del (inter alia) has also submitted a number of other improvements that ‘seem’ to be held off.
“Module file size bloat” would seem to imply that what’s being stored is superfluous or unnecessary—e.g., unused images. I take it that the scenarios are not bloat, because they’re wanted and used.
Not correct. A lot of saved games that may or may not be actually used by players. Thus ‘bloat’, as in something provided that may not actually be used.
Please read my post above regarding storage costs. There is nothing significant to be gained here. If the WiF scenario files were reduced to zero size, we would save around two cents per month on storage. It is not worth having an argument of this length over two cents per month.
This has nothing to do with the cost of storage. This is fully concerned with the Vassal module webpage ‘owners’ having the ability to do their own management of that webpage. You say that your time taken to respond to requests to delete file on a vassal module webpage in miniscule. Why is that? Because very few ask you as it takes a while for you to respond and do that. If everyone who wanted a deletion of a file suddenly realised that there would be a rapid response how many requests might there be?
Lastly, it is the ‘owners’ of the module webpage responsibility to manage their webpage and the files contained therein. Vassal, ie you it seems, is responsible for setting the ‘rules’, but all else is not within your purview. I come back to micromanagement, this is something you are doing. This is something you should not be doing.
I see one open PR from @delatbabel at present, which has significant overlap with another, more extensive PR that’s being actively discussed at the moment. Have I missed something? If so, I’d be grateful if you’d point it out specifically.
Micromanagement is what we used to do. Tim handled multiple requests per day with the old module library. That was an unsustainable amount of work, which he nonetheless did without complaint for a decade. The current module library gives project owners far more freedom to manage their projects as they see fit—they need not wait for moderation, their projects are not squashed together with other projects with which they must jockey for position on the page—and involves us in that only a few times per month. That is not micromanagement.
You disagree with our policy. I understand that. Thank you for your input. I have no intention of changing our policy at this time.
The big one is this one: Index module names for search, and add an exact ?module_name= lookup by delatbabel · Pull Request #34 · vassalengine/game-library-service · GitHub
The workflow subsequent to that would be:
- Once GLS/34 is finalised, that adds database slots to the GLS to store the module “name”. Note that this is different to the file name or the name that the owner gives it in the vassal library, this is the name embedded in the module, visible from the vassal module browser, and, more critically, embedded in every vsav game that was saved from within that module.
- Next step would be a catch-up PR that runs a script in the GLS back end to populate the module name from every module (vmod file) uploaded to the library. That might take some time to run.
- Once that’s done, the vassal desktop could download the correct module and all needed extensions from any given vsav file just by loading the file – just do a ?module= search on the GLS, find the module based on the name embedded in the vsav file, find the list of extensions based on the list embedded in the vsav file, and download what is needed.
That kind of obviates both Add File > Download Module from Library to the Module Manager by delatbabel · Pull Request #15034 · vassalengine/vassal · GitHub and Game Library Browser UI by cholmcc · Pull Request #15103 · vassalengine/vassal · GitHub
The end result would be that you could just point vassal at a vsav file and it would (after prompting I guess) just download the module needed and load the game. A separate module browser would not be needed. A further change would be to enable the same functionality when joining a game online – just see the game and join, the module would then automatically download.
I wasn’t previously aware of Game Library Browser UI by cholmcc · Pull Request #15103 · vassalengine/vassal · GitHub – it seems to duplicate some of my earlier work but also tackles it in a sligtly different way. I’ll have a more detaied look later. Either is fine by me. Currently I have players using a separate program to download the WiF module and the needed extensions – it works fine as it is and reliably detects the extensions needed, their corrrect versions, and downloads those, but having that functionality built into vassal would be useful.
To be fair, module file size bloat isn’t that bad or that wasteful. Yes lots of images do make a module much bigger (and watch this space), those images don’t trigger anything in the vsav files that make them bigger.
Each counter on the map or in the force pool triggers a deep clone of the counter from the counters repository and as Joel pointed out earlier that is going to go away in V4, and could probably be gotten rid of in V3.9 or whatever. That is the number 1 source of the vsav bloat.
Number 2 source of the vsav file size bloat is the escaping or “back-slashing” of traits in the vsav file as discussed here vassal_extension_utility/docs/wif-save-bloat-analysis.md at main · delatbabel/vassal_extension_utility · GitHub – to the point where a little over 50% of the space used in a WiF vsav file is backslashes.
Given that vsav file size is the number 1 complaint that we get from actual players, those are the main issues that are worth addressing IMHO, not the image size or memory footprint of those images – as Don pointed out to me in discussions earlier this week, image file format, i.e. PNG vs JPEG vs WEBP affect the vmod size on disk but not the memory footprint very much.
This is due to SequenceEncoder. It’s been part of Vassal the entire time it’s existed, and was in VASL before that. SequenceEncoder should not have been written the way it is.
The problem you will immediately hit with trying to fix it is that SequenceEncoder appears in every component that contributes to a save file, but also there are some components where SequenceEncoder isn’t used but the component encodes itself in the way that SequenceEncoder would. Rooting all of this out would be a massive job. Supposing you managed that, you would then have a problem with all of the custom module code using SequenceEncoder (or worse, not using SequenceEncoder and writing out sequences manually), which you can’t change.
None of what you found there is new to me—I’ve been aware of the problem since 2006.
Do they ever say why they are complaining about this? I understand that they complain, but in what way does this cause them a problem?
Because players frequently share saved games on services such as Dropbox that have free/paid tier storage limits. With lots of saved games it’s important to have those saves as small as possible to be able to stay within the free tier or whatever paid tier you’re on.
1 Like
A noted elsewhere, I think that for such a specialised task, you are better off with a specialised tool.
BTW, note that storing the module name, as given in the moduledata, in the Game Library database, isn’t necessarily the panacea you are looking for: Very different modules may use the same name (wit the number of Afrika Korps projects). What we’re guaranteed is unique is the projectID in the Game Library. Suppose you have a save file that belongs to the module named A. You then query the DB for the module name A, and it returns 4 projects, most of which contain extensions - some of them may even have the same name - say A artillery extensions, A airwar extension. How would you know which ones to get?
Getting something like you are advocating to work robustly across many different modules, etc. will require many changes. For example, that each module series (releases) have a unique ID, and that unique ID is written in the save files (as well as extensions, logs, …).
Yours,
Christian
You put up a list and ask the user to pick one. Not that hard I would have thought.
I am looking at SequenceEncoder again. Regardless of the outcome, this is the last time I will ever consider modifying SequenceEncoder.
Eventhough SequenceEncoder may produce a lot of repeated \ it is actually not as bad as one might think. The algorithm used by ZIP is pretty good at compressing repeated data, so the many repeated \ probably doesn’t take up much space.
Once the format has been decoded (which happens almost immediately), the data takes up no more memory than needed.
In other words, there may not be a massive gain in terms of storage going from the current format to another format.
That said, the SequenceEncoder is annoying for other reason - kinda hard to get right, and does not lend it self to manual editing. Traits could be encoded in JSON - e.g., for Embellishment (a.k.a. Layer) trait could be
{
"trait": "emb2",
"parameters": {
"images": [ ... ],
"newNames": [ ... ],
"activateName": "Activate",
"activateMask": "CTRL",
"activateChar": "A",
"increaseName": "Increase",
....
},
"states" : {
"level": 1
}
}
Of course, anything that is null can be left out, which will take the size down a bit. If one wanted a slightly less verbose format, one could do
{
"trait": "emb2",
"parameter": [ "Activate", "CTRL", "A", "Increase", ... ],
"states": [1]
}
but then all values must be present - at least up to the point where the rest are null.
Incidentally, the Python module pywargame works with an internal format that is almost the above, and could be used to convert from one to another 
Yours,
Christian
I found a way to stop SequenceEncoder from encoding nested sequences in a way that uses space quadratic in the depth of their nesting.
This should further reduce the size of your saves and logs.
Yup, I saw that. I had already started work on a slightly different approach, so I pushed it anyway in case you want to compare the two.
What I found was that it could be fixed without touching SequenceEncoder.