Technical questions

Development
21

Yet another question from yours truly.

I see a lot of class fields with “protected” visibility. I have not counted but it feels like the majority of the fields are protected. This is very uncommon for me, I have learned long ago that composition is preferable to inheritance, and that fields should be marked protected if and only if the class has been designed with inheritance in mind. In todays corporate Java world I would say 99% of the fields are private, the methods are either public, private, or sometimes with package visibility so they can be accessed by unit tests, and even if there is inheritance, the subclasses often access the private fields of the superclass using the public getters/setters. Might seem like an overhead but the JVM can optimize this away at runtime by inlining these accessor methods, modern JVMs can even turn often used code hotspots into native code. The general idea is to hide as much as possible from the outside world and from subclasses, and to only allow access to fields through accessor methods, never directly.

Are all these protected fields in Vassal a kind of public API for the modules, to allow them to inherit from Vassal’s core classes and add/overwrite behaviour?

Or is this a legacy of the C++ developers and the 80ies/90ies style of OOP where inheritance was always a good thing and let’s make everything protected by default in case we want to inherit from it later on? :smiley:

22

However/whyever they once arose, I can definitely tell you that having most of the stuff be protected is absolutely critical for creating custom classes for modules (so in other words, yes, effectively a public API).

(And yes, I’m old enough to have worked professionally in the all-inheritance-all-the-time world, in C++)

Brian

23

Mostly this.

Early on, little or no thought was given to the design of the classes to allow Module Developers to develop custom code. Many classes had private members without even protected accessibility. It was next to impossible to write properly functioning custom classes except in the simplest of cases.

Like many other parts of Vassal, custom module code was a neat demonstration of capability, but with no formal underpinning.

The primary Vassal ‘API’ is the module editor, I don’t think the designer ever envisaged or wanted people to be writing large amounts of Java to implement their modules.

24

I understand. So things are even worse than if these fields were due to weird OOP practices.

Another question, I have tried several modules, deleted some of them afterwards, now I have 1.3GB in my ~/.VASSAL/tiles directory. I guess each of its subdirectories belongs to a module? Is there any way of telling which of them belongs to which module? Is there any cleanup procedure? Can I delete all of them and just reopen the modules that I kept to rebuild the tiles?

25

I don’t know about other questions, but the answer to this one is definitely “yes”.

26

And you can delete them from the Module Manager Tools → Clear Tile Cache

27

A couple more, this time concerning custom code.

I have looked into what I thought was a quite sophisticated module and simply had to have custom code, but only found images in it, several premade savegames which seem to be the scenarios, and the buildfile XML, no custom code, no Java classes or anything. I’m curious how a module with custom code works, is it subclasses only or are original vassal classes sometimes overridden, perhaps anyone could show me a module that has this so I can dissect it.

Another question is, can an estimation be made as to how many or what percentage of modules use custom code, and also what parts of the “public API” is actually in use by modules? And several other questions that are deducted from this one, are all modules known, are 100% of vassal modules hosted on this site’s wiki or are there rogue modules that are not found on this site? And if it is not known which parts of the public API are actually in use, would it make sense to write a program that parses all the modules and spits out a list of which module uses exactly which class/method of the public API? And another question, probably more for module developers than for vassal developers, are module developers generally aware of the possibilities to extend vassal by writing custom code, or are they happy and content with the module editor and these simple beanshell scripts they can write in the editor?

28

Some modules that have custom code. The first two on the list are mine, so I include the source code in the modules as well.

  • For the People 3.2 - whole bunch of stuff, including Chatter w/ HTML/colors, some UI-code to interpret commands a different way, a “flare map”, a visually tweaked Turn Tracker, and a few miscellaneous other things.
  • Paths of Glory 9.7 - Chatter w/ HTML/colors, a “flare map”, and a couple tweaks to Turn Tracker.
  • Twilight Struggle - a “chess clock” among other things
29

Thus spake Flint1b:

Another question is, can an estimation be made as to how many or what
percentage of modules use custom code, and also what parts of the
“public API” is actually in use by modules?

Sort of.

I can get a list of VASSAL classes which are used by custom code in a
module using jdeps and a bit of shell scripting. I can run that on
all the modules on our site and get a list of all the classes which
are in use by all the modules we host. I did this a few weeks ago
to check if it was safe to remove a few long-deprecated classes.

Unfortunately, jdeps doesn’t give you function or field information,
only class dependencies—at least not that I can see—so I can’t
tell if anything uses particular deprecated public or protected member
functions, which would also be nice to remove. (On that point, I’m not
entirely certain what will happen with subclasses in custom code if
one removes fields or functions they don’t use. I know it wouldn’t
hinder recompilation, but we’re not in a position to recompile those
classes. So I don’t know if this is a viable option.)

This is what you can do with jdeps:

jdeps -verbose:class -e ‘VASSAL.*’ x.jar

Note that it throws an exception if you try to give it a filename which
does not end in ‘.jar’, even though a JAR is just a ZIP archive, and so
is a module. Apparently they did this to be irritating, because it had
to be extra work to reject files which it can otherwise process just fine.

And several other questions
that are deducted from this one, are all modules known, are 100% of
vassal modules hosted on this site’s wiki or are there rogue modules
that are not found on this site?

There are definitely modules we don’t host. It’s hard to know what
proportion of all modules in existence that is, but my guess is that
we host the vast majority.

And if it is not known which parts of
the public API are actually in use, would it make sense to write a
program that parses all the modules and spits out a list of which module
uses exactly which class/method of the public API?

Yes. It would be a nice thing to have, as then we could do what I said
above—find any deprecated functions nothing we hosts uses, and remove
them. (Better might be to remove deprecated functions which aren’t used
by current versions of modules, but that’s trickier, as you can’t tell
just from looking at a module whether it’s the current version.)

–
J.

30

Thanks for taking the time to answer.

Jdeps is indeed a very high-level tool and only works on package- and class-level dependencies, and also seems to have issues with generics and it’s type erasure. There are other good tools though that can find field-level and method-level dependencies, there are also bytecode manipulation frameworks like asm or cglib that can be used to analyze existing bytecode. Some dependencies sadly cannot be detected at all, references to static final fields (constants) of type String for instance are optimized away by the javac compiler and inlined into the bytecode (a glorious “optimization” e.g. when you have few files with huge static final SQL statements for tables with 150+ columns that are referenced from all over the codebase, leading to megabytes worth of strings being duplicated).

Removing unused fields or methods should not break existing bytecode, they should keep working if the fields and methods they reference keep the same interface.

I will look some more into this and try to find a decent solution.

31

This tool looks good: jeantessier.github.io/dependency-finder/

I tried it like this, and got some results, sorry for the messed up formatting, the “code” tag here seems to replace some spaces with tabs:

export PATH=$DEPENDENCY_FINDER_HOME/bin:$PATH

– dependency finder needs JAVA_HOME to be set else it looks in /bin/java

export JAVA_HOME=/usr/lib/jvm/graalvm-java11

– possibly filter modules, based on version number in filename or version number set inside the module

???

– prepare dependency data for modules, this creates a 1-1.5mb xml per module:

DependencyExtractor -xml -out Paths_of_Glory_9.7.xml $VASSAL_MODULES/Paths_of_Glory_9.7.vmod

DependencyExtractor -xml -out For_the_People_3.2.xml $VASSAL_MODULES/For_the_People_3.2.vmod

– extract deprecated elements from vassal, then write a report of deprecated elements in use:

ListDeprecatedElements -out deprecated.txt $VASSAL_HOME/lib/Vengine.jar

DependencyReporter -out report.txt -show-inbounds -scope-includes-list deprecated.txt Paths_of_Glory_9.7.xml For_the_People_3.2.xml

– result is an empty file, now do a cross-check by doing the same for the current 3.3 codebase:

ListDeprecatedElements -out deprecated_3.3.txt $VASSAL_COMPILED_CLASSES

DependencyReporter -out report_3.3.txt -show-inbounds -scope-includes-list deprecated_3.3.txt Paths_of_Glory_9.7.xml For_the_People_3.2.xml

– result:

$ cat report_3.3.txt

VASSAL.build.module *
    Map *
        componentCoordinates(java.awt.Point) *
            <-- ForThePeople.FTPKeyBufferer.mousePressed(java.awt.event.MouseEvent)
            <-- ForThePeople.FTPMenuDisplayer.mouseReleased(java.awt.event.MouseEvent)
        mapCoordinates(java.awt.Point) *
            <-- ForThePeople.FTPKeyBufferer.mouseReleased(java.awt.event.MouseEvent)

Overall a very convenient tool, can take anything from a jar, zip, vmod, filesystem directory with compiled classes, finds and analyzes every piece of bytecode in it, has lots of other options for analysis and reporting.

Is this a viable tool and should I try writing a full bash script? I’m not a professional bash coder and don’t know how the module files are laid out in the filesystem, also don’t know the disk space constraints, and wouldn’t be able to test this script in a production-like environment. Also don’t know if it would be possible to detect the latest versions of the modules by using the filename, the few modules I’ve looked at have their version in the filename but this might not apply to all modules, and the module version set by the module designers might also not be a reliable criteria for deciding which is latest.

32

Thus spake Flint1b:

[This message has been edited.]

This tool looks good:
jeantessier.github.io/dependency-finder/[1]

I’m giving this a try now. Running it on all the modules is going to
take quite a while; I’ll report back when I have some results.

–
J.

33

Thus spake Joel Uckelman:

Thus spake Flint1b:

[This message has been edited.]

This tool looks good:
jeantessier.github.io/dependency-finder/[1]

I’m giving this a try now. Running it on all the modules is going to
take quite a while; I’ll report back when I have some results.

See Bug 13124, where I’ve attached the results:

vassalengine.org/tracker/sho … i?id=13124

–
J.

34

Is this all that is used from the modules, only about 50 methods? If this is true then backwards compatibility is not as big a problem as it sounded like.

35

Thus spake Flint1b:

Is this all that is used from the modules, only about 50 methods? If
this is true then backwards compatibility is not as big a problem as it
sounded like.

That’s all that’s used which is deprecated. I didn’t run the analysis
on what’s used in total.

–
J.

36

Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated fields/methods first, then checks all modules against that file.

I’m curious though, how many modules are there in total, how big is their total size, and how long did it take to run this dependency analysis? Hours? Days?

37

Thus spake Flint1b:

Ahh ok, I misunderstood. Right, that tool makes a file of all deprecated
fields/methods first, then checks all modules against that file.

What I ran over the modules is

DependencyExtractor -xml -out blah.xml blah.vmod

which gets everything, if I understand correctly.

That output was then fed to

DependencyReporter -out depreport -show-inbounds -scope-includes-list deprecated

which is what filters out everything not in the deprecated list.

I’m curious though, how many modules are there in total, how big is
their total size, and how long did it take to run this dependency
analysis? Hours? Days?

Moments ago, there were 7546 files which match *.{mod,vmod,ext,vext}
(many of which don’t have any custom code in them, many of which also
aren’t current, but that’s hard to tell just from running find over
the tree where the files are). The total size of the tree is around
270GB, which also includes some images and other random stuff, but
the vast majority of which is modules. I let the analysis run overnight,
and it produced 2.6GB of text. The total time was hours, but I didn’t
keep track of how many hours.

–
J.

38

That is also only modules that we host though. There are at least some modules using custom code not hosted on vassalengine.org

39

Interesting info, thanks.

I have yet another question, in src/, next to the VASSAL package, we have what looks like a stripped and possibly modified or manually extended version of BeanShell v. 2.x. It has a subset of asm in bsh.org.objectweb.asm which BeanShell 1.3.0 did not have so I guess it is BeanShell >= 2.0.

Why is the BeanShell libraries code manually copied into the project instead of bringing its .jar as a dependency? And which version of BeanShell is this exactly? What is the process of updating to newer BeanShell versions? Are we aware that the class bsh.XThis.Handler that we have in the code has this neat little arbitrary code execution exploit github.com/beanshell/beanshell/ … /tag/2.0b6) ?

More generally speaking, if I made a module which, apart from allowing to play a boardgame, used some kind of network channel to contact a private server and get new orders from me e.g. for scanning the hard drive for sensitive data, taking photographs with the camera, would there be some mechanism in Vassal to prevent that i.e. prevent “evil” module designers from doing evil things?

And by extension, what if I was a “sloppy” (in the sense of software security) module designer and without having bad intentions wrote custom code for my module that would allow a user with bad intentions to do evil things e.g. in a multiplayer or PBEM setting enter some code into a text field which my module’s custom code would blindly execute on another user’s machine?

Are problems like this solvable at all if modules have the whole Java language as their API, I am not even sure myself. I only had to deal with server-based security so far.

40

Because we had to make significant modifications to the source, including parser changes to get it to play nicely with Vassal. We had also been having a load of trouble with upgraded external libraries importing bugs into our project, so we essentially took a fork of Beanshell.

2.0b4

It’s not really required except for any major bug fixes which will need to be done manually.

I don’t believe that affects us, but should probably be patched.