Roadmap for VASSAL 4

Thus spake Michael Kiefte:

It’s also not guaranteed that the next platform will be based on C++. I’m not
sure that it’s my top choice. Right now, my top choice would be wxPython or
even just pyOpenGL on top of tkinter. It runs surprisingly fast and is exce
edingly easy to code.

The problem I ran into in the Python demo I wrote was that the parts
which were not being handled by the video hardware were surprisingly
slow. Loading large images, for example, was much, much slower than
Java, while the same images loaded instantly in C++. I also have
serious doubts about the maintainability of large projects writen in
dynamic languages. No compile-time checking means more bugs to find
at runtime. I think Python is a good choice for our scripting language,
but not a good choice for the engine itself.


J.

Had you compiled the python code to bytecode?

Your other arguments are valid though.

  • M.

On 2011-09-05, at 1:47 PM, Joel Uckelman uckelman@nomic.net wrote:

Thus spake Michael Kiefte:

It’s also not guaranteed that the next platform will be based on C++. I’m not
sure that it’s my top choice. Right now, my top choice would be wxPython or
even just pyOpenGL on top of tkinter. It runs surprisingly fast and is exce
edingly easy to code.

The problem I ran into in the Python demo I wrote was that the parts
which were not being handled by the video hardware were surprisingly
slow. Loading large images, for example, was much, much slower than
Java, while the same images loaded instantly in C++. I also have
serious doubts about the maintainability of large projects writen in
dynamic languages. No compile-time checking means more bugs to find
at runtime. I think Python is a good choice for our scripting language,
but not a good choice for the engine itself.


J.


messages mailing list
messages@vassalengine.org
vassalengine.org/mailman/listinfo/messages

postby mkiefte » 05 Sep 2011 17:25

Custom class modules breaking might be acceptable depending how many there are. Is there anyway to look at the modules on Vassal site to see how many have custom classes (contain .class files)?

I don’t think breaking other modules would be especially as the developers will often have moved on and no longer be interested. If too many modules break then users will remain on Vassal 3.x

Actually major versions don’t break the code, normally. Every new version of OpenGL library doesn’t mean all the programs using it must be re-written. Every new version of c or c++ compiler doesn’t mean rewriting all your source code. Every new version of windows or linux doesn’t mean throwing away all your programs and buying a new set. (Although I’m sure MS would make you do this if it thought it could get away with it!)

If the new version can’t read most of the modules then it is not a version of Vassal. It’s something else.

I hope not. The delivery of software that runs on multiple platforms is a very large piece of work. I did this professionally for many years and the amount of time spent working out why the system doesn’t work on build X of system Y when it does everywhere else can take more time than developing the system in the first place.

Consider just the actual deliverables you need to create: binaries for Windows XP, Vista, Windows 7, linux (RedHat, Ubuntu, Debian), Mac OSx Lion, Leopard, … With 64 bit and 32 bit varieties all with installation packages to ensure all the libraries you need are also installed (in the correct version). For a release you need to build all these, test them, apply any fixes, rebuild , test and so on. It used to take a team of 4 or 5 working full time most of a month to get all our builds for a particular release together and ready to go and someone would always come up with an urgent fix to be squeezed in at the last moment with predictable results for the quality of the release.

Going with C++ or anything that needs platform dependent deliverables is making yourselves a lot of work. Stick with Java. Fix the Java bugs in OpenJDK (which is being ported to Mac).

Thus spake Michael Kiefte:

Had you compiled the python code to bytecode?

I believe so, yes. But it wasn’t my code that was slow, it was the
image loading code I was calling that was slow.


J.

I dont think there is but having gone through most of the modules maintaining the library I would say the number of modules with custom classes is less than 100 modules or 10% there abouts.

Breaking these modules should not be a major issue for most of them as the majority contain custom classes that get incorporated into the core at some point (such as the turn counter for example) down the road and only need to be updated to reflect this after the class is adapted into the core engine.

The remaining few such as your OCS code or Mike’s TS code are the exceptions and will be the problem modules that you’ll have to handle yourselves unless you can make it generic enough to include in Vassal if you think it could be useful for others ( hint - cards in hand count shows in toolbar, looking at you Mike! :smiley: )

Thus spake george973:

A lot of modules are going to break with the next version – there is
no question about that. For example, the Twilight Struggle module is
guaranteed to not work in the next major version because of the custom
code therein. There are a number of other modules with custom code
that also will not work.

Custom class modules breaking might be acceptable depending how many
there are. Is there anyway to look at the modules on Vassal site to see
how many have custom classes (contain .class files)?

I could write a script to run on the server to check on this, but I’d
rather not spend time doing that unless we need to know precisely. My
experience with examining random modules has been that the answer is
“few”.

I don’t think breaking other modules would be especially as the
developers will often have moved on and no longer be interested. If too
many modules break then users will remain on Vassal 3.x

Writing a module converter is one of the projects which I expect we will
pursue nearly from the start with V4. All of the built-in stuff in V3
will map straightforwardly to something in V4.

I hope not. The delivery of software that runs on multiple platforms is
a very large piece of work. I did this professionally for many years and
the amount of time spent working out why the system doesn’t work on
build X of system Y when it does everywhere else can take more time than
developing the system in the first place.

Consider just the actual deliverables you need to create: binaries for
Windows XP, Vista, Windows 7, linux (RedHat, Ubuntu, Debian), Mac OSx
Lion, Leopard, … With 64 bit and 32 bit varieties all with
installation packages to ensure all the libraries you need are also
installed (in the correct version). For a release you need to build all
these, test them, apply any fixes, rebuild , test and so on. It used to
take a team of 4 or 5 working full time most of a month to get all our
builds for a particular release together and ready to go and someone
would always come up with an urgent fix to be squeezed in at the last
moment with predictable results for the quality of the release.

Going with C++ or anything that needs platform dependent deliverables is
making yourselves a lot of work. Stick with Java. Fix the Java bugs in
OpenJDK (which is being ported to Mac).

We have platform-dependent deliverables and platform-specific code with
Java, plus a load of decade-old bugs for which we’ve had to write
workaround code. Java is not platform-independent for nontrival things,
despite what Sun and now Oracle claim. I already spend quite a lot of
time determining why things work on one platform but not another (by
my count, something like 20 hours in the past two weeks alone), a lot
of which is due to Java bugs. I no longer have any faith that these bugs
will be fixed. With C++, we can rely on higher-quality libraries, ones
where you don’t typically encounter bugs which are nearly old enough to
drive.

For more detail, I refer you to points 1, 3, and 5 in this post:


J.

The remaining few such as your OCS code or Mike’s TS code are the
exceptions and will be the problem modules that you’ll have to handle
yourselves unless you can make it generic enough to include in Vassal if
you think it could be useful for others ( hint - cards in hand count
shows in toolbar, looking at you Mike! :smiley: )

It’s in my inbox! (i.e., my todo list)

  • M.

Python Imaging Library?

  • M.

On 5 September 2011 15:29, Joel Uckelman uckelman@nomic.net wrote:

Thus spake Michael Kiefte:

Had you compiled the python code to bytecode?

I believe so, yes. But it wasn’t my code that was slow, it was the
image loading code I was calling that was slow.


J.


messages mailing list
messages@vassalengine.org
vassalengine.org/mailman/listinfo/messages

Thus spake Michael Kiefte:

Python Imaging Library?

I think it might have been, yes. I’ll see about getting you the code
as soon as I can.


J.

Thus spake george973:

The codebase does need fixing. But we don’t need to jump into a
different language etc to do it. Just slog away rewriting it.

I’ve been trying to refactor parts of the current codebase for literally
years now. This experience has led me to conclude that:

  1. This cannot be done with any reasonable amount of effort. E.g., the
    current codebase is so messed up w/r/t thread safety, that it’s just not
    feasible to fix it. It is, I believe, feasible to do it correctly from
    the start, if one is trying to do that. Our problem is that nobody paid
    any attention to these things until the 3.0 release or thereabouts.

  2. This cannot be done while maintaining existing class hierarchies in
    a backwards-compatible way. E.g., doing something as simple as adding
    the capacity for components to report on what images they use will
    break every piece of custom code in existence.

Just to reiterate the main point: you don’t have a blank sheet of paper
on which to design the perfect game system. You’ve got the legacy of the
current system and lots of users to satisfy. Your options are limited.

I don’t intend to abandon existing modules. A working module converter
will be one of the V4 milestones.


J.

by uckelman » 01 Aug 2011 21:03:

Hmmm. It all depends on which part of a multi-platform project you work on. We always estimated that 90% would be platform independent and 10% platform dependent but the effort would be 50-50 for both parts. We were writing in objective-C so had ported the GNU compiler to each platform (so avoiding problems where system X’s compiler optimises your code to not working).

The basic problem is that nothing worth doing ports easy and anything that ports easily is trivial.

Don’t forget the Java implementation on any system stands on the image i/o libraries for that system. The implementors of Java didn’t re-invent the wheel. Any code you write will need to sit on the same libraries and hit the same bugs possibly.

Don’t forget the major testing headache. Now you can assume that Vassal will run on 90% of machines which can install JRE but if you write in C++ you’ll have to build it and test it on all possible environments including old versions of MacOS and Windows and Linux.

You know it works on whatever you’ve built it on. Don’t forget most users are not at the leading edge of Windows, Linux or MaxOS and their machines have old libraries which may have old bugs. You’ll not be able to carry your new libraries around with you as they may not work on older software and/or installing them on older systems may break other programs on the machine. Not to mention 64-bit and 32-bit version of the libraries.

If you don’t want to wait for a bug to be fixed join openJDK and fix the bugs there. They’ll get into mainstream Java and openJDK is porting to Mac so you’ll have 1.6 on the Mac plus bug fixes.

I would hope so!! But if just speed was the goal would it be better to go for Java Native classes for the bits that need speeding up

Don’t forget that no one writes a post on the forum to say his JRE installation worked perfectly! You’ll get a very distorted view if you take the forum posts to represent the general experience. There are thousands using Vassal who’ve had little or no problem.

Do our users care if they are in a code ghetto? They just want their modules to work. As they do now for the most part. The number of bug reports is minimal compared to the number of games played on Vassal.

Hmm, to be a little cynical but perhaps realistics, I suspect someone will be saying the same about your new design in five years time. A design is only clean if you’ve thought everything out beforehand and never extend it. In reality any program grows by accretion after its initial release as users request thing never thought of which don’t quite fit the design or do things with your product which the designers never imagined - and with scripts the module developers will be able to really go to town. Consider what has been done with traits which never crossed anyone’s mind.

But I do agree with the major rewrite, just not with abandoning the Java platform to do it.

by uckelman » 05 Sep 2011 22:58 :

Looking at the code even just quickly shows that it wasn’t written to be multi-threaded and the layout of the packages etc is all wrong. So a major rewrite is called for but this does not mean leaping into something totally different.

Yes, I accept all my custom code will be broken, reluctantly but I do accept it. I suspect that mkiefte and I account for most of it anyway.

I suspect the users would just like the new version to load existing modules and save files without them having to re download new modules (assuming you convert all those on the web site).

Conversion in Java would be easiest as you could just replace existing classes in the buildFile by stubs which collected the data and then created the new classes from that. You could do something like that in C++ but would have to re-write the stubs rather than just stripping surplus code out of existing classes.

I think you are making yourselves a lot of work going to C++ and I don’t think you’ll reap as many benefits as you think especially for portability.

Thus spake george973:

Hmmm. It all depends on which part of a multi-platform project you work
on. We always estimated that 90% would be platform independent and 10%
platform dependent but the effort would be 50-50 for both parts. We were
writing in objective-C so had ported the GNU compiler to each platform
(so avoiding problems where system X’s compiler optimises your code to
not working).

The basic problem is that nothing worth doing ports easy and anything
that ports easily is trivial.

The platform-dependent problems I’ve encountered in Java fall into two
categories: Some are due to the nature of the different platforms, some
are due to the JDK. I don’t believe we can eliminate the ones in the
first category, but the ones in the second category are unique to Java,
so we could avoid them by moving to any other language.

Don’t forget the Java implementation on any system stands on the image
i/o libraries for that system. The implementors of Java didn’t re-invent
the wheel. Any code you write will need to sit on the same libraries and
hit the same bugs possibly.

Actually, the implementers of Java did reinvent the wheel with
ImageIO. I know this because I’ve spent a significant amount of time
looking at the ImageIO PNG loader in particular to determine exactly how
it’s broken. This would not be a problem with libpng. Because virtually
everything else which loads PNGs depends on it, it has received intense
scrutiny over the years to make sure it works. When bugs are found, they
get fixed promptly. (Similarly for libjpeg, libgif, librsvg.) I don’t
have a collection of PNGs on which libpng barfs. I do have such a
collection for ImageIO.

Don’t forget the major testing headache. Now you can assume that Vassal
will run on 90% of machines which can install JRE but if you write in
C++ you’ll have to build it and test it on all possible environments
including old versions of MacOS and Windows and Linux.

Right now we have a problem which so far as I can tell, only occurs on
64-bit Windows 7. We’ve seen plenty of problems specific to old versions
of MacOS in the past. I don’t see that we can predict which OS-JRE
combinations will work now. This is not a win for Java.

You know it works on whatever you’ve built it on. Don’t forget most
users are not at the leading edge of Windows, Linux or MaxOS and their
machines have old libraries which may have old bugs. You’ll not be able
to carry your new libraries around with you as they may not work on
older software and/or installing them on older systems may break other
programs on the machine. Not to mention 64-bit and 32-bit version of the
libraries.

Except for Linux packages built for particular distribution versions
(where we can specify which lib versions we require), I intend that we
will ship our own copies of the libraries on which we depend. We won’t
be interfering with libraries already on the machine because we’ll be
bringing our own.

If you don’t want to wait for a bug to be fixed join openJDK and fix the
bugs there. They’ll get into mainstream Java and openJDK is porting to
Mac so you’ll have 1.6 on the Mac plus bug fixes.

Some of the bugs I’ve encountered have had patches in Sun’s bug tracker
since before OpenJDK existed, yet they are still not fixed. I’m quite
willing to contribute bug fixes to projects I use, but the JDK looks
like a bottomless pit to me. There’s a limit to the amount of effort
that I’m willing to invest in something that is not my project. Using
dependencies which already work is a more efficient use of our time
than fixing the JDK is. The fact that we’re having this discussion about
image loading bugs—bugs which you would expect to have widespread
impact—tells me that hardly anybody else uses Java the way we do.

  1. You’re underestimating just how much faster C++ is than Java when
    working with images. The majority of the computation that VASSAL does
    involves images. The C++ that I wrote for our V4 demo does everything
    essentially instantanously; working with the same images in Java is
    agonizingly slow.

I would hope so!! But if just speed was the goal would it be better to
go for Java Native classes for the bits that need speeding up

No, it wouldn’t, for reasons I’ll give below.

Don’t forget that no one writes a post on the forum to say his JRE
installation worked perfectly! You’ll get a very distorted view if you
take the forum posts to represent the general experience. There are
thousands using Vassal who’ve had little or no problem.

My point is that this shouldn’t happen at all. It’s a point of failure
introduced by Java, and a particularly difficult one to troubleshoot.

Do our users care if they are in a code ghetto? They just want their
modules to work. As they do now for the most part. The number of bug
reports is minimal compared to the number of games played on Vassal.

The impact on users isn’t direct, but it’s real. Less and less attention
goes in to Java on the desktop, so as time goes on, the user experience
will degrade relative to native apps. Java bugs have a real affect on our
users, not just on us developers.

The number of bug reports is always going to be small in comparison with
the number of occurances of bugs. The fact that we have bugs with 50+
reports tells me we have serious problems.

Hmm, to be a little cynical but perhaps realistics, I suspect someone
will be saying the same about your new design in five years time. A
design is only clean if you’ve thought everything out beforehand and
never extend it. In reality any program grows by accretion after its
initial release as users request thing never thought of which don’t
quite fit the design or do things with your product which the designers
never imagined - and with scripts the module developers will be able to
really go to town. Consider what has been done with traits which never
crossed anyone’s mind.

I don’t agree. When VASSAL was started, there wasn’t anything like it
out there. We have a much better idea of what’s needed now than at the
beginning. It might be the case that in the future, we’ll realize that
some of the design decisions in V4 were bad—but they won’t be the same
bad design decisions we’re saddled with now, and because I believe we’ve
learned from working on VASSAL to this point, I don’t believe they’ll
be worse. Moreover, it’s not just bad design decisions which I want to
correct, but also some pervasive bugs which we can be sure not to
reintroduce.

But I do agree with the major rewrite, just not with abandoning the Java
platform to do it.

Looking at the code even just quickly shows that it wasn’t written to be
multi-threaded and the layout of the packages etc is all wrong. So a
major rewrite is called for but this does not mean leaping into
something totally different.

A rewrite in Java only solves the problems in the codebase which are
not caused by Java itself. Rewriting in Java won’t solve the any of
the JDK bugs we suffer from. Rewriting in Java won’t let us get rid
of the 20k+ lines of code I’ve written in order to make it possible
to handle large images (none of which is necessary in C++). Rewriting
in Java won’t let us make the Module Manager and Player run in the
same process. Rewriting in Java won’t let us compile for something like
an iPad. Rewriting in Java won’t change the fact that Java frees the
programmer from doing memory management by forcing the user to fiddle
with heap settings. Rewriting in Java won’t add type punning for
arrays to the language, the lack of which forces us to do things like
make second copies of very large images when we need the data as a
byte array instead of as an int array.

I suspect the users would just like the new version to load existing
modules and save files without them having to re download new modules
(assuming you convert all those on the web site).

We could convert them on the site, but I was intending to provide a
converter that works from within V4 also.

I think you are making yourselves a lot of work going to C++ and I don’t
think you’ll reap as many benefits as you think especially for
portability.

The fact that we have a C++ demo which already works as a virtual
tabletop in under 2500 lines of code tells me that we won’t suffer from
the problems which have consumed our development time when working in
Java, and that we’ll end up with something far more maintainable.

I’m not claiming that portability will be easier than with Java, I just
doubt that it will be any harder. And it would have to be a lot harder
to eat up all of the other advantages I see.


J.

I just read over on ZunTzu forums that ZunTzu 2 development has now shifted from (the abandoned) Silverlight to C++ and OpenGL.

zuntzu.com/forum/viewtopic.php?t=1715

Ooh. Competition. ;)

  • M.

On 6 September 2011 09:28, bdgza bdegroot@gmail.com wrote:

I just read over on ZunTzu forums that ZunTzu 2 development has now
shifted from (the abandoned) Silverlight to C++ and OpenGL.

zuntzu.com/forum/**viewtopic.php?t=1715[1]http://www.zuntzu.com/forum/viewtopic.php?t=1715[1]

[1] zuntzu.com/forum/**viewtopic.php?t=1715http://www.zuntzu.com/forum/viewtopic.php?t=1715

Thus spake bdgza:

I just read over on ZunTzu forums that ZunTzu 2 development has now
shifted from (the abandoned) Silverlight to C++ and OpenGL.

zuntzu.com/forum/viewtopic.php?t=1715[1]

That’s very interesting. It makes me want to have another go at
convincing Jerome to join us…


J.

What it says to me is that C++ and OpenGL is probably the right choice.

ZunTzu doesn’t really use a GUI though. I’ve heard there are problems with
GLUT, though I’ve never used it. Do we need to GUI toolkit?

  • M.

That’s very interesting. It makes me want to have another go at
convincing Jerome to join us…

OK; we are planning to rewrite VASSAL in C++. Seriously, that has to be 2 (very optimistic) to 5+ (more realistic) year project. Is VASSAL 3 going to be frozen to new featues in that time?

Thus spake pgeerkens:

OK; we are planning to rewrite VASSAL in C++. Seriously, that has to be
2 (very optimistic) to 5+ (more realistic) year project.

Based on my experience with the demo, I think that we can have something
usable, but without many features, by the end of the year. Being able
to do everything that V3 does now will take longer, but I don’t expect
it will take as long as 2 years for that. Don’t underestimate the amount
of development time for V3 which has been eaten up by Java bugs and
workarounds. For me, I would say it’s been at least half of my dev time
since we started work on 3.1. That’s a huge drag on the project. We
should be able to move quite quickly once we’re not dealing with those
things.

Is VASSAL 3
going to be frozen to new featues in that time?

I’m divided on that right now. I don’t want to sink dev time into V3
which could be spent on V4 once 3.2 is released (though I do want to
continue with bug fixes as needed). I guess it depends on what the
features are. I’m definitely not planning to work on V3 myself once
V4 is really underway.


J.

Thus spake Michael Kiefte:

What it says to me is that C++ and OpenGL is probably the right choice.

ZunTzu doesn’t really use a GUI though. I’ve heard there are problems with
GLUT, though I’ve never used it. Do we need to GUI toolkit?

I think we do: I forsee needing dialogs, context menus, tabs/panes,
buttons, and scroll bars. I’m not particularly keen on writing any of
those things myself, and I think we’d be better off with a GUI toolkit
which makes them look right on the platforms we support.

ZunTzu takes a completely different approach with this—there’s a post
someplace in their forum where Jerome explains that he sees ZZ as a sort
of GUI experiment, which is why he devises his own widgets.


J.

See the 4th post.
zuntzu.com/forum/viewtopic.php?t=1553