P2P text encoding bug

Technical Support & Bugs
1

As far as I know, this bug is limited to P2P. It’s an old bug, we’ve first encountered it last summer while using 3.1.17, in our only attempt with three player P2P game that was otherwise successful. However, the way our module handles character sheets and notes, the bug made it impossible for us to continue that game session back then, with main server down. (And that was when I started tinkering with private servers to get around main server failures and P2P issues.)

Now, we tried P2P with the new 3.2.5 release and this nasty bug is still there.

This is how a character sheet in our game properly looks like.

For the record, the character sheet is a Property Sheet with single multi-line text field and it updates with every keystroke. The bug affects text labels as well.

This is the same character sheet in a P2P game.

Some ASCII characters got messed up, but that’s not all. See those Polish letters that I added at the top of the sheet (ĄŹĆ…), in the very first line? This is what I see under Ubuntu, they display properly. The other player was on Windows though, and all he could see there were some “Ü” symbols. Those unknown symbol marks at the end of the first line are actually some Polish letters that he added there, and as you can see they didn’t display properly for me.

When we came across this bug last summer, there was also another Windows player present. They could both see their Polish text properly, my text displayed wrong for both of them, and the other way around. Consequently, it’s clearly related to us using different operating systems.

So, seems like special character encoding goes nuts when Windows sends text to Linux or the other way around.

This is an example of the bug affecting text labels.

Look for two white labels starting with “[T2/O1/W1]” tags, just above the blue “Gustav Stark” label. The left one is mine and displays properly. The one to the left is a copy of the same label pasted on Windows. Notice how “ą” in “gorąco” displays as unknown symbol.

I do realize this bug is so damn specific it’s likely not going to come up in any other group’s game. Our games tend to be text-heavy though, and we need those special characters for formatting and, well, because our primary language uses lots of those.

Any ideas how to fix this?

2

Hi Filip,

It is going to be an encoding issue with the non-latin characters, I’m sure we will be able to fix it.

Can you please confirm that Windows to Windows p2p and Linux to Linux p2p works ok, it is just Windows to Linux (and vice versa) that is the problem?

It would help a lot if you could create a mini module that demonstrates this. ie. No map, 1 counter, 1 text label that shows the problem.

Thanks,
Brent.

3

Indeed.

Windows to Windows works ok, unless something changed since last summer.

I never had an opportunity to observe how Linux to Linux p2p works. I believe it should work ok, since Windows to Windows worked fine, but I can’t be completely sure about that.

4

No, that does not follow if the Linux client is the problem.

Do you have access to a Linux machine? You can start two clients on it at the same time listening on different ports and connect them to each other and test this.

5

In fact yes, I do have two other linux machines at home. However, all share the same router for internet connection, so I expect having the same IP will make it impossible to connect properly. In the past, I tried testing my private server fix that way, and it didn’t work, though perhaps P2P is different. I’ll give it a go tomorrow anyway, if time permits.

I hope this is enough.

I pasted some test characters in the label.

Property Sheet is an empty notepad, as I never needed to have any text there at start and consequently I wasn’t sure how to do it. Just open the sheet, resize the window if it’s too small, and paste the text from the label there.

(The 3.1.17 hack that I’m still normally using has some minor changes that make those Property Sheet notepads easier to use for us, but that’s besides the point, as the bug in question appears in 3.2.5 as described above. I wanted to move all those little changes to a more current Vassal version, but the hack also made us independent from the main server, hence my interest in P2P or private servers working smoothly. Sometimes I wish I could do some actual programming rather than just minor hacks on my own, so I didn’t have to bother you with this stuff and perhaps contribute on my own - but I guess it would take me years to gain the necessary proficiency.)

6

While the two machines will share the same external IP address as seen by machines on the other side of your router, they will have separate IP’s on the LAN on your side of the router. For example, My IP address (i.e. the external address of my router) may be 123.456.654.321, but on my LAN, my two machines have IP’s of 192.160.0.1 and 192.168.0.2. I can connect the two machines via Vassal p2p using these IP numbers.

Vassal tells you what your Local (LAN) address is in the P2P server configuration box.

Thanks, I will look into it.

7

This actually worked, I managed to connect with my other linux machine. Note however that internal IP displayed by Vassal was wrong (Vassal displays 127.0.1.1 for both machines, while in order to connect locally I had to use 192.168.0.2).

Also, I uploaded corrected test module, as I forgot to add label edit shortcut in the first version.

Either way, Linux to Linux seems to work fine. It displays exactly what it should on both machines.

So, seems like a Windows to Linux issue, unless there are some other factors causing the bug. For this test, I used machines with different versions of Ubuntu. There’s still a possibility the bug could occur between, say Ubuntu and Fedora or between Win 7 and Win XP.

8

Ok, Thanks. Must be a problem with the way the local IP is found on Linux.

Could you please post what the label is supposed to look like? Under windows, I get a series of accented characters and then some shapes, I;m not sure if that is correct.

9

ABC ĄąŹźĆć ███●►

I hope it displays correctly through the forum. ABC, then AaZzCc with accents (polish characters), then three black bars (or a continuous black bar, depending on letter spacing), black dot and black triangle (we use those as layout elements).

Anyway.

I’ve just tested it Windows 7 to Linux, and what I got were random characters in place of everything but ABC.

Next, I tested Windows 7 to Windows XP. ABC and Polish characters displayed properly, but shapes displayed as question marks. So the bug is not limited to Windows to Linux, it’s a more general bug that may occur with different operating systems or configurations.

I have no idea what’s the reason this does not happen with standard server connections (or with working private server hacks), just with P2P.

10

Oh, and local IP does indeed display correctly under Windows.

11

I believe I have found the problem. The p2p open source package we use does not specify a Charset to use, so it uses the default charset on each system which may not support an extended character set. The standard client forces the use of UTF-8, so I have changed the p2p client to do the same.

It is a lot of trouble for me to set up a test environment for this, but I am pretty sure this should fix the problem. I have committed the change to svn 8705. Can you please test after Joel makes a new Dev build.

I have opened a separate bug

Bug 10170 - Incorrect LAN address reported under Linux

for the LAN address issue.

12

Ok, I have sorted out the LAN address problem in svn 8706.

It’s an ‘issue’ in the way in which Linux reports the local IP address of the machine. Machines have multiple IP addresses. When you ask for the local IP address, Java under Windows assumes you want to know what your local LAN address. Java under Linux just gives you the first IP address it comes across, which is usually not what you are after.

I found some code that looks through all the IP address your machine has and returns the one most likely to be the local LAN address. It works fine on Windows and should resolve the problem on Linux.

13

Thus spake Brent Easton:

I believe I have found the problem. The p2p open source package we use
does not specify a Charset to use, so it uses the default charset on
each system which may not support an extended character set. The
standard client forces the use of UTF-8, so I have changed the p2p
client to do the same.

It is a lot of trouble for me to set up a test environment for this, but
I am pretty sure this should fix the problem. I have committed the
change to svn 8705. Can you please test after Joel makes a new Dev
build.

Merged to trunk@8708. Try 3.2.6-svn8708:

vassalengine.sourceforge.net/builds/

–
J.

14

Thus spake Brent Easton:

Ok, I have sorted out the LAN address problem in svn 8706.

It’s an ‘issue’ in the way in which Linux reports the local IP address
of the machine. Machines have multiple IP addresses. When you ask for
the local IP address, Java under Windows assumes you want to know what
your local LAN address. Java under Linux just gives you the first IP
address it comes across, which is usually not what you are after.

I found some code that looks through all the IP address your machine has
and returns the one most likely to be the local LAN address. It works
fine on Windows and should resolve the problem on Linux.

Merged to trunk@8709… before I noticed that this doesn’t compile. Did
you forget to check in a class called PrivateClientFactory?

–
J.

15

Oops, my bad. Left some Work In Progress in. Clean version committed to 8711

16

Thus spake Brent Easton:

Oops, my bad. Left some Work In Progress in. Clean version committed to
8711

Thanks, Brent. Merged to trunk@8712. New build uploading now.

vassalengine.sourceforge.net/builds/

–
J.

17

I tested Vassal 3.2.6 today (Windows 98 to Linux and Windows 7 to Linux), and the encoding bug seems fixed. Thanks.

Also, I can now see the proper IP under Ubuntu.

18

That is good to hear, thanks Filip