Slow Server Status Window.

RK, is there any chance I could take a look at:

  • The script/s which return the server status list.
  • A few lines from the relevant MySql table/s.

I really want to try and speed it up.

Thus spake “bsmith”:

Related, but from the VASSAL end: The request to the server should be done
in the background with a SwingWorker so as not to block the GUI.


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

From what Rodney has told me, I think the script is parsing the entire dataset each time someone hits “Refresh”, so if I can move all of the work into a single DB query, it should really fly; So even if we don’t thread, it should hopefully be quick enough for users not to care!

Sure. Thanks for taking a look at this. The php script is attached. The table data is just what you would expect, e.g.

Ark of the Covenant eisenraum teisenga 1199942286000 Combat Commander Main Room newbie 1200752833000 Warhammer 40k Module Apocalypse? whodis 1200308854000 Russian Front Fred-Thierry Fred 1200114900000 Russian Front Fred-Thierry 1200114533000 Miniatures Skirmish Game Main Room ElComandante 1201863138000

The table has about 25,000 rows, and no index.

rk

Post generated using Mail2Forum (mail2forum.com)

I think that script just grabs the 20,000 rows and echoes them Rodney, does it normally pass this information to some other application?

For the “Current” window of Server Status, the query to the DB should only be returning a cursor containing the ~30 rows needed to output the running games info to the user.

VASSAL calls that PHP page, then parses the output and displays the info, so the processing is all in Java code. The “Current” status is a different PHP page, which actually just reads a flat file and doesn’t query the DB at all.

rk

Post generated using Mail2Forum (mail2forum.com)

Ah ok, that’s the part that I would really like to fix as it’s the most frequently refreshed, and painfully slow. How many rows in that text file? It must be doing something quite intensive.

Nothing intensive. Script is just:

<?php $connections = fopen("connections/connectionStatus","r"); fpassthru($connections); fclose($connections); ?>

and the data file is just 89 lines long when I last checked.

rk

Post generated using Mail2Forum (mail2forum.com)

If it’s as simple as that, I’m baffled as to why it is so slow Rodney; Unless it puts a read lock on the file, but that wouldn’t explain why the delay is so consistently long. It’s not as if it’s fast on some days and slower on others.

Thus spake “bsmith”:

Is it possible that the slow part is in VASSAL itself?


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

Yeah, could be. May be a little tricky to figure out without setting up some sort of test environment. This may have to go into the too-hard basket for the time being.

I noticed it takes around 9 seconds for me to do a refresh on the website.

Thus spake “bsmith”:

I’ll eventually check this in a profiler if no one else does, but it might
be a while… (All mail like this goes into my VASSAL todo folder so it
doesn’t get lost.)


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

Thus spake “bsmith”:

Hey Ben, I ran this through the profiler a minute ago and found the
following: Almost all of the time you spend waiting is for
CgiServerStatus.retrieveHistory() to finish. Within that method, 56%
of the time is spent in calls to java.io.Reader.read() and 31% is
consumed by StringTokenizer.nextToken(). What I can’t tell yet is
why Reader.read() takes so long—some of it could be due to reading
from reading from an HTTP connection. Also, depending on what
StringTokenizer is doing, it might be possible to replace it with
something faster… still investigating.


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

Thus spake Joel Uckelman:

This becomes about one second faster if we use BufferedReader.readLine()
to get our lines. Using StringTokenizer to split lines is suboptimal.

However, that doesn’t address the main problem, which is that we’re
reading a ton of data from the HTTP connection. A counter I added
tells me that we’re making two connections to the server. The first
one reads 1760 bytes in 45 lines, and the second one reads a whacking
huge 1364480 bytes in 24760 lines.

I have my doubts about whether the Server Status dialog needs to read
1.3MB from the server. The only way to make this faster is to read less
data (or maybe to compress it before sending it).


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

Thus spake Joel Uckelman:

The first, small request is the one which populates the “Current” tab.
The second, huge request populates the other tabs. The getStatus()
method calls retrieveHistory() after getting the data for the Current
tab. retrieveHistory() is called anyway if you bring up one of the
historical tabs and it hasn’t been called yet, so there’s no purpose
whatsoever to fetch that huge chunk of data unless the user tries to
display it.

I’ve checked a fix for this into uckelman-working@3163.

Ben: Give this a try:

nomic.net/~uckelman/tmp/vass … indows.exe

Rodney: Would you take a look at this change before I merge it to the
trunk? I want to be sure this is ok, and I’m unfamiliar with the chat
part of the source tree.


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

That’s true. I think the intended behavior was to only call retrieveHistory after the user clicked on one of the tabs, but an up-front call may have sneaked in. I’ll take a look at your code change.

rk

Post generated using Mail2Forum (mail2forum.com)

Wow, I can’t believe how fast it is! Thanks Joel, fixed. This opens up a ton of new possibilities.

It would be nice, instead of opening a module then seeing only the games in progress for that module, to display the entire server status in that pane instead, then you could go in… chat… load a module and sync to any game on the list. So it would eliminate the need to have downloaded the module; You could go straight into the “Main” area of each game and just chat.

Very nice.

Thus spake “bsmith”:

I can believe how fast it is, because it’s reading 1000x less data now.
While I’m waiting for Rodney to finish the LibraryWindow stuff, I’m
going to further modify this to have the data fetched in the background,
so that a slow internet connection doesn’t block the GUI.

I’ll think about this when I’m doing the Flexdock stuff for 3.2.


J.


Messages mailing list
Messages@forums.vassalengine.org
forums.vassalengine.org/mailman/ … engine.org

Post generated using Mail2Forum (mail2forum.com)

I’ve now fixed this properly:

  • CgiServerStatus now keeps track of the time intervals it’s requested
    and the results it received, so that it will never request results it
    already has. Reading the requests from the server is the most time-
    consuming operation here, so this gets the user results quicker and saves
    a ton of bandwidth. The way it was before, we did a complete 1-month
    request each time a historical tab was raised; now we request just what we
    need, so raising all three tabs is equivalent to one 1-month request
    instead of three. (For example, if the user brings up all three historical
    tabs in 3.0.17 right now, approximately 3.9MB is sent by the server. With
    this change, only 1.3MB is sent.)

  • Requests happen in the background so as not to block the GUI.

  • If the user changes tabs in the middle of a request, that doesn’t screw
    things up.

  • The modules are now listed alphabetically. (I’m shocked that no one
    complained about displaying them in “HashMap” order before!)

svn3224:

nomic.net/~uckelman/tmp/vassal/

Rodney: If you want to further reduce the bandwidth impact, you could have
the CGI script compress the data before sending. It would be trivial to
handle this on the client end. Whether it would be worth it depends on the
relative value of CPU cycles and bandwidth on the server side. It looks
like the server history will compress about 4:1 with gzip, so you could
be sending about 400k instead of 1.3MB if you thought it was worth the
effort.