SOS4Arena/Polyglot/Tablebases

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 11 April 2004 21:06:55:

I was having trouble using Polyglot with SOS4 for Arena. Prior to polyglot I had a similar problem with uci2wb. The SOS4 engine would often stay in ram after a game was over. I had to manually use the task manager to remove SOS4 from ram.
With polyglot it was happening on average about once every 4 games.
Now I hopefully have found the trigger for the problem. Sos4Arena occassionally has memory hangups when using Nalimov tablebases with polyglot. Remove the link to the Nalimov tablebases in the polyglot.ini file and sos4arena does not hang anymore (so far in my testing). I cannot supply an explanation.
Of course it would be nicer if polyglot+sos4 could use nalimovs without hanging, but right now I happy the hanging has stopped.

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 13 April 2004 10:30:34:
Als Antwort auf:/In reply to: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Norm Pollock at 11 April 2004 21:06:55:

I was having trouble using Polyglot with SOS4 for Arena. Prior to polyglot I had a similar problem with uci2wb. The SOS4 engine would often stay in ram after a game was over. I had to manually use the task manager to remove SOS4 from ram.
With polyglot it was happening on average about once every 4 games.
Now I hopefully have found the trigger for the problem. Sos4Arena occassionally has memory hangups when using Nalimov tablebases with polyglot. Remove the link to the Nalimov tablebases in the polyglot.ini file and sos4arena does not hang anymore (so far in my testing). I cannot supply an explanation.
Of course it would be nicer if polyglot+sos4 could use nalimovs without hanging, but right now I happy the hanging has stopped.

It might be related to initialisation time or something.
I can't do anything without log files, of course.
Not really satisfactory ...
Fabien.

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 13 April 2004 15:49:21:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Fabien Letouzey at 13 April 2004 10:30:34:

It might be related to initialisation time or something.
I can't do anything without log files, of course.
XBOARD 97 +0 2 386 Kh7 Rf7+ Kg8 Rf8+
XBOARD 98 +0 2 390 Kh7 Rf7+ Kg8 Rf8+
XBOARD 99 +0 2 394 Kh7 Rf7+ Kg8 Rf8+
XBOARD move g8h7
XBOARD 1/2-1/2 {Draw by repetition}
XBOARD 1/2-1/2 {Draw by repetition}

With nalimov tbs, SOS4 occasionally stays in memory after the game is over and keeps running at 99% of the CPU. It has to be manually removed from memory. Under these circumstances it does not produce a winboard.debug file.
Here is the bottom portion of polyglot.log in a game where sos4 hangs. sos4 is using nalimov tbs in this game.
< ENGINE info score cp 0
< ENGINE info currmove g8h7 currmovenumber 1
< ENGINE info score cp -1
< ENGINE info depth 97 seldepth 2
< ENGINE info depth 98 seldepth 2
< ENGINE info currmove g8h7 currmovenumber 1
< ENGINE info score cp 0 depth 98 seldepth 2 nodes 390 time 16 pv g8h7 f8f7 h7g8 f7f8
< ENGINE info score cp 0
< ENGINE info currmove g8h7 currmovenumber 1
< ENGINE info score cp -1
< ENGINE info depth 98 seldepth 2
< ENGINE info depth 99 seldepth 2
< ENGINE info currmove g8h7 currmovenumber 1
< ENGINE info score cp 0 depth 99 seldepth 2 nodes 394 time 16 pv g8h7 f8f7 h7g8 f7f8
< ENGINE info score cp 0
< ENGINE info currmove g8h7 currmovenumber 1
< ENGINE info score cp -1
< ENGINE info depth 99 seldepth 2
< ENGINE info cpuload 937
< ENGINE bestmove g8h7 ponder f8f7
POLYGLOT MOVE Kh7
POLYGLOT THINKING -> WAITING
< XBOARD result 1/2-1/2 {Draw by repetition}
POLYGLOT GAME END
POLYGLOT WAITING
< XBOARD force
POLYGLOT WAITING
< XBOARD ping 2

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 13 April 2004 16:43:20:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Norm Pollock at 13 April 2004 15:49:21:

With nalimov tbs, SOS4 occasionally stays in memory after the game is over and keeps running at 99% of the CPU. It has to be manually removed from memory. Under these circumstances it does not produce a winboard.debug file.
Here is the bottom portion of polyglot.log in a game where sos4 hangs. sos4 is using nalimov tbs in this game.
...
< ENGINE bestmove g8h7 ponder f8f7

XBOARD move g8h7
XBOARD 1/2-1/2 {Draw by repetition}
XBOARD 1/2-1/2 {Draw by repetition}

POLYGLOT MOVE Kh7
POLYGLOT THINKING -> WAITING
< XBOARD result 1/2-1/2 {Draw by repetition}
POLYGLOT GAME END
POLYGLOT WAITING
< XBOARD force
POLYGLOT WAITING
< XBOARD ping 2

Is there anything after the last line?
This is *very* important!
If there is nothing (or a new start), that means PolyGlot was killed before it had the time to close the engine properly. I have seen this several times (can only happen on Windows), and I have no idea what the source is.
I suspect this is the main reason for the "engine unloading" problem, although it can only happen if the engine has a bug (ignores end-of-file on input, I have written a post about this on CCC). Unfortunately many engine like Shredder and SOS seem to have this bug.
Why is there no Winboard log file?
It's the only place I could get a clue!
Unless the task manager can tell you how PolyGlot died?
Fabien.

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 13 April 2004 17:20:18:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Fabien Letouzey at 13 April 2004 16:43:20:

With nalimov tbs, SOS4 occasionally stays in memory after the game is over and keeps running at 99% of the CPU. It has to be manually removed from memory. Under these circumstances it does not produce a winboard.debug file.
Here is the bottom portion of polyglot.log in a game where sos4 hangs. sos4 is using nalimov tbs in this game.
...
< ENGINE bestmove g8h7 ponder f8f7

XBOARD move g8h7
XBOARD 1/2-1/2 {Draw by repetition}
XBOARD 1/2-1/2 {Draw by repetition}

POLYGLOT MOVE Kh7
POLYGLOT THINKING -> WAITING
< XBOARD result 1/2-1/2 {Draw by repetition}
POLYGLOT GAME END
POLYGLOT WAITING
< XBOARD force
POLYGLOT WAITING
< XBOARD ping 2

Is there anything after the last line?
This is *very* important!
If there is nothing (or a new start), that means PolyGlot was killed before it had the time to close the engine properly. I have seen this several times (can only happen on Windows), and I have no idea what the source is.
I suspect this is the main reason for the "engine unloading" problem, although it can only happen if the engine has a bug (ignores end-of-file on input, I have written a post about this on CCC). Unfortunately many engine like Shredder and SOS seem to have this bug.
Why is there no Winboard log file?
It's the only place I could get a clue!
Unless the task manager can tell you how PolyGlot died?
Fabien.

Nothing after the last line of polyglot.log.
After the game is over, Jori's tournament manager keeps saying that something is accessing the winboard.debug file while sos4 is still in memory. New games cannot start. Once sos4 is manually taken out of memory, there is no winboard.debug file for that game.
All I can suggest is that you recreate the scenario yourself and see what clues you can find. The bottom line is that sos4 exits well without nalimov tbs, so it is the releasing of control from nalimov tbs back to sos4 that is part of the exit problem.

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 13 April 2004 17:37:00:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Norm Pollock at 13 April 2004 17:20:18:

Nothing after the last line of polyglot.log.
After the game is over, Jori's tournament manager keeps saying that something is accessing the winboard.debug file while sos4 is still in memory. New games cannot start. Once sos4 is manually taken out of memory, there is no winboard.debug file for that game.
All I can suggest is that you recreate the scenario yourself and see what clues you can find. The bottom line is that sos4 exits well without nalimov tbs, so it is the releasing of control from nalimov tbs back to sos4 that is part of the exit problem.

Can you reproduce this problem with WinBoard but *not* the TM?
Something kills PolyGlot; it might not be WinBoard (it might also be PolyGlot crashing, bug the log file seems to indicate a murder) ...
I don't have Windows.
There is 0 problem with XBoard.
Also I couldn't find anything suspicious in the WinBoard source code ...
I suspect it is a Windows-specific multitasking thing, maybe a "message".
Just a guess though.
Fabien.

[Leo Dijksman]

Geschrieben von:/Posted by: Leo Dijksman at 13 April 2004 18:32:03:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Fabien Letouzey at 13 April 2004 17:37:00:

Nothing after the last line of polyglot.log.
After the game is over, Jori's tournament manager keeps saying that something is accessing the winboard.debug file while sos4 is still in memory. New games cannot start. Once sos4 is manually taken out of memory, there is no winboard.debug file for that game.
All I can suggest is that you recreate the scenario yourself and see what clues you can find. The bottom line is that sos4 exits well without nalimov tbs, so it is the releasing of control from nalimov tbs back to sos4 that is part of the exit problem.

Can you reproduce this problem with WinBoard but *not* the TM?
Something kills PolyGlot; it might not be WinBoard (it might also be PolyGlot crashing, bug the log file seems to indicate a murder) ...
I don't have Windows.
There is 0 problem with XBoard.
Also I couldn't find anything suspicious in the WinBoard source code ...
I suspect it is a Windows-specific multitasking thing, maybe a "message".
Just a guess though.
Fabien.

Thats because WBTM can not rename the winboard.debug file to g*.debug if a engine hangs, if you find a "hanging" engine then _first_ stop (by typing "stop" in the WBTM console followed by pressing 'enter') WBTM then kill the engine from memory and rename the winboard.debug file before start WBTM again.
Best wishes,
Leo.



WBEC Ridderkerk homepage.

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 13 April 2004 20:31:09:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Leo Dijksman at 13 April 2004 18:32:03:

Fabien,
Sos4/polyglot with nalimov tbs is working fine without the Jori WBTM. Maybe it is the renaming of the winboard.debug to g???.debug that the Jori WBTM does at the end of the game that is causing the problem with sos4/polyglot with nalimov tbs. Yet the WBTM works fine with sos4/polyglot without nalimov tbs. I can try a setting with the WBTM that says "set savedebugfiles never" and see if that helps.
In any case sos4/polyglot works fine with nalimov tbs when used directly with winboard, or used with a batch file contain -mg #. Also -xreuse/-xreuse2 does not affect anything.

Nothing after the last line of polyglot.log.
After the game is over, Jori's tournament manager keeps saying that something is accessing the winboard.debug file while sos4 is still in memory. New games cannot start. Once sos4 is manually taken out of memory, there is no winboard.debug file for that game.
All I can suggest is that you recreate the scenario yourself and see what clues you can find. The bottom line is that sos4 exits well without nalimov tbs, so it is the releasing of control from nalimov tbs back to sos4 that is part of the exit problem.

Can you reproduce this problem with WinBoard but *not* the TM?
Something kills PolyGlot; it might not be WinBoard (it might also be PolyGlot crashing, bug the log file seems to indicate a murder) ...
I don't have Windows.
There is 0 problem with XBoard.
Also I couldn't find anything suspicious in the WinBoard source code ...
I suspect it is a Windows-specific multitasking thing, maybe a "message".
Just a guess though.
Fabien.

Thats because WBTM can not rename the winboard.debug file to g*.debug if a engine hangs, if you find a "hanging" engine then _first_ stop (by typing "stop" in the WBTM console followed by pressing 'enter') WBTM then kill the engine from memory and rename the winboard.debug file before start WBTM again.
Best wishes,
Leo.

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 13 April 2004 20:51:40:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Leo Dijksman at 13 April 2004 18:32:03:

Doing what Leo suggested, I was able to get the winboard.debug after a hanging engine situation with sos4/polyglot using nalimov tbs with Jori wbtm.
I think the winboard.debug and polyglot.log look normal, yet the there was a hanging engine that had to be taken out of memory.
Here is the bottom of the winboard.debug. Following it is the bottom of the polyglot.log.
winboard.debug (bottom):
Interrupting second
92203 >second: time 939
otim 2096
92203 >second: h2h1
92328 first : time 2096
otim 926
92328 >first : usermove 92328 >first : f2f1
92407

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 14 April 2004 12:25:00:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Norm Pollock at 13 April 2004 20:31:09:

Sos4/polyglot with nalimov tbs is working fine without the Jori WBTM. Maybe it is the renaming of the winboard.debug to g???.debug that the Jori WBTM does at the end of the game that is causing the problem with sos4/polyglot with nalimov tbs. Yet the WBTM works fine with sos4/polyglot without nalimov tbs. I can try a setting with the WBTM that says "set savedebugfiles never" and see if that helps.
In any case sos4/polyglot works fine with nalimov tbs when used directly with winboard, or used with a batch file contain -mg #. Also -xreuse/-xreuse2 does not affect anything.

I don't think the renaming has anything to do with it.
But again, my Unix background makes me think differently.
I don't think you can find out what really happens unless you have good knowledge of Windows multitasking ...
Dann reported a similar problem with WinBoard + PolyGlot + Shredder.
So I don't think WBTM is responsible for it.
Something kills PolyGlot (or PolyGlot dies without any trace).
I think the problem is more general that just SOS with/without TBs ...
The problem is just more visible with SOS and Shredder because they have bad I/O behaviour.
But I suspect it's present much more often.
Fabien.

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 14 April 2004 12:30:17:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Norm Pollock at 13 April 2004 20:51:40:

Doing what Leo suggested, I was able to get the winboard.debug after a hanging engine situation with sos4/polyglot using nalimov tbs with Jori wbtm.
I think the winboard.debug and polyglot.log look normal, yet the there was a hanging engine that had to be taken out of memory.
Here is the bottom of the winboard.debug. Following it is the bottom of the polyglot.log.
winboard.debug (bottom):
...
92625 GameEnds(33, Draw by repetition, 1)
92625 >first : result 1/2-1/2 {Draw by repetition}
Interrupting second
92625 >second: result 1/2-1/2 {Draw by repetition}
92641 >first : force
92641 >first : ping 2
92641 >second: force
92641 >second: ping 2
Exiting: Match sos4 vs. Ruffian 1.0.5: final score 0-0-1
GameEnds(0, (null), 2)
92641 >first : quit
92641 >second: quit
polyglot.log (bottom):
...

XBOARD 1/2-1/2 {Draw by repetition}
XBOARD 1/2-1/2 {Draw by repetition}

POLYGLOT THINKING -> WAITING
< XBOARD result 1/2-1/2 {Draw by repetition}
POLYGLOT GAME END
POLYGLOT WAITING
< XBOARD force
POLYGLOT WAITING
< XBOARD ping 2

I can tell you that it is not normal at all, see below!
First the "Exiting" is a bit suspicious ...
Second, notice WinBoard sends "quit" to both engines after the pings (this is somewhat normal, but see below).
PolyGlot says it's received the "ping".
The very next thing it does is to answer with "pong", *always*.
But there's no "pong"!
No "quit".
Nothing ...
This can never happen with XBoard (Unix).
I should read the source code of WinBoard again, but I don't think the answer is there.
I suspect that's a Windows multitasking thing.
Have you had any similar problem with different software (e.g. UCI2WB)?
Fabien.

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 14 April 2004 12:48:35:
Als Antwort auf:/In reply to: Re: SOS4Arena/Polyglot/Tablebases geschrieben von:/posted by: Fabien Letouzey at 14 April 2004 12:30:17:

Have you had any similar problem with different software (e.g. UCI2WB)?

Sos4 with uci2wb was bad. Based on my experience with uci2wb, sos4 remained in memory whenever it lost, but otherwise left memory. On the other hand, polyglot only stays in memory on a random basis, about 1 in 4 times, and only when using both Jori wbtm and nalimov tbs. So clearly polyglot is a big improvement over uci2wb.