Crafty Book Test revisited

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 07 January 2004 20:35:11:

Crafty Generated Book Test
==========================
I have used my recently posted Blitz rating list (posted in Winboard forum, http://f11.parsimony.net/forum16635/messages/58889.htm) as 'infrastructure' for a test aiming to measure how generated books (as opposed to hand-tuned books) impact the playing strength of a chess engine. I have already done a similar test in the past (posted in both CCC and Winboard forum, see http://f11.parsimony.net/forum16635/messages/41900.htm), which suggested that the impact on playing strength is quite small (possibly < 30 ELO points, though results were statistically not significant). This previous test was a self-play experiment at a longer time control, while this new test involves matches against a wide range of different engines at Blitz time controls.

Method:
=======
Crafty 19.6DC has been configured with three different book settings (large book, small book, no book). These three Crafty versions have played against each other (20 games per match) and Gauntlet tournaments (also 20 games per match) against a large and identical set of engines that are listed in my rating list. All duplicate games have been removed and the remaining unique games have been used to calculate a final rating list, hence taking also into account the games between the opponents.
Apart from the opening book, the configuration settings of the three Crafty versions were identical. Book learning has been disabled, all three Crafty configurations have played without EGTBs.

Participants and Settings:
==========================
Crafty v19.06DCntb: Crafty 19.6DC with book created with parameters 50 1 50 from cleanbook.pgn taken from Dann Corbit´s ftp site (size 28.3MB)
Crafty v19.06DCntbsbk: Crafty 19.6DC with book created with parameters 20 3 50 from gm2600.pgn taken from the Crafty ftp site (size 564KB)
Crafty v19.06DCntbnbk: Crafty 19.6DC with no book.
The two configurations with book have played with the standard books.bin file built from the start.pgn file as downloaded from the Crafty ftp site.
crafty.rc (example for Crafty v19.06DCntb):
hash 96M
hashp 8M
drawscore=0
log off
book on
book random 1
book width 5
learn 0
resign 9
no EGTBs for the Crafties; opponents may have used a full set of 5, 4 and 3 piece EGTBs with 8MB TB cache
Time Control: 5min + 2sec
Ponder off
Book leaning off

Platform and Tools:
===================
Athlon Thunderbird 1.1MHz
512 MB RAM
Windows 2000
Crafty 19.6DC (from Dann Corbit´s ftp site)
Elostat 1.1b
PGN-Extract 15.0
Winboard 4.2.3
WB Tourney Manager 0.60 (Jori Ostrovskij)

Results:
========


&#80;&#114;&#111;&#103;&#114;&#97;&#109;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#32;&#69;&#108;&#111;&nbsp;&nbsp;&nbsp;&nbsp;&#43;&nbsp;&nbsp;&#32;&#45;&nbsp;&nbsp;&#32;&#71;&#97;&#109;&#101;&#115;&nbsp;&nbsp;&#32;&#83;&#99;&#111;&#114;&#101;&nbsp;&nbsp;&#32;&#65;&#118;&#46;&#79;&#112;&#46;&nbsp;&nbsp;&#68;&#114;&#97;&#119;&#115;
&#67;&#114;&#97;&#102;&#116;&#121;&#32;&#118;&#49;&#57;&#46;&#48;&#54;&#68;&#67;&#110;&#116;&#98;&nbsp;&nbsp;&nbsp;&nbsp;&#58;&#32;&#50;&#53;&#56;&#48;&nbsp;&nbsp;&#32;&#50;&#49;&nbsp;&nbsp;&#50;&#51;&nbsp;&nbsp;&#32;&#55;&#48;&#50;&nbsp;&nbsp;&nbsp;&nbsp;&#54;&#51;&#46;&#48;&#32;&#37;&nbsp;&nbsp;&#32;&#50;&#52;&#56;&#55;&nbsp;&nbsp;&#32;&#51;&#49;&#46;&#53;&#32;&#37;
&#67;&#114;&#97;&#102;&#116;&#121;&#32;&#118;&#49;&#57;&#46;&#48;&#54;&#68;&#67;&#110;&#116;&#98;&#110;&#98;&#107;&#32;&#58;&#32;&#50;&#53;&#54;&#50;&nbsp;&nbsp;&#32;&#50;&#50;&nbsp;&nbsp;&#50;&#51;&nbsp;&nbsp;&#32;&#55;&#48;&#56;&nbsp;&nbsp;&nbsp;&nbsp;&#54;&#48;&#46;&#55;&#32;&#37;&nbsp;&nbsp;&#32;&#50;&#52;&#56;&#55;&nbsp;&nbsp;&#32;&#50;&#56;&#46;&#52;&#32;&#37;
&#67;&#114;&#97;&#102;&#116;&#121;&#32;&#118;&#49;&#57;&#46;&#48;&#54;&#68;&#67;&#110;&#116;&#98;&#115;&#98;&#107;&#32;&#58;&#32;&#50;&#53;&#52;&#48;&nbsp;&nbsp;&#32;&#50;&#51;&nbsp;&nbsp;&#50;&#48;&nbsp;&nbsp;&#32;&#55;&#49;&#52;&nbsp;&nbsp;&nbsp;&nbsp;&#53;&#55;&#46;&#52;&#32;&#37;&nbsp;&nbsp;&#32;&#50;&#52;&#56;&#56;&nbsp;&nbsp;&#32;&#51;&#51;&#46;&#54;&#32;&#37;



Conclusion:
===========
Even with these quite small error margins of close to 20 the differences in playing strength due to different book configurations are still statistically not significant. Nevertheless, these results confirm the results of the previous self-play book test: Both tests suggest that the impact of size and presence of generated books on playing strength seems to be quite small, probably in the order of 30 to 40 ELO points or even below.
This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions.
Robert Allgeuer



YABRL

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 07 January 2004 21:48:30:
Als Antwort auf: / In reply to: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 07 January 2004 20:35:11:

This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions.
Robert Allgeuer

Hello Robert,
Interesting test. I have been trying to bestir myself to have a look at the impact of timecontrols on playing strength but I am sooooo busy with resting...
I have a couple of suggestions:
(a) Somewhere on Dann Corbit's ftp is a Crafty tournament book. That may be a place to start.
(b) Dann has a pgn of GM games - 2700 and 2600 - on his site. I would imagine that a book, narrowed down to to four or five classical opening lines - e4, d4 and the two knights - c3 and f3 - with perhaps a fianchetto opening thrown in for good measure would be an improvement. GM games are handy ECO summaries for most of us.
(c) Toss out the flank a3, a4 etc. stuff. Arena allows you to remove opening lines as you see fit. Simply check to see what the GMs played and fashion the book accordingly.

(d) You could use a commercial gui to obtain use of a hand tuned book. Chesspartner and Fritz come to mind.

I hope this helped.

Later.

[Volker Pittlik]

Geschrieben von: / Posted by: Volker Pittlik at 07 January 2004 22:30:08:
Als Antwort auf: / In reply to: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 07 January 2004 20:35:11:

...

This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions.

Interesting test IMHO! It's my impression that the importance of books is overestimated. Just make sure that there are not too much blunders in the PGN for the book and the thing should be usefull.
Best regards
Volker

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 07 January 2004 22:52:52:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Volker Pittlik at 07 January 2004 22:30:08:

...

This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions.

Interesting test IMHO! It's my impression that the importance of books is overestimated.

yes, I have the same impression.
Robert

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 07 January 2004 23:01:44:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Roger Brown at 07 January 2004 21:48:30:

This result appears to be in line with results that other people have obtained. Hand-tuned books may, however, be a completely different story. If there is a high quality hand-tuned book for Crafty somewhere out there, I would be happy to extend this test and to test such a configuration under the same conditions.
Robert Allgeuer

Hello Robert,
Interesting test. I have been trying to bestir myself to have a look at the impact of timecontrols on playing strength but I am sooooo busy with resting...
I have a couple of suggestions:
(a) Somewhere on Dann Corbit's ftp is a Crafty tournament book. That may be a place to start.
(b) Dann has a pgn of GM games - 2700 and 2600 - on his site. I would imagine that a book, narrowed down to to four or five classical opening lines - e4, d4 and the two knights - c3 and f3 - with perhaps a fianchetto opening thrown in for good measure would be an improvement. GM games are handy ECO summaries for most of us.
(c) Toss out the flank a3, a4 etc. stuff. Arena allows you to remove opening lines as you see fit. Simply check to see what the GMs played and fashion the book accordingly.

(d) You could use a commercial gui to obtain use of a hand tuned book. Chesspartner and Fritz come to mind.

I hope this helped.

Later.

In fact as I ran with books built from the standard start.pgn Crafty does not choose unreasonable openings in my configurations. So thinking about it, what could be tested is to remove books altogether and see what happens when Crafty is also allowed to choose other less classical opening lines, or alternatively as said to test a top-of-the-line book. Too bad that I do not possess Fritz. I do possess Rebel, but unless I am not completely mistaken Rebel´s book is in Rebel format and not accessible by the chesspartner GUI.
Robert

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 07 January 2004 23:45:11:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 07 January 2004 23:01:44:

Too bad that I do not possess Fritz. I do possess Rebel, but unless I am not completely mistaken Rebel´s book is in Rebel format and not accessible by the chesspartner GUI.

Robert

Hello Robert,
Which Rebel are you talking about? Unless I am mistaken - I am not at my chess machine at home just now - the Chesspartner gui will use the Rebel.mvs book.
Of course, the Chesspartner Rebel - not the DOS one - comes with books in the NN.bk format which is the CP book format.

Later.

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 08 January 2004 00:03:55:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Roger Brown at 07 January 2004 23:45:11:

Too bad that I do not possess Fritz. I do possess Rebel, but unless I am not completely mistaken Rebel´s book is in Rebel format and not accessible by the chesspartner GUI.

Robert

Hello Robert,
Which Rebel are you talking about? Unless I am mistaken - I am not at my chess machine at home just now - the Chesspartner gui will use the Rebel.mvs book.
Of course, the Chesspartner Rebel - not the DOS one - comes with books in the NN.bk format which is the CP book format.

Later.

I have no decent book in *.bk format (only small.bk). The rebel book is under engines\rebel\books\rebel.mvs. It is the latest Rebel 12 with Chesspartner 5.3
Robert

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 08 January 2004 04:09:31:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 08 January 2004 00:03:55:

I have no decent book in *.bk format (only small.bk). The rebel book is under engines\rebel\books\rebel.mvs. It is the latest Rebel 12 with Chesspartner 5.3
Robert

Tsk, tsk Robert,
Might I suggest that you visit the Lokasoft website?
I think if you check their free downloads section you might be pleasantly surprised. Those should suit your purposes just fine.


Later.

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 January 2004 04:49:03:
Als Antwort auf: / In reply to: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 07 January 2004 20:35:11:

this may be slightly off topic.
I concatenated a large pgn file which I then thinned by removing duplicates (as much as possible)using the Scid database. The final result is a pgn file of 97M. The makeup of this file is from quality pgn files on the Internet including cleanbook.pgn, book.pgn, 2500-2550-2600-2650-2700-2750 pgns from the Dann Corbitt site, gm2600.pgn from the Bob Hyatt site, gm2001.pgn from the Arena site, plus game collections of many individual chess superstars both past and present.
I used to have games from correspondence chess, but I eliminated them because my testing showed that they were not justified.
From this pgn file, I make up books for ruffian and crafty and whatever other engine that allows me to make up the book. I like the setting of 30 plys, depth = 3.
My testing shows that these books improve the performance of the engines. If anyone is interested in testing my books, I could make them available for downloading.

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 08 January 2004 11:14:28:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Norm Pollock at 08 January 2004 04:49:03:

this may be slightly off topic.
I concatenated a large pgn file which I then thinned by removing duplicates (as much as possible)using the Scid database. The final result is a pgn file of 97M. The makeup of this file is from quality pgn files on the Internet including cleanbook.pgn, book.pgn, 2500-2550-2600-2650-2700-2750 pgns from the Dann Corbitt site, gm2600.pgn from the Bob Hyatt site, gm2001.pgn from the Arena site, plus game collections of many individual chess superstars both past and present.
I used to have games from correspondence chess, but I eliminated them because my testing showed that they were not justified.
From this pgn file, I make up books for ruffian and crafty and whatever other engine that allows me to make up the book. I like the setting of 30 plys, depth = 3.
My testing shows that these books improve the performance of the engines. If anyone is interested in testing my books, I could make them available for downloading.

Currently I have something else running, but possibly in February.
Did you use your book in combination with the file books (built from start.pgn) which filters out probably quite a few of the opening lines contained in the 97MB file?
How much do you think was the increase in playing strength?
I think an interesting aspect would also be what impact this books file has on playing strength (that is more selective versus less selective book move selection)
Robert

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 08 January 2004 11:15:28:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Roger Brown at 08 January 2004 04:09:31:

I have no decent book in *.bk format (only small.bk). The rebel book is under engines\rebel\books\rebel.mvs. It is the latest Rebel 12 with Chesspartner 5.3
Robert

Tsk, tsk Robert,
Might I suggest that you visit the Lokasoft website?
I think if you check their free downloads section you might be pleasantly surprised. Those should suit your purposes just fine.


Later.

Thanks, will check
Robert

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 January 2004 15:55:36:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Robert Allgeuer at 08 January 2004 11:14:28:

Currently I have something else running, but possibly in February.
Did you use your book in combination with the file books (built from start.pgn) which filters out probably quite a few of the opening lines contained in the 97MB file?
How much do you think was the increase in playing strength?
I think an interesting aspect would also be what impact this books file has on playing strength (that is more selective versus less selective book move selection)
Robert

I built the crafty book with the command "book create 30 3". Does that automatically utilize the start.pgn file to filter?
I don't know how many elo points it improves the engine. Can you suggest a utility I could use to measure this. My testing of ruffian 1.05 v ruffian 1.05 at 1'+1" using different books, over hundreds of games, indicated to me a definite and consistant improvement due to the book.
My 97M pgn file is not directly filtered. Some of the books in the concatenation were filtered. My file covers all the openings A00-E99. When I made the books for ruffian and crafty, my only filter was that the move occurred in a minimun of 3 games (with duplicate games removed). And I limited the openings to 30 plies. Rare openings and gambits were virtually eliminated.
Some of the files in the concatenation were filtered by their authors.
My feeling is that too much filtering will eliminate 10 good moves for each bad move. So in the long run, over hundreds of games, I'm better off with less filtering. I also think that the only reliable filter is that 3 or more GMs 2500+ used the move.

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 08 January 2004 18:10:53:
Als Antwort auf: / In reply to: Re: Crafty Book Test revisited geschrieben von: / posted by: Norm Pollock at 08 January 2004 15:55:36:

Crafty has another book file called books which can be put in the Crafty directory and is built with 'books create start.pgn 50' or similar. start.pgn (I use the standard one from the crafty ftp site) has its own specific syntax and allows to discourage or filter out specific variations, even when those lines are contained in the main book file. I do not have start.pgn here at the moment, but I think the standard one casues Crafty not to choose essentially all gambits and limits openings to e4, d4 and I think f4. It allows to customise which moves contained in the book Crafty will play or not play, so it is a bit like book tuning. I have never tried removing books but I assume Crafty will then choose any line contained in the book file.
Robert