Crafty book test, extended

[Robert Allgeuer]

Geschrieben von:/Posted by: Robert Allgeuer at 30 March 2004 23:41:08:

I have run a test to quantify further the influence of the opening book on playing strength. This is an extension of the test that I have posted previously (see http://f11.parsimony.net/forum16635/messages/60135.htm and also http://f11.parsimony.net/forum16635/messages/41900.htm for test at longer time controls). I have used Crafty 19.6 (as compiled by Dann Corbit) for the test; specifically I have also looked at, whether the use of bookc and books files (which essentially make Crafty´s choice of the book move more selective) has a measurable effect on playing strength.

Method:
=======
Crafty 19.6DC has been configured with five different book settings (combinations of large book, small book, no book; books and bookc present, only books or neither bookc and books). These five Crafty configurations have played against each other (20 games per match) and have performed Gauntlet tournaments (also 20 games per match) against a large and identical set of engines that are listed in my rating list. All duplicate games have been removed and the remaining unique games have been used to calculate a final rating list.
Apart from the opening book settings, the configurations of the five Crafty versions were identical. Book learning has been disabled; all three Crafty configurations have played without EGTBs.

Participants and Settings:
==========================
Crafty v19.06DCntbbookc: Crafty 19.6DC with book created with parameters 50 1 50 from cleanbook.pgn taken from Dann Corbit´s ftp site (size 28.3MB), both bookc and books files present
Crafty v19.06DCntb: as above, but only books file present, no bookc file
Crafty v19.06DCntbnobookcs: as above but neither bookc nor books file present
Crafty v19.06DCntbsbk: Crafty 19.6DC with book created with parameters 20 3 50 from gm2600.pgn taken from the Crafty ftp site (size 564KB), only books present
Crafty v19.06DCntbnbk: Crafty 19.6DC with no book at all
The files books.bin and bookc.bin are taken from the Crafty ftp site.
crafty.rc (example for Crafty v19.06DCntb):
hash 96M
hashp 8M
drawscore=0
log off
book on
book random 1
book width 5
learn 0
resign 9
no EGTBs for the Crafties; opponents may have used a full set of 5, 4 and 3 piece EGTBs with 8MB TB cache
Time Control: 5min + 2sec
Ponder off
Book leaning off

Platform and Tools:
===================
Athlon Thunderbird 1.1GHz
512 MB RAM
Windows 2000
Crafty 19.6DC (from Dann Corbit´s ftp site)
Elostat 1.1b
PGN-Extract 15.0
Winboard 4.2.3
WB Tourney Manager 0.60 (Jori Ostrovskij)

Results:
========


  Program                     Elo    +   -   Games   Score   Av.Op.  Draws
  Crafty v19.06DCntbbookc   : 2592   20  19   825    61.6 %   2510   36.4 %
  Crafty v19.06DCntb        : 2580   21  20   818    59.7 %   2512   32.8 %
  Crafty v19.06DCntbnobookcs: 2575   21  19   810    59.3 %   2510   34.1 %
  Crafty v19.06DCntbnbk     : 2568   21  20   824    58.1 %   2511   31.1 %
  Crafty v19.06DCntbsbk     : 2548   22  18   834    55.1 %   2512   35.1 %


Conclusion:
===========
The influence of bookc and books is small and even after more than 800 games for each configuration statistically not significant. Still, the test results suggest that there may be indeed an increase in playing strength due to the use of a more selective book move selection that favours "engine-friendly" positions, although this increase appears to be quite small.
Even the result of Crafty using no book at all is only a little bit worse than the top. Furthermore, when looking at the error margins, even the lowest scoring configuration with the small book based on gm2600.pgn, could in reality be very close to the top.
These results confirm the results of my previous book tests and also other tests (e.g. Lousy Lines Tournament of Volker Pittlik): the impact of books (i.e. their presence, their size and the applied move selection strategy) on playing strength seems to be relativly small, probably not more than some 40 ELO points or so. And if an engine has to play without a book altogether, this does not appear to be a huge penalty, either.
Robert Allgeuer