Building Opening Books

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 31 March 2004 17:07:58:

I have been fooling around building opening books for the past few months. I have been putting together pgn files and building books from them for engines that allow book building. Like ruffian and crafty for example.
I don't think the size of the book is that important. What is important are the quality of the openings in the book, and enough variety to stay in book for a reasonable number of moves against all types of quality openings by the opponent. Especially important is the avoidance of opening mistakes that another engine can exploit.
At this point I have a pretty good pgn file which I would like to make publicly available if I can find someplace to host it. This book consists of 76K games from 1997 to the present (17M zipped). I used Scid to put together the pgns from many sources, especially from "This Week in Chess" and "Chess Collect". I focused on recent games exclusively because modern players have the advantage of computers to analyze openings.
I included many filters:
(1) All games were played from 1997 to the present.
(2) I limited the games to players of 2400+ elo. Users could restrict this even more using scid, but that reduces the number of games in the pgn. I used scid to insert elo ratings where possible. Games without elo ratings for a player were removed.
(3) I have filtered out 99.99% of the twin games after using scid's player name corrections. I cannot be sure of 100% because sometimes games are recorded with different spellings of the players' names.
(4) I have filtered out short games. Drawn games to 20 moves and won games to 30 moves were removed. These games I felt were more likely to have opening mistakes.
(5) I have filtered out games with a player name missing, and games with just a partial player name.
(6) Also filtered out were blindfold games, blitz games, rapid games, ICCF games, IECC games, Internet games, Email games, Simultaneous games, Fischer Random games, computer-computer games, computer-human games, correspondence games and playchess games.
(7) Also filtered out were pgns that cause "trouble" with the engines. Particularly pgns with FEN, with "..." and with comments. My pgn works well with scid, ruffian and crafty without error messages or hangups.
(8) The games were NOT individually checked for quality of play. Also my filtering with scid is never likely to be 100% effective, but it is still very effective.
(9) Excess tags were removed. This does not affect the opening book but greatly reduces the size of the pgn file.
Other notes:
(1) I recommend that users use 5 plies when creating a book (# of times a move has occurred in the pgn) because repetition is the best assurance that an opening does not have a mistake.
(2) I recommend that programmers create engines that enable the engines to use a "white book" and a "black book" separately, instead of one book for both sides. By eliminating white wins from a "black book" for example, black's opening would not be influenced by pgns in which black lost. That would reduce the likelihood of black making an opening mistake.
(3) Book learning by an engine is also very important.
So if anyone has a place for this 17M file, let me know.

[Koundinya Veluri]

Geschrieben von:/Posted by: Koundinya Veluri at 31 March 2004 18:57:27:
Als Antwort auf:/In reply to: Building Opening Books geschrieben von:/posted by: Norm Pollock at 31 March 2004 17:07:58:

Looks interesting. I can put it on my site if you like. Send me an e-mail.
Koundinya

[Norm Pollock]

Geschrieben von:/Posted by: Norm Pollock at 31 March 2004 19:51:22:
Als Antwort auf:/In reply to: Re: Building Opening Books geschrieben von:/posted by: Koundinya Veluri at 31 March 2004 18:57:27:

Looks interesting. I can put it on my site if you like. Send me an e-mail.
Koundinya

Thanks for your offer. Anyone who wishes to host this file please e-mail me at npollock@hoflink.com . I want to choose the most widely available site.