Opening Book Question

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 21 February 2004 18:32:05:

Why don't chess engines have 2 opening books: one for playing white and one for playing black?
Nalimov endgames have separate white and black files for each endgame material situation. Each "white" file is for playing as white, and each "black" file is for playing black. Yet this is not the case in the vast majority of opening books.
I notice that Aristarch has white/black books for the first few opening moves, then has a general opening book.
The problem as I see it, is that a game in the opening book that ends 1-0 may have good opening moves for white, will also have poor opening moves for black. If a chess engine relies on one book for its opening, this above mentioned game will be helpful when the engine is playing white, but harmful when playing black.
Suppose I have a 100000 game pgn file with 30000 games 1-0, 40000 games =, and 30000 games 0-1. Then 2 opening books could be obtained by filtering into a "white" pgn file with the 30000 games 1-0 and the 40000 games =. Likewise for a "black" pgn. From there 2 opening books could be created.
To me this seems such an obvious way to improve an engines performance.

[Marcus Prewarski]

Geschrieben von: / Posted by: Marcus Prewarski at 21 February 2004 18:40:21:
Als Antwort auf: / In reply to: Opening Book Question geschrieben von: / posted by: Norm Pollock at 21 February 2004 18:32:05:

Why don't chess engines have 2 opening books: one for playing white and one for playing black?
Nalimov endgames have separate white and black files for each endgame material situation. Each "white" file is for playing as white, and each "black" file is for playing black. Yet this is not the case in the vast majority of opening books.
I notice that Aristarch has white/black books for the first few opening moves, then has a general opening book.
The problem as I see it, is that a game in the opening book that ends 1-0 may have good opening moves for white, will also have poor opening moves for black. If a chess engine relies on one book for its opening, this above mentioned game will be helpful when the engine is playing white, but harmful when playing black.
Suppose I have a 100000 game pgn file with 30000 games 1-0, 40000 games =, and 30000 games 0-1. Then 2 opening books could be obtained by filtering into a "white" pgn file with the 30000 games 1-0 and the 40000 games =. Likewise for a "black" pgn. From there 2 opening books could be created.
To me this seems such an obvious way to improve an engines performance.

I think the logical way to handle this is when you are building your opening book only let the wins influence the positions. So if a game ends 1-0 only the white moves get added to the opening book and vice versa. I don't see any advantage to having two opening books aside from having one more file to manage.
-Marcus

[Will Singleton]

Geschrieben von: / Posted by: Will Singleton at 21 February 2004 19:06:45:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Marcus Prewarski at 21 February 2004 18:40:21:

Why don't chess engines have 2 opening books: one for playing white and one for playing black?
Nalimov endgames have separate white and black files for each endgame material situation. Each "white" file is for playing as white, and each "black" file is for playing black. Yet this is not the case in the vast majority of opening books.
I notice that Aristarch has white/black books for the first few opening moves, then has a general opening book.
The problem as I see it, is that a game in the opening book that ends 1-0 may have good opening moves for white, will also have poor opening moves for black. If a chess engine relies on one book for its opening, this above mentioned game will be helpful when the engine is playing white, but harmful when playing black.
Suppose I have a 100000 game pgn file with 30000 games 1-0, 40000 games =, and 30000 games 0-1. Then 2 opening books could be obtained by filtering into a "white" pgn file with the 30000 games 1-0 and the 40000 games =. Likewise for a "black" pgn. From there 2 opening books could be created.
To me this seems such an obvious way to improve an engines performance.

I think the logical way to handle this is when you are building your opening book only let the wins influence the positions. So if a game ends 1-0 only the white moves get added to the opening book and vice versa. I don't see any advantage to having two opening books aside from having one more file to manage.
-Marcus

One problem with adding book moves based on wins-only is that you might have a poor move that wins 3 times, but loses 20 times. If you only count the wins, you've got a problem.
Will

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 22 February 2004 12:56:49:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Will Singleton at 21 February 2004 19:06:45:

One problem with adding book moves based on wins-only is that you might have a poor move that wins 3 times, but loses 20 times. If you only count the wins, you've got a problem.
Will

Hello Will,
At the GM level, isn't that sort of occurrence unlikely?
Later.

[Thomas McBurney]

Geschrieben von: / Posted by: Thomas McBurney at 22 February 2004 14:21:08:
Als Antwort auf: / In reply to: Opening Book Question geschrieben von: / posted by: Norm Pollock at 21 February 2004 18:32:05:

Why don't chess engines have 2 opening books: one for playing white and one for playing black?
Nalimov endgames have separate white and black files for each endgame material situation. Each "white" file is for playing as white, and each "black" file is for playing black. Yet this is not the case in the vast majority of opening books.
I notice that Aristarch has white/black books for the first few opening moves, then has a general opening book.
The problem as I see it, is that a game in the opening book that ends 1-0 may have good opening moves for white, will also have poor opening moves for black. If a chess engine relies on one book for its opening, this above mentioned game will be helpful when the engine is playing white, but harmful when playing black.
Suppose I have a 100000 game pgn file with 30000 games 1-0, 40000 games =, and 30000 games 0-1. Then 2 opening books could be obtained by filtering into a "white" pgn file with the 30000 games 1-0 and the 40000 games =. Likewise for a "black" pgn. From there 2 opening books could be created.
To me this seems such an obvious way to improve an engines performance.

In my case, I have flag in my opening book file that indicates which side to move. Using this flag means I do not need separate files for for black and white. Each record in my opening book contains the following information, a flag to indicate if the record has been deleted or not, board position, side to move, move to make from that board position and a score which can be used to set a priority for different moves.


I have written a special tool for extracting moves from PGN files. The moves of the side that lost the game will not be added to the opening book, this effectively means it adds every second move to the book file. I don't need to store a sequence of moves because I store the board position instead. For games that end in a draw my tool will add the moves for both sides, but I can configure my PGN reading tool to ignore games that end in a draw and only add moves that lead to a win.


What I discovered is that the opening book is full of positions that the engine doesn't understand very well and often does not play as well as it could when out of book. It isn't good enough to store moves from grand masters that lead to a win, because the stupid engine when out of book will not have a clue what the plan is and start moving pieces around in an attempt to get a position it likes. To avoid this situation with my engine, I came up with a way to give higher priority to moves that my engine likes, which is basically open positions with good centre control. It now avoids gambits and flank openings. All this helps a little but in my experience it hasn't made a big difference.
Cheers,
Tom.

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 22 February 2004 14:35:57:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Roger Brown at 22 February 2004 12:56:49:

One problem with adding book moves based on wins-only is that you might have a poor move that wins 3 times, but loses 20 times. If you only count the wins, you've got a problem.
Will

Hello Will,
At the GM level, isn't that sort of occurrence unlikely?
Later.

Even if it's 3 wins and 6-10 losses, which isn't unlikely at any level, it's a problem.
A variation selection based on minimum number of games, minimum number of wins and a reasonable point percentage is probably the most obvious method. But it has a few weaknesses. My preference is black and white repetoire.
Regards,
Mogens

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 22 February 2004 17:22:54:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Thomas McBurney at 22 February 2004 14:21:08:

I have written a special tool for extracting moves from PGN files. The moves of the side that lost the game will not be added to the opening book, this effectively means it adds every second move to the book file. I don't need to store a sequence of moves because I store the board position instead. For games that end in a draw my tool will add the moves for both sides, but I can configure my PGN reading tool to ignore games that end in a draw and only add moves that lead to a win.

Suppose I have a 100000 game pgn file with 30000 games 1-0, 40000 games =, and 30000 games 0-1. Then 2 opening books could be obtained by filtering into a "white" pgn file with the 30000 games 1-0 and the 40000 games =. Likewise for a "black" pgn. From there 2 opening books could be created.
To me this seems such an obvious way to improve an engines performance.

What I discovered is that the opening book is full of positions that the engine doesn't understand very well and often does not play as well as it could when out of book. It isn't good enough to store moves from grand masters that lead to a win, because the stupid engine when out of book will not have a clue what the plan is and start moving pieces around in an attempt to get a position it likes. To avoid this situation with my engine, I came up with a way to give higher priority to moves that my engine likes, which is basically open positions with good centre control. It now avoids gambits and flank openings. All this helps a little but in my experience it hasn't made a big difference.

This sounds equivalent to what I suggested (separate books for white/black based upon wins&draws for white/black).
It could also be that the engine saw a fault in the opening and avoided the line the gm used. But if the engine is "stupid", deviation from gm lines could be the inspiration for making an improvement in the engine.

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 22 February 2004 18:53:39:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Mogens Larsen at 22 February 2004 14:35:57:

A variation selection based on minimum number of games, minimum number of wins and a reasonable point percentage is probably the most obvious method. But it has a few weaknesses. My preference is black and white repetoire.
Regards,
Mogens

Hello Mogens,
Same here. Unfortunately, few engines have this capacity. These are engines such as the Baron which have a black and white book or Yace which can generate the desired repetoire in one book but it seems that in any book from raw pgn that you take the good and the bad.
I had raised this issue some time ago. As a famous amateur book builder, do you have any ideas about how to create a base pgn file that only had Alekhine's opening moves for black and white? What happens is that in a normal book file some of the games will be played by Alekhine's opponent - which I do not want at all.
Is it possible to somehow strain the pgn to eliminate the opponent? It would not be pgn then, would it?
Any thoughts would be welcome.
Later.

[Will Singleton]

Geschrieben von: / Posted by: Will Singleton at 23 February 2004 00:47:01:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Roger Brown at 22 February 2004 18:53:39:

A variation selection based on minimum number of games, minimum number of wins and a reasonable point percentage is probably the most obvious method. But it has a few weaknesses. My preference is black and white repetoire.
Regards,
Mogens

Hello Mogens,
Same here. Unfortunately, few engines have this capacity. These are engines such as the Baron which have a black and white book or Yace which can generate the desired repetoire in one book but it seems that in any book from raw pgn that you take the good and the bad.
I had raised this issue some time ago. As a famous amateur book builder, do you have any ideas about how to create a base pgn file that only had Alekhine's opening moves for black and white? What happens is that in a normal book file some of the games will be played by Alekhine's opponent - which I do not want at all.
Is it possible to somehow strain the pgn to eliminate the opponent? It would not be pgn then, would it?
Any thoughts would be welcome.
Later.

Roger,
The possibilities are extensive, you can easily select any number of parameters from which to choose the moves. But I fail to see why separating the books by color makes any difference whatsoever. Any data in two books you can have in one.
Will

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 23 February 2004 04:28:29:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Will Singleton at 23 February 2004 00:47:01:

Roger,
The possibilities are extensive, you can easily select any number of parameters from which to choose the moves. But I fail to see why separating the books by color makes any difference whatsoever. Any data in two books you can have in one.
Will

Hello Will,
This is coming from a previous thread. At the risk of boring you, my desire is to create a base pgn file from which I could create a book - for any engine which supports the creation of a book from pgn obviously - which featured opening moves exclusively from one player for both black and white.
As my example I cite Alekhine, one of my favourite players all time. Now, when I download a file containing say, 10 of his games as white and 10 as black and use that file to create a book, I am going to get:
10 possible openings when Alekhine opened as white
10 possible games when the opponent played as white
10 possible games when the opponent played as black
10 possible games when Alekhine played as black
I do not want those games by the various opponents of Alekhine in my book.
With an engine that uses two books I can use Scid to create two pgn files - one where Alekhine played with the white pieces, the other when he played with the black ones. Then, when the engine is playing black, it will use the black book and select only moves by Alekhine. Likewise for white.
There were a number of authors who responded with suggestions that worked for their particular engines but if I had a way to create the desired input file - Alekhine.pgn - consisting of only Alekhine's moves as white and black then I would be thrilled. Most amateur engines take the pgn and chew it up whole. The most frequent parameters are win score, frequency played and ply depth.
I want more. Regardless of the above, I want engines that play like Alekhine in the opening. Or Capablanca. Or Reti. Etc.
I simply cannot see how most amateur engines are going to give me what I want in one book without the pgn source being fixed.
Should you have any helpful suggestions please offer them.
Do you now see my need for the two books? Or have I been as clear as mud?
Should it be the latter then please do not be distressed. It really is a minor problem in the grand scheme of things....
Later.

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 23 February 2004 22:19:15:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Roger Brown at 22 February 2004 18:53:39:

Hello Mogens,
Same here. Unfortunately, few engines have this capacity. These are engines such as the Baron which have a black and white book or Yace which can generate the desired repetoire in one book but it seems that in any book from raw pgn that you take the good and the bad.
I had raised this issue some time ago. As a famous amateur book builder, do you have any ideas about how to create a base pgn file that only had Alekhine's opening moves for black and white? What happens is that in a normal book file some of the games will be played by Alekhine's opponent - which I do not want at all.
Is it possible to somehow strain the pgn to eliminate the opponent? It would not be pgn then, would it?

Hey Roger,
Generally everything is mashed into stats.
Well, I assume that you want the engine to play Alekhine's moves even if he lost the game. That's the only way the engine can play the opponents moves, because most book builders take win/loss into account. Otherwise I'm misreading your intention.
The best idea I can think of is making a white and black pgn file with Alekhine wins as repetoire books. They can be used directly with some engines, eg. GLC, Yace and a few others. Probably also Aristarch if you can find a program to convert pgn to coordinate notation (e2e4 c7c5 g1f3 d7d6....) and create a white.boo and black.boo. I might even be tempted to try that. Then what about the other engines? Chessbase programs can merge game collections into a single game with variations. Export it as text if it's possible and type it out with exclamation marks en masse. It would take some work. That's the best I can do before my evening cup of coffee.
Regards,
Mogens

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 24 February 2004 06:50:27:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Mogens Larsen at 23 February 2004 22:19:15:

Well, I assume that you want the engine to play Alekhine's moves even if he lost the game.
Regards,
Mogens

Exactly!

The best idea I can think of is making a white and black pgn file with Alekhine wins as repetoire books. They can be used directly with some engines, eg. GLC, Yace and a few others. Probably also Aristarch if you can find a program to convert pgn to coordinate notation (e2e4 c7c5 g1f3 d7d6....) and create a white.boo and black.boo.

This seems to be where it is going in my opinion. Hence my binary books idea. Incidentally you left out Baron but that's O.K., I will not take it all personally!

I might even be tempted to try that. Then what about the other engines?

How could I possibly tempt you to attempt this task? Please let me know!
Chessbase programs can merge game collections into a single game with variations. Export it as text if it's possible and type it out with exclamation marks en masse. It would take some work.

Tons of work, just the thing that I would love automated!
That's the best I can do before my evening cup of coffee.

Which makes it twice as good as anything as I could come up with fully awake.
Thanks Mogens.
Later.

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 24 February 2004 13:09:26:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Roger Brown at 24 February 2004 06:50:27:

This seems to be where it is going in my opinion. Hence my binary books idea. Incidentally you left out Baron but that's O.K., I will not take it all personally!
How could I possibly tempt you to attempt this task? Please let me know!

:-)
I'll take a look during the weekend. My latest project is handtyping a opening repetoire for Aristarch in coordinate notation. If I can solve that task of reducing repetitive and arduous labour by something more automatic, I'll give it a try. I think I can use Extract to generate something useful by fiddling with its options. Maybe something like:
1) Reduce the gamelength to 30 plies and number of gametags to 1.
2) Change to coordinate notation.
3) Change remaining pgn tag to !.
4) Alekhine with white -> white.boo and Alekhine with black -> black.boo.
Just a theory so far.
I'm afraid that adapting Alekhine to other engines will require other means. My knowledge of Yace's capabilities are rather limited, which is rather embarassing as homepage hoster. Maybe it can be combined with the above to yield something useful in general.
Later .
Regards,
Mogens

[Roger Brown]

Geschrieben von: / Posted by: Roger Brown at 24 February 2004 13:31:48:
Als Antwort auf: / In reply to: Re: Opening Book Question geschrieben von: / posted by: Mogens Larsen at 24 February 2004 13:09:26:

I'll take a look during the weekend.
I'm afraid that adapting Alekhine to other engines will require other means.

I await the fruits of your labours!


You are afraid, I am terrified.

My knowledge of Yace's capabilities are rather limited, which is rather embarassing as homepage hoster. Maybe it can be combined with the above to yield something useful in general.

Dieter posted something for Yace and I had the good sense to copy it. Should you ever want it....

Thanks Mogens.

Later.