possible problem with automating book for movei

[Uri Blass]

Geschrieben von: / Posted by: Uri Blass at 09 March 2004 14:54:27:

Mogens Larsen suggested a way to automate a book for movei
see http://www.talkchess.com/forums/1/message.html?353467
I see based on looking at my code that there may
be a problem if movei have more than 50000 lines.
The problem is that if the first 50000 lines of movei
start in the same opening(for example petrov) then movei will never play different opening with black even if later there are lines with other openings
(movei may play also e4 c6 with black with the default option but it is only because movei does not read it directly from book in the default option and it will never play 1.e4 e5 2.Nf3 Nc6 even if Nc6 is in book but not in the first 50,000 lines of 1.e4 e5 2.Nf3).
This problem can be solved by random order of lines so the chance that the first 50,000 lines start the same way is close to 0 but even in this case rare move that does not appear in the first 50,000 cases that the line was played will never be played.

I may consider releasing a new version without that limitation when the only problem is that movei may be slow in finding the book moves with big book even in the first moves.
Note that Movei does not do efficient search for moves in book and search them line after line.
It is better to have book of positions and for every position to have list of moves with probabilities for every move when later I may do binary search for positions but I need some function to translate book of pgn to book of positions and I did not look at source code of other programs to read how they do it.
Uri

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 09 March 2004 17:27:16:
Als Antwort auf: / In reply to: possible problem with automating book for movei geschrieben von: / posted by: Uri Blass at 09 March 2004 14:54:27:

Mogens Larsen suggested a way to automate a book for movei
see http://www.talkchess.com/forums/1/message.html?353467
I see based on looking at my code that there may
be a problem if movei have more than 50000 lines.
The problem is that if the first 50000 lines of movei
start in the same opening(for example petrov) then movei will never play different opening with black even if later there are lines with other openings
(movei may play also e4 c6 with black with the default option but it is only because movei does not read it directly from book in the default option and it will never play 1.e4 e5 2.Nf3 Nc6 even if Nc6 is in book but not in the first 50,000 lines of 1.e4 e5 2.Nf3).

I've only added about 500-550 (24ply) lines to the existing books for Movei. Nothing revolutionary, just a test. The 50000 lines limit shouldn't be a problem unless you use a large game collection sorted by opening and disregard the chances for including blunders.
The first two games by Movei were lost, but probably not due directly to bad openings.
[Event "KOT2: Qualification (RR)"]
[Site "Dual PIII-866MHZ/384MB"]
[Date "2004.03.09"]
[Round "1"]
[White "Postmodernist v.1010"]
[Black "Movei 00.8.178"]
[Result "1-0"]
[TimeControl "3600"]
[PairNo ""]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8.
c3 O-O 9. h3 Na5 10. Bc2 c5 11. d4 Nc6 12. d5 Na5 13. b3 Nb7 14. Bb2 Bd7
15. Nbd2 Rc8 16. a4 Na5 17. axb5 axb5 18. Qe2 Nh5 19. g3 g6 20. Ra2 c4 21.
b4 Nb7 22. Ra7 Qc7 23. Kh2 Ra8 24. Rea1 Rxa7 25. Rxa7 f5 26. exf5 gxf5 27.
Nd4 Qb6 28. Ra1 exd4 29. Qxe7 d3 30. Kg2 Nf6 31. Bd1 Rf7 32. Qe1 Nxd5 33.
Bf3 Nc7 34. Bh5 Rg7 35. Nf3 Nd5 36. Bc1 Re7 37. Qd2 f4 38. Nd4 Nd8 39. Bf3
Bc6 40. Nxc6 Qxc6 41. Qa2 Kf7 42. gxf4 Ne6 43. f5 Nec7 44. Qd2 Qd7 45. Qg5
Nf6 46. Kh2 d5 47. Bf4 Nce8 48. Be3 Qc6 49. Bc5 Ng7 50. Bxe7 Qc7+ 51. Kg2
Qxe7 52. Ra6 d4 53. cxd4 c3 54. Rb6 d2 55. Rc6 Ne4 56. Qxe7+ Kxe7 57. f6+
Nxf6 58. Rxc3 Ne6 59. Bd1 Nf4+ 60. Kh2 N4d5 61. Rd3 Ne4 62. Bf3 Nec3 63.
Kg3 Kd6 64. Rxd2 Nxb4 65. Kf4 Na6 66. Kg5 Nc7 67. Kh6 Ne6 68. h4 Nb1 69.
Rb2 Nxd4 70. Rxb1 Nxf3 71. Kxh7 Nxh4 72. Rxb5 Ke6 73. Rb4 Nf3 74. Rf4 Nd2
75. Kg7 Kd5 76. Rf8 Nf1 77. f4 Ng3 78. f5 Kd6 79. Kg6 Kc7 80. Re8 Kd7 81.
Re1 Kd6 82. f6
{Movei adjudication} 1-0
Movei was out of book after 11. d4.
[Event "KOT2: Qualification (RR)"]
[Site "Dual PIII-866MHZ/384MB"]
[Date "2004.03.09"]
[Round "2"]
[White "Movei 00.8.178"]
[Black "WildCat 3.0"]
[Result "0-1"]
[TimeControl "3600"]
[PairNo ""]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 g6 6. Be3 Bg7 7. f3 O-O 8.
Qd2 Nc6 9. O-O-O Bd7 10. Nxc6 bxc6 11. Bc4 Qa5 12. Kb1 Rfb8 13. Bb3 Rb7 14.
Bh6 Bxh6 15. Qxh6 Rab8 16. g4 c5 17. e5 dxe5 18. g5 Nh5 19. Rd5 Qb6 20.
Rxe5 c4 21. Nd5 Qd6 22. Nxe7+ Kh8 23. Rhe1 cxb3 24. axb3 Rb5 25. R5e4 Rd8
26. b4 a5 27. c3 axb4 28. Rd4 Qc7 29. Rde4 Qb6 30. Nc6 Rg8 31. Nxb4 Bf5 32.
Kc1 Bxe4 33. Rxe4 Qg1+ 34. Kc2 Rd8 35. Rd4 Rbb8 36. Rxd8+ Rxd8
{Movei adjudication} 0-1
Movei was out of book after 9... Bd7.
Regards,
Mogens

[Uri Blass]

Geschrieben von: / Posted by: Uri Blass at 09 March 2004 23:37:57:
Als Antwort auf: / In reply to: Re: possible problem with automating book for movei geschrieben von: / posted by: Mogens Larsen at 09 March 2004 17:27:16:

Mogens Larsen suggested a way to automate a book for movei
see http://www.talkchess.com/forums/1/message.html?353467
I see based on looking at my code that there may
be a problem if movei have more than 50000 lines.
The problem is that if the first 50000 lines of movei
start in the same opening(for example petrov) then movei will never play different opening with black even if later there are lines with other openings
(movei may play also e4 c6 with black with the default option but it is only because movei does not read it directly from book in the default option and it will never play 1.e4 e5 2.Nf3 Nc6 even if Nc6 is in book but not in the first 50,000 lines of 1.e4 e5 2.Nf3).

I've only added about 500-550 (24ply) lines to the existing books for Movei. Nothing revolutionary, just a test. The 50000 lines limit shouldn't be a problem unless you use a large game collection sorted by opening and disregard the chances for including blunders.
The first two games by Movei were lost, but probably not due directly to bad openings.
[Event "KOT2: Qualification (RR)"]
[Site "Dual PIII-866MHZ/384MB"]
[Date "2004.03.09"]
[Round "1"]
[White "Postmodernist v.1010"]
[Black "Movei 00.8.178"]
[Result "1-0"]
[TimeControl "3600"]
[PairNo ""]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8.
c3 O-O 9. h3 Na5 10. Bc2 c5 11. d4 Nc6 12. d5 Na5 13. b3 Nb7 14. Bb2 Bd7
15. Nbd2 Rc8 16. a4 Na5 17. axb5 axb5 18. Qe2 Nh5 19. g3 g6 20. Ra2 c4 21.
b4 Nb7 22. Ra7 Qc7 23. Kh2 Ra8 24. Rea1 Rxa7 25. Rxa7 f5 26. exf5 gxf5 27.
Nd4 Qb6 28. Ra1 exd4 29. Qxe7 d3 30. Kg2 Nf6 31. Bd1 Rf7 32. Qe1 Nxd5 33.
Bf3 Nc7 34. Bh5 Rg7 35. Nf3 Nd5 36. Bc1 Re7 37. Qd2 f4 38. Nd4 Nd8 39. Bf3
Bc6 40. Nxc6 Qxc6 41. Qa2 Kf7 42. gxf4 Ne6 43. f5 Nec7 44. Qd2 Qd7 45. Qg5
Nf6 46. Kh2 d5 47. Bf4 Nce8 48. Be3 Qc6 49. Bc5 Ng7 50. Bxe7 Qc7+ 51. Kg2
Qxe7 52. Ra6 d4 53. cxd4 c3 54. Rb6 d2 55. Rc6 Ne4 56. Qxe7+ Kxe7 57. f6+
Nxf6 58. Rxc3 Ne6 59. Bd1 Nf4+ 60. Kh2 N4d5 61. Rd3 Ne4 62. Bf3 Nec3 63.
Kg3 Kd6 64. Rxd2 Nxb4 65. Kf4 Na6 66. Kg5 Nc7 67. Kh6 Ne6 68. h4 Nb1 69.
Rb2 Nxd4 70. Rxb1 Nxf3 71. Kxh7 Nxh4 72. Rxb5 Ke6 73. Rb4 Nf3 74. Rf4 Nd2
75. Kg7 Kd5 76. Rf8 Nf1 77. f4 Ng3 78. f5 Kd6 79. Kg6 Kc7 80. Re8 Kd7 81.
Re1 Kd6 82. f6
{Movei adjudication} 1-0
Movei was out of book after 11. d4.
[Event "KOT2: Qualification (RR)"]
[Site "Dual PIII-866MHZ/384MB"]
[Date "2004.03.09"]
[Round "2"]
[White "Movei 00.8.178"]
[Black "WildCat 3.0"]
[Result "0-1"]
[TimeControl "3600"]
[PairNo ""]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 g6 6. Be3 Bg7 7. f3 O-O 8.
Qd2 Nc6 9. O-O-O Bd7 10. Nxc6 bxc6 11. Bc4 Qa5 12. Kb1 Rfb8 13. Bb3 Rb7 14.
Bh6 Bxh6 15. Qxh6 Rab8 16. g4 c5 17. e5 dxe5 18. g5 Nh5 19. Rd5 Qb6 20.
Rxe5 c4 21. Nd5 Qd6 22. Nxe7+ Kh8 23. Rhe1 cxb3 24. axb3 Rb5 25. R5e4 Rd8
26. b4 a5 27. c3 axb4 28. Rd4 Qc7 29. Rde4 Qb6 30. Nc6 Rg8 31. Nxb4 Bf5 32.
Kc1 Bxe4 33. Rxe4 Qg1+ 34. Kc2 Rd8 35. Rd4 Rbb8 36. Rxd8+ Rxd8
{Movei adjudication} 0-1
Movei was out of book after 9... Bd7.
Regards,
Mogens

I think that at least in the second game movei blundered in the first move out of book.
10.Nxc6 does not lose immediately but it is a bad move.
It seems that 16.g4 is the losing error of movei.
before that move wildcat evaluates the position as equal or almost equal.

Movei did not understand that the queen is only a spectator that cannot move after g5 Nh5
I can comment about the first game that movei never plays 2...Nc6 with the original book.
I do not know but maybe it is a bad choice for it.
It seems to me that movei got the advantage in the middle game but blundered.
It seems that it blundered by 38...Nd8 instead of 38...Nc5 and it seem not easy for programs to see it.
Postmodernist v.1010 - Movei 00.8.178
6k1/1n1br2p/1q1p4/1p1n3B/1PpN1p2/2Pp2PP/3Q1PK1/R1B5 b - - 0 1
Analysis by Fritz 8:
38...Re5!
² (0.37) Depth: 6/21 00:00:00 103kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.Kxg3 Qc7 42.Kh2 Nb6 43.Ra6 d5
= (0.19) Depth: 7/25 00:00:00 178kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.Kxg3 Qc7 42.Kh2 Nb6 43.Ra6 d5
= (0.19) Depth: 8/27 00:00:01 317kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.Kxg3 Qc7 42.Kh2 Nb6 43.Ra6 d5
= (0.19) Depth: 9/29 00:00:03 717kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.fxg3 Ne3+ 42.Kh1 Re8 43.g4 d5 44.Bxd5+ Nxd5 45.Qg5+
= (0.22) Depth: 10/31 00:00:07 1845kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.fxg3 Ne3+ 42.Kg1 Bxh3
= (0.22) Depth: 11/33 00:00:17 4787kN
38...Re5--
² (0.50) Depth: 12/35 00:00:35 10578kN
38...Re5 39.Ra8+ Nd8 40.Bf3 fxg3 41.Qh6 d2 42.Bxd2 gxf2 43.Qh4 Ne7 44.Bf4
² (0.59) Depth: 12/52 00:00:53 16616kN
38...Nd8!
² (0.56) Depth: 12/52 00:01:09 22186kN
38...Nd8!
² (0.41) Depth: 12/52 00:01:14 24077kN
38...Nd8 39.Ra8 Re4 40.Bf3 Rxd4 41.cxd4 Bc6 42.Rc8 Bb7 43.Qxd3 cxd3
= (0.22) Depth: 12/52 00:01:46 34310kN
38...Nd8 39.Ra8 Re4 40.Bf3 Rxd4 41.cxd4 Bc6 42.Rc8 Bb7 43.Qxd3 cxd3
= (0.22) Depth: 13/38 00:03:03 61322kN
38...Nc5!
= (0.19) Depth: 13/39 00:06:10 128958kN
(Blass, Tel-Aviv 10.03.2004)

38...Nc5 39.bxc5 dxc5 40.Bf3 Ne3+ seems an interesting line.

Unfortunately I spend too much time on analyzing and finding these lines instead of working about my program.
Uri