First impression of Ruffian 1.0.5

[capgadget]

Geschrieben von: / Posted by: capgadget at 07 November 2003 23:08:11:

I assume everyone except me had already snagged a copy prior to the "Official Release" of Areana 0.95 . I gave up writting chess programs in the TRS-80 Model I days M$ Microchess 1.0 and Sargon where way beyond anything I could write using 16K of mem or less
I am assuming this was not the Lieden version. I also had heard that a Linux version would be released, but I guess it won't see the light of day.
The only big thing I see is the reduction of the size of the opening book by the final pass eliminating rarely played moves.
I created 2 opening books from a pgn file containing ~254,000 2300+ rated games that I have collected over the past couple of years and was exported from a SCID database.
Original pgn file:
178,914,963 bytes (pgnall_20031101a.pgn)
Using the following command to create an opening book with 60 plies (Actually run
from cygwin):
1.0.1 Version
ruffian -D 60 -I d:pgnall_20031101a.pgn -B e:/ruffian101.2300.20031101a.book
Resultant book file:
266,728,740 bytes (ruffian101.2300.20031101a.book)
-------------
1.0.5 Version
ruffian_105 -D 60 -I d:pgnall_20031101a.pgn -B e:/ruffian105.2300.20031101a.book
Resultant book file:
35,373,828 bytes (ruffian105.2300.20031101a.book)

I played 1.0.1 against 1.0.5 overnight on a 800 Mhz PIII which I run as ruffiantestb on freechess.org with the following ruffian.cfg files (Note that some try KEEP_HASH 2. I don't know if it really does anything)


Ruffian 1.0.1
RESIGN 900
KEEP_HASH 1
HASH 32M
EGTB_HASH_SIZE 4M
LOGFNAME "f:/ruffianlogs/ruffian_test.log"
BOOK "e:/ruffian_book/ruffian101.2300.20031103.book"
TB_PATH "c:/tb345"

Ruffian 1.0.5
RESIGN 900
KEEP_HASH 1
HASH 32M
EGTB_HASH_SIZE 4M
LOGFNAME "f:/ruffianlogs/ruffian_105.log"
BOOK "e:/ruffian_book/ruffian105.2300.20031103.book"
TB_PATH "c:/tb345"

The tablebase files are all 3-4 table bases and some 5. Ruffian doesn't seem to want to use the 5 man table bases unless you have a certain number of them. I don't recall the threshhold. I have watched the file opens via TaskInfo www.iarsn.com to see what is actually being opened. I just keep mostly the Pawn and Rook ones and what ever else is needed to avoid underpromotion since I really don't have the 7GB to spare much less those Expensive Humongous Mass Storage Device (EHMSD) drives that Dr. Hyatt must have to store the 6 man tables at the moment until a long overdue hardware upgrade. Crafty et al seems to "wuv" what few 5 man egtb it can get.

I let the two play in the 2300+ mudpile overnight using Winboard. I really should have used a smaller 2600 book for to get a better comparison with so few games played, plus this will probably take a few hundred games. Just for grins at 5 min 3s time controls. I personally blame the people associated with Arena 0.95 for causing me not to go to bed early last night but I didn't check the message board till late:
Computer chess game
R1F6J2, 2003.11.07
Score 123456789012345678901234567890123
-----------------------------------------------------------------
1: Ruffian 1.0.1 19.0 / 33 =1100===1=1====1===101=======011=
2: Ruffian 1.0.5 14.0 / 33 =0011===0=0====0===010=======100=
-----------------------------------------------------------------
33 games: +7 =20 -6

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 November 2003 03:34:08:
Als Antwort auf: / In reply to: First impression of Ruffian 1.0.5 geschrieben von: / posted by: capgadget at 07 November 2003 23:08:11:

Original pgn file:
178,914,963 bytes (pgnall_20031101a.pgn)
Using the following command to create an opening book with 60 plies (Actually run
from cygwin):
1.0.1 Version
ruffian -D 60 -I d:pgnall_20031101a.pgn -B e:/ruffian101.2300.20031101a.book
Resultant book file:
266,728,740 bytes (ruffian101.2300.20031101a.book)
-------------
1.0.5 Version
ruffian_105 -D 60 -I d:pgnall_20031101a.pgn -B e:/ruffian105.2300.20031101a.book
Resultant book file:
35,373,828 bytes (ruffian105.2300.20031101a.book)

I played 1.0.1 against 1.0.5 overnight on a 800 Mhz PIII which I run as ruffiantestb on freechess.org with the following ruffian.cfg files (Note that some try KEEP_HASH 2. I don't know if it really does anything)
Ruffian 1.0.1
RESIGN 900
KEEP_HASH 1
HASH 32M
EGTB_HASH_SIZE 4M
LOGFNAME "f:/ruffianlogs/ruffian_test.log"
BOOK "e:/ruffian_book/ruffian101.2300.20031103.book"
TB_PATH "c:/tb345"

Ruffian 1.0.5
RESIGN 900
KEEP_HASH 1
HASH 32M
EGTB_HASH_SIZE 4M
LOGFNAME "f:/ruffianlogs/ruffian_105.log"
BOOK "e:/ruffian_book/ruffian105.2300.20031103.book"
TB_PATH "c:/tb345"

I created an opening book for Ruffian 1.05 from cleanbook.pgn.
I used the following command:
ruffian -D 60 -I cleanbook.pgn -B cleanbook.book
It game me an 11M opening book, larger than the 1.8M book that came with Ruffian.
I'm running 50 games (2'1") between Ruffian 1.01 with the original book aginst Ruffian 1.05 with cleanbook.book

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 November 2003 07:57:05:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Norm Pollock at 08 November 2003 03:34:08:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

[Dann Corbit]

Geschrieben von: / Posted by: Dann Corbit at 08 November 2003 09:57:02:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Norm Pollock at 08 November 2003 07:57:05:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).




my ftp site {remove http:// unless you like error messages}

[Uri Blass]

Geschrieben von: / Posted by: Uri Blass at 08 November 2003 10:23:36:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Dann Corbit at 08 November 2003 09:57:02:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

Another problem is that being superior against itself does not mean being superior against other programs.
Uri

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 08 November 2003 15:15:43:
Als Antwort auf: / In reply to: First impression of Ruffian 1.0.5 geschrieben von: / posted by: capgadget at 07 November 2003 23:08:11:

It is possible to reduce the size of a selfmade Ruffian book. The available book options are:
-B FILE Use FILE as opening book
-I PGN-file Build opening book from PGN-file
-D DEPTH Cut book lines at DEPTH
-P MINPLAYED Add moves played more then n-times
-o Disable opening book
Adding a P (>= 3) value should make it easier to use large PGN files.
Regards,
Mogens

[Uri Blass]

Geschrieben von: / Posted by: Uri Blass at 08 November 2003 15:29:44:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Uri Blass at 08 November 2003 10:23:36:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

Another problem is that being superior against itself does not mean being superior against other programs.
Uri

I also forgot about the time control.
Big book is probably better for blitz when the engine cannot find superior moves relative to moves of masters so the book save time.
small book is probably better for longer time control when the engine probably can find superior moves to moves of players with rating of 2300.
Uri

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 November 2003 17:28:29:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Dann Corbit at 08 November 2003 09:57:02:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

I just noticed that the ruffian book (ruffian.bok) has different versions even though internally it says "Sept 29, 2002 59116 book moves". I have 3 different "ruffian.bok" because of 3 different downloads of ruffian 1.01. They have comparison mismatches even though file size is the same.
In my next experiment I will be using the version of ruffian.bok that I downloaded today. I will use it with Ruffian 1.05 (which does not come with a book). The opponent will be Ruffian 1.05 with cleanbook.bok, which I created yesterday from cleanbook.pgn.
I have an AMD2000, 732M machine using 128M hashtables per engine. The match will be 50 games.

[Mogens Larsen]

Geschrieben von: / Posted by: Mogens Larsen at 08 November 2003 18:26:07:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Norm Pollock at 08 November 2003 17:28:29:

I just noticed that the ruffian book (ruffian.bok) has different versions even though internally it says "Sept 29, 2002 59116 book moves". I have 3 different "ruffian.bok" because of 3 different downloads of ruffian 1.01. They have comparison mismatches even though file size is the same.

The date discrepancy is due to learning, which modifies the book.
Regards,
Mogens

[capgadget]

Geschrieben von: / Posted by: capgadget at 08 November 2003 19:16:16:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Mogens Larsen at 08 November 2003 15:15:43:

It is possible to reduce the size of a selfmade Ruffian book. The available book options are:
-B FILE Use FILE as opening book
-I PGN-file Build opening book from PGN-file
-D DEPTH Cut book lines at DEPTH
-P MINPLAYED Add moves played more then n-times
-o Disable opening book
Adding a P (>= 3) value should make it easier to use large PGN files.
Regards,
Mogens

Kewl, I hadn't dug into the new flags yet.
One thing I have noticed about Ruffian is that it does not add moves as
part of the learning heuristics. My assuption is because the book size
does not change, especially when creating a dummy book containing only
1 move like e4. It seems to benifit with overkill that is using something
like a 2300+ book rather than a 2600+ book. I've playing hundreds if not
thousands of games between books that were extracted from various sources
i.e. everything onthe corbit site, the large book from Dr. Hyatt's crafty site,
the SCID books, the TWIC books, games played on freechess.org
Tuning the book weights by playing hundreds of games is EXTREMELY important
with ruffian. When I have added new "strong" engines with their own books like Aristarch 4.21 LG, Gandalf. It is a struggle for Ruffian until it get's it's
book tuned from the new tricks up the sleaves of it's newly found opponents,
then it starts to slowly pull ahead. Building a new book for Ruffian then playing 4-5 games for a comparison is essentially meaningless. It takes hundreds if not thousands of games. This is probably true with most engines with book learning.
When I create a new book I usually make 2 copies then make 2 ruffians and set them up to play each other. Usually starting with 1m 1s time controls, playing several hundred or a few thousand games, then slowly ramping up the time controls as time permits. i.e. 2m 1s, 3m 1s, 5m 3s, 10m 2s, 15m 3s, 15m 5s, maybe 2m 15s. Of course this takes weeks mostly played at night. Well, Seti@home did get boring after a few years of not finding any LGM

[Norm Pollock]

Geschrieben von: / Posted by: Norm Pollock at 08 November 2003 22:46:07:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Dann Corbit at 08 November 2003 09:57:02:

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

I ran the experiment with the same engine, Ruffian 1.0.5, and it came out equal. 13+,24=,13- between Ruffian 1.0.5 with cleanbook versus Ruffian 1.0.5 with ruffian.bok. Perhaps each of the books is better against certain engines. {erhaps one is better with longer time controls.

[capgadget]

Geschrieben von: / Posted by: capgadget at 09 November 2003 04:34:43:
Als Antwort auf: / In reply to: Re: First impression of Ruffian 1.0.5 geschrieben von: / posted by: Uri Blass at 08 November 2003 15:29:44:

Interesting thought about extremely well built small book for long time controls. Worth testing. I basically have to hide behind a large book because my measely 800Mhz PIII has to go up against 2+Ghz Fritzs and Shredders. it still survives sometimes especially at 30 minute+ games. Time seems to be the great equalizer unless the chess program is lame.

ruffiantestb on freechess.org

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

Another problem is that being superior against itself does not mean being superior against other programs.
Uri

I also forgot about the time control.
Big book is probably better for blitz when the engine cannot find superior moves relative to moves of masters so the book save time.
small book is probably better for longer time control when the engine probably can find superior moves to moves of players with rating of 2300.
Uri

[capgadget]

Geschrieben von: / Posted by: capgadget at 09 November 2003 04:38:35:
Als Antwort auf: / In reply to: 2nd experiment geschrieben von: / posted by: Norm Pollock at 08 November 2003 22:46:07:

I was getting something like 5-2-5 at 5 3 with a large custom book against the stock 1.0.5 book but once again probably because the weights arent tuned yet.
I think cleanbook is in my 500K chess database, but definitely not clean after mixing in the TWIC junk.

The results:
Ruffian 1.0.5 with cleanbook 17+, 23=, 10- or 57% versus 43% for Ruffian 1.0.1 with the standard Ruffian book (1.8M). I believe that cleanbook.pgn, even though it is so large, is superior to other opening books.

You changed two things at once. Was it the program or the book that improved?
I suggest repeating the experiment with the same version of Ruffian (take your pick).

I ran the experiment with the same engine, Ruffian 1.0.5, and it came out equal. 13+,24=,13- between Ruffian 1.0.5 with cleanbook versus Ruffian 1.0.5 with ruffian.bok. Perhaps each of the books is better against certain engines. {erhaps one is better with longer time controls.