Ruffian 2.0.0 versus Ruffian Leiden

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 17 January 2004 13:05:53:

Following some posts in CCC claiming that Ruffian Leiden were stronger than Ruffian 2.0.0 I let those two play a match of 20 games against each other (win2k, ponder off, 5 piece egtbs, time control 300+2, winboard 4.2.3, wbtm 0.60, elostat 1.1b, 96Mb hash, default books, Athlon TB 1.1Mhz).


Ruffian v2.0.0 playing 20 games against each other engine. Score: 9.5 / 20 (47%)
Rank|No  |Name          |                    |Pts             |
----|----|--------------|--------------------|----------------|
  1.|  2.|Ruffian Leiden|=0==01=11===100==101|  10.5 / 20  52%|


As this match was very close I extended the test and ran a second match with 50 games under the same conditions:


Ruffian v2.0.0 playing 50 games against each other engine. Score: 22.5 / 50 (45%)
Rank|No  |Name          |                                                  |Pts             |
----|----|--------------|--------------------------------------------------|----------------|
  1.|  2.|Ruffian Leiden|=======0==10===11=11=0=10101=1=001=0==1=01==10=11=|  27.5 / 50  55%|


All 70 games were unique, the overall result is as follows:


    Program                     Elo    +   -   Games   Score   Av.Op.  Draws
  1 Ruffian Leiden            : 2695   80  51    70    54.3 %   2665   48.6 %
  2 Ruffian v2.0.0            : 2665   51  80    70    45.7 %   2695   48.6 %


This result is not statistically signifcant, but it may indeed be true that Ruffian Leiden has a slight edge over Ruffian 2.0.0.
Possibly I will include Ruffian Leiden in my YABRL rating list in order to get a better estimation.
Robert

[Kurt Utzinger]

Geschrieben von: / Posted by: Kurt Utzinger at 18 January 2004 07:48:32:
Als Antwort auf: / In reply to: Ruffian 2.0.0 versus Ruffian Leiden geschrieben von: / posted by: Robert Allgeuer at 17 January 2004 13:05:53:

Matches between different versions of the
same program do not always show which is
the better one. To find out about real
strength of a version, it is moch more
important to play games vs other programs
and to compare. I have stopped playing
program v.1 against program v.2 as I had
sometime a very bad experience with CM8k.
Here I found a setting that has beaten
all other settings with an overall score
of 65 % but the same setting made much
worse results against other programs
in comparision with some other CM8k-
settings.
Kurt
Kurt & Rolf Chess

[Uri Blass]

Geschrieben von: / Posted by: Uri Blass at 18 January 2004 13:11:26:
Als Antwort auf: / In reply to: Re: Ruffian 2.0.0 versus Ruffian Leiden geschrieben von: / posted by: Kurt Utzinger at 18 January 2004 07:48:32:

Matches between different versions of the
same program do not always show which is
the better one. To find out about real
strength of a version, it is moch more
important to play games vs other programs
and to compare. I have stopped playing
program v.1 against program v.2 as I had
sometime a very bad experience with CM8k.
Here I found a setting that has beaten
all other settings with an overall score
of 65 % but the same setting made much
worse results against other programs
in comparision with some other CM8k-
settings.
Kurt
Kurt & Rolf Chess

I think that both results are interesting.
It seems based on other results that Ruffian Leiden is also stronger against other programs.
I want a new program to get more than 50% against previous version and not only get better results against other programs.
Uri

[Robert Allgeuer]

Geschrieben von: / Posted by: Robert Allgeuer at 18 January 2004 16:43:23:
Als Antwort auf: / In reply to: Re: Ruffian 2.0.0 versus Ruffian Leiden geschrieben von: / posted by: Kurt Utzinger at 18 January 2004 07:48:32:

Matches between different versions of the
same program do not always show which is
the better one. To find out about real
strength of a version, it is moch more
important to play games vs other programs
and to compare. I have stopped playing
program v.1 against program v.2 as I had
sometime a very bad experience with CM8k.
Here I found a setting that has beaten
all other settings with an overall score
of 65 % but the same setting made much
worse results against other programs
in comparision with some other CM8k-
settings.
Kurt
Kurt & Rolf Chess

I am aware of this, that is why I wrote "may indeed be stronger" and that I consider including Ruffian Leiden in my rating list for getting a better estimate (which involves matches against a wide range of other engines).
I agree, at the moment we do not know, whether Ruffian 2.0.0 or Ruffian Leiden are stronger, but there are at least some indications - matches against Shredder 8 posted in CCC and my self play test - that Ruffian 2.0.0 may even be a bit weaker than Ruffian Leiden.
Robert

[Kurt Utzinger]

Geschrieben von: / Posted by: Kurt Utzinger at 18 January 2004 16:51:29:
Als Antwort auf: / In reply to: Re: Ruffian 2.0.0 versus Ruffian Leiden geschrieben von: / posted by: Uri Blass at 18 January 2004 13:11:26:

Matches between different versions of the
same program do not always show which is
the better one. To find out about real
strength of a version, it is moch more
important to play games vs other programs
and to compare. I have stopped playing
program v.1 against program v.2 as I had
sometime a very bad experience with CM8k.
Here I found a setting that has beaten
all other settings with an overall score
of 65 % but the same setting made much
worse results against other programs
in comparision with some other CM8k-
settings.
Kurt
Kurt & Rolf Chess

I think that both results are interesting.
It seems based on other results that Ruffian Leiden is also stronger against other programs.
I want a new program to get more than 50% against previous version and not only get better results against other programs.
Uri

I understand this intention but if the internal
stronger version produces worse results vs other
engines this does not serve the general goal to
improve an engine.
Kurt