Romichess Gauntlet, Edition 4

[swami]

Code: Select all

Engine            Score Ro
01: RomiChessNG5      0.0/0 · 
01: Kiwi              0.0/0   
01: Hamsters043devE   0.0/0   
01: Crafty-21.5-win32 0.0/0   
01: Danasah313        0.0/0   
01: Homer20_P4        0.0/0   
01: NanoSzachy        0.0/0   
01: Delphil_18        0.0/0   
01: Muse0899b         0.0/0   
01: Lime_v62          0.0/0   
01: Xpdnt_061120      0.0/0   
01: Djinn925x         0.0/0   

0 of 330 games played
Name of the tournament: Romichess NG 5
Site/ Country: ADMIN, United States
Level: Blitz 1/1
Hardware: Intel(R) Pentium(R) 4 CPU 2.60GHz  with 248 MB Memory
Operating system: Microsoft Windows XP Professional Service Pack 2 (Build 2600)
PGN-File: D:\Arena 1.99\Tournaments\Romichess NG 5 .pgn
Website: 
E-Mail Address: 

Noomen 30 games PGN as opening book.

[swami]

After 220 Games and 20 Rounds:

Code: Select all

  Engine            Score                       Ro
01: RomiChessNG5      124.0/220 ···················· 
02: Delphil_18        12.0/20   =1110=001=11=001=1=1 
02: Xpdnt_061120      12.0/20   0=1001101010011=1111 
04: Hamsters043devE   11.5/20   10001111==1=01010011 
05: Muse0899b         11.0/20   11=10=00=1=000111101 
06: Danasah313        9.0/20    100==01=1==100=10100 
06: Crafty-21.5-win32 9.0/20    1001=101=00==0=0101= 
08: Kiwi              8.0/20    1001=0=00===0=000111 
08: Djinn925x         8.0/20    =0000100=0==0=1=0111 
10: NanoSzachy        6.5/20    0=0=001=1==0===0=000 
11: Homer20_P4        5.5/20    000101010=0001000010 
12: Lime_v62          3.5/20    00=10001000010000000 

220 of 330 games played
Name of the tournament: Romichess NG 5
Site/ Country: ADMIN, United States
Level: Blitz 1/1
Hardware: Intel(R) Pentium(R) 4 CPU 2.60GHz  with 248 MB Memory
Operating system: Microsoft Windows XP Professional Service Pack 2 (Build 2600)
PGN-File: D:\Arena 1.99\Tournaments\Romichess NG 5 .pgn
Website: 
E-Mail Address: 

Romi NG5 playing better than stronger players but little weaker than weaker players.

Maybe you can use NG4 in the division of equal strength players, and when it gets promoted, use NG 5

Regards.

[Michael Sherwin]

After 220 Games and 20 Rounds:

Code: Select all

  Engine            Score                       Ro
01: RomiChessNG5      124.0/220 ···················· 
02: Delphil_18        12.0/20   =1110=001=11=001=1=1 
02: Xpdnt_061120      12.0/20   0=1001101010011=1111 
04: Hamsters043devE   11.5/20   10001111==1=01010011 
05: Muse0899b         11.0/20   11=10=00=1=000111101 
06: Danasah313        9.0/20    100==01=1==100=10100 
06: Crafty-21.5-win32 9.0/20    1001=101=00==0=0101= 
08: Kiwi              8.0/20    1001=0=00===0=000111 
08: Djinn925x         8.0/20    =0000100=0==0=1=0111 
10: NanoSzachy        6.5/20    0=0=001=1==0===0=000 
11: Homer20_P4        5.5/20    000101010=0001000010 
12: Lime_v62          3.5/20    00=10001000010000000

360 Games:TOURNAMENT FINISHED

Code: Select all

Engine            Score                                 Ro
01: RomiChessNG5      208.5/360 ······························ 
02: Crafty-21.5-win32 24.0/30   110==110111110==1111=1=1111111 
03: Kiwi              17.5/30   11=01101=1100=0=110=01001==111 
04: Muse0899b         17.0/30   1101010=01=1=0=110010111=01=01 
05: Hamsters 0.3      16.0/30   000=0=101=1010==101=101=0111=1 
06: Delphil_18        14.0/30   =00=0=0===110=0110==001110=1== 
07: Djinn925x         13.5/30   0110=01100010=0001=1100==1=10= 
08: Danasah313        10.5/30   100=1===0000000==0010010=0=1=1 
08: Homer20_P4        10.5/30   1000010011000010010==0=1===0=0 
10: GreKo             9.5/30    1=00=00=0=0==10000001==0==1=00 
11: NanoSzachy        9.0/30    0=0=1000=0=0==0=0000==1=010=0= 
12: Xpdnt_061120      7.0/30    010000=0101=00=10001000000000= 
13: Lime_v62          3.0/30    0000=001000==00000000000000=00 

Hi Swami,

Seems to be something wrong. But, if things are as they seem then you have unwittingly conducted a very interesting experiment. This shows the wide variance in results that can be seen in a gauntlet of this size. That is why Bob says that 2560 games are needed per opponent to make a determination.

I believe that you were supposed to run NG4 in this latest gauntlet!

But, this is also very interesting, so please let it finish.

Thanks!
Mike

[swami]

Hi Michael,

Seems to be something wrong. But, if things are as they seem then you have unwittingly conducted a very interesting experiment. This shows the wide variance in results that can be seen in a gauntlet of this size. That is why Bob says that 2560 games are needed per opponent to make a determination.

I don't think the result will be the same if you run 2 different tests of 2500 games with the same version.There's some sort of variance maybe it's due to Romi's learning features?

I believe that you were supposed to run NG4 in this latest gauntlet!

Ouch, Sorry Yes This was an error.Anyway, From next edition if you haven't got a newer beta then i can make tests on Romichess NG4.

But, this is also very interesting, so please let it finish.
Thanks!
Mike

Yeah,Let's see what the difference in the scores are.

[Michael Sherwin]

It looks like Graham's trophy edition of RomiChess will be playing in your next edition. Sofar, it's kicking ass. Not only have I got it tuned up really well, I also added a little something from Glaurung that seems to help. Like Tord, I now "shave" the last two bits off of the eval to decrease the granularity of the scores returned. This allows for more efficient null move prunning and also quicker more frequent beta-cuts. Or at least I think that that is what it does.

[swami]

It looks like Graham's trophy edition of RomiChess will be playing in your next edition.

That's the cool prize to award the winner,Thanks

I heard from Alessandro yesterday that his new Hamsters beta has surpassed Kiwi's strength and has shown progress in 300 games he ran. Romi has to catch up with Hamsters!

[Michael Sherwin]

It looks like Graham's trophy edition of RomiChess will be playing in your next edition.

That's the cool prize to award the winner,Thanks

I heard from Alessandro yesterday that his new Hamsters beta has surpassed Kiwi's strength and has shown progress in 300 games he ran. Romi has to catch up with Hamsters!

Once I reach my goal of 70% vs Hamsters 0.2 then I will tackle the latest available version of Hamsters. Sofar, after 28 games of 100 in a 4'4 match it is Romi +16 -3 =9 for 73%!

When I first saw Alessandro's Hamsters, I knew that it has the potential to be one of the great programs. That is why I am so glad that Alessandro is working on it again.

+17 -3 =9

[swami]

Testing it against the antique version of Hamsters would be pointless,Michael. It's similar to testing the Latest beta of Hamsters against the Public old Romichess proto.
Get the prize and see if you can report the similar results.

[Michael Sherwin]

Testing it against the antique version of Hamsters would be pointless,Michael. It's similar to testing the Latest beta of Hamsters against the Public old Romichess proto.
Get the prize and see if you can report the similar results.

Everyone thinks that it is pointless!

It is a known quantity that I can use to judge an improvement. A better score is a better score. That is why I still test against Crafty 19.19 instead of moving up to 21.5. I have a 'yardstick' that I use to measure with. If I change the 'yardstick' then I have to go back and test previous versions of RomiChess against the new 'yardstick' to tell if my new results are an improvement and that kind of time, I do not have. I change the 'yardstick' as infrequently as possible for this reason.

+18 -4 =9

[swami]

Everyone thinks that it is pointless!

Yeah Except few members who have every single version of engines

I only use updated or the best version of the engine and that's why It seems pointless to me, but it would seem useful to you

It is a known quantity that I can use to judge an improvement. A better score is a better score. That is why I still test against Crafty 19.19 instead of moving up to 21.5. I have a 'yardstick' that I use to measure with. If I change the 'yardstick' then I have to go back and test previous versions of RomiChess against the new 'yardstick' to tell if my new results are an improvement and that kind of time, I do not have. I change the 'yardstick' as infrequently as possible for this reason.

I see, you do have a good way of testing engines, but I don't have much interest in testing old versions except in the case where old version is better than the current one. Crafty 19 and Hamsters 0.2 are ancient.