MCEC anyone?

[mvanthoor]

I have a very bad experience with fast TC selfplay. I developed a version that outsmarted both latest dev and stable in super fast TC but tanked in slower ones. Even though all I had changed is the eval.

Here, Zahak 0.2.1 and 0.3.0 underperform against Rustic on 10s+0.1s (i.e., Rustic Alpha 2 is clearly stronger than Zahak 0.2.1, while according to CCRL, the Elo difference should be only +10, for Zahak.) It doesn't matter too much for the gauntlet though, because they don't lose on time or by disconnects, so they can be used for testing. The only thing is that they won't survive long in the fast gauntlet.

Strangely enough, Loki 1.0.2 and 1.2.0 have exactly the same problem.

I switch back to slower time control with less games (1200 in total)

I started a gauntlet with 10 engines for Alpha 1 to play against. Then, I removed the weakest 4, and replaced them with 4 engines that are clearly stronger than Alpha 1, and then I tested Alpha 1.5. Then I replaced 2 engines, and tested Alpha 2. Now I've replaced 2 engines again for Alpha 2.1.100, and again for Alpha 2.2.100.

You can see the pattern. The overlapping engines between the Rustic versions are responsible for the rating of the new Rustic version. The new engines get their rating according to the performance against the new Rustic version. That way, the gauntlet keeps creeping up in strength while Rustic's strength increases.

If an engine underperforms, but otherwise plays normally (no crashes, no forfeits, etc), it'll just be replaced earlier. If a new engine does REALLY bad in a fast time gauntlet, I'll just remove its games from the database (that first run will be 500 games), replace it with a different engine, do a 1-on-1, and then add those games to the database.

I'm keeping two gauntlets this way: a 10s+0.1s, and a 60s+0.6s, so I can use both to determine the approximate gain in strength per version. To get a (very) rough idea, I run a 1-on-1 between the latest and the previous version for 2000 games, but I intend to replace that with an SPRT run.

[lithander]

Quick question: All your engines can report a score that is then displayed as M6, -M3 etc... MinimalChess just returns a score of +/-9999 whenever the game is lost and that always written as a +/-M1 in the tournament schedule view. What's the right way to report a mate?

[amanjpro]

Quick question: All your engines can report a score that is then displayed as M6, -M3 etc... MinimalChess just returns a score of +/-9999 whenever the game is lost and that always written as a +/-M1 in the tournament schedule view. What's the right way to report a mate?

+M16 means the side that reports it is able to deliver mate in 16 full moves (a full move is two plies), -M16 means the side that reports it will get mated at the very best in 16 full moves.

The way we implement it, is instead of reporting Mate Score, we report Mate Score - the current depth, and in report time we basically do the reverse

[amanjpro]

Both your download links for FiaryMax and KingSlayer binaries are broken, can you please post them here?

This was because I gave the links before I actually uploaded the files there. The links should work now.

Upon second try, both of them can be downloaded. But I cannot run them:

Code: Select all

amanj@zahaktop:~/Documents/engines$ ./kingslayer 
uci
Error: unknown command
\q
Error: unknown command
^C
amanj@zahaktop:~/Documents/engines$ ./fairymax 
tellics say     Fairy-Max 5.0b
tellics say     by H.G. Muller
telluser piece-description file '/usr/local/share/games/fairymax/fmax.ini'  not found

Maybe kingslayer uses XBoard? If so, I need to teach myself how to set hash size

Both Fairy-Max and KingSlayer are XBoard engines.

And I forgot to supply the fmax.ini data file, sorry about that. I uploaded that now to http://hgm.nubati.net/fmax.ini . You would have to put it in the directory mentioned in the error message.

Hello,

I am able to run FairyMax, but wasn't able to run KingSlayer, here is the engine logs, kingslayer gets disconnected right away:

Code: Select all

<fairymax(3): tellics say     Fairy-Max 5.0b
<fairymax(3): tellics say     by H.G. Muller
<fairymax(3): feature myname="Fairy-Max 5.0b"
>fairymax(3): accepted myname
<fairymax(3): feature memory=1 exclude=1
>fairymax(3): accepted memory
>fairymax(3): rejected exclude
<fairymax(3): feature setboard=0 xedit=1 ping=1 done=0
>fairymax(3): accepted setboard
>fairymax(3): rejected xedit
>fairymax(3): accepted ping
>fairymax(3): accepted done
<fairymax(3): feature variants="normal,nocastle,shatranj,asean,makruk,cambodian,ai-wok,courier,knightmate,capablanca,gothic,janus,falcon,cylinder,berolina,super,seirawan,spartan,great,light-brigade,king-of-the-hill,bifurcator,team-mate,los-alamos,ciccolini,mexican,grande-acedrex,roman,almost-wildebeest,fairy"
>fairymax(3): accepted variants
<fairymax(3): feature option="Resign -check 0"
>fairymax(3): accepted option
<fairymax(3): feature option="Resign Threshold -spin 800 200 1200"
>fairymax(3): accepted option
<fairymax(3): feature option="Claim draw after -spin 50 0 200"
>fairymax(3): accepted option
<fairymax(3): feature option="Ini File -file /usr/local/share/games/fairymax/fmax.ini"
>fairymax(3): accepted option
<fairymax(3): feature option="Multi-PV Margin -spin 0 0 1000"
>fairymax(3): accepted option
<fairymax(3): feature option="Variant fairy selects -combo FIDE-Clobberers /// Clobberers-FIDE /// FIDE-Nutters /// Nutters-FIDE /// Clobberers-Nutters /// Nutters-Clobberers /// FIDE-Rookies /// Rookies-FIDE /// Clobberers-Rookies /// Rookies-Clobberers /// Nutters-Rookies /// Rookies-Nutters"
>fairymax(3): accepted option
<fairymax(3): feature option="Makruk rules -combo makruk /// Cambodian /// Ai-wok"
>fairymax(3): accepted option
<fairymax(3): feature option="Dummy Slider Example -slider 20 0 100"
>fairymax(3): accepted option
<fairymax(3): feature option="Dummy String Example -string happy birthday!"
>fairymax(3): accepted option
<fairymax(3): feature option="Dummy Path Example -path ."
>fairymax(3): accepted option
<fairymax(3): feature option="Automatic persistent-hash dialog -check 0"
>fairymax(3): accepted option
<fairymax(3): feature option="Info -button"
>fairymax(3): accepted option
<fairymax(3): feature option="Save in hash file -button"
>fairymax(3): accepted option
<fairymax(3): feature option="Clear Hash -button"
>fairymax(3): accepted option
<fairymax(3): feature done=1
>fairymax(3): accepted done
>fairymax(3): option Automatic persistent-hash dialog=0
>fairymax(3): option Claim draw after=50
>fairymax(3): option Dummy Path Example=.
>fairymax(3): option Dummy Slider Example=20
>fairymax(3): option Dummy String Example=happy birthday!
>fairymax(3): option Ini File=/usr/local/share/games/fairymax/fmax.ini
>fairymax(3): option Makruk rules=makruk
>fairymax(3): option Multi-PV Margin=0
>fairymax(3): option Resign=0
>fairymax(3): option Resign Threshold=800
>fairymax(3): option Variant fairy selects=FIDE-Clobberers
>fairymax(3): memory 256
>fairymax(3): new
>fairymax(3): force
>fairymax(3): level 40 5 0
>fairymax(3): post
>fairymax(3): easy
>kingslayer(4): new
>kingslayer(4): force
>kingslayer(4): level 40 5 0
>kingslayer(4): post
>kingslayer(4): easy
>fairymax(3): ping 16
<fairymax(3): pong 16
>fairymax(3): time 30000
otim 30000
>fairymax(3): go
<fairymax(3): # times @ 3647498532
<fairymax(3): # 16+16 pieces, centr = (1,1) R=40
<fairymax(3):  1      6        0          7 a2a4
<fairymax(3):  1     15        0          9 c2c4
<fairymax(3):  2      0        0         17 c2c4 c7c5
<fairymax(3):  3      4        0        406 c2c4 d7d5 b2b3
<fairymax(3):  3     13        0        605 c2c3 c7c5 d2d4
<fairymax(3):  3     15        0        756 d2d4 d7d5 b1c3
<fairymax(3):  4      0        0       1171 d2d4 d7d5 b1c3 b8c6
<fairymax(3):  5     15        1      21499 d2d4 d7d5 b1c3 b8c6 c1f4
<fairymax(3):  6      0        3      36671 d2d4 d7d5 b1c3 b8c6 c1f4 c8f5
<fairymax(3):  6      2       18     224109 c2c4 d7d6 d2d4 b8c6 b1c3 c8f5
<fairymax(3):  7     12       33     406033 c2c4 g8f6 d2d3 d7d5 b1d2 b8c6 g1f3
<fairymax(3):  7     13       56     692015 g1f3 b8c6 d2d4 d7d5 b1c3 c8f5
<fairymax(3):  8      0       71     881342 g1f3 b8c6 d2d4 d7d5 b1c3 c8f5 c1f4 g8f6
<fairymax(3):  8      1      153    1908256 c2c4 c7c5 g1f3 d7d6 d2d4 c8e6 d1a4 b8c6 b1c3
<fairymax(3):  9      7      444    5438797 c2c4 g8f6 b1c3 b8c6 g1f3 d7d5 c4d5
<fairymax(3):  9     11      690    8474386 g1f3 b8c6 d2d4 d7d5 c2c3 c8f5 c1f4 g8f6 b1d2
<fairymax(3): # times @ 3647505494: real=6962 cpu=6950
<fairymax(3): # promo = 0 (`) GT = 0
<fairymax(3): move g1f3
>kingslayer(4): usermove g1f3
>kingslayer(4): ping 3
<kingslayer(4): pong 3
>kingslayer(4): time 30000
otim 29303
>kingslayer(4): go
>fairymax(3): force
>fairymax(3): result 1-0 {Black disconnects}
>fairymax(3): ping 7
<fairymax(3): Error (unknown command): result
<fairymax(3): pong 7

And here is what I experience in the command line:

Code: Select all

amanj@zahaktop:~/Documents/engines$ ./kingslayer 
new
force
level 40 5 0
post
easy
ping
pong
move g1f3
Error: unknown command
usermove g1f3
ping 3
pong 3
time 30000
go
1 -20 0 2 a7a5
1 -19 0 6 c7c5
1 6 0 18 b8c6
2 -28 0 44 b8c6 b1c3
3 0 0 195 b8c6 b1c3 g8f6
4 -8 0 342 b8c6 b1c3 g8f6 d2d4
5 0 0 1695 b8c6 b1c3 g8f6 d2d4 d7d5
6 -26 0 7418 b8c6 b1c3 d7d5 d2d4 c8f5 c1f4
7 0 3 55929 b8c6 b1c3 d7d5 d2d4 c8f5 c1f4 g8f6
8 -3 4 68856 b8c6 b1c3 d7d5 d2d4 g8f6 c1f4
9 -6 11 180638 b8c6 b1c3 e7e5 d2d3 g8f6 c1e3 f6g4 e3g5 f8e7
10 -10 32 556906 b8c6 e2e4 g8f6 b1c3 e7e5 f1d3 f8e7 e1g1
11 -8 55 932167 b8c6 e2e4 g8f6 b1c3 e7e5 f1d3 f8e7 c3d5 f6d5
12 -8 112 1955775 b8c6 e2e4 g8f6 b1c3 e7e5 f1d3 f8e7 c3d5 e7c5 e1g1
13 -4 628 11108334 b8c6 e2e4 e7e6 f1e2 g8f6 b1c3 f8e7 e4e5 f6g4 d2d4
move b8c6
^C

I haven't restarted the test tournament (to include fairymax), because I am waiting for a reply from you regarding kingslayer

[amanjpro]

Actually, disabling the optional evaluation settings as default makes kingslayer play... probably you have a bug in one of the settings

Now both FairyMax and KingSlayer are playing in the test run...

I couldn't find either of them in CCRL rating list, can you please provide a rating that they have somewhere?

[hgm]

Fairy-Max is a multi-variant version of micro-Max, and should be about equally strong in orthodox Chess. The Linux compilation I made is not nearly as fast as the Windows compilation of micro-Max that is tested at CCRL, though. So I would estimate Fairy-Max' rating as about 1900.

KingSlayer is somewhere between Fairy-Max and Joker. So around 2100.

I tried to reproduce the problem with KingSlayer, but I couldn't. With the default eval settings this version ran fine both from he command line and in a self-play ame under XBoard. It seemed to run fine when you ran it from the command line too.

The log you posted from the game where it disconnected is not complete; it hardly contains any commands that where sent to KingSlayer. This makes it impossible for me to recreate the circumstances of the disconnect. KingSlayer would lose a lot of strength when almost its entire evaluation is switched off.

[amanjpro]

Fairy-Max is a multi-variant version of micro-Max, and should be about equally strong in orthodox Chess. The Linux compilation I made is not nearly as fast as the Windows compilation of micro-Max that is tested at CCRL, though. So I would estimate Fairy-Max' rating as about 1900.

KingSlayer is somewhere between Fairy-Max and Joker. So around 2100.

Thank you, I'll update the Tournament Info accordingly

I tried to reproduce the problem with KingSlayer, but I couldn't. With the default eval settings this version ran fine both from he command line and in a self-play ame under XBoard. It seemed to run fine when you ran it from the command line too.

The log you posted from the game where it disconnected is not complete; it hardly contains any commands that where sent to KingSlayer. This makes it impossible for me to recreate the circumstances of the disconnect. KingSlayer would lose a lot of strength when almost its entire evaluation is switched off.

I copied the entire debug log of CuteChess, not sure why it doesn't have enough info, sorry. As for the strength, I didn't disable the entire evaluation, I went with the default, that means:

- Contempt = 0
- Evaluate piece mobility = false
- Evaluate trapped pieces = false
- Evaluate drawish material = true
- Evaluate specific endings = true
- Evaluate pawn structure = true
- Evaluate king shelter = true
- Evaluate king seige = false

[hgm]

I am confused now. The case where it disconnected was when you had changed the default settings? If so, how?

It is best to use the default settings; the terms that are switched off by default in the current form did deteriorate play. And since I have not used them since, it is possible bugs crept in. In the monthly on-line blitz tournament I always run it with the default settings.

[amanjpro]

I am confused now. The case where it disconnected was when you had changed the default settings? If so, how?

Not sure how, why... all I see is engine disconnected

It is best to use the default settings; the terms that are switched off by default in the current form did deteriorate play. And since I have not used them since, it is possible bugs crept in. In the monthly on-line blitz tournament I always run it with the default settings.

OK then we are fine...

[Henk]

I think it is more spectacular to watch how a weak engine gets slaughered by a strong engine.
That means no boring draws. Also engines should play fast. So you don't have to wait for infinity until something is happening.