Then I wanted to see the scaling. I haven't chosen regular chess, as draw rate at longer TC between them is going high, and in 100 games matches I will have weak Wins and Losses statistics. I have chosen the playable 3-Queens Chess:
8 | |||||||||
7 | |||||||||
6 | |||||||||
5 | |||||||||
4 | |||||||||
3 | |||||||||
2 | |||||||||
1 | |||||||||
a | b | c | d | e | f | g | h |
rnqqkqnr/pppppppp/8/8/8/8/PPPPPPPP/RNQQKQNR w KQkq - 0 1
The result at 1s/move in 100 games is (engines on 1 i7 thread):
Code: Select all
Games Completed = 100 of 100 (Avg game length = 138.082 sec)
Settings = Gauntlet/64MB/1000ms per move/M 1500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\3QEPD_Trim.epd(112)
Time = 3521 sec elapsed, 0 sec remaining
1. Komodo 67.5/100 44-9-47 (L: m=9 t=0 i=0 a=0) (D: r=26 i=19 f=0 s=0 a=2) (tpm=972.5 d=24.10 nps=1914802)
2. Komodo MCTS 32.5/100 9-44-47 (L: m=43 t=0 i=0 a=1) (D: r=26 i=19 f=0 s=0 a=2) (tpm=990.3 d=9.22 nps=2812)
Code: Select all
Games Completed = 100 of 100 (Avg game length = 1397.608 sec)
Settings = Gauntlet/64MB/10000ms per move/M 1500cp for 3 moves, D 140 moves/EPD:C:\LittleBlitzer\3QEPD_Trim.epd(112)
Time = 36779 sec elapsed, 0 sec remaining
1. Komodo 71.0/100 44-2-54 (L: m=2 t=0 i=0 a=0) (D: r=13 i=39 f=0 s=0 a=2) (tpm=9672.4 d=31.39 nps=1925693)
2. Komodo MCTS 29.0/100 2-44-54 (L: m=41 t=0 i=0 a=3) (D: r=13 i=39 f=0 s=0 a=2) (tpm=9832.4 d=14.26 nps=2941)
And here are the results for Normalized Elo:
1s/move: 0.548 sigma=0.1
10s/move: 0.789 sigma=0.1
The difference is 0.241 and 1 sigma of the difference is 0.141. The likelihood that K MCTS scales worse than K AB is 95%.
To note that this result is for this particular game with 3 Queens, but I saw no differences in behavior with respect to different variants of chess or chess board games between K AB and K MCTS.