Still a very good performance, around 2550-2600 CCRL Elo points.Werewolf wrote:Laskos wrote:Well, my initial assessments of this engine were complete rubbish, but 2750 CCRL Elo seems already achieved, say on a very strong GPU like Nvidia 1080 and LTC of say 1 minute per move.duncan wrote:still on target ?Laskos wrote: As of now, the progress does not seem to be plateauing. If it continues in the same fashion, and the framework continues its work, by the 20th of April we can expect LCZero to be about 2750 CCRL Elo points.
As of now, it seems to progress again slowly, by my preliminary results with meager CPU means and short time control of 1s/move, maybe these results are not very representative. I will post my results maybe this night or tomorrow.
I had to finish my match early because I need my PC back.
Sadly it didn't go as well as I expected.
LCZero 127 on Nvidia 1060 @ 15 sec/move
Colossus 2008b @15 sec/move
31.5/86
LCZero was dominating in the first third of the match but then went downhill really badly. Made me think there was either a bug or the hash (or whatever it uses) needs clearing.
I am getting very bad "progress" on my CPU testing with v0.6 from ID124 to ID134. The strongest "bignet" was ID124, and since then it seems to go downhill. I left overnight some testing batches, and the results came very bad as progression goes. Either my CPU testing at fast TC (1s/move and 10s/move) is unrepresentative, or the chosen opponent, a stable engine Jabba 1.0 (about 2050 CCRL) is a bad choice as an opponent, but here I am:
1/ Games at 1s/move on 1 CPU core (about 200 playouts per move on average) against Jabba 1.0 standard, stable engine:
ID124 (the second "bignet")
172.5/500
Code: Select all
Games Completed = 500 of 500 (Avg game length = 103.231 sec)
Settings = Gauntlet/64MB/1000ms per move/M 5500cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 14275 sec elapsed, 0 sec remaining
1. LCZero CPU ID124 172.5/500 116-271-113 (L: m=271 t=0 i=0 a=0) (D: r=82 i=13 f=13 s=2 a=3) (tpm=948.5 d=12.49 nps=230)
2. Jabba 1.0 327.5/500 271-116-113 (L: m=116 t=0 i=0 a=0) (D: r=82 i=13 f=13 s=2 a=3) (tpm=802.5 d=9.11 nps=0)
111.0/500
Code: Select all
Games Completed = 500 of 500 (Avg game length = 89.922 sec)
Settings = Gauntlet/64MB/1000ms per move/M 5500cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 12964 sec elapsed, 0 sec remaining
1. LCZero CPU ID133 111.0/500 69-347-84 (L: m=347 t=0 i=0 a=0) (D: r=68 i=10 f=5 s=0 a=1) (tpm=945.7 d=12.49 nps=151)
2. Jabba 1.0 389.0/500 347-69-84 (L: m=69 t=0 i=0 a=0) (D: r=68 i=10 f=5 s=0 a=1) (tpm=803.7 d=8.94 nps=0)
95.0/500
Code: Select all
Games Completed = 500 of 500 (Avg game length = 90.115 sec)
Settings = Gauntlet/64MB/1000ms per move/M 5500cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 12647 sec elapsed, 0 sec remaining
1. LCZero CPU ID134 95.0/500 59-369-72 (L: m=369 t=0 i=0 a=0) (D: r=60 i=4 f=5 s=0 a=3) (tpm=943.8 d=12.52 nps=172)
2. Jabba 1.0 405.0/500 369-59-72 (L: m=59 t=0 i=0 a=0) (D: r=60 i=4 f=5 s=0 a=3) (tpm=804.8 d=8.72 nps=0)
I am not sure if this is representative for strong GPU results. But I tested also at 10s/move, or about 2000 playouts per move on my CPU, this is already not that few, and the results show again a regression almost outside error margins:
2/
ID124
13.0/20
Code: Select all
Games Completed = 20 of 20 (Avg game length = 1261.311 sec)
Settings = Gauntlet/64MB/10000ms per move/M 5500cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 6754 sec elapsed, 0 sec remaining
1. LCZero CPU ID124 13.0/20 12-6-2 (L: m=6 t=0 i=0 a=0) (D: r=2 i=0 f=0 s=0 a=0) (tpm=7238.3 d=16.19 nps=185)
2. Jabba 1.0 7.0/20 6-12-2 (L: m=12 t=0 i=0 a=0) (D: r=2 i=0 f=0 s=0 a=0) (tpm=9803.3 d=11.84 nps=0)
7.5/20
Code: Select all
Games Completed = 20 of 20 (Avg game length = 948.178 sec)
Settings = Gauntlet/64MB/10000ms per move/M 5500cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5484 sec elapsed, 0 sec remaining
1. LCZero CPU ID134 7.5/20 6-11-3 (L: m=11 t=0 i=0 a=0) (D: r=3 i=0 f=0 s=0 a=0) (tpm=7094.3 d=16.44 nps=639)
2. Jabba 1.0 12.5/20 11-6-3 (L: m=6 t=0 i=0 a=0) (D: r=3 i=0 f=0 s=0 a=0) (tpm=9781.1 d=11.55 nps=0)
I am unable to test with v0.6 EPD test suites, either with Polyglot, or any GUI. Seems that the PV is not outputted in standard form. I would be curious to see where this regression comes from, from tactical or positional factors (I have one very positional test suite and several very tactical). Maybe there is some conflict in the training between policy part of the network, which gives a probability distribution over possible moves, and the value part of the network, which gives probability of winning given the board.