I don't know what the hell is going on with the amount of nodes, if it's the fault of the engine or Arena, but not mine.Laskos wrote: ↑Thu Sep 13, 2018 8:01 am
Wow, that's VERY interesting! So, you get about equal results against SF8 on 4 cores from positions used in AlphaZero paper, and +0 -9 =11 result, or some -170 Elo points from my 3-mover balanced positions? My positions are balanced and collected from GM and IM human games, so they are not some nutty positions from random 2-mover openings of SF framework. I have about 900 3-mover positions in my suite, and I am quite sure it's a good set to test on. The conclusion would be: AlphaZero team choose on purpose openings suitable to AlphaZero to boost significantly the performance, by more than 100 Elo points. That's remarkable, as I didn't expect some regular 3-movers to change dramatically the result. One has to play diverse openings, because, first, other engines might not reply the way Lc0 plays, second, other engines might use a book. In fact CCCC results, which show, like me, Lc0 the level of Fire 7.1, from initial standard opening position, show that other engines do not play the openings as Lc0 would like them to play.
But anyway, I don't understand the total nodes issue, the depth issue (this is not only about initial depth, but the depths reached after say 10s), NPS issue. Your NPS seem very stable from the beginning, I even saw peaks in NPS at 1-2s in a Youtube video, larger than after 10s, while my NPS are increasing slowly, being almost twice higher after 10s search than after less than 1s search. Hash issue as observed by Milos seemed quite plausible to me.
Interesting, and my advice would be to not use AlphaZero openings, they seem to be chosen not quite fairly against SF8.
"Hash issue as observed by Milos seemed quite plausible to me." I have shown that repeating the games doesn't help much lc0, as accidentaly has happened in my last test. You can see from the above score
Lc01711261 - Stockfish_8_x64_bmi2 : 5,5/20 0-9-11 (=0 0= =0 =0 =0 == 00 =0 == 0=) 28%
Stockfish_8_x64_bmi2 - Lc01711261 : 14,5/20 9-0-11 (=1 1= =1 =1 =1 == 11 =1 == 1=) 73%
First two games Lc0 0.5-Sto8 1.5, were replayed in games 3-4 with the same result Lc0 0.5-Sto8 1.5, but different colours,
Games 5-6, Lc0 0.5-Sto8 1.5, replayed in games 7-8 with the same result Lc0 0.5-Sto8 1.5and colours,
Games 9-10, Lc0 0.5-Sto8 1.5, replayed in games 11-12 Lc0 1-Sto8 1, here lc0 improves,
Games 13-14, Lc0 0-Sto8 2, replayed in games 15-16 Lc0 0.5-Sto8 1.5, here lc0 improves,
Games 17-18, Lc0 1-Sto8 1, replayed in games 19-20 Lc0 0.5-Sto8 1.5, here Stockfish8 improves.
"The conclusion would be: AlphaZero team choose on purpose openings suitable to AlphaZero to boost significantly the performance, by more than 100 Elo points."
No, the conclusion is that some initial positions lead more probably than others to tactical play, where lc0 fails. Until now, we don't know about the tactical abilities of AlphaZero. We must see the complete 100 games and the 1200 with the other initial positions. These 12 initial positions were chosen because they were the most played, as they said in their paper:
"12 most popular human openings (played more than 100,000 times in an online database)"
From my experience with lc0, if you put complicated tactical initial positions the better Stochfish scores, if you put starting position or quiet positional position the better lc0 scores.
leela is official(?) better than sf9??
Positionally probably, tactically of course not and globally either.
But it is very interesting to see how with its low NPS, lc0 is able to stop Stockfish deep search in some games.
Recommendation for immature Stockfish fanboys, they just have to look for balanced positions with a lot of tactic so that Stockfish scores much more than lc0, even 100% I can assure.
Now I am testing lc011261 against Stockfish 8, same conditions as before, starting from the 10 Nunn positions
After 12 games of the 20, the score is
Engine Score Lc
1: Stockfish_8_x64_bmi2 8,0/12 ===1==1101=1
2: Lc01711261 4,0/12 ············
You can see some games at youtube:
https://youtu.be/3fogVPj0RRk
https://youtu.be/Fh-QF07oiXQ
