Time handicap tournament. LC0 and SF11

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Time handicap tournament. LC0 and SF11

Post by mmt » Sat Feb 08, 2020 12:46 am

I'll be running various time handicap matches at different time controls, starting with self-play. Probably limited to LC0 and SF 11. I'll keep the log here. Related to viewtopic.php?f=2&t=73000.

Started with:
Match: LC0 30s + 1s vs LC0 15s + 0.5s
Settings: Default settings + 90000000 cache, net 59098, 4 threads, fp16, roundrobin
Hardware: Titan V + 2080Ti
Book: Silver-Suite-Large random with reversed color
Tablebases: Complete 6-piece and around 40% of 7-piece
Software: Arena 3.5.1, Windows
Result after 100 games:

lc0 59x 30s+1s - lc0 59x 15s+0.5s : 54.0/100 10-2-88 (=====11=============1==0==1=======================1===========1===1===11================1====0======) 54% +28

lc0 59x 15s+0.5s - lc0 59x 30s+1s : 46.0/100 2-10-88 (=====00=============0==1==0=======================0===========0===0===00================0====1======) 46% -28

mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Re: Time handicap tournament. LC0 and SF11

Post by mmt » Sat Feb 08, 2020 11:11 am

200 games:
lc0 59x 30s+1s - lc0 59x 15s+0.5s :114.0/200 30-2-168
Elo: +49 -49
Ordo:
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 lc0 59x 30s+1 : 2324.7 17.9 114.0 200 57
2 lc0 59x 2 15s+0.5s : 2275.3 17.9 86.0 200 43

Guenther
Posts: 3280
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Time handicap tournament. LC0 and SF11

Post by Guenther » Sat Feb 08, 2020 11:17 am

mmt wrote:
Sat Feb 08, 2020 12:46 am
I'll be running various time handicap matches at different time controls, starting with self-play. Probably limited to LC0 and SF 11. I'll keep the log here. Related to viewtopic.php?f=2&t=73000.

Started with:
Match: LC0 30s + 1s vs LC0 15s + 0.5s
Settings: Default settings + 90000000 cache, net 59098, 4 threads, fp16, roundrobin
Hardware: Titan V + 2080Ti
Book: Silver-Suite-Large random with reversed color
Tablebases: Complete 6-piece and around 40% of 7-piece
Software: Arena 3.5.1, Windows
Result after 100 games:

lc0 59x 30s+1s - lc0 59x 15s+0.5s : 54.0/100 10-2-88 (=====11=============1==0==1=======================1===========1===1===11================1====0======) 54% +28

lc0 59x 15s+0.5s - lc0 59x 30s+1s : 46.0/100 2-10-88 (=====00=============0==1==0=======================0===========0===0===00================0====1======) 46% -28
I don't see how this is related to the thread you give above and what you are trying to measure here?
(we already know rough values for time doubling with various programs over thousands of games depending on tc)

BTW are the games available?
(Usually, if the same entity plays against itself with time odds and the handicapped 'clone' loses some games, it might indicate
bad start positions. Moreover I am not sure time odds implementation in Arena is safe, I would use CuteChess-cli or WB here)
https://rwbc-chess.de
https://rwbc-chess.de/chronology.htm
--------------------------------------------------
The troll explosion at talkchess:
https://docs.google.com/spreadsheets/d/ ... KSptBx9AUs

Alayan
Posts: 312
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: Time handicap tournament. LC0 and SF11

Post by Alayan » Sat Feb 08, 2020 12:45 pm

Guenther wrote:
Sat Feb 08, 2020 11:17 am
Usually, if the same entity plays against itself with time odds and the handicapped 'clone' loses some games, it might indicate
bad start positions.
This doesn't make any sense, the version with more time plays stronger, it's expected for the one with less time to lose to it.

Guenther
Posts: 3280
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Time handicap tournament. LC0 and SF11

Post by Guenther » Sat Feb 08, 2020 1:09 pm

Alayan wrote:
Sat Feb 08, 2020 12:45 pm
Guenther wrote:
Sat Feb 08, 2020 11:17 am
Usually, if the same entity plays against itself with time odds and the handicapped 'clone' loses some games, it might indicate
bad start positions.
This doesn't make any sense, the version with more time plays stronger, it's expected for the one with less time to lose to it.
Ofc I wanted to write ...the not handicapped... - should be self explaining - just a typo, sorry.
https://rwbc-chess.de
https://rwbc-chess.de/chronology.htm
--------------------------------------------------
The troll explosion at talkchess:
https://docs.google.com/spreadsheets/d/ ... KSptBx9AUs

mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Re: Time handicap tournament. LC0 and SF11

Post by mmt » Sat Feb 08, 2020 1:50 pm

Guenther wrote:
Sat Feb 08, 2020 11:17 am
I don't see how this is related to the thread you give above and what you are trying to measure here?
(we already know rough values for time doubling with various programs over thousands of games depending on tc)
I'm looking to see how the Elo/Ordo difference will change with longer time controls. I saw this done before but not with the latest LC0 nets and SF 11. I think it'll also be interesting to see how the curves differ for LC0 and SF.
Guenther wrote:
Sat Feb 08, 2020 11:17 am
BTW are the games available?
(Usually, if the same entity plays against itself with time odds and the handicapped 'clone' loses some games, it might indicate
bad start positions. Moreover I am not sure time odds implementation in Arena is safe, I would use CuteChess-cli or WB here)
I'll make games available periodically in bigger batches or just make a google drive directory - too much pain to upload each time I update.
The Arena is doing ok so far with self-play but it seems impossible to use it for SF vs LC0 matches because the number of threads is global. And it's definitely buggy. I didn't see the time handicap possibility in CuteChess GUI (1.0) so I didn't consider it for this match yet.

mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Re: Time handicap tournament. LC0 and SF11

Post by mmt » Tue Feb 11, 2020 1:26 am

Match: LC0 5s + 0.5s vs LC0 2.5s + 0.25s
Settings: Default settings + 90000000 cache, net 59098, 4 threads, fp16, roundrobin
Hardware: Titan V + 2080Ti
Book: Silver-Suite-Large random with reversed color
Tablebases: Complete 6-piece and around 40% of 7-piece
Software: Arena 3.5.1, Windows

Result after 1000 games:
lc0 59x 5s/0.5s 617.5/1000 249-14-73 +85
lc0 59x 2.5s/0.25s 382.5/100 14-249-737 -85

Ordo (baseline 2300):
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 lc0 5s/0.5s : 2342.0 7.5 617.5 1000 62
2 lc0 2.5s/0.25s : 2258.0 7.5 382.5 1000 38
Difference : 84
Last edited by mmt on Tue Feb 11, 2020 1:49 am, edited 2 times in total.

mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Re: Time handicap tournament. LC0 and SF11

Post by mmt » Tue Feb 11, 2020 1:45 am

Match: Stockfish 11 5s + 0.5s vs Stockfish 11 2.5s + 0.25s
Settings: 31 threads, 20 GB cache
Hardware: 2x Xenon Gold 6246
Book: Silver-Suite-Large random. With reversed colors
Tablebases: Complete 6-piece and a good part of 7-piece TBs
Software: Arena 3.5.1, Windows

Results after 1000 games:
Stockfish 11 5s + 0.5s: 544.0/1000 125-37-838 +28
Stockfish 11 2.5s + 0.25s: 456.0/1000 37-125-838 -28

Ordo (baseline 2300):
# PLAYER : RATING ERROR POINTS PLAYED (%)
1 Stockfish 11 5s + 0.5s : 2315.5 7.4 544.0 1000 54
2 Stockfish 11 2.5s + 0.25s : 2284.5 7.4 456.0 1000 46
Difference: 31

Alayan
Posts: 312
Joined: Tue Nov 19, 2019 7:48 pm
Full name: Alayan Feh

Re: Time handicap tournament. LC0 and SF11

Post by Alayan » Tue Feb 11, 2020 1:48 am

20GB hash for ultra-bullet ? That's unreasonable.

mmt
Posts: 216
Joined: Sun Aug 25, 2019 6:33 am
Full name: .

Re: Time handicap tournament. LC0 and SF11

Post by mmt » Tue Feb 11, 2020 1:51 am

Hmm, I don't think a different value would make much of a difference, it's self-play with the same cache setting for both sides. But if less is better at these time controls then SF should figure it out itself and accordingly use less RAM.

Post Reply