CCRL Testing (@Testers)

[Desperado]

Both engines handle the 10s time control easily and i did not see any time losses.

But maybe one engines uses 9.9 seconds in very few moves and than running on increment not losing on time but barely leaving the "pruning at shallow depth" area (if you really used 10ms increment) which will result in blunders.

Well, that is possible. I started a match with 60s+100ms right now. The goal is to see a significant change in the match result.
If the binaries are correct, i assume it is the case, we should be far away from 70 Elo advantage of Demolito. I save the games as pgn for further
investigation. Maybe Xiphos 0.2 does not handle the time control 10s very well and there is only a problem with the engine related to time control.
This would be important to know, because the engine is in my test pool.

Ok, i stopped the match with 60s + 100ms:

Score of Xiphos2 vs Demolito: 301 - 455 - 427 [0.435] 1183
ELO difference: -45.49 +/- 15.87

demolito_20181029-cygwin-64-popcnt vs. Xiphos 0.2 SSE

Although using more time gives some extra points, it is absolutely clear that Xiphos is not 50 Elo ahead of Demolito
but still more than 40 Elo points weaker in a direct competition. If the scaling would improve further, i strongly doubt that using
the time conrol by ccrl will result in another 100 Elo.

For now i am sure that the super fast games 10s+0.01s are "not" the reason for the gap!

[Graham Banks]

Both engines handle the 10s time control easily and i did not see any time losses.

But maybe one engines uses 9.9 seconds in very few moves and than running on increment not losing on time but barely leaving the "pruning at shallow depth" area (if you really used 10ms increment) which will result in blunders.

Well, that is possible. I started a match with 60s+100ms right now. The goal is to see a significant change in the match result.
If the binaries are correct, i assume it is the case, we should be far away from 70 Elo advantage of Demolito. I save the games as pgn for further
investigation. Maybe Xiphos 0.2 does not handle the time control 10s very well and there is only a problem with the engine related to time control.
This would be important to know, because the engine is in my test pool.

Ok, i stopped the match with 60s + 100ms:

Score of Xiphos2 vs Demolito: 301 - 455 - 427 [0.435] 1183
ELO difference: -45.49 +/- 15.87

demolito_20181029-cygwin-64-popcnt vs. Xiphos 0.2 SSE

Although using more time gives some extra points, it is absolutely clear that Xiphos is not 50 Elo ahead of Demolito
but still more than 40 Elo points weaker in a direct competition. If the scaling would improve further, i strongly doubt that using
the time conrol by ccrl will result in another 100 Elo.

For now i am sure that the super fast games 10s+0.01s are "not" the reason for the gap!

Did you miss this post?

The CEGT rating lists show a similar rating difference between Xiphos 0.2 and Demolito 2018-10-29 as the CCRL rating lists.

[Desperado]

Both engines handle the 10s time control easily and i did not see any time losses.

But maybe one engines uses 9.9 seconds in very few moves and than running on increment not losing on time but barely leaving the "pruning at shallow depth" area (if you really used 10ms increment) which will result in blunders.

Well, that is possible. I started a match with 60s+100ms right now. The goal is to see a significant change in the match result.
If the binaries are correct, i assume it is the case, we should be far away from 70 Elo advantage of Demolito. I save the games as pgn for further
investigation. Maybe Xiphos 0.2 does not handle the time control 10s very well and there is only a problem with the engine related to time control.
This would be important to know, because the engine is in my test pool.

Ok, i stopped the match with 60s + 100ms:

Score of Xiphos2 vs Demolito: 301 - 455 - 427 [0.435] 1183
ELO difference: -45.49 +/- 15.87

demolito_20181029-cygwin-64-popcnt vs. Xiphos 0.2 SSE

Although using more time gives some extra points, it is absolutely clear that Xiphos is not 50 Elo ahead of Demolito
but still more than 40 Elo points weaker in a direct competition. If the scaling would improve further, i strongly doubt that using
the time conrol by ccrl will result in another 100 Elo.

For now i am sure that the super fast games 10s+0.01s are "not" the reason for the gap!

Did you miss this post?

The CEGT rating lists show a similar rating difference between Xiphos 0.2 and Demolito 2018-10-29 as the CCRL rating lists.

And what is the essence of that information ? Does it make the game results wrong that i posted ? Isn't it possible to check my observation by
running an own test ? Of course, it does not mean that some guys did something wrong, but isn't it contradictory what the head to head match results and the lists provide as information? Are you sure that it isn't worth to figure out what the matter is?

Well, we are not talking of 10 or 20 Elo, but a gap of more than 100 Elo!

There might be a good or even a simple reason but it is not obvious reason.
Even if everything is ok, than it would be very (very very) suprising, that an engine with +50 Elo in the list should have -40 to -70 in a
direct competition. I never seen somethin like this before, do you ?!

So, i did not miss this post.

[Desperado]

@All
Well, just for your information, i looked into my match script and i did use 0.1s increment instead of 0.01s. (of course i have several different scripts...). I already reported that the time control cannot be the reason (at least not exclusive) comparing the result with 60s+0.1s
That just confirms that it is impossible that the 10ms increment is a (the) reason. Other statements that i made in the 10ms
context are still valid.

Regards

[Guenther]

IMHO you very much understimate the difference of mps and base+inc time controls and differences in time management for both.
As I was therefore not satisfied with your second test which just multiplied your first tc I started my own test yesterday!

In my test Xiphos 0.22 (I already explained that 0.2.2 and 0.2 are identically regarding this test and their results are also gathered in CCRL accordingly - the reason is, I just had no working 0.2 compilation handy and was too lazy to build for this test) played against Demolito 20181029, both 16MB. but with a tc of 40/10, which has a similar average time per move as your original 10+0.1.
Of course I was prepared that my thesis could be incorrect and the result would show it.

Actually the goal was to play 1000 games and I will at least let the match run as long, but the current result (>700 games) already with its error bars
and LOS leaves no doubt that my thesis seems correct.

Code: Select all

Score of Xiphos_022-64 vs Demolito_20181029-64: 288 - 195 - 248 [0.564]
...      Xiphos_022-64 playing White: 168 - 84 - 114  [0.615] 366
...      Xiphos_022-64 playing Black: 120 - 111 - 134  [0.512] 365
...      White vs Black: 279 - 204 - 248  [0.551] 731
Elo difference: 44.4 +/- 20.5, LOS: 100.0 %, DrawRatio: 33.9 %
731 of 1000 games finished.

The 'interim' result hovers nearly precisely around the difference CCRL gives for both.

So either your tests are somehow wrong, or in fact both programs behave very different in those ditrimental time controls.
May be good time to create a new thread about time management, because I think a lot of programs dump some low hanging fruits here.

My games will be available after the match is finished.

[Desperado]

IMHO you very much understimate the difference of mps and base+inc time controls and differences in time management for both.
As I was therefore not satisfied with your second test which just multiplied your first tc I started my own test yesterday!

In my test Xiphos 0.22 (I already explained that 0.2.2 and 0.2 are identically regarding this test and their results are also gathered in CCRL accordingly - the reason is, I just had no working 0.2 compilation handy and was too lazy to build for this test) played against Demolito 20181029, both 16MB. but with a tc of 40/10, which has a similar average time per move as your original 10+0.1.
Of course I was prepared that my thesis could be incorrect and the result would show it.

Actually the goal was to play 1000 games and I will at least let the match run as long, but the current result (>700 games) already with its error bars
and LOS leaves no doubt that my thesis seems correct.

Code: Select all

Score of Xiphos_022-64 vs Demolito_20181029-64: 288 - 195 - 248 [0.564]
...      Xiphos_022-64 playing White: 168 - 84 - 114  [0.615] 366
...      Xiphos_022-64 playing Black: 120 - 111 - 134  [0.512] 365
...      White vs Black: 279 - 204 - 248  [0.551] 731
Elo difference: 44.4 +/- 20.5, LOS: 100.0 %, DrawRatio: 33.9 %
731 of 1000 games finished.

The 'interim' result hovers nearly precisely around the difference CCRL gives for both.

So either your tests are somehow wrong, or in fact both programs behave very different in those ditrimental time controls.
May be good time to create a new thread about time management, because I think a lot of programs dump some low hanging fruits here.

My games will be available after the match is finished.

Thanks Guenther, that is useful information! I have to come back later because no time left now.
I guess pondering was disabled and what book did you use ? I would like to use the same book too and make
my own test for the mentioned time control.

Well, it would be very interesting if the outcome would say, that although the average time per move is the same
the modus can make such a big difference. Lets' see...

Thank you for your effort, i appreciate it very much.

Regards.

[Guenther]

IMHO you very much understimate the difference of mps and base+inc time controls and differences in time management for both.
As I was therefore not satisfied with your second test which just multiplied your first tc I started my own test yesterday!

...

Code: Select all

Score of Xiphos_022-64 vs Demolito_20181029-64: 288 - 195 - 248 [0.564]
...      Xiphos_022-64 playing White: 168 - 84 - 114  [0.615] 366
...      Xiphos_022-64 playing Black: 120 - 111 - 134  [0.512] 365
...      White vs Black: 279 - 204 - 248  [0.551] 731
Elo difference: 44.4 +/- 20.5, LOS: 100.0 %, DrawRatio: 33.9 %
731 of 1000 games finished.

...

Thanks Guenther, that is useful information! I have to come back later because no time left now.
I guess pondering was disabled and what book did you use ? I would like to use the same book too and make
my own test for the mentioned time control.

Well, it would be very interesting if the outcome would say, that although the average time per move is the same
the modus can make such a big difference. Lets' see...

Thank you for your effort, i appreciate it very much.

Regards.

Michael, you can download the start positions pgn used (6 plies) from my download site.
https://rwbc-chess.de/download.htm

And yes ofc it was ponder off and 1 core each. Except hash 16MB, all settings default.
(CuteChessGUI-dev build with various enhancements from alweys' EXPER1 branch and a few tiny changes by myself)

Standing now:

Code: Select all

Score of Xiphos_022-64 vs Demolito_20181029-64: 350 - 232 - 298 [0.567]
...      Xiphos_022-64 playing White: 200 - 98 - 142  [0.616] 440
...      Xiphos_022-64 playing Black: 150 - 134 - 156  [0.518] 440
...      White vs Black: 334 - 248 - 298  [0.549] 880
Elo difference: 46.9 +/- 18.7, LOS: 100.0 %, DrawRatio: 33.9 %
880 of 1000 games finished.

[Guenther]

The final result (calculated directly in ordo now with 3000 base and 200 simuls):

Code: Select all

   # PLAYER                  :   RATING  ERROR  POINTS  PLAYED    (%)  CFS(%)
   1 Xiphos_022-64           :  3023.13   9.13   565.0    1000  56.50     100
   2 Demolito_20181029-64    :  2976.87   9.13   435.0    1000  43.50     ---
White advantage = 34.33 +/- 8.71
Draw rate (equal opponents) = 34.08 % +/- 1.55

Code: Select all

Head to head statistics:

1) Xiphos_022-64        3023.13 :   1000 (+398,=334,-268),  56.5 %

   vs.                          :  games (   +,   =,   -),   (%) :     Diff,     SD, CFS (%)
   Demolito_20181029-64         :   1000 ( 398, 334, 268),  56.5 :   +46.26,   9.31,  100.0

2) Demolito_20181029-64 2976.87 :   1000 (+268,=334,-398),  43.5 %

   vs.                          :  games (   +,   =,   -),   (%) :     Diff,     SD, CFS (%)
   Xiphos_022-64                :   1000 ( 268, 334, 398),  43.5 :   -46.26,   9.31,    0.0

I noticed that during the last 200 games, five time losses (only Demolito) appeared, may be it was after I had reactivated
my kaspersky (in game mode though of course) for posting here and doing some internet related work.
There were zero time losses in the first 800 games over night, while practically all other apps were closed and kaspersky was disabled.
It should be mentioned that Demolito 20181029s time management appears to be very risky anyway for mps.

Also there was one illegal move from Demolito, which remembered me, it could send an illegal bestmove 0000,
(and/or ponder 0000, while not pondering at all) under seldom circumstances.
See this thread:
http://talkchess.com/forum3/viewtopic.p ... 20#p778664

[Event "Test@CuteCGdv"]
[Site "RWBC-CAPPUCCINO Win7U64 Q8200 2.33Ghz + Nvidia GT 710"]
[Date "2021.03.26"]
[Round "432"]
[White "Xiphos_022-64"]
[Black "Demolito_20181029-64"]
[Result "1-0"]
[ECO "D00"]
[Opening "Levitsky attack (Queen's Bishop attack)"]
[PlyCount "69"]
[Termination "illegal move"]
[TimeControl "40/10"]

1. d4 {book} d5 {book}
2. Bg5 {book} c6 {book}
3. e3 {book} h6 {book}
4. Bh4 {-0.12/14 0.40} Qb6 {+0.26/11 0.48}
5. b3 {-0.04/14 0.39} Nf6 {+0.18/10 0.49}
6. Nd2 {+0.06/13 0.36} c5 {+0.21/10 0.50}
7. dxc5 {-0.03/14 0.36} Qxc5 {+0.21/11 0.46}
8. Ngf3 {-0.03/14 0.34} g5 {+0.15/10 0.45}
9. Bg3 {-0.04/12 0.35} Bg7 {+0.21/9 0.47}
10. c4 {-0.16/12 0.32} O-O {+0.02/9 0.43}
11. a3 {+0.22/11 0.30} Bf5 {+0.02/9 0.43}
12. b4 {+0.43/13 0.29} Qc8 {+0.40/10 0.40}
13. Rc1 {+0.60/12 0.28} Ne4 {+0.40/11 0.37}
14. cxd5 {+1.62/13 0.31} Qd8 {-0.99/10 0.39}
15. Nxe4 {+1.41/14 0.26} Bxe4 {-0.72/10 0.33}
16. h4 {+1.48/13 0.26} Qxd5 {-0.77/9 0.40}
17. hxg5 {+1.86/12 0.25} Qxd1+ {-1.13/10 0.34}
18. Rxd1 {+2.19/13 0.24} Bc3+ {-1.10/10 0.30}
19. Ke2 {+2.28/13 0.25} Bc6 {-0.82/11 0.28}
20. Nd4 {+2.01/15 0.25} Bxd4 {-1.22/11 0.26}
21. Rxd4 {+2.00/15 0.24} Bb5+ {-1.32/11 0.25}
22. Kf3 {+2.00/16 0.25} Bxf1 {-1.17/12 0.25}
23. Rxf1 {+1.99/16 0.25} hxg5 {-1.23/12 0.22}
24. Rc1 {+2.30/14 0.25} b6 {-1.38/12 0.21}
25. Rc7 {+2.51/12 0.25} f6 {-0.84/11 0.25}
26. b5 {+2.91/14 0.24} Re8 {-1.20/12 0.24}
27. a4 {+2.60/13 0.24} Kf8 {-1.25/10 0.22}
28. Rb7 {+2.31/14 0.24} Rc8 {-1.32/12 0.20}
29. Rd1 {+2.49/16 0.24} Ke8 {-1.77/11 0.18}
30. Bh2 {+2.50/17 0.23} a6 {-1.54/10 0.19}
31. g4 {+3.32/14 0.23} axb5 {-2.06/10 0.19}
32. axb5 {+3.10/16 0.23} e5 {-1.91/10 0.16}
33. Rd6 {+3.26/16 0.23} Ra2 {-2.05/10 0.15}
34. Rxf6 {+3.43/16 0.33} Nd7 {-2.58/10 0.14}
35. Rd6 {+3.43/16 0.21}
{Black makes an illegal move: 0000}
1-0

Of course those things don't make the test invalid.
(games available on request)

[Desperado]

The final result (calculated directly in ordo now with 3000 base and 200 simuls):

Code: Select all

   # PLAYER                  :   RATING  ERROR  POINTS  PLAYED    (%)  CFS(%)
   1 Xiphos_022-64           :  3023.13   9.13   565.0    1000  56.50     100
   2 Demolito_20181029-64    :  2976.87   9.13   435.0    1000  43.50     ---
White advantage = 34.33 +/- 8.71
Draw rate (equal opponents) = 34.08 % +/- 1.55

Code: Select all

Head to head statistics:

1) Xiphos_022-64        3023.13 :   1000 (+398,=334,-268),  56.5 %

   vs.                          :  games (   +,   =,   -),   (%) :     Diff,     SD, CFS (%)
   Demolito_20181029-64         :   1000 ( 398, 334, 268),  56.5 :   +46.26,   9.31,  100.0

2) Demolito_20181029-64 2976.87 :   1000 (+268,=334,-398),  43.5 %

   vs.                          :  games (   +,   =,   -),   (%) :     Diff,     SD, CFS (%)
   Xiphos_022-64                :   1000 ( 268, 334, 398),  43.5 :   -46.26,   9.31,    0.0

I noticed that during the last 200 games, five time losses (only Demolito) appeared, may be it was after I had reactivated
my kaspersky (in game mode though of course) for posting here and doing some internet related work.
There were zero time losses in the first 800 games over night, while practically all other apps were closed and kaspersky was disabled.
It should be mentioned that Demolito 20181029s time management appears to be very risky anyway for mps.

Also there was one illegal move from Demolito, which remembered me, it could send an illegal bestmove 0000,
(and/or ponder 0000, while not pondering at all) under seldom circumstances.
See this thread:
http://talkchess.com/forum3/viewtopic.p ... 20#p778664

[Event "Test@CuteCGdv"]
[Site "RWBC-CAPPUCCINO Win7U64 Q8200 2.33Ghz + Nvidia GT 710"]
[Date "2021.03.26"]
[Round "432"]
[White "Xiphos_022-64"]
[Black "Demolito_20181029-64"]
[Result "1-0"]
[ECO "D00"]
[Opening "Levitsky attack (Queen's Bishop attack)"]
[PlyCount "69"]
[Termination "illegal move"]
[TimeControl "40/10"]

1. d4 {book} d5 {book}
2. Bg5 {book} c6 {book}
3. e3 {book} h6 {book}
4. Bh4 {-0.12/14 0.40} Qb6 {+0.26/11 0.48}
5. b3 {-0.04/14 0.39} Nf6 {+0.18/10 0.49}
6. Nd2 {+0.06/13 0.36} c5 {+0.21/10 0.50}
7. dxc5 {-0.03/14 0.36} Qxc5 {+0.21/11 0.46}
8. Ngf3 {-0.03/14 0.34} g5 {+0.15/10 0.45}
9. Bg3 {-0.04/12 0.35} Bg7 {+0.21/9 0.47}
10. c4 {-0.16/12 0.32} O-O {+0.02/9 0.43}
11. a3 {+0.22/11 0.30} Bf5 {+0.02/9 0.43}
12. b4 {+0.43/13 0.29} Qc8 {+0.40/10 0.40}
13. Rc1 {+0.60/12 0.28} Ne4 {+0.40/11 0.37}
14. cxd5 {+1.62/13 0.31} Qd8 {-0.99/10 0.39}
15. Nxe4 {+1.41/14 0.26} Bxe4 {-0.72/10 0.33}
16. h4 {+1.48/13 0.26} Qxd5 {-0.77/9 0.40}
17. hxg5 {+1.86/12 0.25} Qxd1+ {-1.13/10 0.34}
18. Rxd1 {+2.19/13 0.24} Bc3+ {-1.10/10 0.30}
19. Ke2 {+2.28/13 0.25} Bc6 {-0.82/11 0.28}
20. Nd4 {+2.01/15 0.25} Bxd4 {-1.22/11 0.26}
21. Rxd4 {+2.00/15 0.24} Bb5+ {-1.32/11 0.25}
22. Kf3 {+2.00/16 0.25} Bxf1 {-1.17/12 0.25}
23. Rxf1 {+1.99/16 0.25} hxg5 {-1.23/12 0.22}
24. Rc1 {+2.30/14 0.25} b6 {-1.38/12 0.21}
25. Rc7 {+2.51/12 0.25} f6 {-0.84/11 0.25}
26. b5 {+2.91/14 0.24} Re8 {-1.20/12 0.24}
27. a4 {+2.60/13 0.24} Kf8 {-1.25/10 0.22}
28. Rb7 {+2.31/14 0.24} Rc8 {-1.32/12 0.20}
29. Rd1 {+2.49/16 0.24} Ke8 {-1.77/11 0.18}
30. Bh2 {+2.50/17 0.23} a6 {-1.54/10 0.19}
31. g4 {+3.32/14 0.23} axb5 {-2.06/10 0.19}
32. axb5 {+3.10/16 0.23} e5 {-1.91/10 0.16}
33. Rd6 {+3.26/16 0.23} Ra2 {-2.05/10 0.15}
34. Rxf6 {+3.43/16 0.33} Nd7 {-2.58/10 0.14}
35. Rd6 {+3.43/16 0.21}
{Black makes an illegal move: 0000}
1-0

Of course those things don't make the test invalid.
(games available on request)

Thx again.

Here are some quenstions on the time control and cutechess-cli. (Only used fixed time with and without inc so far).
Let's say i use 40/60 (40 moves in a 60s "repeating"), how is the behaviour in cutechess-cli ?

1. Move 40 in 55s -> move 40...80 in 65 seconds ?
2. Move 40 in 55s -> move 40...80 in 60 seconds again (wasting 5s) ?
3. Will the time control repeated automatically (how often) if move 40 was in time?
4. Can the time control be combined like 40/60s and 20s for the rest?

[Guenther]

Thx again.

Here are some quenstions on the time control and cutechess-cli. (Only used fixed time with and without inc so far).
Let's say i use 40/60 (40 moves in a 60s "repeating"), how is the behaviour in cutechess-cli ?

1. Move 40 in 55s -> move 40...80 in 65 seconds ?
2. Move 40 in 55s -> move 40...80 in 60 seconds again (wasting 5s) ?
3. Will the time control repeated automatically (how often) if move 40 was in time?
4. Can the time control be combined like 40/60s and 20s for the rest?

I am not sure I understand your question, mps means always repeating the same tc forever, the is no 'behaviour' of cutechess or cutechess-cli or any GUI, e.g. 40/60 = 1-40 in 60s, 41-80 in 60s, 81-120 in 60s ad nauseam.

Edit: (now I realized your problem with '2.')
Of course there is nothing to be wasted, rest time always remains and new time will be added to it.
Did you really never test this 'standard' tc?

Yes it can be combined at least with inc (it seems not with a rest, at least not in cute + cute-cli + WB, IIRC Arena can set up to three different mps in a game, but who needs this?), but that is a relatively new thing and I guess a lot of programs have problems with this, newer programs probably not(?), moreover I have (nearly) zero experience with this mixed tcs (for me those are still exotic).