Ordo calculating incorrect number of games played

[KhepriChess]

Hey all. Sorry for the noob question here, but I can't quite understand this.

I'm trying to test a change to my engine, so I'm doing a head-to-head using the cutechess-cli (just the new version playing the old/current version). The test completed (sprt stopped at about 800) and gave it said "Elo difference: 41.7 +/- 21.9" (and for what it's worth, I ran another test of 10,000 games and got round a 35 Elo difference reported then).

But then when I take that pgn file and run it in Ordo, it gives me back only a 5 Elo difference and the number of games it says has been played is way off (it says 2300 instead of 800). But I'm not sure why that is. I've looked at the pgn file and there are 803 completed games, plus 2 incomplete games.

If it helps, my cutechess command:

Code: Select all

cutechess-cli -tournament gauntlet -concurrency 3 -pgnout result.pgn -engine conf=khepricandidate tc=0/10+0.1 -engine conf=khepribase tc=0/10+0.1 -draw movenumber=40 movecount=5 score=8 -resign movecount=5 score=1000 -each proto=uci -openings file="openings-6ply-1000.pgn" order=random policy=round -repeat -rounds 2000 -games 2 -sprt elo0=0 elo1=10 alpha=0.05 beta=0.05

[gaard]

What's the ordo command look like? Can you link the pgn?

[op12no2]

2300 just happens to be the default Ordo rating...

[KhepriChess]

What's the ordo command look like? Can you link the pgn?

Just a simple "ordo -p cutechess.pgn". I've tried variations on that command but they always return roughly the same thing. The file:
https://download1589.mediafire.com/qwhj ... echess.pgn

2300 just happens to be the default Ordo rating...

It's the number of games that Ordo says has been played (2300) which isn't correct. Not sure why there'd be default number of games.

[Guenther]

What's the ordo command look like? Can you link the pgn?

Just a simple "ordo -p cutechess.pgn". I've tried variations on that command but they always return roughly the same thing. The file:
https://download1589.mediafire.com/qwhj ... echess.pgn

2300 just happens to be the default Ordo rating...

It's the number of games that Ordo says has been played (2300) which isn't correct. Not sure why there'd be default number of games.

I get this (avg. rating set to 2000) with 200 simuls:

Code: Select all

   # PLAYER             :   RATING  ERROR  POINTS  PLAYED    (%)  CFS(%)
   1 khepricandidate    :  2002.72   6.06  1212.0    2387  50.78      81
   2 khepribase         :  1997.28   6.06  1175.0    2387  49.22     ---

White advantage = 9.25 +/- 6.10
Draw rate (equal opponents) = 19.37 % +/- 0.80

It shows 2387 games are considered, the pgn has 2391 games (I just did a quick count for the 'Event' tag), but four games are unfinished...
Thus it seems you have appended games from several tests (all on the same day though) to the same pgn file.

BTW if I am not mistaken you use the Pesto PST. IMHO this should be commented somehow and somewhere.

Edit:
It seems also, you have deleted versions 0.50 and 0.60, which were already released? At least one of them was already tested at CCRL.

[Guenther]

Removing the other samples from the pgn file, the output (again with 200 simuls) is this:
That's probably what you expected ;-)

Code: Select all

   # PLAYER             :   RATING  ERROR  POINTS  PLAYED    (%)  CFS(%)
   1 khepricandidate    :  2021.20  10.83   449.5     803  55.98     100
   2 khepribase         :  1978.80  10.83   353.5     803  44.02     ---

White advantage = 31.03 +/- 10.06
Draw rate (equal opponents) = 17.80 % +/- 1.40

[KhepriChess]

Ah thanks. Seems like a "duh", that I should have just looked at the pgn to see how many games were in there. Not entirely sure why there's more, but thanks for figuring that out.

It seems also, you have deleted versions 0.50 and 0.60, which were already released? At least one of them was already tested at CCRL.

I did, as I decided to rework how I'm doing releases (basically I realized that it'll be hard to keep track of the version number if I'm checking in single changes at a time). Should the previous versions still be available? Technically the 1.0.0 is the exact same as 0.6.

BTW if I am not mistaken you use the Pesto PST. IMHO this should be commented somehow and somewhere.

You're right and I will.

[Guenther]

Ah thanks. Seems like a "duh", that I should have just looked at the pgn to see how many games were in there. Not entirely sure why there's more, but thanks for figuring that out.

It seems also, you have deleted versions 0.50 and 0.60, which were already released? At least one of them was already tested at CCRL.

I did, as I decided to rework how I'm doing releases (basically I realized that it'll be hard to keep track of the version number if I'm checking in single changes at a time). Should the previous versions still be available? Technically the 1.0.0 is the exact same as 0.6.

I checked CCRL again and I would prefer the initial release 0.5.0., which was already tested, would be still there and I guess CCRL
will conclude the same.

Code: Select all

567	KhepriChess 0.5.0 64-bit	1716	+22	−21	47.6%	+20.0	19.1%	843

Thanks for considering. If you are very unhappy with 0.5.0, I can understand though that you won't show it under releases.
I can still get it as it was under the github history from here, but people who don't know github probably won't find it:
https://github.com/kurt1288/KhepriChess ... 5b498a1944

BTW if I am not mistaken you use the Pesto PST. IMHO this should be commented somehow and somewhere.

You're right and I will.

That's fine.

BTW does it need a minimum version of Nodes, as I am limited to versions, which still run under Win7?

[KhepriChess]

BTW does it need a minimum version of Nodes, as I am limited to versions, which still run under Win7?

My build files are targeting ES2020, which I believe is fully supported in Node 14 and above.

[Guenther]

BTW does it need a minimum version of Nodes, as I am limited to versions, which still run under Win7?

My build files are targeting ES2020, which I believe is fully supported in Node 14 and above.

I tested it right now (from cmd only) and it worked ok with Nodes 11.0.0
This reminded me though that I had built LiGround a while ago and I see from there that I could use up to 13.14.
forum3/viewtopic.php?f=2&t=76682&p=9149 ... js#p914914

Not too much diff between 0.5.0 and 1.0.0, but starting with depth 5 the newer one needs to search
some nodes less for same depth.

Code: Select all

1642529428.503 POLYGLOT *** Switching to UCI mode ***
1642529428.503 Adapter->GUI: id name KhepriChess 1.0.0
1642529428.504 Adapter->GUI: id author Kurt Peters
1642529428.504 Adapter->GUI: option name Hash type spin default 32 min 1 max 512
1642529428.505 Adapter->GUI: option name Polyglot Log type check default true
1642529428.505 Adapter->GUI: option name Polyglot LogFile type string default KhepriChess_100-64.log
1642529428.506 Adapter->GUI: option name Polyglot UseNice type check default false
1642529428.506 Adapter->GUI: option name Polyglot NiceValue type spin default 5 min 0 max 20
1642529428.506 Adapter->GUI: option name Polyglot Book type check default false
1642529428.507 Adapter->GUI: option name Polyglot BookFile type string default book.bin
1642529428.507 Adapter->GUI: option name Polyglot BookRandom type check default true
1642529428.507 Adapter->GUI: option name Polyglot BookDepth type spin default 256 min 0 max 256
1642529428.507 Adapter->GUI: option name Polyglot BookTreshold type spin default 5 min 0 max 1000
1642529428.508 Adapter->GUI: uciok
1642529430.388 GUI->Adapter: go
1642529430.388 Adapter->Engine: go
1642529430.390 Engine->Adapter: Move time: 0
1642529430.390 Adapter->GUI: Move time: 0
1642529430.403 Engine->Adapter: info score cp 36 depth 1 nodes 41 time 11 pv g1f3 
1642529430.403 Adapter->GUI: info score cp 36 depth 1 nodes 41 time 11 pv g1f3 
1642529430.419 Engine->Adapter: info score cp 0 depth 2 nodes 140 time 27 pv g1f3 g8f6 
1642529430.419 Adapter->GUI: info score cp 0 depth 2 nodes 140 time 27 pv g1f3 g8f6 
1642529430.452 Engine->Adapter: info score cp 35 depth 3 nodes 412 time 60 pv g1f3 g8f6 d2d4 
1642529430.452 Adapter->GUI: info score cp 35 depth 3 nodes 412 time 60 pv g1f3 g8f6 d2d4 
1642529430.843 Engine->Adapter: info score cp 0 depth 4 nodes 3015 time 450 pv g1f3 g8f6 d2d4 d7d5 
1642529430.843 Adapter->GUI: info score cp 0 depth 4 nodes 3015 time 450 pv g1f3 g8f6 d2d4 d7d5 
1642529431.070 Engine->Adapter: info score cp 33 depth 5 nodes 5993 time 678 pv g1f3 g8f6 d2d4 d7d5 b1c3 
1642529431.070 Adapter->GUI: info score cp 33 depth 5 nodes 5993 time 678 pv g1f3 g8f6 d2d4 d7d5 b1c3 
1642529432.023 Engine->Adapter: info score cp 0 depth 6 nodes 17437 time 1632 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 
1642529432.023 Adapter->GUI: info score cp 0 depth 6 nodes 17437 time 1632 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 
1642529433.136 Engine->Adapter: info score cp 28 depth 7 nodes 33810 time 2745 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 c1e3 
1642529433.136 Adapter->GUI: info score cp 28 depth 7 nodes 33810 time 2745 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 c1e3 
1642529441.779 Engine->Adapter: info score cp 8 depth 8 nodes 153817 time 11388 pv e2e4 b8c6 g1f3 e7e5 d2d4 g8f6 b1c3 f8d6 
1642529441.780 Adapter->GUI: info score cp 8 depth 8 nodes 153817 time 11388 pv e2e4 b8c6 g1f3 e7e5 d2d4 g8f6 b1c3 f8d6 

Code: Select all

1642529198.734 POLYGLOT *** Switching to UCI mode ***
1642529198.734 Adapter->GUI: id name KhepriChess 0.5.0
1642529198.735 Adapter->GUI: id author Kurt Peters
1642529198.735 Adapter->GUI: option name Hash type spin default 32 min 1 max 512
1642529198.735 Adapter->GUI: option name Polyglot Log type check default true
1642529198.736 Adapter->GUI: option name Polyglot LogFile type string default KhepriChess_050-64.log
1642529198.736 Adapter->GUI: option name Polyglot UseNice type check default false
1642529198.737 Adapter->GUI: option name Polyglot NiceValue type spin default 5 min 0 max 20
1642529198.737 Adapter->GUI: option name Polyglot Book type check default false
1642529198.737 Adapter->GUI: option name Polyglot BookFile type string default book.bin
1642529198.738 Adapter->GUI: option name Polyglot BookRandom type check default true
1642529198.738 Adapter->GUI: option name Polyglot BookDepth type spin default 256 min 0 max 256
1642529198.739 Adapter->GUI: option name Polyglot BookTreshold type spin default 5 min 0 max 1000
1642529198.739 Adapter->GUI: uciok
1642529203.085 GUI->Adapter: go
1642529203.085 Adapter->Engine: go
1642529203.087 Engine->Adapter: Move time: 0
1642529203.087 Adapter->GUI: Move time: 0
1642529203.099 Engine->Adapter: info score cp 36 depth 1 nodes 41 time 12 pv g1f3 
1642529203.099 Adapter->GUI: info score cp 36 depth 1 nodes 41 time 12 pv g1f3 
1642529203.114 Engine->Adapter: info score cp 0 depth 2 nodes 140 time 27 pv g1f3 g8f6 
1642529203.114 Adapter->GUI: info score cp 0 depth 2 nodes 140 time 27 pv g1f3 g8f6 
1642529203.144 Engine->Adapter: info score cp 35 depth 3 nodes 412 time 57 pv g1f3 g8f6 d2d4 
1642529203.144 Adapter->GUI: info score cp 35 depth 3 nodes 412 time 57 pv g1f3 g8f6 d2d4 
1642529203.479 Engine->Adapter: info score cp 0 depth 4 nodes 3015 time 392 pv g1f3 g8f6 d2d4 d7d5 
1642529203.479 Adapter->GUI: info score cp 0 depth 4 nodes 3015 time 392 pv g1f3 g8f6 d2d4 d7d5 
1642529203.706 Engine->Adapter: info score cp 33 depth 5 nodes 6050 time 618 pv g1f3 g8f6 d2d4 d7d5 b1c3 
1642529203.706 Adapter->GUI: info score cp 33 depth 5 nodes 6050 time 618 pv g1f3 g8f6 d2d4 d7d5 b1c3 
1642529204.674 Engine->Adapter: info score cp 0 depth 6 nodes 17620 time 1587 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 
1642529204.674 Adapter->GUI: info score cp 0 depth 6 nodes 17620 time 1587 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 
1642529205.790 Engine->Adapter: info score cp 28 depth 7 nodes 34140 time 2703 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 c1e3 
1642529205.790 Adapter->GUI: info score cp 28 depth 7 nodes 34140 time 2703 pv g1f3 g8f6 d2d4 d7d5 b1c3 b8c6 c1e3 
1642529214.593 Engine->Adapter: info score cp 8 depth 8 nodes 156686 time 11506 pv e2e4 b8c6 g1f3 e7e5 d2d4 g8f6 b1c3 f8d6 
1642529214.593 Adapter->GUI: info score cp 8 depth 8 nodes 156686 time 11506 pv e2e4 b8c6 g1f3 e7e5 d2d4 g8f6 b1c3 f8d6 
1642529228.074 Engine->Adapter: info score cp 35 depth 9 nodes 341426 time 24987 pv e2e4 b8c6 d2d4 e7e6 g1f3 g8f6 b1c3 d7d5 c1g5 
1642529228.074 Adapter->GUI: info score cp 35 depth 9 nodes 341426 time 24987 pv e2e4 b8c6 d2d4 e7e6 g1f3 g8f6 b1c3 d7d5 c1g5