Pedantic

[JoAnnP38]

I am getting some unexpected results in a 500 game match between Pedantic 0.30 and 0.2.1.

Cutechess with time control of 10+0.1 seconds and all adjudication turned off and no opening book used. I have the PGN if it may be useful.

Code: Select all

Score of Pedantic 0.3.0 vs Pedantic 0.2.1: 187 - 232 - 81 [0.455]
...      Pedantic 0.3.0 playing White: 93 - 105 - 52  [0.476] 250
...      Pedantic 0.3.0 playing Black: 94 - 127 - 29  [0.434] 250
...      White vs Black: 220 - 199 - 81  [0.521] 500
Elo difference: -31.4 +/- 28.0, LOS: 1.4 %, DrawRatio: 16.2 %
500 of 500 games finished.

Player: Pedantic 0.3.0
   "Draw by 3-fold repetition": 13
   "Draw by fifty moves rule": 17
   "Draw by insufficient mating material": 51
   "Loss: Black mates": 105
   "Loss: White mates": 127
   "Win: Black mates": 94
   "Win: White mates": 93
Player: Pedantic 0.2.1
   "Draw by 3-fold repetition": 13
   "Draw by fifty moves rule": 17
   "Draw by insufficient mating material": 51
   "Loss: Black mates": 94
   "Loss: White mates": 93
   "Win: Black mates": 105
   "Win: White mates": 127

What is your cpu?

Intel i7-12700K 16GB memory all settings for Pedantic default (other than no Own Book). Concurrent games 6 but Ponder is off so with 8 performance cores should be not an issue.

Can you rerun at longer TC (like 40+0.4?) You can drop the number of games to 100-200.

[JoAnnP38]

While I almost never test at such short TC, I did rerun your test with 10+0.1 to make sure I didn't miss something. My results seem normal. Perhaps the win% is a little lower than at longer TC but since this release has a redesigned game clock I can probably attribute it to that. Here is what typical results might look like:

Code: Select all

Score of Pedantic 0.3 vs Pedantic 0.2: 385 - 46 - 69  [0.839] 500
...      Pedantic 0.3 playing White: 192 - 25 - 33  [0.834] 250
...      Pedantic 0.3 playing Black: 193 - 21 - 36  [0.844] 250
...      White vs Black: 213 - 218 - 69  [0.495] 500
Elo difference: 286.8 +/- 36.0, LOS: 100.0 %, DrawRatio: 13.8 %

Double check that the EXE you are using for Pedantic 0.3 is the correct one. I do not rename Pedantic for each version so its possible to get them mixed up. If you are using an Engines.json with cutechess, make sure it has been updated correctly. If that all checks out here as some things you can do to make sure you are using the correct exe. If you run Pedantic 0.3 from the command line this is what you will see:

Code: Select all

e:\releases\Pedantic 0.3>pedantic
Pedantic 0.3.0

Also, if from Windows File Explorer, you right-click on what you believe is the Pedantic 0.3 exe and select "properties", you can navigate to the Details tab and check that the file version is 0.3.0.0 and the product version is 0.3.0.

Your CPU fully supports BMI2 processor extensions and there are no issues with the PEXT/PDEP instructions as with some AMD processors, so my suspicion is that somehow you are not referencing the correct exe. Unfortunately, the only CPUs I have available to me are Ryzen 9 6900HX and a Ryzen 4500U. If it does end-up having something to do with a processor incompatibility I would never be able to check that from my end.

[adnoh]

While I almost never test at such short TC, I did rerun your test with 10+0.1 to make sure I didn't miss something. My results seem normal. Perhaps the win% is a little lower than at longer TC but since this release has a redesigned game clock I can probably attribute it to that. Here is what typical results might look like:

Code: Select all

Score of Pedantic 0.3 vs Pedantic 0.2: 385 - 46 - 69  [0.839] 500
...      Pedantic 0.3 playing White: 192 - 25 - 33  [0.834] 250
...      Pedantic 0.3 playing Black: 193 - 21 - 36  [0.844] 250
...      White vs Black: 213 - 218 - 69  [0.495] 500
Elo difference: 286.8 +/- 36.0, LOS: 100.0 %, DrawRatio: 13.8 %

Double check that the EXE you are using for Pedantic 0.3 is the correct one. I do not rename Pedantic for each version so its possible to get them mixed up. If you are using an Engines.json with cutechess, make sure it has been updated correctly. If that all checks out here as some things you can do to make sure you are using the correct exe. If you run Pedantic 0.3 from the command line this is what you will see:

Code: Select all

e:\releases\Pedantic 0.3>pedantic
Pedantic 0.3.0

Also, if from Windows File Explorer, you right-click on what you believe is the Pedantic 0.3 exe and select "properties", you can navigate to the Details tab and check that the file version is 0.3.0.0 and the product version is 0.3.0.

Your CPU fully supports BMI2 processor extensions and there are no issues with the PEXT/PDEP instructions as with some AMD processors, so my suspicion is that somehow you are not referencing the correct exe. Unfortunately, the only CPUs I have available to me are Ryzen 9 6900HX and a Ryzen 4500U. If it does end-up having something to do with a processor incompatibility I would never be able to check that from my end.

500 games at 40+0.4 TC

Code: Select all

Score of Pedantic 0.3.0 vs Pedantic 0.2.1: 117 - 241 - 142 [0.376]
...      Pedantic 0.3.0 playing White: 77 - 108 - 65  [0.438] 250
...      Pedantic 0.3.0 playing Black: 40 - 133 - 77  [0.314] 250
...      White vs Black: 210 - 148 - 142  [0.562] 500
Elo difference: -88.0 +/- 26.3, LOS: 0.0 %, DrawRatio: 28.4 %
500 of 500 games finished.

Player: Pedantic 0.3.0
   "Draw by 3-fold repetition": 50
   "Draw by fifty moves rule": 44
   "Draw by insufficient mating material": 46
   "Draw by stalemate": 2
   "Loss: Black mates": 108
   "Loss: White mates": 133
   "Win: Black mates": 40
   "Win: White mates": 77
Player: Pedantic 0.2.1
   "Draw by 3-fold repetition": 50
   "Draw by fifty moves rule": 44
   "Draw by insufficient mating material": 46
   "Draw by stalemate": 2
   "Loss: Black mates": 40
   "Loss: White mates": 77
   "Win: Black mates": 108
   "Win: White mates": 133

I keep each version of Pedantic in its own subdirectory and the Properties tab does show it as File version and Product version 0.3.0. If anything, I noticed for both Pedantic 0.1 and 0.2.1 it shows as version 0.1.0

Code: Select all

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.2.1]Pedantic.exe uci
Pedantic v0.2.0
uci
id name Pedantic v0.2.0
id author JoAnn D. Peeler
option name OwnBook type check default true
option name Ponder type check default true
option name Hash type spin default 64 min 1 max 2048
option name Clear Hash type button
option name MaxThreads type spin default 1 min 1 max 18
option name UCI_EngineAbout type string default Pedantic v0.2.0 by JoAnn D. Peeler, see https://github.com/JoAnnP38/Pedantic
option name Evaluation_ID type string default <empty>
option name Random_Search type check default false
option name Collect_Stats type check default false
uciok
quit

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.2.1]pedan\

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.3]Pedantic.exe
Pedantic 0.3.0
uci
id name Pedantic 0.3.0
id author JoAnn D. Peeler
option name OwnBook type check default true
option name Ponder type check default true
option name Hash type spin default 64 min 1 max 2048
option name Clear Hash type button
option name UCI_EngineAbout type string default Pedantic 0.3.0 by JoAnn D. Peeler, see https://github.com/JoAnnP38/Pedantic
option name Evaluation_ID type string default <empty>
option name Random_Search type check default false
option name Collect_Stats type check default false
uciok
quit

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.3]

Cutechess engines.json

Code: Select all

	{
		"command" : "Pedantic.exe uci",
		"name" : "Pedantic 0.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.1"
	},

	{
		"command" : "Pedantic.exe uci",
		"name" : "Pedantic 0.2.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1"
	},

	{
		"command" : "Pedantic.exe",
		"name" : "Pedantic 0.3.0",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3"
	},

[Guenther]

I am getting some unexpected results in a 500 game match between Pedantic 0.30 and 0.2.1.

Cutechess with time control of 10+0.1 seconds and all adjudication turned off and no opening book used. I have the PGN if it may be useful.
...

Yes, please offer the pgn for download, I would like to check for validation.
(own book = true for both versions of it is an unusual testing BTW, no book/start position at all for both like you told above would be also
quite unusual testing and they might play just a lot of doubles or very few lines)

[Gabor Szots]

Intel i7-12700K 16GB memory all settings for Pedantic default (other than no Own Book).

I have the same CPU, currently running a blitz tournament for CCRL, and interim results for Pedantic 0.3 are even better than expected.

I have to add that this CPU is tricky. I did a lot of experiments till I arrived at my current setup: all cores (even P-cores) are set to 3.6 GHz and Windows 11 power scheme is set to highest performance. This was the only way I could find to ensure all cores run at the same speed. Even then, when I test on more than 6 threads simultaneously, benchmarks start to worsen.

[JoAnnP38]

500 games at 40+0.4 TC

Code: Select all

Score of Pedantic 0.3.0 vs Pedantic 0.2.1: 117 - 241 - 142 [0.376]
...      Pedantic 0.3.0 playing White: 77 - 108 - 65  [0.438] 250
...      Pedantic 0.3.0 playing Black: 40 - 133 - 77  [0.314] 250
...      White vs Black: 210 - 148 - 142  [0.562] 500
Elo difference: -88.0 +/- 26.3, LOS: 0.0 %, DrawRatio: 28.4 %
500 of 500 games finished.

Player: Pedantic 0.3.0
   "Draw by 3-fold repetition": 50
   "Draw by fifty moves rule": 44
   "Draw by insufficient mating material": 46
   "Draw by stalemate": 2
   "Loss: Black mates": 108
   "Loss: White mates": 133
   "Win: Black mates": 40
   "Win: White mates": 77
Player: Pedantic 0.2.1
   "Draw by 3-fold repetition": 50
   "Draw by fifty moves rule": 44
   "Draw by insufficient mating material": 46
   "Draw by stalemate": 2
   "Loss: Black mates": 40
   "Loss: White mates": 77
   "Win: Black mates": 108
   "Win: White mates": 133

I keep each version of Pedantic in its own subdirectory and the Properties tab does show it as File version and Product version 0.3.0. If anything, I noticed for both Pedantic 0.1 and 0.2.1 it shows as version 0.1.0

Code: Select all

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.2.1]Pedantic.exe uci
Pedantic v0.2.0
uci
id name Pedantic v0.2.0
id author JoAnn D. Peeler
option name OwnBook type check default true
option name Ponder type check default true
option name Hash type spin default 64 min 1 max 2048
option name Clear Hash type button
option name MaxThreads type spin default 1 min 1 max 18
option name UCI_EngineAbout type string default Pedantic v0.2.0 by JoAnn D. Peeler, see https://github.com/JoAnnP38/Pedantic
option name Evaluation_ID type string default <empty>
option name Random_Search type check default false
option name Collect_Stats type check default false
uciok
quit

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.2.1]pedan\

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.3]Pedantic.exe
Pedantic 0.3.0
uci
id name Pedantic 0.3.0
id author JoAnn D. Peeler
option name OwnBook type check default true
option name Ponder type check default true
option name Hash type spin default 64 min 1 max 2048
option name Clear Hash type button
option name UCI_EngineAbout type string default Pedantic 0.3.0 by JoAnn D. Peeler, see https://github.com/JoAnnP38/Pedantic
option name Evaluation_ID type string default <empty>
option name Random_Search type check default false
option name Collect_Stats type check default false
uciok
quit

[C:\Games\Chess\Engines\Pedantic\Pedantic-0.3]

Cutechess engines.json

Code: Select all

	{
		"command" : "Pedantic.exe uci",
		"name" : "Pedantic 0.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.1"
	},

	{
		"command" : "Pedantic.exe uci",
		"name" : "Pedantic 0.2.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1"
	},

	{
		"command" : "Pedantic.exe",
		"name" : "Pedantic 0.3.0",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3"
	},

Please double check the cutechess engines.json file. I use this file for my testing as well and my entries for pedantic look like this:

Code: Select all

    {
        "name":"Pedantic_0.3B",
        "command":"C:/chess_engines/Pedantic_0.3B/Pedantic.exe",
        "protocol":"uci",
        "workingDirectory":"C:/chess_engines/Pedantic_0.3B",
        "stderrFile":"C:/chess_engines/errors/pedantic_0.3b_errors.txt",
        "initStrings": [
            "setoption name Hash value 128",
            "setoption name Ponder value false",
			"setoption name OwnBook value false",
			"setoption name Collect_Stats value false"
        ],
        "whitepov":false,
        "ponder":false
    },
    {
        "name":"Pedantic_0.2",
        "command":"C:/chess_engines/Pedantic_0.2/Pedantic.exe",
        "protocol":"uci",
        "workingDirectory":"C:/chess_engines/Pedantic_0.2",
        "stderrFile":"C:/chess_engines/errors/pedantic_0.2_errors.txt",
        "initStrings": [
            "setoption name Hash value 128",
            "setoption name Ponder value false",
			"setoption name OwnBook value false",
			"setoption name Collect_Stats value false"
        ],
        "whitepov":false,
        "ponder":false
    },
    {
        "name":"Pedantic_0.1",
        "command":"C:/chess_engines/Pedantic_0.1/Pedantic.exe",
        "protocol":"uci",
        "workingDirectory":"C:/chess_engines/Pedantic_0.1",
        "stderrFile":"C:/chess_engines/errors/pedantic_0.1_errors.txt",
        "initStrings": [
            "setoption name Hash value 128",
            "setoption name Ponder value false",
			"setoption name OwnBook value false",
			"setoption name Collect_Stats value false"
        ],
        "whitepov":false,
        "ponder":false
    },

Notice that I include the full path to the exe. Also not that the "workingDirectory" can be set to anywhere not just where the EXE resides, so I am not sure if these is helping you resolve where the EXE is.

[Gabor Szots]

May I ask what the role of the json file included in the package is? Is it part of the evaluation? Is it necessary to run Pedantic?

[JoAnnP38]

May I ask what the role of the json file included in the package is? Is it part of the evaluation? Is it necessary to run Pedantic?

Pedantic.json is not required. If you delete it, it will be recreated with default values. This file contains the default evaluation weights and is used to store revised weights after a training session.

If you are referring to engines.json that we were discussing earlier in this thread, that is for cutechess-cli.

[adnoh]

Thanks to everyone who has given feedback so far.

It was useful to hear from multiple testers that no one else is seeing these unusual results, especially from Gabor since he has the same processor as I do which removes that as a possible explanation. Presumably something on my setup but what?

I have my own little private rating list with several hundred combinations of engines and their versions tested using my own methodology and this is only the second time an engine/version combination has shown such expected behavior, well below what the engine author would expect.

I use Cutechess-GUI and normally use Stefan Pohl's SuperGM_4mvs.pgn book for variety with each engine playing white and black for each opening and use short TC and adjudication to get quick results. The results are "good enough" for me as it is generally consistent to the various rating lists out there. The issue with Pedantic 0.3.0 is it is performing several hundred ELO points below expectations, especially against 0.2.1.

To address the comment so far.

Guenther: Here are the PGN, including a new 500 game match with Pedantic using its own opening book and full command line to the EXE

https://1drv.ms/f/s!Aqe2-2i2H5hegoBxZeC ... g?e=2eCfyI

Gabor: I use the Windows "Balanced" power plan is I find "High Performance" sometimes "hangs" Windows for my day to day other uses on the computer. What I do see in task manager is that when chess engines are running it immediately goes up to ~4.5GHz from the base 3.6GHz and seems to be using the P-cores and not the E-cores. Are you saying you have a way to lock the clock speed? Through the BIOS? Have you considered disabling the E-cores? I read some motherboard manufacturers allow that but I am not aware ASUS supports it.

JoAnn: I have updated the settings in CuteChess-GUI so that it uses the full path to the EXE but did not make any difference.

Code: Select all

	{
		"command" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1\\Pedantic.exe uci",
		"name" : "Pedantic 0.2.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1"
	},
	{
		"command" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3\\Pedantic.exe",
		"name" : "Pedantic 0.3.0",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3"
	},

I am very inspired by your motivation to create Pedantic. I also went through University in the early 80s with an aspiration to create a chess engine but life kept me busy enough. After working my entire career as a programmer in business applications and now retired, I thought I would make an effort to do this so decided to start small and did create a decent Othello program. That seemingly has satisfied my craving to program for fun and now onto other non-programming interests.

So, I ran another 500 game match at 10+0.1 TC with Pedantic using its own opening book, no adjudication and full path to the EXE. Still unexpected results.

Code: Select all

Score of Pedantic 0.3.0 vs Pedantic 0.2.1: 156 - 243 - 101 [0.413]
...      Pedantic 0.3.0 playing White: 86 - 115 - 49  [0.442] 250
...      Pedantic 0.3.0 playing Black: 70 - 128 - 52  [0.384] 250
...      White vs Black: 214 - 185 - 101  [0.529] 500
Elo difference: -61.1 +/- 27.5, LOS: 0.0 %, DrawRatio: 20.2 %
500 of 500 games finished.

Player: Pedantic 0.3.0
   "Draw by 3-fold repetition": 24
   "Draw by fifty moves rule": 25
   "Draw by insufficient mating material": 49
   "Draw by stalemate": 3
   "Loss: Black mates": 115
   "Loss: White mates": 128
   "Win: Black mates": 70
   "Win: White mates": 86
Player: Pedantic 0.2.1
   "Draw by 3-fold repetition": 24
   "Draw by fifty moves rule": 25
   "Draw by insufficient mating material": 49
   "Draw by stalemate": 3
   "Loss: Black mates": 70
   "Loss: White mates": 86
   "Win: Black mates": 115
   "Win: White mates": 128

[JoAnnP38]

Thanks to everyone who has given feedback so far.

It was useful to hear from multiple testers that no one else is seeing these unusual results, especially from Gabor since he has the same processor as I do which removes that as a possible explanation. Presumably something on my setup but what?

I have my own little private rating list with several hundred combinations of engines and their versions tested using my own methodology and this is only the second time an engine/version combination has shown such expected behavior, well below what the engine author would expect.

I use Cutechess-GUI and normally use Stefan Pohl's SuperGM_4mvs.pgn book for variety with each engine playing white and black for each opening and use short TC and adjudication to get quick results. The results are "good enough" for me as it is generally consistent to the various rating lists out there. The issue with Pedantic 0.3.0 is it is performing several hundred ELO points below expectations, especially against 0.2.1.

To address the comment so far.

Guenther: Here are the PGN, including a new 500 game match with Pedantic using its own opening book and full command line to the EXE

https://1drv.ms/f/s!Aqe2-2i2H5hegoBxZeC ... g?e=2eCfyI

Gabor: I use the Windows "Balanced" power plan is I find "High Performance" sometimes "hangs" Windows for my day to day other uses on the computer. What I do see in task manager is that when chess engines are running it immediately goes up to ~4.5GHz from the base 3.6GHz and seems to be using the P-cores and not the E-cores. Are you saying you have a way to lock the clock speed? Through the BIOS? Have you considered disabling the E-cores? I read some motherboard manufacturers allow that but I am not aware ASUS supports it.

JoAnn: I have updated the settings in CuteChess-GUI so that it uses the full path to the EXE but did not make any difference.

Code: Select all

	{
		"command" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1\\Pedantic.exe uci",
		"name" : "Pedantic 0.2.1",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : 1,
				"max" : 18,
				"min" : 1,
				"name" : "MaxThreads",
				"type" : "spin",
				"value" : 1
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.2.1"
	},
	{
		"command" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3\\Pedantic.exe",
		"name" : "Pedantic 0.3.0",
		"options" : [
			{
				"alias" : "",
				"default" : true,
				"name" : "OwnBook",
				"type" : "check",
				"value" : true
			},
			{
				"alias" : "",
				"default" : 64,
				"max" : 2048,
				"min" : 1,
				"name" : "Hash",
				"type" : "spin",
				"value" : 64
			},
			{
				"name" : "Clear Hash",
				"type" : "button"
			},
			{
				"alias" : "",
				"default" : "<empty>",
				"name" : "Evaluation_ID",
				"type" : "text",
				"value" : "<empty>"
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Random_Search",
				"type" : "check",
				"value" : false
			},
			{
				"alias" : "",
				"default" : false,
				"name" : "Collect_Stats",
				"type" : "check",
				"value" : false
			}
		],
		"protocol" : "uci",
		"stderrFile" : "",
		"timeout_scale_factor" : 1,
		"workingDirectory" : "C:\\Games\\Chess\\Engines\\Pedantic\\Pedantic-0.3"
	},

I am very inspired by your motivation to create Pedantic. I also went through University in the early 80s with an aspiration to create a chess engine but life kept me busy enough. After working my entire career as a programmer in business applications and now retired, I thought I would make an effort to do this so decided to start small and did create a decent Othello program. That seemingly has satisfied my craving to program for fun and now onto other non-programming interests.

So, I ran another 500 game match at 10+0.1 TC with Pedantic using its own opening book, no adjudication and full path to the EXE. Still unexpected results.

Code: Select all

Score of Pedantic 0.3.0 vs Pedantic 0.2.1: 156 - 243 - 101 [0.413]
...      Pedantic 0.3.0 playing White: 86 - 115 - 49  [0.442] 250
...      Pedantic 0.3.0 playing Black: 70 - 128 - 52  [0.384] 250
...      White vs Black: 214 - 185 - 101  [0.529] 500
Elo difference: -61.1 +/- 27.5, LOS: 0.0 %, DrawRatio: 20.2 %
500 of 500 games finished.

Player: Pedantic 0.3.0
   "Draw by 3-fold repetition": 24
   "Draw by fifty moves rule": 25
   "Draw by insufficient mating material": 49
   "Draw by stalemate": 3
   "Loss: Black mates": 115
   "Loss: White mates": 128
   "Win: Black mates": 70
   "Win: White mates": 86
Player: Pedantic 0.2.1
   "Draw by 3-fold repetition": 24
   "Draw by fifty moves rule": 25
   "Draw by insufficient mating material": 49
   "Draw by stalemate": 3
   "Loss: Black mates": 70
   "Loss: White mates": 86
   "Win: Black mates": 115
   "Win: White mates": 128

There is a switch in cutechess to restart the engines after every game. You might want to try that to see if it makes a difference. I believe you can provide it on the -engine switch or the -each switch (i.e. restart=on.) The other thing you could try is to specify the "-debug" switch to cutechess, capture all the output to a file and then see if there is anything out of the ordinary. I would love to see that file myself. Also, to make sure the cutechess config/setup/version isn't part of the problem is to try using a different means to run a game/tournament. Have you used Banksia or Arena? Banksia is pretty good and it lets you run games concurrently, but Arena does not. But right now I'm just trying to get handle on what could possibly be the problem.