How to test engine performance and strenght

[lithander]

My main testing method besides letting the engine play thousands of games is that I have set of 300 positions for which the best move is known.
While testing I search each position to fixed depth like for example 9.

Now I get 3 important data points:
1) number of solved positions
2) total runtime
3) number of visited nodes
and the combination of 2 and 3:
4) nodes per second. (NPS)

With each change you make you can now verify if you get the expected results:
- Improved the evaluation? Number of solved positions should go up. Nps goes down a little maybe.
- Added some loss-less pruning like PVS or better move ordering? Number of visited nodes needs to go down! Solved positions should stay the exact same.
- Added null move pruning? Runtime and visited nodes should drop massively but maybe you solve a few less positions now.
....and so on!

[pedrojdm2021]

You can check Cosette if you want some inspiration - it's also written in C# (not Unity tho, but shouldn't make any difference), and perft is about 20 MN/s (which is still rather slow compared to other engines, but "good enough").

If you're the developer of Cosette i have to say that your engine is amazing and super fast ! it for sure is an inspiration for us who are "beginners" in the world of chess programming, that said, one for sure can look at the code and took some inspiration from it, but not everything done there is compatible with unity, as it uses . net 5-6 only features in some parts, and Unity supports Up to .net 4.x.

[pedrojdm2021]

Hello, i'm very happy to see other Unity users here!

Also, another good idea is to get the test fen's from chessprogramming wiki:
https://www.chessprogramming.org/Perft_Results

And run the AI on these, for example in depth 9 or 10:

Code: Select all

position fen fenStringHere
go deph 9

Good luck!

Nice, Unity is really good for developing your own GUI as well which I find fun

I have done some smaller Perft for all thinkable critical cases which succeeds.

The problem right now is that my engine is calculating around 70-80k nodes/s which is pretty slow. How many is yours doing? And is there any chance I can see your move gen code and search code for some inspiration?
I have only developed in Python before so C# is pretty new to me..

Yeah, you have to think like that. Unity is only for developing the GUI for your chess game. the AI and engine is enterly another project that are outside Unity.

Without stats is hard to say what you are doing wrong.

Few points to mention:
- cache data as much as possible, for example in the move generator, you don't have to create a new list dinamycally each time you generate moves, is better to take an array and only update the values.

- perform perft tests, based on the ones of chess programming wiki.

For example my perft in startposition for depth 5 is:

Code: Select all

a2a3: 181046
a2a4: 217832
b1a3: 198572
b1c3: 234656
b2b3: 215255
b2b4: 216145
c2c3: 222861
c2c4: 240082
d2d3: 328511
d2d4: 361790
e2e3: 402988
e2e4: 405385
f2f3: 178889
f2f4: 198473
g1f3: 233491
g1h3: 198502
g2g3: 217210
g2g4: 214048
h2h3: 181044
h2h4: 218829

Nodes searched: 4865609 in 616ms

on a single core task in my old amd fx 8350.
You should take more or less than that, with basic optimizations is not hard to reach that speed.

- To me is more simple to develop a bitboard based engine, rather than an array based engine, and also easier to optimize.

If you don't know what i mean, i highly recommend the video series of "Code Monkey King" and "BlueFeverSoft" on chess engines:

Code monkey kind video series on a bitboard engine:


BlueFeverSoft video series:


They will give you a solid base on how is structured and developed an bitboard chess engine.

I know, is not something that you can learn in a weekend, if you are a beginner in can take time, but is a start, take your time and don't rush.

Good luck with the development, and enjoy the journey

[JohnWoe]

If I don't change bench signature. Then I'll just measure a speedup.
If I do. I rarely do. Then I run a few 1000 1s FRC games.
If it's good then longer Blitz games vs Crafty / Glaurung / Stockfish

Here is startpos depth: 6. 4800U

Code: Select all

perft
1. a2a3 -> 4463267 (90 ms)
2. a2a4 -> 5363555 (96 ms)
3. b2b3 -> 5310358 (97 ms)
4. b2b4 -> 5293555 (95 ms)
5. c2c3 -> 5417640 (99 ms)
6. c2c4 -> 5866666 (107 ms)
7. d2d3 -> 8073082 (145 ms)
8. d2d4 -> 8879566 (160 ms)
9. e2e3 -> 9726018 (178 ms)
10. e2e4 -> 9771632 (179 ms)
11. f2f3 -> 4404141 (80 ms)
12. f2f4 -> 4890429 (88 ms)
13. g2g3 -> 5346260 (97 ms)
14. g2g4 -> 5239875 (94 ms)
15. h2h3 -> 4463070 (81 ms)
16. h2h4 -> 5385554 (98 ms)
17. b1a3 -> 4856835 (88 ms)
18. b1c3 -> 5708064 (104 ms)
19. g1f3 -> 5723523 (104 ms)
20. g1h3 -> 4877234 (88 ms)

===========================

Nodes:    119060324
Time(ms): 2168
NPS:      54917123

[eligolf]

Thanks all for the input.

Regarding perft, I already have a successful perft so there is no need to do that again. Or do you mean to use it as a way of testing move generation speed?

Cosette seems very nice, I have looked through the code for a week or so and there is much that I don't understand. But some parts are great inspiration for sure! Some parts not compatible with Unity so I am developing my own setup, with some parts inspired by Cosette such as file structure.

[pedrojdm2021]

Thanks all for the input.

Regarding perft, I already have a successful perft so there is no need to do that again. Or do you mean to use it as a way of testing move generation speed?

Cosette seems very nice, I have looked through the code for a week or so and there is much that I don't understand. But some parts are great inspiration for sure! Some parts not compatible with Unity so I am developing my own setup, with some parts inspired by Cosette such as file structure.

I think yes, perft is a good tool the measure only the move generation speed.
Remember that the AI search will be much slower than pure move generation, as it envolves much more stuff than just making and un-making moves.

[pedrojdm2021]

Guys i have a question:

If i run engine vs engine tests but no time control, instead it's depth fixed, it is not a fair "test" right?

[Ras]

If i run engine vs engine tests but no time control, instead it's depth fixed, it is not a fair "test" right?

Yeah, that's pretty pointless. The only times where you'd want depth fixed is for debugging or limiting strength.