How to test engine performance and strenght

[eligolf]

Hi,

I am currently developing a chess engine in C#/Unity which seems to be working as expected and playing decent chess with just simple PST as evaluation. It is pretty basic and reaches a depth of 6 in 2.5s and depth of 7 in 15s from the starting position as white.

Now is the time to start implementing more optimizations and a better evaluation function. It seems like many of the techniques can yield good results in some cases and game states while giving worse in others. So my questions are:

1. How do you test if an added feature/better evaluation function gives better general strength during a whole game of chess and not just in some tactical position?
2. How do I test if an optimized search feature such as PVS or nullmove gives shorter thinking times during a whole game of chess and not just in some chosen positions?

[yeni_sekme]

Run engine vs engine matches if you have implemented UCI protocol.
500-1000 games would be enough to detect big changes( >30 ELO).

[eligolf]

Yes that is an idea, I have not yet implemented that though.
Wouldn't it take super long to run 500-1000 games? With time control of 3 minutes per side that is max 100 hours of constant playing.

[yeni_sekme]

3 minutes for each side is very long. you can use 20sec+ 0.2sec and go even lower time control when your engine become faster to reach high depths. Also, you can run more than one game at the same time depending on how many threads your computer have.

[mvanthoor]

Yes that is an idea, I have not yet implemented that though.
Wouldn't it take super long to run 500-1000 games? With time control of 3 minutes per side that is max 100 hours of constant playing.

It is enough to run matches with something like 10s+0.1s or 20s+0.2s per game. If you use CuteChess, you can run game per core. I have n older quad-core, and a gauntlet of 5-6000 games takes about 8 hours or so with the time control I'm using. I intend to upgrade to a 16-core machine as soon as Zen 4 is out (or Intel releases something with 16 or more P-cores that can hold its own against a Zen4 CPU). That will cut testing time to 1/4th, and it will be a computer that will be fast enough for my use for a decade.

[eligolf]

Well with 10 or 20 s I will have to reach max depth 3-4 or something. Not sure my engine will be MUCH faster than it is now, maybe an extra depth or something.

[Chessnut1071]

Hi,

I am currently developing a chess engine in C#/Unity which seems to be working as expected and playing decent chess with just simple PST as evaluation. It is pretty basic and reaches a depth of 6 in 2.5s and depth of 7 in 15s from the starting position as white.

Now is the time to start implementing more optimizations and a better evaluation function. It seems like many of the techniques can yield good results in some cases and game states while giving worse in others. So my questions are:

1. How do you test if an added feature/better evaluation function gives better general strength during a whole game of chess and not just in some tactical position?
2. How do I test if an optimized search feature such as PVS or nullmove gives shorter thinking times during a whole game of chess and not just in some chosen positions?

Most optimization is done by testing with many games. I agree with that; however, I use a different approach before I test with game performance. I optimize 58 variables, 38 white & 20 black, specific to the opening and defense. I assume that Grandmasters know best, so, I optimize each of the parameters by the depth that the master's move falls from the top of the legal move list. The objective is to minimize the number of moves from the top. Those optimized 50 parameters differ depending on the opening & defense, so, there's tremendous time required for that approach. I use a modified Hooke & Jeeves nonlinear optimization program that attaches to the chess engine. This approach only works on the middle game, but, the opening and end game strategy is all databases.

[pedrojdm2021]

Hello, i'm very happy to see other Unity users here!

i've developed a engine too in my work, that i using it in unity as well.

This is the workflow that i'm using:

I have 3 projects:
- Engine Class library (.dll file) , you need to set the target framework version to .net 4.7.2

- Engine Console app: the C# chess engine were you're going to implement the UCI protocol, and other engine-only related features.
With this console app and "engine application" is the tool that you're going to use to test the engine like any other "normal" chess engines.

- Unity project: this is where you're going to use your engine .dll file and use it as a library, you only need to export the .dll into unity Only when you're sure that your chess engine is working as expected.

Your project structure should look like this:

Code: Select all

/ROOT FOLDER/
 - Assets
 - ProjectSettings 
 - /Engine Solution folder/
       - solutionName.sln
      - /Engine Console App folder/
          - csharp project file
          - .cs code files
          - program.cs
      - /Engine Class library folder/
          - csharp project file 
          - .cs code files

So after you setup your project like that, you should be ready to develop your engine just as any other engine.

The tests are mostly doing by testing your UCI engine console app vs other engines, and running various number of games at cretain depth, you get the stats of :
- number of nodes visited
- time that took the search
- evaluation

with that information you can test performance.

Also, another good idea is to get the test fen's from chessprogramming wiki:
https://www.chessprogramming.org/Perft_Results

And run the AI on these, for example in depth 9 or 10:

Code: Select all

position fen fenStringHere
go deph 9

Good luck!

[eligolf]

Hello, i'm very happy to see other Unity users here!

Also, another good idea is to get the test fen's from chessprogramming wiki:
https://www.chessprogramming.org/Perft_Results

And run the AI on these, for example in depth 9 or 10:

Code: Select all

position fen fenStringHere
go deph 9

Good luck!

Nice, Unity is really good for developing your own GUI as well which I find fun

I have done some smaller Perft for all thinkable critical cases which succeeds.

The problem right now is that my engine is calculating around 70-80k nodes/s which is pretty slow. How many is yours doing? And is there any chance I can see your move gen code and search code for some inspiration?
I have only developed in Python before so C# is pretty new to me..

[Tearth]

You can check Cosette if you want some inspiration - it's also written in C# (not Unity tho, but shouldn't make any difference), and perft is about 20 MN/s (which is still rather slow compared to other engines, but "good enough").