Engine results: a surprise! (again)

[xmas79]

Hi everyone,
here I'm back with a more appropriate test. This time I ran it at 10s+0.1s. I tried lower time controls like 5+0.05 or 3+0.03 as other people suggested me, but my engine really suffers at this very fast TC as it has a "slow start": at 3+0.03 it would not even start the first iteration, so it will play the first generated move, while at 5+0.05 it will iterate but will lose every game on time. I need to change something in the search bootstrap code, but it has a very low priority now.

I think I hit the wall most programmers hit: time management... I really had to think about it, trying to figure out how to manage this... I finally came to a solution that works, but it's very far from being perfect and sure needs a lot of work.


Back to the match, I randomized the first 8 moves of my engine, and fairymax randomizes as well, so here's some stats about the games actually played:

pgn-extract output after removing duplicated games:
3172 games matched out of 3179.

So only 7 games were duplicate.

9 games out of 3172 were unfinished, so valid games are 3163 (I had to quit cutechess three times)

Fairymax lost only 15 games by time forfait.

Elostat says:

Games : 3163 (finished)

White Wins : 1415 (44.7 %)
Black Wins : 1325 (41.9 %)
Draws : 423 (13.4 %)
Unfinished : 9

White Perf. : 51.4 %
Black Perf. : 48.6 %

ECO A = 2626 Games (83.0 %)
ECO B = 0 Games ( 0.0 %)
ECO C = 28 Games ( 0.9 %)
ECO D = 506 Games (16.0 %)
ECO E = 3 Games ( 0.1 %)

With an ELO base of 2200:
1 MyEngine : 2297 13 13 3163 75.4 % 2103 13.4 %
2 FairyMax : 2103 13 13 3163 24.6 % 2297 13.4 %

so the estimated ELO difference is 194 points.

Other stats:
Longest valid game: 518 plies
Shortest valid game: 16 plies (a draw by repetition)
Next shortest valid game: 30 plies

Average game length: around 127 plies

Looking at these numbers, I'm still not satisfied... Next thing to do is an opening book... I really don't want to "randomize too much" in the opening like I do now (add +/- 50cp to every evaluation).



What do you think about this match?



Best regards,
Natale.

[jdart]

as it has a "slow start"

If your engine has significant start-up code to execute you should put it before sending "readyok" (for UCI) or use the "feature done" mechanism if Winboard, so the GUI does not count startup time as thinking time.

Losing on time is also fixable. I am not clear how you are measuring time usage. If you are polling you may need to adjust frequency of polling upwards if using a very short time control.

--Jon

[xmas79]

as it has a "slow start"

If your engine has significant start-up code to execute you should put it before sending "readyok" (for UCI) or use the "feature done" mechanism if Winboard, so the GUI does not count startup time as thinking time.

Losing on time is also fixable. I am not clear how you are measuring time usage. If you are polling you may need to adjust frequency of polling upwards if using a very short time control.

--Jon

Hi Jon,
the problem I think I have is "start of the search", not start of the engine. I actually initialize everything even before accepting things on stdin.

My problem can be split in two: one part of the problem is of course in the time management code. I see that the time allocation policy has a great influence on the games, it's a very difficult thing to get right.
The second part of the problem is the "slow start" I mentioned (speaking about search start): even if I allocate 1 centiseconds to EVERY move, I see my clock going down. I do time checks on every node and q-node and that means the flow

1) receive the move from winboard
2) start search (and pratically abort immediately as assigned 1 centiseconds to every move)
3) output move

is not a 1cs (or so) time consuming flow. I tested this, and I pratically need at least a +0.09 increment to not lose time on clock by doing nothing (send the first generated move)... I thought about it, and simply changed my mind: what's the point on getting so fast? Yes I know, big numbers and games to reduce error bars, but I really don't know if is it worth (in my case) fixing these things. At these TC (3'+0.03 or less), the test goes towards the static evaluation, not playing real chess. So if I want to tune static evaluation I just do a 1-ply depth search and I'm done... A more conservative TC (10'+0.1 or 30'+0.3) will just allow me to play decent chess and test chess knowledge AND time management, which I saw is a critical component.

That's my opinion, what do you think?

Best regards,
Natale.

[jdart]

I think 10s+0.1s is a reasonable time control but it sounds like you have some additional problem that may prevent you from reaching reasonable depth even at that time control.

If you are actually using winboard, you might try instead cutechess-cli, which supports the same protocol but has no UI, and therefore probably less overhead.

--Jon

[jd1]

ECO A = 2626 Games (83.0 %)
ECO B = 0 Games ( 0.0 %)
ECO C = 28 Games ( 0.9 %)
ECO D = 506 Games (16.0 %)
ECO E = 3 Games ( 0.1 %)

...

Looking at these numbers, I'm still not satisfied... Next thing to do is an opening book... I really don't want to "randomize too much" in the opening like I do now (add +/- 50cp to every evaluation).



What do you think about this match?

Hi Natalie,

Well done writing your own engine!

In my opinion it is best to use a fixed test suite for testing. The advantages:
- Wider coverage of all ECO openings (83% of yours are ECO A)
- Each opening position can be repeated with opposite colours
- Less repeated games

It is really quite easy:
- download LittleBlitzer or cutechess-cli (as Jon said there is much less overhead)
- download a test suite of starting positions

I guess your engine will probably do better against Fairy-Max without having to add randomness in the opening (you can make a quite bad mistake with +- 50 cp).

Regarding time management, IMO it is not that important until your engine gets to 2700 or above. I would just use a simple scheme like using move_time = time_left / 30 + increment. If you like, you can get a few more elo by making sure you quit searching after a complete iteration.

Jerry

[hgm]

If you are actually using winboard, you might try instead cutechess-cli, which supports the same protocol but has no UI, and therefore probably less overhead.

Note that if you run WinBoard with the option -noGUI to suppress the update of the displayed board and task-bar icon on every move, (and don't open the engine-output window to see all PVs the engine spits out), it is not significantly slower than cutechess-cli. Which on a slow machine (1.3GHz Celeron-M) was about 1 msec, (engine-to-engine) and should be far below any other speed problems you might encounter (e.g. that the system clock ticks only every 10 or 16 msec).

The main reason that people encounter problems in fast games with WinBoard is that they forget to switch move animation off. With move animation on, there is a minimum time per move (depending how vast you set the animation speed), to allow the spectators to see all moves, which could cause a time loss if the opponent already replied with a move before the animation of the previous move is completed, and WinBoard can start animating the reply move. (And there of course always is the general problem that if you fully load your system, i.e. put engines on every available hyper thread, even the fastest program can get slow response time because the OS simply denies it CPU for some time.)

[xmas79]

If you are actually using winboard, you might try instead cutechess-cli, which supports the same protocol but has no UI, and therefore probably less overhead.

Note that if you run WinBoard with the option -noGUI to suppress the update of the displayed board and task-bar icon on every move, (and don't open the engine-output window to see all PVs the engine spits out), it is not significantly slower than cutechess-cli. Which on a slow machine (1.3GHz Celeron-M) was about 1 msec, (engine-to-engine) and should be far below any other speed problems you might encounter (e.g. that the system clock ticks only every 10 or 16 msec).

The main reason that people encounter problems in fast games with WinBoard is that they forget to switch move animation off. With move animation on, there is a minimum time per move (depending how vast you set the animation speed), to allow the spectators to see all moves, which could cause a time loss if the opponent already replied with a move before the animation of the previous move is completed, and WinBoard can start animating the reply move. (And there of course always is the general problem that if you fully load your system, i.e. put engines on every available hyper thread, even the fastest program can get slow response time because the OS simply denies it CPU for some time.)

Aaahhh yes, that was the real problem! I used winboard to "tune" time management "the dirty way": by looking live at games.... Well, the thing is that I thought winboard would make the move on the board with all its bells&whistles, keeping the clocks stopped for both engines during this process... I was clearly wrong. Turning off animations & co. showed me that my engine is already capable of 1'+0.01 without losing on time..

Thanks,
Natale.

[xmas79]

...
Hi Natalie,

Actually my name is Natale

Well done writing your own engine!

In my opinion it is best to use a fixed test suite for testing. The advantages:
- Wider coverage of all ECO openings (83% of yours are ECO A)
- Each opening position can be repeated with opposite colours
- Less repeated games

It is really quite easy:
- download LittleBlitzer or cutechess-cli (as Jon said there is much less overhead)
- download a test suite of starting positions

Ok, I did it, and used "cutechess-cli -openings file=silversuite.pgn -repeat" as command arguments, I used silversuite openings (link from the CPW). But even with this, is randomness not needed? I mean, if my engine plays white, the first move out the book (supposing opening X) my engine will start thinking and in the end will spite out what it thinks the best move is, and this move will always be the same next time cutechess-cli will make my engine play white with the same opening X, isn't it? What am I missing?

I guess your engine will probably do better against Fairy-Max without having to add randomness in the opening (you can make a quite bad mistake with +- 50 cp).

Regarding time management, IMO it is not that important until your engine gets to 2700 or above. I would just use a simple scheme like using move_time = time_left / 30 + increment. If you like, you can get a few more elo by making sure you quit searching after a complete iteration.

Jerry

I already do something similar. Thanks anyway for your suggestion.

Best regards,
Natale.

[hgm]

Well, the thing is that I thought winboard would make the move on the board with all its bells&whistles, keeping the clocks stopped for both engines during this process...

Normally that would be wasting time, as the animation can be done in parallel with the opponent thinking. It is just that when the opponent replies before the animation is finished that there is a problem. This is not so easy to fix, and it also makes little sense to operate in this regimes. The idea of having very fast TC is that you can play many games per hour. If most of the time the clocks would be stopped because of animation being in progress, it sort of defeats the purpose. You would be better of having the engine think during the animation.

For this reason WinBoard relays the move instantly, which means it also sets the clock running instantly.

[Rein Halbersma]

Note that if you run WinBoard with the option -noGUI to suppress the update of the displayed board and task-bar icon on every move, (and don't open the engine-output window to see all PVs the engine spits out), it is not significantly slower than cutechess-cli. Which on a slow machine (1.3GHz Celeron-M) was about 1 msec, (engine-to-engine) and should be far below any other speed problems you might encounter (e.g. that the system clock ticks only every 10 or 16 msec).

Ddi you ever experiment with the C++ std::chrono::high_resolution_clock? It is substantially better than system_clock and has a resolution at the sub-microsecond scale. (It's a C++11 library function, but there is also the Boost equivalent older compilers).