Frustrations of a mediocre chess programmer. What next?

[hgm]

Doing a re-write seems only useful if you know what you want to do fundamentally different (or a necessary evil when your code has become such a patched mess that it has become unreadable even for you...).

The others are right: try to figure out what makes you lose against equal opponents. Is it because you are outsearched? If they don't have a higher nps, then it must be your move ordering. If they do, it could be some of your algorithms (move generation, sorting, evaluation) are just slow, and need speed up (e.g. evaluating things differentially rather than from scratch). If you outsearch your opponent, you might still lose because the 'gain' you saw at the horizon evaporates on deeper search, as evaluation scores are replaced by search scores at the former leaves. You would have to tune your evaluation for better consistency, then (e.g. doing a linear regression tpo determine the best weights of the eval terms). If the main eval discrepancies are of tactical nature, you should consider adding extensions that would capture them.

[Uri Blass]

I need a fast way to test if a change need more testing.

The simple way that I think is having many games at super fast time control of 1 second per game and if I see significant result to test at slower time control.

The main problem with this approach is that arena interface is very slow and does not allow games at that time control.

It is possible to use winboard for that purpose by using a batch file but in that case based on my experience I cannot follow the total result of the match based on the screen and I dislike looking at the pgn and count to find out and I want to see the result of the match during the match and not after it.

Edit:note that I plan to use fixed number of nodes per move instead of
time control of one second per game because I prefer to have deterministic results and I do not like to have losses on time in the fast testing.

Uri

[hgm]

I am not sure Winboard would be much faster; it might all be OS delays.

I always use Winboard_x under PSWBTM to set up gauntlets. There you can see the results while the match is running.

[Uri Blass]

I am not sure Winboard would be much faster; it might all be OS delays.

I always use Winboard_x under PSWBTM to set up gauntlets. There you can see the results while the match is running.

I do not know what is PSWBTM

practically I am not interested in book in my testing so my testing is from the noomen positions(I would like to increase the number of positions so I can get 1000 games or in chess960 1920 games from all positions.

I would like a tool that allows me to run the same match hopefully with 1000 games(I need to decide about pgn file) and repeat it with different number of nodes

I can start at 1000 nodes per move and later at 2000 nodes per move and later at 3000 nodes per move when I can follow the results.

I think that an interface that only save the games as pgn and show the results of every match with no graphic can be good for programmers.

Uri

[hgm]

PSWBTM does allow you to start from arbitrary positions: you just have to specify a file with the moves leading to them in PGN, or FENs, and your match uses those as initial position. I always run matches from the Nunn positions for testing. If the specified file contains 10 positions, and you ask for 200 games, it plays every position 10 times with white and 10 times with black.

I think others already suggested how you could disable the display in Winboard.

I am not sure if you could make it very fast, though, as it starts up the engines anew for each game. On some Operating Systems starting up an engine takes dozens of seconds.

[pijl]

I think there are multiple ways to deal with this:

- Rewrite the engine entirely. Only a valid approach if you regret some of the basic design principles of your old engine which cannot be corrected without this rewrite, or because your old engine source needs a big cleanup activity as code has grown to be ugly. You'll have to be sure that rewriting is worth the trouble, so you must have a good reason to choose this path instead of trying to improve your existing program.

- Do something else for a while. When you return to your chess source code you see different things and perhaps immediately spot a bug or two that were preventing you from making progress.

- Watch your program play, or use it to analyze some games, and check out its output carefully. Do you see jumps in score/instabilities etc. Try to find out what causes it as usually it means there is a bug to solve.

- Play matches against weaker engines. A loss against a weaker engine probably means you hit a weak spot of your engine (or selected a crappy bookline, which obviously needs fixing too). As your program is as strong as its weakest component, this is a sure way to improve it

Richard.

Hello TalkChess!

I wonder how many of you have had the same frustrating experience as me.

When i started writing my engine, Pupsi, it was steady improvements for roughly a year and a half! It was wonderful! Thrilling! Exciting!

But after the latest release, v0.18, I hit a road block.

Nothing gives any improvement! I have tried many, many things, but all of them resultet in roughly zero ELO points, so they were all discarded. Countless hours down the drain.

The frustration have set in!

So to all of you wonderful programmers out there:
How do you keep going, when the going gets tough?

And a second question:
How do you decide what to try to improve next?
How do you analyse games, in order to find out which weakness to concentrate on? Any techniques?

What are your secrets?

Kind regards,
Jesper

[Vempele]

ord Romstad, the author of Glaurung has his third engine already.
Edsel

I guess the Viper is an abridged version of Glaurung, So it's hardly a rewrite or a new engine,isn't it?Tord can clarify this.

Gothmog, Glaurung 1 and Glaurung 2 (a complete rewrite in everything except name). Viper is indeed a simplified Glaurung 1.

[MartinBryant]

I'd definitely agree with Richard's second point there about taking a break for a while.
Shocking as it may seem I do do other stuff besides computer chess!
Sometimes I take months off.
Then when you come back you can see all sorts of stuff to try and the enthusiasm's back.
Hell, we only do this for the fun, and if you're not enjoying it today, leave it until it is fun again.

Another thing I'd try is adding loads of verification code to your program (for use in debug only of course) to try to catch all those bugs you don't even know you've got. I've found that very productive in the past!

[Uri Blass]

PSWBTM does allow you to start from arbitrary positions: you just have to specify a file with the moves leading to them in PGN, or FENs, and your match uses those as initial position. I always run matches from the Nunn positions for testing. If the specified file contains 10 positions, and you ask for 200 games, it plays every position 10 times with white and 10 times with black.

I think others already suggested how you could disable the display in Winboard.

I am not sure if you could make it very fast, though, as it starts up the engines anew for each game. On some Operating Systems starting up an engine takes dozens of seconds.

The problem in arena is not only starting a new game but also that the game itself is slowed down so winboard is better than arena.

The problem with PSWBTM is that I do not see an option to tell the engines fixed number of nodes(I can make special exe of movei that will use fixed number of nodes and use slow time control but if I want to test movei against other engines that support fixed number of nodes I cannot use that idea)

I also do not see an option to tell the engine to play game in 1 second(note that it is possible to play game in 1 second under winboard if you use a batch file).

Uri

[hgm]

Doesn't PSWBTM accept time in the format mm:ss ? If not, we should inform Pradu.

As concerning the fixed number of nodes: Is it possible to ask for this in the Winboard protocol at all?

If you don't want any graphics, an adapter to play two programs a number of times is almost trivial to write.