Frustrations of a mediocre chess programmer. What next?

[jesper_nielsen]

Hello TalkChess!

I wonder how many of you have had the same frustrating experience as me.

When i started writing my engine, Pupsi, it was steady improvements for roughly a year and a half! It was wonderful! Thrilling! Exciting!

But after the latest release, v0.18, I hit a road block.

Nothing gives any improvement! I have tried many, many things, but all of them resultet in roughly zero ELO points, so they were all discarded. Countless hours down the drain.

The frustration have set in!

So to all of you wonderful programmers out there:
How do you keep going, when the going gets tough?

And a second question:
How do you decide what to try to improve next?
How do you analyse games, in order to find out which weakness to concentrate on? Any techniques?

What are your secrets?

Kind regards,
Jesper

[Matthias Gemuh]

Hi Jesper,
my BigLion is so weak that I have no advice for you.
Just a question: how do I get Pupsi to use less than 100 MB memory ?
When told to use 32 MB or 64 MB in ChessGUI, it uses 100+ MB.
Best,
Matthias.

[brianr]

I've been working on and off (mostly off) on Tinker and prior incarnations since about 1970.
While Tinker has been roughly a "middle" or "top half of bottom third" strength engine
(I'm on the verge of a total rewrite), here are some things that are more incremental.

First, put your engine up on ICC. Then, watch it from time to time, especially in a losing game.
Make sure you have instrumented it with suitable logging information.

Analyze the losing game by replaying it and going thru the log file to see at what point your engine's eval significantly differs from the opponent's eval.

Then get a FEN for that position and recompile your engine with "debug" info, particularly in the eval function.

Look at the static eval to see if for example, the king safety portion seems way off,
or the pawn eval component, or postition, mobility or whatever.

You can compare "reasonable" static evaluation values from a vastly stronger benchmark engine, such as Rybka.

If the eval looks ok, then another possibility is that your engine was simply tactically outsearched.
Clues to this are sometimes a long sequence of correctly predicted moves, but no sudden score drop until "its too late".

Fiddling with search is more complicated, and generally I suggest only changing either the search or the eval, but _not_ both at the same time.

Hashing is one place to try alternatives, along with killers, history pruning, move ordering, and of course extensions.

In both cases, for eval or search changes, it is helpful to have a testing process.
This is another whole topic by itself...ee the nondeterministic testing thread started back on 8/15 for suggestions.

[jesper_nielsen]

Hi Matthias,

A quick test on my Windows XP machine gives 88MB memory usage, when using 64MB hash.

And 56 MB, when using 32 MB hash.

It is a bit tricky though, since Pupsi is a dotnet application. Meaning the memory used have to be garbage collected before it is freed.

So if you start Pupsi with 64MB hash, and make a search. Then set the hash size to 32, I have no real control over when the 64 MB previously used is freed. And for a while, at least, the memory consumption will be 96 MB.

But the case where the hash size is reset before the "isready" message, the memory usage should be ok. So the usage could also depend on how ChessGUI initialises the engines, and changes any default settings.

Please also note that the pawn hash table default size is 8 mb.

Memory usage is very interesting.
You can specify the memory usage of the hash table, but it would be much more "fair", if the total allowed memory could be set. And then the engines could themselvs decide how much to use for hash, pawn hash, EGTB hash and other tables, etc.

Kind regards,
Jesper

[Michael Sherwin]

Hi Jesper,

I am at the same place that you are. What to do next? I think that I could work night and day on my old code base and not make much progress. I have a mindset concerning my old code that prevents me from doing that wich is needed to make RomiChess into a Fruit/Toga or Glaurung killer. I think that a rewrite is the way to go. If Tord had still been able to improve Gothmog easily then I imagime that he would have done so. If Fabien was able to improve Fruit easily, I wonder if he would now be working on Chess64. Also Allesandro's Hamsters has passed up his older program Kiwi. Starting over from scratch allows us to avoid all the bad compromises that we made when writing our first program.

Mike

[Matthias Gemuh]

Hi Jesper,
I shall change engine initialization procedure in ChessGUI and report the effect in first subform (we are here in 3rd), but not today.
As for fairness in memory useage, ChessGUI has a feature to reduce hash size of UCI engines that use too much memory.
Best,
Matthias.

[Edsel Apostol]

Hi Jesper,

I am also an engine author (Twisted Logic). If you had achieved the peak strenght of your engine, consider a rewrite from scratch. It would take you around a month or two to achieve the strenght of your engine now but you will see that the new engine could surpass the old one.

Learn from your old engine, what are the things that you need to revise, what are its shortcomings. Fix it in the new rewrite. Tord Romstad, the author of Glaurung has his third engine already. Daniel Shawul of Scorpio has his second engine.

Keep the rewrite simple yet efficient. Genius is the ability to make complex things simple.

Edsel

[jesper_nielsen]

Hi Brian,

Thanks for the advice!

It makes sense to me, to examine the situations, where the evaluation of my engine differs from the opponent, when it is losing. Could be an indication of an evaluation problem.

I will try that!

The last thing i have tried is different replacement schemes for the hash table. The 0.18 version has a simple "always replace" scheme, so i am experimenting with having 1/2 or 1/3 depth preferred entries. Not a big improvement, but i need to test more, before being able to see how much.

Kind regards,
Jesper

[jesper_nielsen]

It seems there are many votes for a re-write!

I am not quite at that point yet. But i might be getting to it real soon, unless I start finding improvements again.

Writing a completely new engine would certainly re-ignite my motivation.

I have considered making a 0x88 engine, as an alternative to the rotated bitboard approach of Pupsi.

I did implement a very specialized 0x88 type "thing", for generating simple 3-4 man endgame tables. And no, they have not, so far, improved the play of Pupsi!

Kind regards,
Jesper

[swami]

ord Romstad, the author of Glaurung has his third engine already.
Edsel

I guess the Viper is an abridged version of Glaurung, So it's hardly a rewrite or a new engine,isn't it?Tord can clarify this.