What's the trick ?

[bob]

If it is true that you need to cross a certain threshold of inherent depth before aggressive reductions and pruning starts to work then you get a testing problem. For you can only test it using large time controls, which means it costs many days or weeks of computing time. I cannot afford that.
And if it is no improvement it's a total waste of time and money (electricity) and inconvenience for your computer is busy all the time.

I think is not that terrible. I tried some aggressive LMR approaches, of course seeing other modules open source ideas and my own ones, and I was able to reach some working ideas without much work. Then I improved and changed it many times.
Sure some ours of testing are necessary, but if you can afford some hundred games, you will see at least if is more or less working. It's not necessary days of testing. With less you see it.

Dangerous thinking. With a few hundred games, the error bar is very large. You will make mistakes by thinking a change is good when it is not, or thinking it is bad when it is not. There is no substitute for enough games to get the error bar down to something that lies well inside the expected gain or loss for your new idea.

[cdani]

Dangerous thinking. With a few hundred games, the error bar is very large. You will make mistakes by thinking a change is good when it is not, or thinking it is bad when it is not. There is no substitute for enough games to get the error bar down to something that lies well inside the expected gain or loss for your new idea.

Yes, you have all the reason. I tell this of few hundred games only because he tells that maybe can't afford longer matches, and instead of just get stuck, it's better this than nothing.

[bob]

Dangerous thinking. With a few hundred games, the error bar is very large. You will make mistakes by thinking a change is good when it is not, or thinking it is bad when it is not. There is no substitute for enough games to get the error bar down to something that lies well inside the expected gain or loss for your new idea.

Yes, you have all the reason. I tell this of few hundred games only because he tells that maybe can't afford longer matches, and instead of just get stuck, it's better this than nothing.

It might not be "better than nothing". I play matches all the time where after 1000 games, version A looks better, but after 30,000 games, version B is clearly better. What you are doing is not much better than just randomly accepting or rejecting a new version...

Once you wrap your head around that (and it sometimes takes a while, to be sure) you will begin to make forward progress rather than random steps forward and backward...

[Henk]

So serious testing is impossible unless you own 100 processors. I think I can afford testing matches up to 10 milliseconds per move only. I wonder if there are any serious tests done on games lasting 4 hours or so for it would take 30000 * 4 /24 days = 500 days using one processor.

[cdani]

We have two truths. The one of that serious testing is impossible without tens of thousands of games, and the one that worked, not only for me, to push the engine from 1700 elo to +2600 in some months, may be letting one computer work many nights but doing less than 3000 games.

I want to believe that the first is the one and only, but apparently is not that simple.

Of course is not that simple because many thousand games are substituted by the knowledge of that something will work, because you see the concept working in other engines and, some of them, working when you play chess.

It's the second strategy limiting the future of the engine? In some ways sure.

[Henk]

I wonder how they were testing fifteen or twenty years ago when computers were much slower.

Maybe the first step is to make operations as fast as possible. For if input and output is the same not much testing is needed.