What happens using egbb

[Daniel Shawul]

When I decided to abandon Scorpio bitbases it was not because Houdini messed them up but because I saw many not particularly strong engines mess it up.
Now I understand that it was not the inherent fault of the Scorpio bitbases but their incorrect implementation.
This thread has been instructive for me. In the future I am going back to Scorpio egbb's, watch engines more closely and disable egbb's only with those engines that cannot benefit from their use.

However, the tone of discussion could improve.

Please Gabor stop the lecture, you would be surprized by the number of people that want to do that specially after doing their best to cause a flame and failing flat. I am not saying you did this intentionally BUT lets look at the facts: you looked at one instance and said you wanted to drop egbbs, then _ignored my reply_ (like Kai did) which is weird, chose to listen to competitor's reply. Well if you don't listen to the guy who created the Egbbs, then one would be inclined to conclude you weren't interested in getting a feedback but just adding fuel and reinforcing a false belief that egbbs had a problem to the extent that they should be dropped.

So let me make it loud and clear, I don't give a hoot that the test your relied up on to show this to the world crashed and burned, specially after these treatment I got. The least you should do to finish up this discussion is answer the question that I asked you in the beginning and provide positions that scorpio failed at, which you would find is very very rare. At least that makes up for something you ignored up until now it becomes so clear who was at fault. But hey no, lets comment on the tone, try to extend disussions, and I don't get a clear "I made a mistake" which goes a long way. Well I guess in the future I should take anything what 'CCRL testing group' says with a grain of salt, even though I repeatedly said I don't have anything against CCRL testing group...

[Daniel Shawul]

here you will see a perfect example of scorpio egtb failing to close in won positions. plenty of games to look at...

http://rybkaforum.net/cgi-bin/rybkaforu ... pid=444588

Yep Houdart should clear the bad mouthing of egbbs that ensued with his screwed up implementation as evidenced with the thread you quoted. It took until today to see for him the elephant in his room (hint: look at his mirror), instead of pointing fingers. If you think about it, my job is finished once I provided the WDL(Win,Loss,Draw) data!! It is up to the engine author to know how to use them, but in fact I go far more than that to help the engine authors unlike many other WDL databases. It takes a little to use them just like Nalimov's endgames but apparently some fall flat on that little requirement...

[Laskos]

LOS 99.9% as a stopping rule.

This rule is utter nonsense and makes your results completely void.

Also, are you sure you are talking about LOS (Bayesian model)? Or p-value?

I cannot begin to count the number of times when I've seen results like that after 100 or 200 games, and things change after 10000 games...

That being said, it is very possible that your conclusion is correct (Scorpio EGBB hurt more in terms of perf than they help). But what you show is no evidence whatsoever.

Here you again. I already posted results that a LOS of 99.9% with >50 (and <50,000) games has Type II error of less than 5%, and a stopping rule with LOS 99.9% for 50-50,000 games is valid, although usually slower than SPRT. SPRT would probably need even less than 220 games to stop with alpha, beta 0.05. LOS has unbounded Type II error for number of games -> infinity, that's all, go re-read that thread about it.

[Laskos]

Kai, the least you could do after your experiment's fault is pointed out is NOT end it with a rhetorical question, which sure is even if you deny it. Fact is you swiftly made ridiclous conclusions, ignored my replies since you made a point, so no I don't belive you had honest intentions. Asking age old questions that do EGTB/EGBBs help to make a point is nothing but rhetorical, seemingly because your intentions went flat.

Come on, what intentions? If I hoped to see an ELO point from endgame bases, this was from egbb loaded into RAM, and it became obvious in this thread that Houdini is not the engine to test with. I was very surprised to see such abysmal results with Houdini, it's an anomaly and I posted it. It turned out to be Houdini's fault. Do you read intentions of people posting here, and do you think that all participate in great wars about endgame bases?

[Daniel Shawul]

Ok Kai at least now you said clearly 'Houdini is at fault' which is important for the casual reader, which is what I am concerned with. If you don't believe me, look at a rybka thread that someone posted here. So lets not ask any more questions, and get back to our friendly ways

[Houdini]

Without DTM or DTZ information there is no fail-proof procedure that can guarantee the win in difficult endgames like KQPvKQ or KQvKNN.

Sorry but Houdini drew a won krpkr, it's not what I would call a difficult endgame.
Should be no problem when driven by eval, especially if Houdini can win this position by itself. I wouldn't blame the bitbases.

I'm not sure what the fuzz is about, all this has been reported for about a year now and is written in the user's manual.
The Houdini 3 egbb implementation was not intended for match play. If you want to play matches with Houdini 3 and EGTB please use Gaviota or Nalimov table bases.

[Ferdy]

Yes, it seems a Houdini problem. Is the ELO benefit from egbb measurable in Deuterium?

I start testing the latest released Deuterium with egbb at tc 40moves/60sec, with different positions and color reversed.

(1) From positions with mixed pieces on late middle game but more than 5-men. Target games is 1600
Bayeselo:

Code: Select all

Rank Name                             Elo     Diff     +     -      Games  Score    Oppo.   Draws     Win          W-L-D 
   1 Deuterium-113-egbb              7.85     0.00   7.06   7.06     1091  52.25%   -7.85  57.93%  23.28%       254-205-632
   2 Deuterium-v13.1.31.113-64bit   -7.85   -15.69   7.06   7.06     1091  47.75%    7.85  57.93%  18.79%       205-254-632

SPRT:

Code: Select all

Engine: Deuterium-113-egbb
SPRT: elo0 = -1.5, elo1 = +4.5, a = +0.05, b = +0.05
LLR = +1.25278 (-2.94444, +2.94444)
T = +1091, W = +254, L = +205, D = +632, WNet = +49

Next will be the following, positions generated from Ed's protools.
(2) Rook and pawn ending but more than 5-men. Target games is 200
(3) Pawn ending but more than 5-men. Target games is 200
(4) Queen and pawn ending but more than 5-men. Target games is 200
(5) Bishop and knight and pawn ending but more than 5-men, Target games is 200

[Daniel Shawul]

Oh well look at that, what a surprise . Hmm... egbbs seem to help Deuterium gain some elos that implemented them correctly, in fact more than I expected. I hope this will clear up the FUD that is being spread about them hurting engines. They help as much as any other bitbase out there, but many don't want to blind themselves with facts, instead chose to repeat a self-reinforcing belief that they hurt... Not much I can do about that.

[Laskos]

Yes, it seems a Houdini problem. Is the ELO benefit from egbb measurable in Deuterium?

I start testing the latest released Deuterium with egbb at tc 40moves/60sec, with different positions and color reversed.

(1) From positions with mixed pieces on late middle game but more than 5-men. Target games is 1600
Bayeselo:

Code: Select all

Rank Name                             Elo     Diff     +     -      Games  Score    Oppo.   Draws     Win          W-L-D 
   1 Deuterium-113-egbb              7.85     0.00   7.06   7.06     1091  52.25%   -7.85  57.93%  23.28%       254-205-632
   2 Deuterium-v13.1.31.113-64bit   -7.85   -15.69   7.06   7.06     1091  47.75%    7.85  57.93%  18.79%       205-254-632

SPRT:

Code: Select all

Engine: Deuterium-113-egbb
SPRT: elo0 = -1.5, elo1 = +4.5, a = +0.05, b = +0.05
LLR = +1.25278 (-2.94444, +2.94444)
T = +1091, W = +254, L = +205, D = +632, WNet = +49

Next will be the following, positions generated from Ed's protools.
(2) Rook and pawn ending but more than 5-men. Target games is 200
(3) Pawn ending but more than 5-men. Target games is 200
(4) Queen and pawn ending but more than 5-men. Target games is 200
(5) Bishop and knight and pawn ending but more than 5-men, Target games is 200

Thanks, so it seems than there is a real ELO benefit from egbb in late middlegames. Your error margins are 1SD, right? 15 ELO points are a lot, even if it reduces with more games to 5 ELO points, it's highly significant. Would be even better if you wait until SPRT stop, to have clear uncertainties.

[Ferdy]

Oh well look at that, what a surprise . Hmm... egbbs seem to help Deuterium gain some elos that implemented them correctly, in fact more than I expected. I hope this will clear up the FUD that is being spread about them hurting engines. They help as much as any other bitbase out there, but many don't want to blind themselves with facts, instead chose to repeat a self-reinforcing belief that they hurt... Not much I can do about that.

I was surprised by this current performance so far. That would only mean that I screwed up my ending heuristics