statistics, testing and frustration

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
sandermvdb
Posts: 139
Joined: Sat Jan 28, 2017 12:29 pm
Location: The Netherlands

Re: statistics, testing and frustration

Post by sandermvdb » Wed Sep 12, 2018 8:45 pm

xr_a_y wrote:
Wed Sep 12, 2018 8:35 pm
Really appreciate you take time to check weini game. May I ask what analysis tool you are using on the pgn?
I use Arena to open the pgn and looked for the game where Weini lost the quickest :P That game was imported in lichess which easily spotted the blunder.

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: statistics, testing and frustration

Post by xr_a_y » Thu Sep 13, 2018 5:32 am

Edsel Apostol wrote:
Wed Sep 12, 2018 7:12 pm
I think most strong engines doesn't check for && score < beta anymore.
Usually you don't store a bestmove in the TT if score is equal or below alpha.
I think Xiphos and Vajolet is doing both.Didn't check others.
And when I remove that, Weini is losing elo (and does not solve fine70 anymore, I mean fast enough ...).

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: statistics, testing and frustration

Post by xr_a_y » Fri Sep 14, 2018 5:56 pm

Testing some advices given in this threat this week, I finally found something that wins some elo.

In Weini, I used to use the TT move only if

Code: Select all

!fromPV || ttt.t_type == Transposition::tt_exact
I now switch as (almost) Xiphos to (both in search and qsearch)

Code: Select all

!fromPV ||
                (ttt.t_type == Transposition::tt_exact
                   && ttt.score > alpha
                   && ttt.score < beta) ||
                (ttt.t_type == Transposition::tt_alpha
                   && ttt.score <= alpha) ||
                (ttt.t_type == Transposition::tt_beta
                   && ttt.score >= beta) ) )
Moreover, I used to not use the TT move directly (i.e. return a score immediatly) if it leads to a draw (I was afraid of false draw score due to 3 reps but now I use only 1 rep so probably no need to worry anymore). I now use the TT move (under the previous conditions) also if it leads to a draw.

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: statistics, testing and frustration

Post by asanjuan » Fri Sep 28, 2018 7:36 am

xr_a_y wrote:
Fri Sep 14, 2018 5:56 pm

Code: Select all

!fromPV ||
                (ttt.t_type == Transposition::tt_exact
                   && ttt.score > alpha
                   && ttt.score < beta) ||
                (ttt.t_type == Transposition::tt_alpha
                   && ttt.score <= alpha) ||
                (ttt.t_type == Transposition::tt_beta
                   && ttt.score >= beta) ) )
Moreover, I used to not use the TT move directly (i.e. return a score immediatly) if it leads to a draw (I was afraid of false draw score due to 3 reps but now I use only 1 rep so probably no need to worry anymore). I now use the TT move (under the previous conditions) also if it leads to a draw.
I think you are messing around a bit with TT.
This is what I do, I think that is what everyone is using, and it is pretty simple:

Code: Select all

	THashEntry entry;
	if (Hash.hash_get(Current->key, &entry)){
		
		if (TO_MOVE(entry.move) != NULL_MOVE && is_pseudolegal(TO_MOVE(entry.move))){			
			trans_move = TO_MOVE(entry.move);
			pv_copy( trans_move );			
		}
		
		entry.eval = hash_value_get(entry.eval, root_distance() );
		if (entry.ok_to_prune(depth, alpha, beta)){
			return entry.eval;
		}
		
	}
the ok_to_prune function looks like this:

Code: Select all

	
	bool THashEntry::ok_to_prune(int depth, int alpha, int beta){
	return (
		(isMate(this->eval) && this->type == typeExact) 		
		|| (
			(this->depth >= depth) && 
				(
				(this->type == typeExact) ||
				(this->type == typeAlpha && this->eval <= alpha) ||
				(this->type == typeBeta  && this->eval >= beta)
				)
			)
		);
	}
the isMate condition is not a real improvement, but shows shorter mates.
regards.

User avatar
Guenther
Posts: 2463
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: statistics, testing and frustration

Post by Guenther » Fri Sep 28, 2018 10:12 am

sandermvdb wrote:
Wed Sep 12, 2018 8:15 pm
I just downloaded the games Weini 0.0.20 played in the CCRL 40/4. This is the game it has lost in the least number of moves:
https://lichess.org/aoEmr338

It looks like a 3-fold repetition bug where it just throws away its queen and therefore loses the match.
Sander is quite right. You have at least a serious repetition bug!
I downloaded the commented CCRL 40/40 for Weini 0.023 and cleaned it with a script,
thus Toms GameAnalyser can read it (filtered out won games before with Scid because
we are not interested in those)

Fiddled a bit with some settings in TGA and found at least 3 games already in a few minutes,
which have the same symptom.
When a rep is on the horizon even when eval is worse it suddenly starts to blunder a piece
or worsens its position instead of trying to force a repetition and take the draw. (or even 50 moves draw)

Here are two examples from the first sample (Weini White).

Example1:
Asymptotes eval also is wrong for a long while and Weini showing around -0.84 has a comfortable
positionally draw and could have just sat with its K around the Black passer on e3 awaiting a 3 time rep
or a 50 moves draw as Black can do nothing.
Instead it suddenly created a decisive weakness with 119.g6??
I could understand it if would show a positive score and attempts to win.



Example 2:
In this game it is even more visible. There was already a two time repetition after 57...Qf5, but suddenly
it played Nxb4?? with a slightly positive score though for only one ply (before always slightly negative).


User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: statistics, testing and frustration

Post by xr_a_y » Fri Sep 28, 2018 3:05 pm

Thanks a lot for this analysis. I guess 0.0.24 (not released yet) doesn't show the same issue but i'll check that twice. There are indeed strange things going on with draw even in the dev version...

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: statistics, testing and frustration

Post by xr_a_y » Fri Sep 28, 2018 3:07 pm

asanjuan wrote:
Fri Sep 28, 2018 7:36 am

Code: Select all

	
				(this->type == typeExact) ||
				(this->type == typeAlpha && this->eval <= alpha) ||
				(this->type == typeBeta  && this->eval >= beta)

Thanks for your inputs. So you suggest to use the TT move and return a score with it as soon as it is an exact score, even at a PV node ?

Sven
Posts: 3699
Joined: Thu May 15, 2008 7:57 pm
Location: Berlin, Germany
Full name: Sven Schüle
Contact:

Re: statistics, testing and frustration

Post by Sven » Sat Sep 29, 2018 3:39 pm

Guenther wrote:
Fri Sep 28, 2018 10:12 am
Example1:
Asymptotes eval also is wrong for a long while and Weini showing around -0.84 has a comfortable
positionally draw and could have just sat with its K around the Black passer on e3 awaiting a 3 time rep
or a 50 moves draw as Black can do nothing.
Instead it suddenly created a decisive weakness with 119.g6??
I could understand it if would show a positive score and attempts to win.
119.g6?! is not optimal but still a draw. The decisive error is 120.Ke2? where the only move that holds the draw is 120.Kc3! Therefore another possible explanation of the symptom would be that the search depth was not sufficient to see that Kc3 draws but Ke2 loses. The score 0.00 calculated for 120.Ke2, compared to -2.99 after the next move, makes this quite likely for me. I do not see a clear connection to the repetition topic in this case.
Sven Schüle (engine author: Jumbo, KnockOut, Surprise)

User avatar
Guenther
Posts: 2463
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: statistics, testing and frustration

Post by Guenther » Sat Sep 29, 2018 4:03 pm

Sven wrote:
Sat Sep 29, 2018 3:39 pm
Guenther wrote:
Fri Sep 28, 2018 10:12 am
Example1:
Asymptotes eval also is wrong for a long while and Weini showing around -0.84 has a comfortable
positionally draw and could have just sat with its K around the Black passer on e3 awaiting a 3 time rep
or a 50 moves draw as Black can do nothing.
Instead it suddenly created a decisive weakness with 119.g6??
I could understand it if would show a positive score and attempts to win.
119.g6?! is not optimal but still a draw. The decisive error is 120.Ke2? where the only move that holds the draw is 120.Kc3! Therefore another possible explanation of the symptom would be that the search depth was not sufficient to see that Kc3 draws but Ke2 loses. The score 0.00 calculated for 120.Ke2, compared to -2.99 after the next move, makes this quite likely for me. I do not see a clear connection to the repetition topic in this case.
Hi Sven, I wrote this because 119.g6 was played with negative score (-0.72), why should one move a pawn, when the the 50 moves counter
already is at 20 and there was already a 2 time rep before I think? I don't think this requires a kind of fortress detection?
I don't know though what other programs do when eval is negative and there is a chance to accumulate the counter?
OTH there might be a problem with 50 moves draws too, may be I check more games later.

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: statistics, testing and frustration

Post by asanjuan » Mon Oct 01, 2018 9:11 am

xr_a_y wrote:
Fri Sep 28, 2018 3:07 pm
asanjuan wrote:
Fri Sep 28, 2018 7:36 am

Code: Select all

	
				(this->type == typeExact) ||
				(this->type == typeAlpha && this->eval <= alpha) ||
				(this->type == typeBeta  && this->eval >= beta)

Thanks for your inputs. So you suggest to use the TT move and return a score with it as soon as it is an exact score, even at a PV node ?
Don't forget the depth condition in order to return a TT cutoff.
if entry->depth >= depth AND (the above conditions) then CUTOFF. returning alpha, beta or tt-value is irrelevant in a PVS search since the upper level of the search will discard the move or research it with more depth.

The only reason to not to prune and return in a PV node is because you want to print the whole pv in a re-search. But this is something cosmetic.

Just in case you couldn't prune the tree with a TT cutoff, if there is a move in your TT, then use it first in your move loop, even before you start to generate moves.

And yes, at PV nodes is the same.

Post Reply