Progress On Bricabrac

[lithander]

The very early beginning of an evaluator that starts each search with zero knowledge of chess other than material that learns to evaluate during the search.

Does it forget everything it learned after returning a best move and learns from scratch again the next time a search is started (on a different position)?

And just generally, how does that work? I think I understood your idea of creating PSTs at runtime that are tailored to the current position but this is something else, entirely, right?

[Mike Sherwin]

The very early beginning of an evaluator that starts each search with zero knowledge of chess other than material that learns to evaluate during the search.

Does it forget everything it learned after returning a best move and learns from scratch again the next time a search is started (on a different position)?

And just generally, how does that work? I think I understood your idea of creating PSTs at runtime that are tailored to the current position but this is something else, entirely, right?

Hi Thomas,

Yes, right now before each search for a new position it zero's the pst. It won't always be that way. It is that way now because I want to see just how well it can evaluate from scratch. Once it is working really well then there are options going forward. It can retain its learning between searches or it can start with either a dynamic pst or a static pst. It works by collecting statistics during the search and using those statistics to modify the pst.

There was some interesting strategical play in those games, were there not? And that attack in the first game by black being a piece down looked like it was well planned once black found its footing. Not bad for a search that starts off as a material only searcher, every search!

[Mike Sherwin]

I am forking Bricabrac into three.
1. Bricabrac with RomiChess style dynamic pst or as the CPW says, an Oracle type evaluation.
2. Bricabrac with zero chess knowledge (zeroed pst) that learns the pst during search.
3. Bricabrac with both 1 & 2.

The quick and not very good yet Oracle version is at present about 1400. The zeroed pst is about 1500. Both combined (the sum is greater than its parts) is about 1750.

[lithander]

In some other thread you mentioned you had also implemented PeSTO tapered eval as a reference. What was the playing strength of that version?

[Mike Sherwin]

In some other thread you mentioned you had also implemented PeSTO tapered eval as a reference. What was the playing strength of that version?

Well, it was a bugged version that gave many points away because of the bugs. So it is impossible to say with any confidence what the playing strength was. It's best result against TSCP was + 191 in a 100 game match. On the other hand the slightest change to the code while hunting down the bugs caused the results to be all over the place, from +56 to well, +191. The one RR that included RomiChess, MCE Chili, Rustic alpha 2 and TSCP had these results.

Bricabrac - MinimalChessEngine : 39.0/100 16-38-46 (00=0=0=0=0=0000==00===0000=1=0=1=0=0=001001===1==11=1==0==011=01==0======0011=00000===001===0=101===) 39% -78
Bricabrac - RomiChess64P3n : 6.0/100 1-89-10 (0000=000000000=000=00000=00==0000000000=00000000000000000000000000000000000000000000=000=00000001=00) 6% -478
Bricabrac - Rustic-alpha-2_64-bit-popcnt : 46.0/100 25-33-42 (=0=011=0==10=0====10=0=0=001=0=11=0010============1=11=101=01000010==0=0=01010110010==11==001=10==10) 46% -28
Bricabrac - Tscp181 : 65.0/100 50-20-30 (111101001=0==1011111==0111==0111=11==11=11=01==0011=111011==0=11=1=01111=1010=111001==1=1==00==01=11) 65% +108

It is problematic to infer just how many points were lost due to the bugs. Could be anywhere from 50 to 100 elo was lost.

[Mike Sherwin]

After a couple thousand test games with no rules infarctions and no time forfeits I finally can say that Bricabrac is free of any major bugs!

As of now the new evaluation is not complete so I decided to revisit PeSTO for these test and upload. The revisit faithfully (or tries to at least) follow exactly the design of PeSTO. Because, material in Bricabrac is not included in the piece square tables only the material difference between PeSTO and Bricabrac was included in the PeSTO tables. The other difference is that the evaluation is not end-leaf meaning that the PeSTO tables are only set up before the search begins.

The previous use of PeSTO did not take material into consideration and it did not interpolate between middlegame and endgame tables. Instead it just did a hard switch, so that at less than 4000 total material it switched to the endgame tables. And it was stronger. Even with all the bugs and lack of 3-fold detection it had better results. So maybe that can be tried by anyone interested.

Even though the "correct" implementation is not as strong it is fine as a reference to test my new evaluation against when it is ready. Other than for testing I am done with PeSTO and there will be no further work done on it. However, if someone can show me that I made a mistake in its implementation I would like to make a corrected version.

In the following test "simple" PeSTO and PeSTO zygote was tested. Zygote is the name of the most simple implementation of my new search function to evaluation function feedback learning mechanism. It is in the source code as an // #ifdef zygote directive. To experiment with it just remove the forward slashes. It was compiled with MSVS 2019. If another compiler is used some work will need to be done.

A little reasoning about the test results. It appears but might not be true that MCE Chile may been heavily influenced by PeSTO. Maybe Thomas will clarify that. With correct PeSTO that would mean that the following results would make sense.

BricPESTO1 - MinimalChessEngine : 31.5/100 17-54-29 (=0=000=1=000=0=001101==00==0000=000=0000=00000=0=00111000====0101=000=10=10==001000=100=110=1000==10) 32% -131
BricPESTO1 - Rustic-alpha-2_64-bit-popcnt : 58.0/100 51-35-14 (1011100=10001001==1010100100110000=11=10=111111=101=01001=1111101=1101101=1000111101111=1=001=101001) 58% +56
BricPESTO1 - Tscp181 : 59.5/100 54-35-11 (=11101=11010000=1=1110111001=1=01=011111000010111100111=11011010110110=1101100011110101100=0100=0111) 60% +70

1939 - 131 = 1808
1815 + 56 = 1871
1724 + 70 = 1794
performance = 1824

In the next test PeSTO is modified by zygote. It looks like that maybe zygote changing PeSTO had a positive effect against MCE Chili but that it did not do as well against Rustic is counter intuitive. But, then 300 games in the test can just mean that it is just noise.

BricPESTOz1 - MinimalChessEngine : 44.0/100 33-45-22 (1010=01011==000110==0=001001=11100001000=011000=0=011110=01=111=0011==000=10==010==0=000100011101=01) 44% -42
BricPESTOz1 - Rustic-alpha-2_64-bit-popcnt : 47.5/100 40-45-15 (10110011111000==1001111101111==01=0=00==000000100=001=001=1000100000=00100011011=100100011=011=11111) 48% -14
BricPESTOz1 - Tscp181 : 60.5/100 54-33-13 (100=010=111=01=11=00010100010111101100111001111=11=010=110110111101==010011111000011=1110=110111=011) 61% +78

1939 - 42 = 1897
1815 - 14 = 1801
1724 + 78 = 1802
performance = 1835

PeSTO + zygote over PeSTO, +11. It might just be noise but it is a start!
https://www.mediafire.com/file/eraa5fx4 ... O.zip/file

[mvanthoor]

The Zygote version scored -14 against Rustic, so that is the best version (for me)

The reason why a version does worse against Rustic can be because Rustic is a very tactical engine. If an engine is around the same strength, it wins against Rustic mostly by better positional play, or by positional tricks.

We can see that against TSCP. Rustic doesn't know anything about passed pawns. If Alpha 1 without TT plays against TSCP, then it's performance is something like -100. If the version with the TT plays, the performance is something like +150, because TSCP's one-trick pony of passed pawn promotion doesn't work anymore. (Because of the TT, Rustic will see the pawn promotion 20 moves ahead and move to stop it.)

That is why I haven't really put any more work into the eval up until now. If you could analyze all games right up to the end, you would only need to know if a position is a draw, win, or lost, without even having to know anything else about positional play. That's not possible, so we need an evaluation for the intermediate positions, to lead an engine from one good position to the next, so it can hopefully do that better than the opponent and win.

[Mike Sherwin]

The Zygote version scored -14 against Rustic, so that is the best version (for me)

The reason why a version does worse against Rustic can be because Rustic is a very tactical engine. If an engine is around the same strength, it wins against Rustic mostly by better positional play, or by positional tricks.

We can see that against TSCP. Rustic doesn't know anything about passed pawns. If Alpha 1 without TT plays against TSCP, then it's performance is something like -100. If the version with the TT plays, the performance is something like +150, because TSCP's one-trick pony of passed pawn promotion doesn't work anymore. (Because of the TT, Rustic will see the pawn promotion 20 moves ahead and move to stop it.)

That is why I haven't really put any more work into the eval up until now. If you could analyze all games right up to the end, you would only need to know if a position is a draw, win, or lost, without even having to know anything else about positional play. That's not possible, so we need an evaluation for the intermediate positions, to lead an engine from one good position to the next, so it can hopefully do that better than the opponent and win.

My goal with Bricabrac, if I live long enough, is to change that by taking RomiChess reinforcement learning to Realtime learning instead of after game learning. Wish me luck!

[mvanthoor]

My goal with Bricabrac, if I live long enough, is to change that by taking RomiChess reinforcement learning to Realtime learning instead of after game learning. Wish me luck!

Good luck Live long and prosper

[Mike Sherwin]

My goal with Bricabrac, if I live long enough, is to change that by taking RomiChess reinforcement learning to Realtime learning instead of after game learning. Wish me luck!

Good luck Live long and prosper

My calculations indicate that it will take me exactly another 20 months, 8 weeks, 1 day, 14 hours, 11 minutes and 19 seconds to complete.