Position learning test for Critter 1.4

[rodolfoleoni]

This test is being run on 100 games, 1+1 TCs, 1 CPU and ponder off with this thematic position, where both white and black can blunder:

r1b1kb1r/2q2p1p/p2p1P2/2nP4/1p1N4/5Q2/PPP4P/2KR1BR1 b kq - 3 1

The opponent is Houdini 1.5a, which has no learning features.

Critter uses a 32 MB session file, strategy=score, resolve score drops checked on. This is Critter progression after 70 games:

1: Critter_1.4_64bit_sse4 36.5/70 0==0===00=00=000=0101=101010101011=110=1111====0100=1=1=1=1===1=1=1===

I had time to watch only first few games, and I got the impression of a wrong (or too fast) backwards propagation. Critter was making some silly moves as a consequence, so we have +0 -10 =8 in first 18 games.

Test is going on. After that, I'll try to turn "resolve score drops" off, then I'll try the depth strategy option.

[Vinvin]

Amazing nice feature !!

14/35 first half
22.5/35 second half !

[rodolfoleoni]

Yes, nice indeed. But my impression is that Critter gets weakened for a while because of unsafe score propagation. At least, in this setting. I'll try others tonight.

This is final score and progression of Critter in this test:

1: Critter_1.4_64bit_sse4 54.0/100 0==0===00=00=000=0101=101010101011=110=1111====0100=1=1=1=1===1=1=1=====0=1=1==1====1=1=====110=0=1=

A good result by a monster against a monster...

[rodolfoleoni]

Step 2

Critter started with an empty session file, same conditions as before except that resolve score drops option was checked off. This is Critter progression in first 60 games.

Critter_1.4_64bit_sse4 25.5/60 0=101=001=1=10=00010101010=010101010=0=====0==1==010==0=0=0=

Learning seems much slower than before. I watched no games, anyway, as I just got home, so I cannot have any impression.

[rodolfoleoni]

Critter_1.4_64bit_sse4 41.5/100 0=101=001=1=10=00010101010=010101010=0=====0==1==010==0=0=0=0101=1===0010=0=1011000=010=0=0=010=01=0

I noticed something that could be wrong, bit I'm too sleepy now and it requires some time to report it. So I start step 3 (strategy=depth + resolve score drops=on) and I go to sleep...

[rodolfoleoni]

Critter_1.4_64bit_sse4 51.0/100
0=1=10==10=0100000=0=010101000000000=00=0=1=1=====1=1=1=0=011=111===1111==1=111=1=====1=1==0101===1=

[rodolfoleoni]

.............................
I noticed something that could be wrong, bit I'm too sleepy now and it requires some time to report it.
............................

So, what was wrong? Playing white, Critter encountered many times (25 or more) the following position:

r3kbr1/1bq2p1p/p1Np1P2/3Pn3/1p6/5Q1B/PPP4P/2KRR3 w q - 7 5

(sorry, I don't know how to build a diagram)

Every time, he performed the same moves, showing the same PVs, with Houdini making the expected reply, and having score falling after two moves. (Critter plays white)

5. Qf4 {(Qf3f4 Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.65/18 10}

Bxc6 {(Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1g1 Qc7b6 Rg1f1 Bh6g7 a2a3 Ne5xd7 c6xd7 Ra8b8 b2b3 Qb6a5 Rd1d3 Qa5e5 Rd3f3) -0.22/15 2}

6. dxc6 {(d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.73/14 0}

Rg6 {(Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Re1xe5 d6xe5 Bd7g4+ Kd8e8 Bg4d7+ Ke8d8 Bd7g4+) -0.22/16 1}

7. Kb1 {(Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.10/15 2}

No scoring backwards here. Now, I know so little about this learning system and I cannot be sure on anything. Maybe, it's due to the "resolve score drops" being unchecked. But I guess a simple hash hit should have caused that backwards propagation, and that didn't happen.

[rodolfoleoni]

Step 4

Session file cleared, strategy=depth, resolve score drops=off, same test conditions against Houdini 1.5a.

Unexpected 50 games performance.....

Critter_1.4_64bit_sse4 30.0/50
0=000101=01=11===11011=110001011101=1=1=1=0=111110

Let us see what happens on next 50!

[Sylwy]

.............................
I noticed something that could be wrong, bit I'm too sleepy now and it requires some time to report it.
............................

So, what was wrong? Playing white, Critter encountered many times (25 or more) the following position:

r3kbr1/1bq2p1p/p1Np1P2/3Pn3/1p6/5Q1B/PPP4P/2KRR3 w q - 7 5

(sorry, I don't know how to build a diagram)

Every time, he performed the same moves, showing the same PVs, with Houdini making the expected reply, and having score falling after two moves. (Critter plays white)

5. Qf4 {(Qf3f4 Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.65/18 10}

Bxc6 {(Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1g1 Qc7b6 Rg1f1 Bh6g7 a2a3 Ne5xd7 c6xd7 Ra8b8 b2b3 Qb6a5 Rd1d3 Qa5e5 Rd3f3) -0.22/15 2}

6. dxc6 {(d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.73/14 0}

Rg6 {(Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Re1xe5 d6xe5 Bd7g4+ Kd8e8 Bg4d7+ Ke8d8 Bd7g4+) -0.22/16 1}

7. Kb1 {(Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.10/15 2}

No scoring backwards here. Now, I know so little about this learning system and I cannot be sure on anything. Maybe, it's due to the "resolve score drops" being unchecked. But I guess a simple hash hit should have caused that backwards propagation, and that didn't happen.

A bit of help

[D]r3kbr1/1bq2p1p/p1Np1P2/3Pn3/1p6/5Q1B/PPP4P/2KRR3 w q - 7 5

Maestro Rodolfo , The Baron is now an uncleless child ?
Or is a STASI concerted spy action versus Critter 1.4 learning function ?

Have a nice evening,
SilvianR

[rodolfoleoni]

.............................
I noticed something that could be wrong, bit I'm too sleepy now and it requires some time to report it.
............................

So, what was wrong? Playing white, Critter encountered many times (25 or more) the following position:

r3kbr1/1bq2p1p/p1Np1P2/3Pn3/1p6/5Q1B/PPP4P/2KRR3 w q - 7 5

(sorry, I don't know how to build a diagram)

Every time, he performed the same moves, showing the same PVs, with Houdini making the expected reply, and having score falling after two moves. (Critter plays white)

5. Qf4 {(Qf3f4 Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.65/18 10}

Bxc6 {(Bb7xc6 d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1g1 Qc7b6 Rg1f1 Bh6g7 a2a3 Ne5xd7 c6xd7 Ra8b8 b2b3 Qb6a5 Rd1d3 Qa5e5 Rd3f3) -0.22/15 2}

6. dxc6 {(d5xc6 Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.73/14 0}

Rg6 {(Rg8g6 Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Re1xe5 d6xe5 Bd7g4+ Kd8e8 Bg4d7+ Ke8d8 Bd7g4+) -0.22/16 1}

7. Kb1 {(Kc1b1 Bf8h6 Qf4xb4 Rg6xf6 Bh3d7+ Ke8d8 Qb4h4 Kd8e7 Re1f1 Bh6g7 b2b3 Bg7h8 Rd1e1 Ra8g8 a2a3 Qc7a5 Kb1b2 Ke7f8 Rf1xf6 Bh8xf6 Qh4xf6 Qa5xe1 c6c7 Ne5d3+ Kb2a2 Nd3c1+ Ka2b2 Nc1d3+) +0.10/15 2}

No scoring backwards here. Now, I know so little about this learning system and I cannot be sure on anything. Maybe, it's due to the "resolve score drops" being unchecked. But I guess a simple hash hit should have caused that backwards propagation, and that didn't happen.

A bit of help

[D]r3kbr1/1bq2p1p/p1Np1P2/3Pn3/1p6/5Q1B/PPP4P/2KRR3 w q - 7 5

Maestro Rodolfo , The Baron is now an uncleless child ?
Or is a STASI concerted spy action versus Critter 1.4 learning function ?

Have a nice evening,
SilvianR

Hi Silvian,

thanks for help with diagram.

It was 6 years ago when I joined the Baron team, and it'll always be my only real passion. While running this test with Critter learning feature, all remaining cpus of my i7 are running a test with 16 different Baron configurations...

I like very much to test learning systems, anyway, since the time of the good old Fritz 3 and its 5 KBs max size of learning file... And I tested many, many different systems. Chess Genius 5... Knightcap and its innovative TD-Lambda eval learning... Romichess with its learning/building game book... Maybe, someone will remember the matches I posted here with Romichess beating Rybka in a 100 games match, thanks to its learning. Spike, Alaric, Frenzee, Smarthink... many engines with their systems. And I ran more than dozens learning tests with the Baron since them. I must say, Baron has a special learning feature no other engine has.