LCZero: Progress and Scaling. Relation to CCRL Elo

[Laskos]

That's my point: it might not be finding it later, it might just be choosing to play it later, which is very different. it might see e5, and see there is no way of preventing it, and somehow decide that a bishop move first improves its evaluation by 0.01 pawns on some invisible evaluation scale. This is common in all engines, and I have seen tons of examples of some unimportant zwischenzug thrown in before the final killing blow.

For the record, Leela played e5 in a second.

So we'd really need to look at the PVs. Is that possible with Leela?

Of course, why would you think it wasn't?

But that's also true of normal engines. Do you think WAC does not rank normal engines accurately in practice?

WAC is like an IQ test: it does not test intelligence, but the ability to score in IQ tests. In this case it is a bit of a lithmus test for easy tactics, but only so long as those positions have a concrete solution. If it has three winning moves (for example), then you are not only testing if it finds one of the three, but which one. As far as I am concerned, a proper tactics test does not allow for multiple solutions.

Albert, I myself was curious about game-changing, unique solutions tactical suite. I managed to collect from ECM middlegame test suite 60 such tactical positions, if I checked them correctly. Many, almost all easier (old) suites, have multiple or non game-changing solutions, or even wrong solutions altogether.

Check these out:

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";
8/2k3p1/2p4p/5P2/2K3PP/8/8/8 w - - bm g5; id "ECM.603";
8/1kp1b3/1p4K1/4P2p/P1P3p1/5pP1/P4P2/4B3 b - - bm h4; id "ECM.604";
8/8/3K1k2/5p1p/4p1p1/4P1P1/5PP1/8 b - - bm f4; id "ECM.606";
6r1/1p3k2/pPp4R/K1P1p1p1/1P2Pp1p/5P1P/6P1/8 w - - bm Rxc6; id "ECM.608";
1r3qk1/pb3p1p/1pn2PpQ/2pN4/3r4/5B2/PPP4P/4RRK1 w - - bm Ne7+; id "ECM.611";
4r1k1/5p1p/3q2p1/1p1P4/1P6/2p4P/2Q1nPB1/4RK2 b - - bm Ng3+; id "ECM.612";
7k/4b2p/5p1P/5PP1/1pNp1P2/1P1B4/2P2K2/r7 w - - bm g6; id "ECM.619";
3q1k2/5p2/p5pN/1b2Q2P/8/8/5PPK/8 w - - bm Qh8+; id "ECM.622";
6k1/p3b1pp/4p3/4Pp2/Pp1r1P1P/1P4P1/2p2R2/5RK1 b - - bm Rc4; id "ECM.623";
rn1q2k1/pp3pb1/3p2pp/2pP2N1/3r1P2/7Q/PP4PP/R1B2RK1 w - - bm Nxf7; id "ECM.628";
8/6Bp/6p1/2k1p3/4PPP1/1pb4P/8/2K5 b - - bm b2+; id "ECM.629";
6k1/5pbp/6p1/2p1r3/1pPr4/3n2NP/1PB2PP1/1R1R2K1 b - - bm Nxb2; id "ECM.633";
r4rk1/ppq3pp/2p1Pn2/4p1Q1/8/2N5/PP4PP/2KR1R2 w - - bm Rxf6; id "ECM.636";
6k1/p4pp1/Pp2r3/1QPq3p/8/6P1/2P2P1P/1R4K1 w - - bm cxb6; id "ECM.641";
6k1/p4pbp/Bp2p1p1/n2P4/q3P3/B1rQP3/P5PP/5RK1 w - - bm dxe6; id "ECM.642";
8/2k5/2p5/2pb2K1/pp4P1/1P1R4/P7/8 b - - bm Bxb3; id "ECM.646";
2r5/1r5k/1P3p2/PR2pP1p/4P2p/2p1BP2/1p2n3/4R2K b - - bm Nd4; id "ECM.647";
4r2k/p2qr1pp/1pp2p2/2p1nP1N/4R3/1P1P2RP/1PP2QP1/7K w - - bm Rxg7; id "ECM.648";
8/1R2P3/6k1/3B4/2P2P2/1p2r3/1Kb4p/8 w - - bm Be6; id "ECM.650";
2kr2r1/pp2bQ1p/2b1P3/2qN4/8/1B2p2P/PPP3P1/3R1R1K b - - bm e2; id "ECM.651";
r1b2rk1/1p2qppp/p3p3/2n5/3N4/3B1R2/PPP1Q1PP/R5K1 w - - bm Bxh7+; id "ECM.652";
6rk/3nrpbp/p1bq1npB/1p2p1N1/4P1PQ/P2B3R/1PP1N2P/5R1K w - - bm Nxh7; id "ECM.655";
1rb2rk1/3nqppp/p1n1p3/1p1pP3/5P2/2NBQN2/PPP3PP/2KR3R w - - bm Bxh7+; id "ECM.656";
2k5/ppp3pp/8/NQ2n2q/2Pp1n2/R4bP1/1P3P1P/4R1K1 b - - bm Qxh2+; id "ECM.657";
2r2r2/p2qppkp/3p2p1/3P1P2/2n2R2/7R/P5PP/1B1Q2K1 w - - bm Rxh7+; id "ECM.662";
r4rk1/pp2q1p1/4b2p/2ppb3/6n1/2P3N1/PPQBBPPP/R4RK1 b - - bm Nxh2; id "ECM.667";
2r1qrk1/3n3p/b3pPp1/4P3/1pp1nBN1/pP4PQ/P1P2PK1/3RR3 w - - bm Qxh7+; id "ECM.669";
r1b1rnk1/pp1nb1pp/2p1pp2/q3N3/2PP1P2/3BP1N1/PBQ3PP/R4RK1 w - - bm Bxh7+; id "ECM.670";
3rr1k1/1pq1nppp/p1p2b2/4pB2/2QPP3/P1P1B3/1P4PP/3R1RK1 w - - bm Bxh7+; id "ECM.680";
2rrn1k1/2q2ppp/p2pp3/1p2P1P1/4B3/P5Q1/1PP3PP/R4R1K w - - bm Bxh7+; id "ECM.682";
r2q3r/2pkb1p1/p2p1n2/4p1p1/Pp2P1P1/1QP5/1P1P2PP/RNB2RK1 b - - bm Rxh2; id "ECM.683";
r4rk1/pp1n1ppp/3qp3/3nN1P1/b2P4/P2B1Q2/3B1P1P/1R2R1K1 w - - bm Bxh7+; id "ECM.687";
r5k1/6bp/2q1p1p1/p2pP3/3P4/1rP2QP1/3B1PK1/2R4R w - - bm Rxh7; id "ECM.689";
r6r/4ppk1/p2p1bp1/B2p4/3P2p1/QP2P3/P1R1qPPP/2R3K1 b - - bm Rxh2; id "ECM.690";
r3k2r/1b1n1p2/pq1p1bp1/1p4p1/P3P3/1NN5/1PP3PP/R2QRB1K b kq - bm Rxh2+; id "ECM.692";
r2qrnk1/4bppp/b1p5/1p1p2P1/p2P1N1P/2NBP3/PPQ2P2/2K3RR w - - bm Bxh7+; id "ECM.693";
rn1q1rk1/pppbb1pp/4p3/3pP1p1/3P3P/2NB4/PPP2PP1/R2QK2R w KQ - bm Bxh7+; id "ECM.694";
r2q1rk1/3n1ppp/8/1pbP2P1/p1N4P/PnBBPQ2/5P2/R3K2R w KQ - bm Bxh7+; id "ECM.697";
3r2k1/p1R2p2/4pQp1/1q5p/5P1P/1PR5/2Pr2P1/6K1 b - - bm Rxg2+; id "ECM.700";
3r2k1/pb5p/1p2qpp1/8/2p5/1P1nP3/P1N2PPP/1Q1R1R1K b - - bm Bxg2+; id "ECM.703";
4rrk1/2qb2pp/p5P1/1p2p3/1b2P3/2N5/PPPQ4/1K1R2R1 w - - bm gxh7+; id "ECM.704";
2r1r1k1/5ppp/p3pn2/1pb1N3/2P5/1PQ3R1/PB2qPPP/3R2K1 w - - bm Rxg7+; id "ECM.708";
r4rk1/p2n2p1/1q1Qpn1p/1P6/P6B/2p5/2B1KP1P/R5R1 w - - bm Rxg7+; id "ECM.711";
r5k1/pn1q1rpp/2pp4/5R1N/bP6/4BQ2/P4PPP/2R3K1 w - - bm Nxg7; id "ECM.714";
r1qb1r1k/2p3pp/p1n1bp2/1p1Np2Q/P3P3/1BP3R1/1P3PPP/R1B3K1 w - - bm Rxg7; id "ECM.717";
r2r3k/5bp1/2p2N2/5P1p/3q3Q/3B2R1/n5PP/3R3K w - - bm Rxg7; id "ECM.720";
r3r1k1/p3bppp/q1b2n2/5Q2/1p1B4/1BNR4/PPP3PP/2K2R2 w - - bm Rg3; id "ECM.722";
r4rk1/1p1q1ppp/p1b4B/8/2R3R1/P2P4/1b1N1QPP/6K1 w - - bm Bxg7; id "ECM.723";
rq3rk1/3b1ppp/p2bp3/3pB2Q/8/1B5P/PP3PP1/2RR2K1 w - - bm Bxg7; id "ECM.724";
2rr2k1/4bppp/p1n1p3/3q4/1p1P2N1/2P3R1/P3QPPP/2B2RK1 w - - bm Nh6+; id "ECM.727";
rq1r1bk1/1b3pp1/3pn2p/1n2BN1P/1P2P3/3R1NP1/3Q1PB1/2R3K1 w - - bm Bxg7; id "ECM.728";
r1b5/ppqn2bk/3R2pp/2p2p2/2P1rN2/4BN2/PPQ2PPP/4R1K1 w - - bm Rxg6; id "ECM.729";
r1bqkbnr/pp2ppp1/2p4p/3n2N1/2BP4/5N2/PPP2PPP/R1BQK2R w KQkq - bm Nxf7; id "ECM.731";
r2qr1k1/1ppb1p1p/p1np2p1/7Q/3PP2b/1B2N2P/PP3PP1/R1B2RK1 w - - bm Bxf7+; id "ECM.732";
r3r1k1/1bqn1ppp/1pp2p2/8/3P4/1B4N1/PP3PPP/R2QR1K1 w - - bm Bxf7+; id "ECM.734";
2rq1rk1/1b2bppp/p1n5/1p1BN3/5B2/P7/1P3PPP/R2Q1RK1 w - - bm Nd7; id "ECM.740";
r2qr1k1/pb2bp1p/1pn1p1pB/8/2BP4/P1P2N2/4QPPP/3R1RK1 w - - bm d5; id "ECM.742";
r3r1k1/1bq1nppp/p1np4/1ppBpN2/4P3/2PP1N2/PP3PPP/R2QR1K1 w - - bm Bxf7+; id "ECM.743";
r5k1/p4ppp/3qpb2/1P2N3/1nBP4/1P5P/4QPP1/4R1K1 w - - bm Nxf7; id "ECM.744";

My first results on CPU:

Code: Select all

Stockfish 9  1s 4 threads: 58/60
Fruit 2.1    1s 1 thread:  44/60
LC0_08 ID271 6s 4 threads: 25/60
Pred 2.2.1   1s 1 thread:  24/60

I put 6s for LC0 to mimic a GTX 1060 card performance.
Predateur 2.2.1 is 1800 CCRL Elo engine, Fruit 2700 CCRL Elo.

Edit:
and it seems now the result for LC0 doesn regress from ID227:

Code: Select all

LC0_08 ID227 6s 4 threads: 23/60

[jp]

So we'd really need to look at the PVs. Is that possible with Leela?

Of course, why would you think it wasn't?

So did Leela give the same line as Komodo in your example? Don't want to make Komodo judge, but whatever.
Or does Komodo say Leela's line is okay at least?

[Albert Silver]

So we'd really need to look at the PVs. Is that possible with Leela?

Of course, why would you think it wasn't?

So did Leela give the same line as Komodo in your example? Don't want to make Komodo judge, but whatever.
Or does Komodo say Leela's line is okay at least?

You may not be aware, but Arena keeps a detailed file with the results. Here is an excerpt:

Code: Select all

11 Bxc6; id "WAC.011"; 
    Searching move: Bf3xc6
    Best move (Lc0-win-cuda90-may9-p30): Bf3xc6
    Identical moves! Found in: 00:09
     2/13	00:00	          1k	3k	+1.06	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5
     2/14	00:00	          2k	3k	+1.08	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5
     2/15	00:00	          2k	3k	+1.12	Bb6-d4  Qd7-c7  Bd4-e5  Qc7-a7+  Kg1-h1  Bc8-d7  c2-c4  b4xc3/ep  Na4xc3  Bf8-e7  Nc3-e4
     2/16	00:00	          3k	3k	+1.09	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+
     2/17	00:01	          4k	3k	+1.08	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+  Kg1-h1
     2/18	00:01	          5k	3k	+1.07	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+  Kg1-h1
     2/19	00:01	          6k	4k	+1.07	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+  Kg1-h1  Bc8-b7
     2/20	00:02	          7k	4k	+1.05	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+  Qe2-f2  Qa7xa2  Rc1xc6
     2/21	00:02	          8k	3k	+1.04	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  O-O  Nc3xb5  a6xb5  Ra1-c1  Qd7-a7+  Kg1-h1  Bc8-b7  Be5-c3
     2/22	00:02	         10k	4k	+1.03	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Nf6-d5  Ra1-d1  f7-f6  Be5-d4  Bf8-e7  c2-c4  b4xc3/ep  Na4xc3  Nd5xc3  Bd4xc3  Qd7-c7
     2/23	00:03	         12k	4k	+0.99	Bb6-d4  Ra8-b8  Bd4-e5  Rb8-b5  Qd1-e2  Nf6-d5  c2-c4  b4xc3/ep  Na4xc3  f7-f6  Nc3xb5  a6xb5  Be5-d4  Nd5xf4  Qe2-e4  Nf4-d5  a2-a4  b5xa4  Ra1xa4  c6-c5  Bd4-f2  Ke8-f7  Rf1-a1
     2/23	00:09	         34k	4k	+6.77	Bf3xc6  Bf8-e7  Bc6xd7+  Bc8xd7  b2-b3  Bd7-b5  Rf1-e1  O-O  Bb6-c5  Be7xc5+  Na4xc5  Ra8-c8  Nc5-e4  Nf6xe4  Re1xe4  a6-a5  Re4-e5  Bb5-c6
   5/9/2018 7:44:49 PM, Time for this analysis: 00:00:10, Rated time: 00:39

 12 Qxf3+; id "WAC.012" 
    Searching move: Qg4xf3
    Best move (Lc0-win-cuda90-may9-p30): Qg4xf3
    Identical moves! Found in: 00:02
     2/13	00:00	         789	2k	-2.08	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Qh3-g4  Qf2-g2  c7-c5  d4xc5
     2/14	00:00	          1k	2k	-2.52	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Qh3-f5  Qf2-e2  Ke8-d7  Kh1-g2  Qf5-g4+
     2/14	00:00	          2k	2k	-2.80	Rg8-f8  Qc2-e2  Rg7-f7  Rc6-c3  Ke8-d8  Rc3-e3  Rf7-f4  Rf1-g1  Qg4-f5  Nf3-g5
     2/14	00:00	          2k	2k	-2.60	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4
     2/14	00:00	          2k	2k	-2.86	Rg8-f8  Qc2-e2  Rg7-f7  Rc6-c3  Ke8-d8  Rc3-e3  Rf7-f4  Rf1-g1  Qg4-f5  Nf3-g5
     2/14	00:00	          2k	2k	-2.78	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+
     2/15	00:00	          2k	3k	-2.93	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+
     2/15	00:00	          2k	3k	-3.02	Rg8-f8  Rc6-c3  Rf8-f4  Rf1-g1  Rf4xf3  Rg1xg4  Rf3-f1+  Kh1-g2  Rg7xg4+  Kg2xf1  Rg4-f4+  Kf1-e1
     2/16	00:00	          3k	3k	-3.08	Rg8-f8  Rc6-c3  Rf8-f4  Rf1-g1  Rf4xf3  Rg1xg4  Rf3-f1+  Kh1-g2  Rg7xg4+  Kg2xf1  Rg4-f4+  Kf1-e1
     2/16	00:00	          3k	3k	-3.01	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+  Rf8-f7
     2/16	00:01	          3k	3k	-3.11	Rg8-f8  Rc6-c3  Rf8-f4  Rf1-g1  Rf4xf3  Rg1xg4  Rf3-f1+  Kh1-g2  Rg7xg4+  Kg2xf1  Rg4-f4+  Kf1-e1
     2/17	00:01	          3k	3k	-3.14	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+  Rf8-f7
     2/17	00:01	          3k	3k	-3.21	Rg8-f8  Rc6-c3  Rg7-f7  Qc2-g2  Qg4-e4  h2-h3  Ke8-d7  Rf1-c1  c7-c5
     2/18	00:01	          4k	3k	-3.16	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Qh3-f5  Qf2-e2  Qf5-b1+  Kh1-g2  Qb1-g6+  Nf3-g5  Qg6-f7
     2/19	00:01	          4k	3k	-3.25	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+  Rf8-f7  Qg7-g3
     2/20	00:02	          8k	3k	-3.43	Qg4-h3  Qc2-e2  Rg7-g2  Rf1-f2  Rg2xf2  Qe2xf2  Rg8-f8  Rc6-c3  Ke8-d7  Qf2-e2  Qh3-g4  Qe2-g2  Qg4-f4  Qg2-g7+  Rf8-f7  Qg7-g3  Qf4-e4  h2-h3
     2/20	00:02	         12k	5k	+81.28	Qg4xf3+  Rf1xf3  Rg7-g1+
   5/9/2018 7:45:00 PM, Time for this analysis: 00:00:10, Rated time: 00:41

The default filename is analyses.log and is found in the root Arena folder.

[peter]

Albert, I myself was curious about game-changing, unique solutions tactical suite. I managed to collect from ECM middlegame test suite 60 such tactical positions, if I checked them correctly. Many, almost all easier (old) suites, have multiple or non game-changing solutions, or even wrong solutions altogether.

Check these out:

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";

Did check the first one only till now, here's SF- Output:

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - 0 1

Analysis by asmFishW_2018-05-07_popcnt:

1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4
-+ (-132.71) Depth: 7/8 00:00:00 6kN
...
1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4 d3 6.c3 d2 7.f5 d1Q 8.f6 Qxa4 9.f7 Qa3 10.f8Q Qxf8
-+ (-132.71) Depth: 59/20 00:00:03 129MN, tb=2361407

And here after "next best move" command:

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - 0 1

Analysis by asmFishW_2018-05-07_popcnt:

1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4 d3 6.c3 d2 7.f5 d1Q 8.f6 Qxa4 9.f7 Qa3 10.f8Q Qxf8
-+ (-132.71) Depth: 60/20 74769kN, tb=536875
y in Fritz GUI
1...Kb7 2.f4 b5 3.axb5 a4 4.bxa4 c4
+- (1.80) Depth: 7/7 00:00:00 153kN
...
1...Kb7 2.Ke6 Kc6 3.f4 c4 4.bxc4 Kc5 5.f5 Kb4 6.f6 Kc3 7.Kd5 d3 8.cxd3 b5 9.cxb5 Kb2 10.f7 Kb3 11.f8Q Kxa4
+- (132.70) Depth: 57/22 00:01:06 2276MN, tb=56820665

1...b5 doesn't seem to be the only winning move. At second output line after 1...Kb7 11...Kxa4 won 6 some is reached.

And then 60 (59, if I was as sceptical as Albert about WAC, having seen one wrong right at the start, assuming 200 of 300 were correct only maybe, would mean here 40 of 60 would remain ) isn't very much for a set as easy as this one obviouly is, isn't it?

BTW 25 out of 60 is even worse then 150 out of 300.

[Laskos]

Albert, I myself was curious about game-changing, unique solutions tactical suite. I managed to collect from ECM middlegame test suite 60 such tactical positions, if I checked them correctly. Many, almost all easier (old) suites, have multiple or non game-changing solutions, or even wrong solutions altogether.

Check these out:

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";

Did check the first one only till now, here's SF- Output:

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - 0 1

Analysis by asmFishW_2018-05-07_popcnt:

1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4
-+ (-132.71) Depth: 7/8 00:00:00 6kN
...
1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4 d3 6.c3 d2 7.f5 d1Q 8.f6 Qxa4 9.f7 Qa3 10.f8Q Qxf8
-+ (-132.71) Depth: 59/20 00:00:03 129MN, tb=2361407

And here after "next best move" command:

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - 0 1

Analysis by asmFishW_2018-05-07_popcnt:

1...b5 2.axb5+ Kb6 3.Ke6 a4 4.bxa4 c4 5.f4 d3 6.c3 d2 7.f5 d1Q 8.f6 Qxa4 9.f7 Qa3 10.f8Q Qxf8
-+ (-132.71) Depth: 60/20 74769kN, tb=536875
y in Fritz GUI
1...Kb7 2.f4 b5 3.axb5 a4 4.bxa4 c4
+- (1.80) Depth: 7/7 00:00:00 153kN
...
1...Kb7 2.Ke6 Kc6 3.f4 c4 4.bxc4 Kc5 5.f5 Kb4 6.f6 Kc3 7.Kd5 d3 8.cxd3 b5 9.cxb5 Kb2 10.f7 Kb3 11.f8Q Kxa4
+- (132.70) Depth: 57/22 00:01:06 2276MN, tb=56820665

1...b5 doesn't seem to be the only winning move. At second output line after 1...Kb7 11...Kxa4 won 6 some is reached.

And then 60 (59, if I was as sceptical as Albert about WAC, having seen one wrong right at the start, assuming 200 of 300 were correct only maybe, would mean here 40 of 60 would remain ) isn't very much for a set as easy as this one obviouly is, isn't it?

BTW 25 out of 60 is even worse then 150 out of 300.

Isn't 1...b5 win for Black and 1...Kb7 win for White? Am I missing something?

About the result, at least it is above 1800 Elo level on this very tactical suite at short time control.

[peter]

Isn't 1...b5 win for Black and 1...Kb7 win for White? Am I missing something?

You're fully right of course, Kai, shame on me.

[peter]

As Kai already wrote, this was a real shame not seeing one output was -+, the other one +-, maybe I should have looked at the positions themselves too a little.

That was probably just too early in the morning for me, at least for this one morning.

[peter]

Isn't 1...b5 win for Black and 1...Kb7 win for White? Am I missing something?

About the result, at least it is above 1800 Elo level on this very tactical suite at short time control.

Awake enough now hopefully after a little early in the morning disgrace, let's talk about nr. 6 maybe?

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";
8/2k3p1/2p4p/5P2/2K3PP/8/8/8 w - - bm g5; id "ECM.603";
8/1kp1b3/1p4K1/4P2p/P1P3p1/5pP1/P4P2/4B3 b - - bm h4; id "ECM.604";
8/8/3K1k2/5p1p/4p1p1/4P1P1/5PP1/8 b - - bm f4; id "ECM.606";
6r1/1p3k2/pPp4R/K1P1p1p1/1P2Pp1p/5P1P/6P1/8 w - - bm Rxc6; id "ECM.608";
1r3qk1/pb3p1p/1pn2PpQ/2pN4/3r4/5B2/PPP4P/4RRK1 w - - bm Ne7+; id "ECM.611";

1r3qk1/pb3p1p/1pn2PpQ/2pN4/3r4/5B2/PPP4P/4RRK1 w - -

Of course, 1...Ne7 is best move, but isn't 1...Qg5 (?) winning too probably?

After that one of course much lesser good move, 1...Qg5 (?):

1r3qk1/pb3p1p/1pn2Pp1/2pN2Q1/3r4/5B2/PPP4P/4RRK1 b - -

Engine: asmFishW_2018-05-07_popcnt0 (32768 MB)
by TypingALot


43/69 10:07 +2.43 1...h6 2.Qg2 Nd8 3.Qg3 Bxd5 4.Bxd5 Rxd5
5.Qxb8 h5 6.Re7 Qh6 7.Qf4 Qxf4
8.Rxf4 Nc6 9.Rb7 g5 10.Rf2 Nd8
11.Rxa7 Ne6 12.Kg2 b5 13.Re7 Nf4+
14.Kg3 Rd4 (18.266.689.616) 30082
TB:1.934.409


Don't get me wrong, of course I now, this is just cherry picking, but where is the difference to what you and Albert do with WAC?

It's simply a matter of definition of "best move".

To me best move doesn't necessarily mean "only one winning move".

Many clearly best moves could be denied then, mate in 2 against other anyhow anywhen also winning moves e.g., as Albert wrote about WAC nr.1. here:
http://talkchess.com/forum3/viewtopic.p ... 80#p761680

[Laskos]

Isn't 1...b5 win for Black and 1...Kb7 win for White? Am I missing something?

About the result, at least it is above 1800 Elo level on this very tactical suite at short time control.

Awake enough now hopefully after a little early in the morning disgrace, let's talk about nr. 6 maybe?

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";
8/2k3p1/2p4p/5P2/2K3PP/8/8/8 w - - bm g5; id "ECM.603";
8/1kp1b3/1p4K1/4P2p/P1P3p1/5pP1/P4P2/4B3 b - - bm h4; id "ECM.604";
8/8/3K1k2/5p1p/4p1p1/4P1P1/5PP1/8 b - - bm f4; id "ECM.606";
6r1/1p3k2/pPp4R/K1P1p1p1/1P2Pp1p/5P1P/6P1/8 w - - bm Rxc6; id "ECM.608";
1r3qk1/pb3p1p/1pn2PpQ/2pN4/3r4/5B2/PPP4P/4RRK1 w - - bm Ne7+; id "ECM.611";

1r3qk1/pb3p1p/1pn2PpQ/2pN4/3r4/5B2/PPP4P/4RRK1 w - -

Of course, 1...Ne7 is best move, but isn't 1...Qg5 (?) winning too probably?

After that one of course much lesser good move, 1...Qg5 (?):

1r3qk1/pb3p1p/1pn2Pp1/2pN2Q1/3r4/5B2/PPP4P/4RRK1 b - -

Engine: asmFishW_2018-05-07_popcnt0 (32768 MB)
by TypingALot


43/69 10:07 +2.43 1...h6 2.Qg2 Nd8 3.Qg3 Bxd5 4.Bxd5 Rxd5
5.Qxb8 h5 6.Re7 Qh6 7.Qf4 Qxf4
8.Rxf4 Nc6 9.Rb7 g5 10.Rf2 Nd8
11.Rxa7 Ne6 12.Kg2 b5 13.Re7 Nf4+
14.Kg3 Rd4 (18.266.689.616) 30082
TB:1.934.409


Don't get me wrong, of course I now, this is just cherry picking, but where is the difference to what you and Albert do with WAC?

It's simply a matter of definition of "best move".

To me best move doesn't necessarily mean "only one winning move".

Many clearly best moves could be denied then, mate in 2 against other anyhow anywhen also winning moves e.g., as Albert wrote about WAC nr.1. here:
http://talkchess.com/forum3/viewtopic.p ... 80#p761680

Anyway, I tried to purge and to add some new positions. 2 or 3 were even worse than the one you have showed, at longer analysis. Here are 64 maybe better (more decisive) game-changers. Maybe you can find again some ambiguity as game-changing goes for some positions, I didn't analyze to such depths as yours.

Code: Select all

8/5K2/kp6/p1p5/P2p4/1P3P2/2P5/8 b - - bm b5; id "ECM.602";
8/2k3p1/2p4p/5P2/2K3PP/8/8/8 w - - bm g5; id "ECM.603";
8/1kp1b3/1p4K1/4P2p/P1P3p1/5pP1/P4P2/4B3 b - - bm h4; id "ECM.604";
8/8/3K1k2/5p1p/4p1p1/4P1P1/5PP1/8 b - - bm f4; id "ECM.606";
4r1k1/5p1p/3q2p1/1p1P4/1P6/2p4P/2Q1nPB1/4RK2 b - - bm Ng3+; id "ECM.612";
3q1k2/5p2/p5pN/1b2Q2P/8/8/5PPK/8 w - - bm Qh8+; id "ECM.622";
6k1/p3b1pp/4p3/4Pp2/Pp1r1P1P/1P4P1/2p2R2/5RK1 b - - bm Rc4; id "ECM.623";
rn1q2k1/pp3pb1/3p2pp/2pP2N1/3r1P2/7Q/PP4PP/R1B2RK1 w - - bm Nxf7; id "ECM.628";
8/6Bp/6p1/2k1p3/4PPP1/1pb4P/8/2K5 b - - bm b2+; id "ECM.629";
r4rk1/ppq3pp/2p1Pn2/4p1Q1/8/2N5/PP4PP/2KR1R2 w - - bm Rxf6; id "ECM.636";
6k1/p4pp1/Pp2r3/1QPq3p/8/6P1/2P2P1P/1R4K1 w - - bm cxb6; id "ECM.641";
6k1/p4pbp/Bp2p1p1/n2P4/q3P3/B1rQP3/P5PP/5RK1 w - - bm dxe6; id "ECM.642";
8/2k5/2p5/2pb2K1/pp4P1/1P1R4/P7/8 b - - bm Bxb3; id "ECM.646";
8/1R2P3/6k1/3B4/2P2P2/1p2r3/1Kb4p/8 w - - bm Be6; id "ECM.650";
2kr2r1/pp2bQ1p/2b1P3/2qN4/8/1B2p2P/PPP3P1/3R1R1K b - - bm e2; id "ECM.651";
r1b2rk1/1p2qppp/p3p3/2n5/3N4/3B1R2/PPP1Q1PP/R5K1 w - - bm Bxh7+; id "ECM.652";
6rk/3nrpbp/p1bq1npB/1p2p1N1/4P1PQ/P2B3R/1PP1N2P/5R1K w - - bm Nxh7; id "ECM.655";
1rb2rk1/3nqppp/p1n1p3/1p1pP3/5P2/2NBQN2/PPP3PP/2KR3R w - - bm Bxh7+; id "ECM.656";
2k5/ppp3pp/8/NQ2n2q/2Pp1n2/R4bP1/1P3P1P/4R1K1 b - - bm Qxh2+; id "ECM.657";
2r2r2/p2qppkp/3p2p1/3P1P2/2n2R2/7R/P5PP/1B1Q2K1 w - - bm Rxh7+; id "ECM.662";
r4rk1/pp2q1p1/4b2p/2ppb3/6n1/2P3N1/PPQBBPPP/R4RK1 b - - bm Nxh2; id "ECM.667";
3rr1k1/1pq1nppp/p1p2b2/4pB2/2QPP3/P1P1B3/1P4PP/3R1RK1 w - - bm Bxh7+; id "ECM.680";
2rrn1k1/2q2ppp/p2pp3/1p2P1P1/4B3/P5Q1/1PP3PP/R4R1K w - - bm Bxh7+; id "ECM.682";
r2q3r/2pkb1p1/p2p1n2/4p1p1/Pp2P1P1/1QP5/1P1P2PP/RNB2RK1 b - - bm Rxh2; id "ECM.683";
r4rk1/pp1n1ppp/3qp3/3nN1P1/b2P4/P2B1Q2/3B1P1P/1R2R1K1 w - - bm Bxh7+; id "ECM.687";
r5k1/6bp/2q1p1p1/p2pP3/3P4/1rP2QP1/3B1PK1/2R4R w - - bm Rxh7; id "ECM.689";
r2qrnk1/4bppp/b1p5/1p1p2P1/p2P1N1P/2NBP3/PPQ2P2/2K3RR w - - bm Bxh7+; id "ECM.693";
rn1q1rk1/pppbb1pp/4p3/3pP1p1/3P3P/2NB4/PPP2PP1/R2QK2R w KQ - bm Bxh7+; id "ECM.694";
r2q1rk1/3n1ppp/8/1pbP2P1/p1N4P/PnBBPQ2/5P2/R3K2R w KQ - bm Bxh7+; id "ECM.697";
3r2k1/p1R2p2/4pQp1/1q5p/5P1P/1PR5/2Pr2P1/6K1 b - - bm Rxg2+; id "ECM.700";
3r2k1/pb5p/1p2qpp1/8/2p5/1P1nP3/P1N2PPP/1Q1R1R1K b - - bm Bxg2+; id "ECM.703";
4rrk1/2qb2pp/p5P1/1p2p3/1b2P3/2N5/PPPQ4/1K1R2R1 w - - bm gxh7+; id "ECM.704";
2r1r1k1/5ppp/p3pn2/1pb1N3/2P5/1PQ3R1/PB2qPPP/3R2K1 w - - bm Rxg7+; id "ECM.708";
r4rk1/p2n2p1/1q1Qpn1p/1P6/P6B/2p5/2B1KP1P/R5R1 w - - bm Rxg7+; id "ECM.711";
r5k1/pn1q1rpp/2pp4/5R1N/bP6/4BQ2/P4PPP/2R3K1 w - - bm Nxg7; id "ECM.714";
r1qb1r1k/2p3pp/p1n1bp2/1p1Np2Q/P3P3/1BP3R1/1P3PPP/R1B3K1 w - - bm Rxg7; id "ECM.717";
r2r3k/5bp1/2p2N2/5P1p/3q3Q/3B2R1/n5PP/3R3K w - - bm Rxg7; id "ECM.720";
r4rk1/1p1q1ppp/p1b4B/8/2R3R1/P2P4/1b1N1QPP/6K1 w - - bm Bxg7; id "ECM.723";
rq3rk1/3b1ppp/p2bp3/3pB2Q/8/1B5P/PP3PP1/2RR2K1 w - - bm Bxg7; id "ECM.724";
2rr2k1/4bppp/p1n1p3/3q4/1p1P2N1/2P3R1/P3QPPP/2B2RK1 w - - bm Nh6+; id "ECM.727";
rq1r1bk1/1b3pp1/3pn2p/1n2BN1P/1P2P3/3R1NP1/3Q1PB1/2R3K1 w - - bm Bxg7; id "ECM.728";
r1bqkbnr/pp2ppp1/2p4p/3n2N1/2BP4/5N2/PPP2PPP/R1BQK2R w KQkq - bm Nxf7; id "ECM.731";
r2qr1k1/1ppb1p1p/p1np2p1/7Q/3PP2b/1B2N2P/PP3PP1/R1B2RK1 w - - bm Bxf7+; id "ECM.732";
r3r1k1/1bq1nppp/p1np4/1ppBpN2/4P3/2PP1N2/PP3PPP/R2QR1K1 w - - bm Bxf7+; id "ECM.743";
2r1r1k1/1pq1bp1p/p3pnp1/P2n2N1/7R/2P4P/1PB1QPP1/2B1R1K1 w - - bm Nxf7; id "ECM.748";
r1bq2k1/pp1n1ppp/3b1n2/PQ1B3r/3N1P2/2N5/1PP3PP/R1B2RK1 w - - bm Bxf7+; id "ECM.749";
2r1r1k1/5ppp/pq3b2/2pB1P2/2p2B2/5Q1P/Pn3PP1/2R1R1K1 w - - bm Bxf7+; id "ECM.750";
r4rk1/ppRn1p2/6pb/2P1pq1p/3N4/P1QPn1Pb/1B1NPP1P/4R1KB b - - bm Qxf2+; id "ECM.751";
r3kr2/1b2qp2/pp2p2N/4p2Q/8/2n5/P3B1PP/3R1R1K w q - bm Nxf7; id "ECM.752";
b2r1rk1/pq2bpp1/1p2p2p/4N2n/2P2R2/1PB2N2/1P2QPPP/4R1K1 w - - bm Rxf7; id "ECM.753";
rqb1k2r/1p1nbp1p/p4pp1/8/1PBN1P2/P1N1P3/7P/2RQ1RK1 w kq - bm Bxf7+; id "ECM.754";
1r2q1k1/p3pp2/3p1bp1/2pP2N1/8/P5PB/2Q2PK1/1rBR4 w - - bm Nxf7; id "ECM.756";
r3r1k1/pbq1ppbp/1pp2np1/4N3/3P4/2P5/PPB1QPPP/R1B1R1K1 w - - bm Nxf7; id "ECM.762";
1qr1b1k1/4bpp1/pn2p2p/1p1nN3/3P4/P2BBN1Q/1P3PPP/4R1K1 w - - bm Bxh6; id "ECM.772";
rr1q2k1/1p2bpp1/2p1p2p/P1Pn4/2NP4/3Q1RP1/5PKP/2B1R3 w - - bm Bxh6; id "ECM.773";
2r5/1p4bk/3p2rp/4pN2/1P2P1pR/2P2q2/QP6/1K5R w - - bm Rxh6+; id "ECM.775";
r1b1r3/pp2Npbk/3pp2p/q5p1/2QNPP2/6P1/PPP3P1/2KR3R w - - bm Ndf5; id "ECM.776";
4r1k1/p1pq1pp1/2p5/3p1b2/Q7/2P1B2P/P1P1rPP1/2R2RK1 b - - bm Bxh3; id "ECM.778";
6rk/3b1n1p/1p1q3b/1PpNp3/2P1Pp2/2Q2NrP/5RP1/2R2B1K b - - bm Bxh3; id "ECM.783";
r2q1rk1/ppp2pp1/1b2b2p/3n3Q/2Bp4/3P1N2/PPP2PPP/R1B1R1K1 w - - bm Bxh6; id "ECM.784";
r3rbk1/1bp1qpp1/p6p/np2p2Q/4P2N/1BP4P/PP3PP1/R1B1R1K1 w - - bm Bg5; id "ECM.785";
4q3/p2r1ppk/R6p/3n4/3B1Q2/4P2P/5PP1/6K1 w - - bm Rxh6+; id "ECM.786";
2r1r1k1/pb1n1pp1/1p1qpn1p/4N1B1/2PP4/3B4/P2Q1PPP/3RR1K1 w - - bm Bxh6; id "ECM.789";
r1b2rk1/pp2bpp1/4p2p/2q4Q/5nNB/2PB4/PP3PPP/2KR3R w - - bm Nxh6+; id "ECM.794";

Fast testing and scaling:

Code: Select all

                          0.2s          5s
                          
Stockfish 9   4 threads: 57/64       64/64
Fruit 2.1     1 thread:  37/64       53/64
Pred 2.2.1    1 thread:  18/64       35/64
LC0_08 ID271  4 threads: 18/64       32/64

LC0 seems to perform at 1800 CCRL Elo level in this "decisive tactics ", and scale no better than AB engines.

[peter]

Many clearly best moves could be denied then, mate in 2 against other anyhow anywhen also winning moves e.g., as Albert wrote about WAC nr.1. here:
http://talkchess.com/forum3/viewtopic.p ... 80#p761680

Same could be said about your position nr.8 and nr.18 e.g., further on i didn't check till now.

Yet I'd be confident with this one tactical suite too, (I was with WAC also anyhow), even if 60 positions isn't very much, that's what I meant before, not the easy level, that's ok, just comparable with WAC, I guess.

And if you think your result with LC0 with it would be 1800 Elo, as much as I doubt measuring that in such a way (you'd have to call it TCelo at least for Tactical Computer Elo, even more exact LTTCelo, LaskosTacticalTestCelo), your 1800 still would be about 1000 Elo less than what's said about LC0's game playing level at the moment, isn't it?

As for this one engine- engine- game- playing Celo- measurement, it depends on opening- books or starting-positions too very much of course.
Ever tried e.g. Jeroen Noomen's Gambit-Lines.ctg for testing LC0 playing against other engines? Or even better his Sharp Gambit Lines starting-positions-collection?

That brings rather different results too, I can tell, having tried only a little till now, cause of course nobody would be interested in such results, at least not as for LC0 right now.


Edit: didn't see your new posting answering mine while writing this one, will have a look at your new suite.