How good is your engine?

[Ovyron]

The best is Bluefish-XI with a couple small tweaks, which has no rivals so far in my testing.

Would you be willing to share it?

[Dann Corbit]

The best is Bluefish-XI with a couple small tweaks, which has no rivals so far in my testing.

Would you be willing to share it?

You will want to turn off logging in the UCI options.

[Dann Corbit]

It is very similar to : Bluefish XI-LP FD (Tactical=2; defensive=off)
the only functional difference being prune less when mobility is extremely cramped and I prune more when mobility is extremely high.

For reference:
http://dorszcz.blogspot.com/p/ttt1.html
Look at the distance between both bluefish entries and the next one, with the tweaked version doing incredibly well.

[Dann Corbit]

Here are the distinct results of the experiment:
wlt |sort|uniq>wlt.txt

Code: Select all

losses: 36 wins: 64 ties: 0
losses: 37 wins: 63 ties: 0
losses: 38 wins: 62 ties: 0
losses: 39 wins: 61 ties: 0
losses: 40 wins: 60 ties: 0
losses: 41 wins: 59 ties: 0
losses: 42 wins: 58 ties: 0
losses: 43 wins: 57 ties: 0
losses: 44 wins: 56 ties: 0
losses: 45 wins: 55 ties: 0
losses: 46 wins: 54 ties: 0
losses: 47 wins: 53 ties: 0
losses: 48 wins: 52 ties: 0
losses: 49 wins: 51 ties: 0
losses: 50 wins: 50 ties: 0
losses: 51 wins: 49 ties: 0
losses: 52 wins: 48 ties: 0
losses: 53 wins: 47 ties: 0
losses: 54 wins: 46 ties: 0
losses: 55 wins: 45 ties: 0
losses: 56 wins: 44 ties: 0
losses: 57 wins: 43 ties: 0
losses: 58 wins: 42 ties: 0
losses: 59 wins: 41 ties: 0
losses: 60 wins: 40 ties: 0
losses: 61 wins: 39 ties: 0
losses: 62 wins: 38 ties: 0
losses: 63 wins: 37 ties: 0

Here is the program:

Code: Select all

#include <random>
#include <iostream>
using namespace std;
class results {
	public:
    int lt_h;
    int gt_h;
    int ties;
    results()
    {
        lt_h = 0;
        gt_h = 0;
        ties = 0;
    }
};

int main(void)
{
    std::mt19937 generator (17);
    std::uniform_real_distribution<double> urd(0.0, 1.0);
    results * contests = new results[1000];
    for (int contest = 0; contest < 1000; contest++)
    {
        for (int result = 0; result < 100; result++)
        {
            double value = urd(generator);
            if (value < 0.5) contests[contest].lt_h++;
            else if (value > 0.5) contests[contest].gt_h++;
            else  contests[contest].ties++;
        }
    }
    for (int contest = 0; contest < 1000; contest++)
    {
        std::cout << "losses: " << contests[contest].lt_h  << " wins: " <<  contests[contest].gt_h  << " ties: " <<  contests[contest].ties << std::endl;
    }
    delete [] contests;
    return 0;
}

You will probably get slightly different results if you change the value of the seed.

So, if you had seen this:
losses: 36 wins: 64 ties: 0
would you have concluded that engine A was enormously stronger than engine B

And if you had seen this:
losses: 63 wins: 37 ties: 0
would you have concluded that engine A was enormously weaker than engine B

As we know, these two "engines" are exactly the same strength.
If you increase the game count, you will see things drifting sensibly towards the mean.

[Dann Corbit]

In case you are wondering, "Why so few draws?"
That was only when the coin landed on its edge (value == 0.5)

[Dann Corbit]

Result from changing game count per contest to 1000 (AFTER filtering with |sort|uniq):

Code: Select all

losses: 450 wins: 550 ties: 0
losses: 452 wins: 548 ties: 0
losses: 455 wins: 545 ties: 0
losses: 456 wins: 544 ties: 0
losses: 457 wins: 543 ties: 0
losses: 458 wins: 542 ties: 0
losses: 459 wins: 541 ties: 0
losses: 460 wins: 540 ties: 0
losses: 464 wins: 536 ties: 0
losses: 465 wins: 535 ties: 0
losses: 466 wins: 534 ties: 0
losses: 467 wins: 533 ties: 0
losses: 468 wins: 532 ties: 0
losses: 469 wins: 531 ties: 0
losses: 470 wins: 530 ties: 0
losses: 471 wins: 529 ties: 0
losses: 472 wins: 528 ties: 0
losses: 473 wins: 527 ties: 0
losses: 474 wins: 526 ties: 0
losses: 475 wins: 525 ties: 0
losses: 476 wins: 524 ties: 0
losses: 477 wins: 523 ties: 0
losses: 478 wins: 522 ties: 0
losses: 479 wins: 521 ties: 0
losses: 480 wins: 520 ties: 0
losses: 481 wins: 519 ties: 0
losses: 482 wins: 518 ties: 0
losses: 483 wins: 517 ties: 0
losses: 484 wins: 516 ties: 0
losses: 485 wins: 515 ties: 0
losses: 486 wins: 514 ties: 0
losses: 487 wins: 513 ties: 0
losses: 488 wins: 512 ties: 0
losses: 489 wins: 511 ties: 0
losses: 490 wins: 510 ties: 0
losses: 491 wins: 509 ties: 0
losses: 492 wins: 508 ties: 0
losses: 493 wins: 507 ties: 0
losses: 494 wins: 506 ties: 0
losses: 495 wins: 505 ties: 0
losses: 496 wins: 504 ties: 0
losses: 497 wins: 503 ties: 0
losses: 498 wins: 502 ties: 0
losses: 499 wins: 501 ties: 0
losses: 500 wins: 500 ties: 0
losses: 501 wins: 499 ties: 0
losses: 502 wins: 498 ties: 0
losses: 503 wins: 497 ties: 0
losses: 504 wins: 496 ties: 0
losses: 505 wins: 495 ties: 0
losses: 506 wins: 494 ties: 0
losses: 507 wins: 493 ties: 0
losses: 508 wins: 492 ties: 0
losses: 509 wins: 491 ties: 0
losses: 510 wins: 490 ties: 0
losses: 511 wins: 489 ties: 0
losses: 512 wins: 488 ties: 0
losses: 513 wins: 487 ties: 0
losses: 514 wins: 486 ties: 0
losses: 515 wins: 485 ties: 0
losses: 516 wins: 484 ties: 0
losses: 517 wins: 483 ties: 0
losses: 518 wins: 482 ties: 0
losses: 519 wins: 481 ties: 0
losses: 520 wins: 480 ties: 0
losses: 521 wins: 479 ties: 0
losses: 522 wins: 478 ties: 0
losses: 523 wins: 477 ties: 0
losses: 524 wins: 476 ties: 0
losses: 525 wins: 475 ties: 0
losses: 526 wins: 474 ties: 0
losses: 527 wins: 473 ties: 0
losses: 528 wins: 472 ties: 0
losses: 529 wins: 471 ties: 0
losses: 530 wins: 470 ties: 0
losses: 531 wins: 469 ties: 0
losses: 532 wins: 468 ties: 0
losses: 533 wins: 467 ties: 0
losses: 534 wins: 466 ties: 0
losses: 535 wins: 465 ties: 0
losses: 536 wins: 464 ties: 0
losses: 537 wins: 463 ties: 0
losses: 538 wins: 462 ties: 0
losses: 539 wins: 461 ties: 0
losses: 540 wins: 460 ties: 0
losses: 541 wins: 459 ties: 0
losses: 545 wins: 455 ties: 0
losses: 547 wins: 453 ties: 0
losses: 548 wins: 452 ties: 0
losses: 551 wins: 449 ties: 0

[Dann Corbit]

And 10,000:

Code: Select all

losses: 4857 wins: 5143 ties: 0
losses: 4861 wins: 5139 ties: 0
losses: 4865 wins: 5135 ties: 0
losses: 4873 wins: 5127 ties: 0
losses: 4874 wins: 5126 ties: 0
losses: 4875 wins: 5125 ties: 0
losses: 4876 wins: 5124 ties: 0
losses: 4879 wins: 5121 ties: 0
losses: 4883 wins: 5117 ties: 0
losses: 4885 wins: 5115 ties: 0
losses: 4887 wins: 5113 ties: 0
losses: 4889 wins: 5111 ties: 0
losses: 4890 wins: 5110 ties: 0
losses: 4894 wins: 5106 ties: 0
losses: 4896 wins: 5104 ties: 0
losses: 4897 wins: 5103 ties: 0
losses: 4898 wins: 5102 ties: 0
losses: 4900 wins: 5100 ties: 0
losses: 4901 wins: 5099 ties: 0
losses: 4902 wins: 5098 ties: 0
losses: 4903 wins: 5097 ties: 0
losses: 4904 wins: 5096 ties: 0
losses: 4905 wins: 5095 ties: 0
losses: 4906 wins: 5094 ties: 0
losses: 4907 wins: 5093 ties: 0
losses: 4908 wins: 5092 ties: 0
losses: 4909 wins: 5091 ties: 0
losses: 4910 wins: 5090 ties: 0
losses: 4911 wins: 5089 ties: 0
losses: 4912 wins: 5088 ties: 0
losses: 4914 wins: 5086 ties: 0
losses: 4915 wins: 5085 ties: 0
losses: 4916 wins: 5084 ties: 0
losses: 4918 wins: 5082 ties: 0
losses: 4919 wins: 5081 ties: 0
losses: 4921 wins: 5079 ties: 0
losses: 4922 wins: 5078 ties: 0
losses: 4923 wins: 5077 ties: 0
losses: 4924 wins: 5076 ties: 0
losses: 4925 wins: 5075 ties: 0
losses: 4926 wins: 5074 ties: 0
losses: 4927 wins: 5073 ties: 0
losses: 4928 wins: 5072 ties: 0
losses: 4929 wins: 5071 ties: 0
losses: 4930 wins: 5070 ties: 0
losses: 4931 wins: 5069 ties: 0
losses: 4932 wins: 5068 ties: 0
losses: 4933 wins: 5067 ties: 0
losses: 4934 wins: 5066 ties: 0
losses: 4935 wins: 5065 ties: 0
losses: 4936 wins: 5064 ties: 0
losses: 4937 wins: 5063 ties: 0
losses: 4938 wins: 5062 ties: 0
losses: 4939 wins: 5061 ties: 0
losses: 4940 wins: 5060 ties: 0
losses: 4941 wins: 5059 ties: 0
losses: 4942 wins: 5058 ties: 0
losses: 4943 wins: 5057 ties: 0
losses: 4944 wins: 5056 ties: 0
losses: 4945 wins: 5055 ties: 0
losses: 4946 wins: 5054 ties: 0
losses: 4947 wins: 5053 ties: 0
losses: 4948 wins: 5052 ties: 0
losses: 4949 wins: 5051 ties: 0
losses: 4950 wins: 5050 ties: 0
losses: 4951 wins: 5049 ties: 0
losses: 4952 wins: 5048 ties: 0
losses: 4953 wins: 5047 ties: 0
losses: 4954 wins: 5046 ties: 0
losses: 4955 wins: 5045 ties: 0
losses: 4956 wins: 5044 ties: 0
losses: 4957 wins: 5043 ties: 0
losses: 4958 wins: 5042 ties: 0
losses: 4959 wins: 5041 ties: 0
losses: 4960 wins: 5040 ties: 0
losses: 4961 wins: 5039 ties: 0
losses: 4962 wins: 5038 ties: 0
losses: 4963 wins: 5037 ties: 0
losses: 4964 wins: 5036 ties: 0
losses: 4965 wins: 5035 ties: 0
losses: 4966 wins: 5034 ties: 0
losses: 4967 wins: 5033 ties: 0
losses: 4968 wins: 5032 ties: 0
losses: 4969 wins: 5031 ties: 0
losses: 4970 wins: 5030 ties: 0
losses: 4971 wins: 5029 ties: 0
losses: 4972 wins: 5028 ties: 0
losses: 4973 wins: 5027 ties: 0
losses: 4974 wins: 5026 ties: 0
losses: 4975 wins: 5025 ties: 0
losses: 4976 wins: 5024 ties: 0
losses: 4977 wins: 5023 ties: 0
losses: 4978 wins: 5022 ties: 0
losses: 4979 wins: 5021 ties: 0
losses: 4980 wins: 5020 ties: 0
losses: 4981 wins: 5019 ties: 0
losses: 4982 wins: 5018 ties: 0
losses: 4983 wins: 5017 ties: 0
losses: 4984 wins: 5016 ties: 0
losses: 4985 wins: 5015 ties: 0
losses: 4986 wins: 5014 ties: 0
losses: 4987 wins: 5013 ties: 0
losses: 4988 wins: 5012 ties: 0
losses: 4989 wins: 5011 ties: 0
losses: 4990 wins: 5010 ties: 0
losses: 4991 wins: 5009 ties: 0
losses: 4992 wins: 5008 ties: 0
losses: 4993 wins: 5007 ties: 0
losses: 4994 wins: 5006 ties: 0
losses: 4995 wins: 5005 ties: 0
losses: 4996 wins: 5004 ties: 0
losses: 4997 wins: 5003 ties: 0
losses: 4998 wins: 5002 ties: 0
losses: 4999 wins: 5001 ties: 0
losses: 5000 wins: 5000 ties: 0
losses: 5001 wins: 4999 ties: 0
losses: 5002 wins: 4998 ties: 0
losses: 5003 wins: 4997 ties: 0
losses: 5004 wins: 4996 ties: 0
losses: 5005 wins: 4995 ties: 0
losses: 5006 wins: 4994 ties: 0
losses: 5007 wins: 4993 ties: 0
losses: 5008 wins: 4992 ties: 0
losses: 5009 wins: 4991 ties: 0
losses: 5010 wins: 4990 ties: 0
losses: 5011 wins: 4989 ties: 0
losses: 5012 wins: 4988 ties: 0
losses: 5013 wins: 4987 ties: 0
losses: 5014 wins: 4986 ties: 0
losses: 5015 wins: 4985 ties: 0
losses: 5016 wins: 4984 ties: 0
losses: 5017 wins: 4983 ties: 0
losses: 5018 wins: 4982 ties: 0
losses: 5019 wins: 4981 ties: 0
losses: 5020 wins: 4980 ties: 0
losses: 5021 wins: 4979 ties: 0
losses: 5022 wins: 4978 ties: 0
losses: 5023 wins: 4977 ties: 0
losses: 5024 wins: 4976 ties: 0
losses: 5025 wins: 4975 ties: 0
losses: 5026 wins: 4974 ties: 0
losses: 5027 wins: 4973 ties: 0
losses: 5028 wins: 4972 ties: 0
losses: 5029 wins: 4971 ties: 0
losses: 5030 wins: 4970 ties: 0
losses: 5031 wins: 4969 ties: 0
losses: 5032 wins: 4968 ties: 0
losses: 5033 wins: 4967 ties: 0
losses: 5034 wins: 4966 ties: 0
losses: 5035 wins: 4965 ties: 0
losses: 5036 wins: 4964 ties: 0
losses: 5037 wins: 4963 ties: 0
losses: 5038 wins: 4962 ties: 0
losses: 5039 wins: 4961 ties: 0
losses: 5040 wins: 4960 ties: 0
losses: 5041 wins: 4959 ties: 0
losses: 5042 wins: 4958 ties: 0
losses: 5043 wins: 4957 ties: 0
losses: 5044 wins: 4956 ties: 0
losses: 5045 wins: 4955 ties: 0
losses: 5046 wins: 4954 ties: 0
losses: 5047 wins: 4953 ties: 0
losses: 5048 wins: 4952 ties: 0
losses: 5049 wins: 4951 ties: 0
losses: 5050 wins: 4950 ties: 0
losses: 5051 wins: 4949 ties: 0
losses: 5052 wins: 4948 ties: 0
losses: 5053 wins: 4947 ties: 0
losses: 5054 wins: 4946 ties: 0
losses: 5055 wins: 4945 ties: 0
losses: 5056 wins: 4944 ties: 0
losses: 5057 wins: 4943 ties: 0
losses: 5058 wins: 4942 ties: 0
losses: 5059 wins: 4941 ties: 0
losses: 5060 wins: 4940 ties: 0
losses: 5062 wins: 4938 ties: 0
losses: 5063 wins: 4937 ties: 0
losses: 5064 wins: 4936 ties: 0
losses: 5065 wins: 4935 ties: 0
losses: 5066 wins: 4934 ties: 0
losses: 5067 wins: 4933 ties: 0
losses: 5068 wins: 4932 ties: 0
losses: 5069 wins: 4931 ties: 0
losses: 5070 wins: 4930 ties: 0
losses: 5071 wins: 4929 ties: 0
losses: 5072 wins: 4928 ties: 0
losses: 5073 wins: 4927 ties: 0
losses: 5074 wins: 4926 ties: 0
losses: 5075 wins: 4925 ties: 0
losses: 5076 wins: 4924 ties: 0
losses: 5077 wins: 4923 ties: 0
losses: 5078 wins: 4922 ties: 0
losses: 5079 wins: 4921 ties: 0
losses: 5080 wins: 4920 ties: 0
losses: 5081 wins: 4919 ties: 0
losses: 5082 wins: 4918 ties: 0
losses: 5083 wins: 4917 ties: 0
losses: 5084 wins: 4916 ties: 0
losses: 5085 wins: 4915 ties: 0
losses: 5086 wins: 4914 ties: 0
losses: 5087 wins: 4913 ties: 0
losses: 5088 wins: 4912 ties: 0
losses: 5089 wins: 4911 ties: 0
losses: 5090 wins: 4910 ties: 0
losses: 5091 wins: 4909 ties: 0
losses: 5092 wins: 4908 ties: 0
losses: 5095 wins: 4905 ties: 0
losses: 5096 wins: 4904 ties: 0
losses: 5097 wins: 4903 ties: 0
losses: 5098 wins: 4902 ties: 0
losses: 5099 wins: 4901 ties: 0
losses: 5100 wins: 4900 ties: 0
losses: 5101 wins: 4899 ties: 0
losses: 5104 wins: 4896 ties: 0
losses: 5105 wins: 4895 ties: 0
losses: 5106 wins: 4894 ties: 0
losses: 5107 wins: 4893 ties: 0
losses: 5108 wins: 4892 ties: 0
losses: 5109 wins: 4891 ties: 0
losses: 5111 wins: 4889 ties: 0
losses: 5112 wins: 4888 ties: 0
losses: 5114 wins: 4886 ties: 0
losses: 5115 wins: 4885 ties: 0
losses: 5118 wins: 4882 ties: 0
losses: 5120 wins: 4880 ties: 0
losses: 5121 wins: 4879 ties: 0
losses: 5122 wins: 4878 ties: 0
losses: 5123 wins: 4877 ties: 0
losses: 5126 wins: 4874 ties: 0
losses: 5132 wins: 4868 ties: 0
losses: 5147 wins: 4853 ties: 0
losses: 5156 wins: 4844 ties: 0

So now you know why CCRL and CEGT run all those games.

[corres]

Some arbitrarily chose position is not suitable do decide about an engine is good or not.
In the evaluation of every engine there are "dark stains".
So only engine-engine matches can show the real power of engines.

Capablanca was a genius for playing game endings and was also a Chess World Champion.
The problem I left on this topic was solved by Tahl and he was also the Chess World Champion.
So a good engine is one that can solve a problem and also be a Champion in engine vs engine match.

This is only your idea.
The old time Chess World Champions were not machines and they were also not perfect.

What I mean is that one thing does not remove the other. That the deficit of the engines. The better they are for competition the worse they are at solving problems and vice versa. Older, lower-range engines perform better than current ones.

If you want to solve problems you need other engine than Stockfish but if you analyse positions of a running party you need Stockfish (together with Leela) and not a weak one.
There will be never such engine what totally universal. You would know that.

[Ajedrecista]

Hello Dann:

And 10,000:

Code: Select all

losses: 4857 wins: 5143 ties: 0
[...]
losses: 5000 wins: 5000 ties: 0
[...]
losses: 5156 wins: 4844 ties: 0

So now you know why CCRL and CEGT run all those games.

Wouldn't it be the direct application of the binomial distribution with P = 0.5? Then:

Code: Select all

n = wins + losses
(n >> 1)

Mean = n*P = 0.5*n
Standard deviation = sqrt[n*P*(1 - P)] = 0.5*sqrt(n)

z = (wins - mean)/(standard deviation) = 2*wins/sqrt(n) - sqrt(n)

------------

For n = 10^4

z = wins/50 - 100

Where z is the z-score. Chauvenet's criterion might be of interest for you but you may know it.

Regards from Spain.

Ajedrecista.

[Ovyron]

The best is Bluefish-XI with a couple small tweaks, which has no rivals so far in my testing.

Would you be willing to share it?

You will want to turn off logging in the UCI options.

Thanks Dann. I wonder if some pseudo-MultiPV 4 setting would outperform everything on the future, one that starts with Tactical=2 but reduces it to 0 as the search gets deeper (and the best move has already been caught.)