Post
by bob » Sun Oct 27, 2019 5:01 am
This is very difficult. Problem starts with different hardware platforms that will make the same program perform at different Elo levels. Just reducing NPS produces a very odd chess opponent. IE a program might have a GM-level evaluation that really understands pawn structure and such, but it will be tactically less skillful. That doesn't feel right as an opponent. Doing the opposite also doesn't work well at all. IE leave the search alone but phase out chess knowledge. Now you get a program that is positionally weak, but tactically a GM. We (Crafty group) spent a lot of time playing with this. I would LOVE to have a command something like "elo 1500" and suddenly Crafty would play like the average 1500 player. Easy to say. Very difficult to do and avoid the pitfalls above.
It is certainly a worthy topic for investigation, but the solution will likely be much more than a few lines of code. And there are oddities. IE if you take a good search + a purely random evaluation, Don Beal wrote a paper on this. I have label the result "the Beal effect." Bottom line is that on good hardware, a purely random evaluation will still play at a 2000 level or so. Why? This turns into a cheap mobility evaluation. At any node in the tree where you have lots of legal moves, you end up with lots of opportunity to get a larger random score returned. If you have few moves, you have a lower probability of getting a large number. I did exactly this several years ago and got LOTS of complaints from Crafty users saying "Bob, even with a purely random evaluation, I can't beat this thing ever. And as I phase out the random eval, it only gets stronger.
Current approach does both proportionally. I did three things. The command is "skill N" where N is a number between 0 and 100.
1. The skill number is taken as a percentage, so that the evaluation looks like this:
val = Evaluate(...)
return (skill * val / 100 + (100 - skill) * random() / 100);
IE skill 0 is purely random, skill 100 is purely total eval.
2. The skill number is used to lower the NPS. But rather than moving quicker (which would be the result of reducing the nodes searched) I introduced a "busy loop" so that for each node searched, the computational cost would ramp up as skill is lowered, so that as skill reduces, search depth reduces.
3. Finally, skill was applied to the various search extensions and reductions so that the clever search stuff phases out as skill lowers. Overall effect was actually pretty good (still in current Crafty versions, but has to be compiled in with -DSKILL). But the problem was, skill 50 means something different on each different platform since faster hardware will ramp up NPS.
4. And then there is parallel search. I did not disable parallel search, but if you think about it, before playing with the skill command, you would obviously NOT use multiple threads which only ramps search speed up.
I would not begin to claim what I did was optimal. But at least the complaints stopped. Just be aware that if you find a skill setting that gives you a good game, but with chances to win, the next time you change hardware, you will have to adjust skill again. If you do as I do and run on different machines (IE a bigger machine for testing, but my MacBook more frequently since I carry it with me. The same skill can't be used on both machines...