EGTB value

[bob]

I had promised, for some time, to do an EGTB/noEGTB test on my cluster. I finally got around to copying the 3-4-5 piece files to each cluster node (to use the local disk rather than NFS-mounted remote filesystems, and have started the first run.

I am using 23.4, and playing my normal fast games first, one with, one without. Then I am going to play slower games, same thing.

For the fast games, which has not quite finished, EGTBs seems to be a negative thing:

Code: Select all

    Crafty-23.4-1        2668    4    4 30000   61%  2578   20% 
    Crafty-23.4-2        2666    4    4 30000   61%  2578   21% 
    Crafty-23.4R09       2660    4    4 26507   61%  2578   20% 

R09 is the version using local 3/4/5 piece files (no way to test with 6's as there is not enough space on any single node, and network-mounted disks are hopeless due to performance issues, particularly on a cluster where all nodes would be competing for access)

I have two of these tests queued up to run. Once both R09's finish, I am going to queue up two more runs, using 23.4 and 23.4R09, but with maybe 1min+1sec time controls. And then finally with 5min+5sec time controls. Maybe I can once and for all determine if EGTBs help, hurt, or break-even. At least for fast games, they appear to hurt very slightly.

More as the tests are completed...

[Mangar]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

[bob]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

[bob]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

Am trying some more restrictive probe limits. So far, 1/2 of iteration depth has not made any significant difference. These should all finish by later today. I strongly suspect that at least at this fairly quick time control, EGTBs are either break-even or a slight loss. Once these finish, I will try longer games, but suspect the results will be the same.

[michiguel]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I do not believe this is biggest help provided by EGTBs. See below.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

Am trying some more restrictive probe limits. So far, 1/2 of iteration depth has not made any significant difference. These should all finish by later today. I strongly suspect that at least at this fairly quick time control, EGTBs are either break-even or a slight loss. Once these finish, I will try longer games, but suspect the results will be the same.

EGTBs are supposed to help more at slower time controls because their main power is their (perfectly sound) pruning abilities. The deeper you search, the more impact you have. If you prune close to the root the savings are more pronounced in deeper trees. The keep cutting the same branch despite the draft is getting bigger and bigger.

Miguel

[bob]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I do not believe this is biggest help provided by EGTBs. See below.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

Am trying some more restrictive probe limits. So far, 1/2 of iteration depth has not made any significant difference. These should all finish by later today. I strongly suspect that at least at this fairly quick time control, EGTBs are either break-even or a slight loss. Once these finish, I will try longer games, but suspect the results will be the same.

EGTBs are supposed to help more at slower time controls because their main power is their (perfectly sound) pruning abilities. The deeper you search, the more impact you have. If you prune close to the root the savings are more pronounced in deeper trees. The keep cutting the same branch despite the draft is getting bigger and bigger.

Miguel

I am not convinced, but have a test ready to go when the current test completes.

If you get cutoffs near the root, you are "already there" anyway. The real benefit is those hits that occur way out in the search, because then you actually have a chance to choose whether or not to go toward that path.

More when results of slower game are finished. I've always been interested in the trade-off these things offer. You get much reduced depth overall, in return for perfect information on the nodes where you get a hit. If those hits are not important, the overall reduced depth will hurt. I am becoming more and more certain that this is not a particularly good trade-off.

[michiguel]

I had promised, for some time, to do an EGTB/noEGTB test on my cluster. I finally got around to copying the 3-4-5 piece files to each cluster node (to use the local disk rather than NFS-mounted remote filesystems, and have started the first run.

I am using 23.4, and playing my normal fast games first, one with, one without. Then I am going to play slower games, same thing.

For the fast games, which has not quite finished, EGTBs seems to be a negative thing:

Code: Select all

    Crafty-23.4-1        2668    4    4 30000   61%  2578   20% 
    Crafty-23.4-2        2666    4    4 30000   61%  2578   21% 
    Crafty-23.4R09       2660    4    4 26507   61%  2578   20% 

R09 is the version using local 3/4/5 piece files (no way to test with 6's as there is not enough space on any single node, and network-mounted disks are hopeless due to performance issues, particularly on a cluster where all nodes would be competing for access)

I have two of these tests queued up to run. Once both R09's finish, I am going to queue up two more runs, using 23.4 and 23.4R09, but with maybe 1min+1sec time controls. And then finally with 5min+5sec time controls. Maybe I can once and for all determine if EGTBs help, hurt, or break-even. At least for fast games, they appear to hurt very slightly.

More as the tests are completed...

"Maybe I can once and for all determine if Nalimov EGTBs help, hurt, or break-even"

Bold is mine
Miguel

[michiguel]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I do not believe this is biggest help provided by EGTBs. See below.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

Am trying some more restrictive probe limits. So far, 1/2 of iteration depth has not made any significant difference. These should all finish by later today. I strongly suspect that at least at this fairly quick time control, EGTBs are either break-even or a slight loss. Once these finish, I will try longer games, but suspect the results will be the same.

EGTBs are supposed to help more at slower time controls because their main power is their (perfectly sound) pruning abilities. The deeper you search, the more impact you have. If you prune close to the root the savings are more pronounced in deeper trees. The keep cutting the same branch despite the draft is getting bigger and bigger.

Miguel

I am not convinced, but have a test ready to go when the current test completes.

If you get cutoffs near the root, you are "already there" anyway. The real benefit is those hits that occur way out in the search, because then you actually have a chance to choose whether or not to go toward that path.

More when results of slower game are finished. I've always been interested in the trade-off these things offer. You get much reduced depth overall, in return for perfect information on the nodes where you get a hit. If those hits are not important, the overall reduced depth will hurt. I am becoming more and more certain that this is not a particularly good trade-off.

That is the point I was trying to make: you do not get reduced depth. Thanks to EGTBs, if done properly, you search deeper.

Miguel

[bob]

Hi Bob,

in Spike we have an UCI parameter for EGTB access. We can select how many plys away from horizont EGTB lookups will be done. The standard setting is 4 or 5 ply away from horizont (this is an old value optimized before some radical lmr and pruning parameters, with our new version it should be about 8).

In our tests we didn´t loose or win elo´s by using such a parameter. How far away from horizont do you access EGTB´s?

Greetings Volker

I use "iteration depth" as a limit. If I am doing a search at iteration 20, I do not probe beyond ply=20, regardless of how deep the search gets extended. I did a ton of testing on this a couple of years ago, and found (for Crafty) that this was the best value. I have not re-tuned since the last year's worth of pruning/reduction changes. But I had been suspicious for a couple of years that this is at best, a break-even deal. My two runs have completed, with the following results:

Code: Select all

    Crafty-23.4-1        2664    4    4 30000   61%  2574   20%
    Crafty-23.4R09-2     2664    4    4 30000   61%  2574   21%
    Crafty-23.4-2        2662    4    4 30000   61%  2574   21%
    Crafty-23.4R09-1     2657    4    4 30000   61%  2574   21%

They are so close, more games are needed. And I suspect that the R09-2 run is on the upper end of reality based on the error bars. I might try a few more restrictive limits on probing, but really believe that until we get to hits from the starting position, this is not going to work out very well. As it is, we have to commit to some path in the game, before we get the first hit. Once we are committed, it doesn't matter whether the egtb says win or lose, we are already going down that path, which means mostly luck as to whether we reach a won or lost position.

I do not believe this is biggest help provided by EGTBs. See below.

I'm going to do more testing, but as of right now, my intent is going to be to remove all the EGTB code since it is not providing any Elo boost, unless I happen to find some setting/limit that does improve results.

Am trying some more restrictive probe limits. So far, 1/2 of iteration depth has not made any significant difference. These should all finish by later today. I strongly suspect that at least at this fairly quick time control, EGTBs are either break-even or a slight loss. Once these finish, I will try longer games, but suspect the results will be the same.

EGTBs are supposed to help more at slower time controls because their main power is their (perfectly sound) pruning abilities. The deeper you search, the more impact you have. If you prune close to the root the savings are more pronounced in deeper trees. The keep cutting the same branch despite the draft is getting bigger and bigger.

Miguel

I am not convinced, but have a test ready to go when the current test completes.

If you get cutoffs near the root, you are "already there" anyway. The real benefit is those hits that occur way out in the search, because then you actually have a chance to choose whether or not to go toward that path.

More when results of slower game are finished. I've always been interested in the trade-off these things offer. You get much reduced depth overall, in return for perfect information on the nodes where you get a hit. If those hits are not important, the overall reduced depth will hurt. I am becoming more and more certain that this is not a particularly good trade-off.

That is the point I was trying to make: you do not get reduced depth. Thanks to EGTBs, if done properly, you search deeper.

Miguel

You _do_ get reduced depth. Just take _any_ position and run with tables on and tables off. Assuming you get a reasonable number of table hits, compare the final depth for each. Even an ending like fine 70, which has a lot of locked pawns, loses 4-5-6 plies depending on how deep you want to go, with tables on.

So, the question is, is that shallower search still better because it has _some_ perfect information in it? In fast games, the answer is certainly "no".

I am now running 1m+1s games. I finagled my script a bit so that I am running both with and without tables at the same time so I can compare partial results as the test progresses. Each node has 2 cpus, and here I am running the same 2 opponents twice, one with a version of crafty with endgame table support compiled in, one with it compiled out, so that on a single node, there will only be one "prober" active.

I'll post partial results from time to time, once enough games are played.

After only 150 games each, it is almost dead heat so far.

Code: Select all

    Crafty-23.4          2631   46   46   135   56%  2587   24% 
    Crafty-23.4R10       2622   45   45   142   55%  2588   29% 

23.4 has no egtb support, 23.4R10 does. They have played the same opponents, same positions (roughly)

[bob]

I had promised, for some time, to do an EGTB/noEGTB test on my cluster. I finally got around to copying the 3-4-5 piece files to each cluster node (to use the local disk rather than NFS-mounted remote filesystems, and have started the first run.

I am using 23.4, and playing my normal fast games first, one with, one without. Then I am going to play slower games, same thing.

For the fast games, which has not quite finished, EGTBs seems to be a negative thing:

Code: Select all

    Crafty-23.4-1        2668    4    4 30000   61%  2578   20% 
    Crafty-23.4-2        2666    4    4 30000   61%  2578   21% 
    Crafty-23.4R09       2660    4    4 26507   61%  2578   20% 

R09 is the version using local 3/4/5 piece files (no way to test with 6's as there is not enough space on any single node, and network-mounted disks are hopeless due to performance issues, particularly on a cluster where all nodes would be competing for access)

I have two of these tests queued up to run. Once both R09's finish, I am going to queue up two more runs, using 23.4 and 23.4R09, but with maybe 1min+1sec time controls. And then finally with 5min+5sec time controls. Maybe I can once and for all determine if EGTBs help, hurt, or break-even. At least for fast games, they appear to hurt very slightly.

More as the tests are completed...

"Maybe I can once and for all determine if Nalimov EGTBs help, hurt, or break-even"

Bold is mine
Miguel

I don't think it matters. bitbases might be a different issue since they are memory-resident and don't add in the msecs/read I/O penalty. But for any other type of "tablebases" I think this test will be pretty definitive, whether we like the answer or not. I might then try bitbases to see if there is any gain by using them in the search, and perhaps EGTBs at the first ply or 2 only to make progress...