Clean

[Daniel Anulliero]

If you haven't seen the source code you don't understand what's going on. Strangely null move does not give any improvement anymore and I did not change the code. So I don't understand why. Might be it never gave any improvement.

These tests are so tremendously boring and wasting so much computer time. Also what holds on one minute games may not hold on five minutes games.

Well, it's simple, nullmove gives ~70+ elo, YMMV. If you get regression then it means that you don't test your code at all or or's pure crap (both is possible)
- note that disabling nullmove will lose elo BUT will make your engine solve some testpostions where others need randomly lobotomized piece of shite like "matesolvers" (troll trademark) and so on.
So yeah, cry ad lib. Basing conclusions on one game, c'mon. You can do better Anyway - you got what you asked for.

If time control is less than about three minutes null move reductions gives no improvements in my code.

How many games did you play , to say " null move in short TC did not improvments at all" ?
If it's toushand games , It's ok , for you , null-move do not give any improvement at short TC
If you've played 1 or 2 games ,then I must say to you (if you didn't understand yet) TOO FEW GAMES !!

[Henk]

If you haven't seen the source code you don't understand what's going on. Strangely null move does not give any improvement anymore and I did not change the code. So I don't understand why. Might be it never gave any improvement.

These tests are so tremendously boring and wasting so much computer time. Also what holds on one minute games may not hold on five minutes games.

Well, it's simple, nullmove gives ~70+ elo, YMMV. If you get regression then it means that you don't test your code at all or or's pure crap (both is possible)
- note that disabling nullmove will lose elo BUT will make your engine solve some testpostions where others need randomly lobotomized piece of shite like "matesolvers" (troll trademark) and so on.
So yeah, cry ad lib. Basing conclusions on one game, c'mon. You can do better Anyway - you got what you asked for.

If time control is less than about three minutes null move reductions gives no improvements in my code.

How many games did you play , to say " null move in short TC did not improvments at all" ?
If it's toushand games , It's ok , for you , null-move do not give any improvement at short TC
If you've played 1 or 2 games ,then I must say to you (if you didn't understand yet) TOO FEW GAMES !!

100-150 one and two minute games. Might still be that on three minutes games null moves does not give any improvement too for I only played fifty games. Three minutes game tests take too long. Getting sick of all these slow tests.

[Daniel Anulliero]

100-150 is too few too .
Now I test Isa with selftests , I test only at TC 1 minute + 250 miliseconds increment . I play 500 games each version against each other.
500 games take around 1200 minutes = 20 hours say, one day .
I test only at short TC because I know Isa is stronger at long TC, and it is released so , the Testers can test it at more long TC.
I have a pgn with 250 opennings ( with ECO codes A01 , A03 etc ... same for ECO codes B C D and E ) who are played reversed colors .
With always the same opennings you can see if it is improved or not

[flok]

If running tons of games takes too long for you, consider running it against the win-at-chess epd-file and see if your change gained more results.

[Henk]

If running tons of games takes too long for you, consider running it against the win-at-chess epd-file and see if your change gained more results.

Yes good idea. At least its fast.

[Henk]

100-150 is too few too .
Now I test Isa with selftests , I test only at TC 1 minute + 250 miliseconds increment . I play 500 games each version against each other.
500 games take around 1200 minutes = 20 hours say, one day .
I test only at short TC because I know Isa is stronger at long TC, and it is released so , the Testers can test it at more long TC.
I have a pgn with 250 opennings ( with ECO codes A01 , A03 etc ... same for ECO codes B C D and E ) who are played reversed colors .
With always the same opennings you can see if it is improved or not

Maybe you have a faster engine or computer. Might be that if you used thirty or fifteen second games null move reductions would harm too.

[Henk]

Skipper can't solve this one (Undermine 049). I took an arbitrary test position and already the first one it can't solve. Right move should be a3-a4.

[d] 3k1r1r/p2n1p1p/q2p2pQ/1p2P3/2pP4/P4N2/5PPP/2R1R1K1 w - -

Code: Select all

Depth  Value   Time(seconds)   Nodes
120    0.7320     659.51   172269791   e5d6 f7f6 h6f4 d8c8 c1b1 c8b8 e1e7 d7b6 d6d7 b8a8 f4e4 a8b8 
110    0.8390      56.60    24500051   e5d6 f7f6 h6f4 d8c8 c1c3 c8b8 e1b1 f8c8 f4e4 a6d6 b1b5 
100    0.7730      50.41    14789377   e5d6 f7f6 h6f4 d8c8 c1b1 c8b8 f4e4 f8d8 e4d5 a6a3 
 90    0.5900       7.55     4299197   e5d6 d8c8 h6g5 c8b8 g5e7 d7b6 f3g5 a6a3 g5f7 
 80    0.4630       4.87     1996944   e5d6 d8c8 h6g5 c8b8 g5e7 d7b6 f3g5 a6a3 
 70    0.3010       1.21      775090   e5d6 d8c8 h6f4 f7f5 c1c3 f8f6 f3e5 
 60    0.7620       0.33      171149   e5d6 d8c8 h6f4 c8b8 e1e7 d7b6 
 50    0.1970       0.28      167890   e5d6 d8c8 h6e3 a6d6 h2h3 
 40    0.7310       0.04       19500   e5d6 d8c8 h6g5 c8b7 
 30    0.8550       0.02        6273   e5d6 f7f6 h6f4 
 20    0.8270       0.01        1067   e5d6 a6d6 
 10    0.9490       0.04         179   e5d6 

[matthewlai]

I suggest you buy 31 (or better: 32) raspberry pi 2 or 3s. Then you can run 124 (or 128) games in parallel. Speeds up testing quite a bit.

I have actually done the math on this, and as it turned out, Raspberry Pis (even RPi 3) are terrible from a perf/$ point of view.

A Raspberry Pi 3 (with power supply and small SD card) is about $50.

The quad core 1.2 GHz Cortex-A53 is dual-issue, in-order CPU with an 8 stages pipeline. In practice that means it's about 1/4 the speed of a modern Intel CPU, per clock. So equivalent to about 300 MHz quad core.

You can build an i5-6600 (quad core 3.3 GHz) computer for about $390 ($200 CPU, $140 for 16GB RAM, $50 for power supply). 11 times faster for 8 times the price.

In practice it's even worse for the Pi because there's distributed computing overhead (though not that big of a problem in this specific case), and you also need to buy network switches and lots of cables, as well as extension cords.

So I would just run it on a regular PC, but use faster time control.

[flok]

Time control can be tricky.
And let's say a rpi2 uses 0.5A (it doesn't, it uses less) then that's 16A is 80 watt. That's probably less than a regular pc.
And hey, you can solder LEDs to the gpio-pins and watch them flicker relatively to the system load (I did that, quite a lot of fun).
Regarding the distributed comp. overhead: I just run 31 cutechess instances and after they finish I combine the pgn resultfiles. Just a bunch of shell scripts.
For the price: I buy them second hand which is way cheaper than 50 dollar.

[matthewlai]

Time control can be tricky.
And let's say a rpi2 uses 0.5A (it doesn't, it uses less) then that's 16A is 80 watt. That's probably less than a regular pc.
And hey, you can solder LEDs to the gpio-pins and watch them flicker relatively to the system load (I did that, quite a lot of fun).
Regarding the distributed comp. overhead: I just run 31 cutechess instances and after they finish I combine the pgn resultfiles. Just a bunch of shell scripts.
For the price: I buy them second hand which is way cheaper than 50 dollar.

It is true that RPis are more power efficient.

I have played many games at 0.1s per game or less. It's not really a problem if the engine and the interface you use are well designed (I wrote my own match arbitrater optimized for very high speed games).

RPi 3 is about 60% faster than RPi 2, so if you are using RPi 2 the difference is even greater. You need about 18 of them to match the same PC.

For engine testing there's not a lot of overhead, but with a PC you can do other things as well, and most things aren't so easily distributable.

You can buy second hand PCs for much lower price, too.