Open Source Bullet: Rodent 0.12, Faile 1.4

[lucasart]

Testing still running, so these results aren't final yet. However it seems clearer and clearer as the number of games increases, that Rodent 0.12 is actually worse than 0.11

Faile 1.4 is of course a much weaker program, but it has the benefit of being portable source code and stable (no crashes or time losses so far). A rare quality among weak programs.

Ratings

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws 
   1 Critter 1.4          3245   32   31   400   76%  3026   30% 
   2 IvanHoe 999946h      3184   31   30   400   67%  3041   33% 
   3 Stockfish 2.2.2      3179   29   29   450   68%  3030   29% 
   4 Protector 1.4        2928   28   28   500   49%  2936   23% 
   5 Umko 1.2             2871   26   25   600   53%  2857   26% 
   6 Toga 1.4.1           2858   24   24   700   57%  2811   24% 
   7 Glaurung 2.2         2822   29   30   450   39%  2923   28% 
   8 Daydreamer 1.75      2746   23   23   650   60%  2675   30% 
   9 Fruit 2.1            2700   22   22   700   51%  2697   25% 
  10 Crafty 23.4          2689   25   25   550   47%  2709   24% 
  11 Cheng3 1.07          2659   26   25   506   55%  2627   25% 
  12 Arasan 13.4          2650   25   25   550   49%  2660   22% 
  13 GNU Chess 5.07.173b  2650   24   24   600   48%  2668   27% 
  14 Scorpio 2.7          2636   22   22   700   45%  2678   26% 
  15 Pepito 1.59          2599   24   24   600   52%  2586   22% 
  16 Rodent 0.12          2597   40   41   206   30%  2737   32% 
  17 Sloppy 0.2.2         2555   22   23   700   42%  2613   24% 
  18 DoubleCheck 2.6      2554   26   26   500   49%  2560   20% 
  19 EXchess 6.10         2540   26   26   550   59%  2463   22% 
  20 Greko 9.0            2494   24   24   650   38%  2586   24% 
  21 Pawny 0.3.1          2488   28   28   500   56%  2429   16% 
  22 Sungorus 1.4         2326   27   27   549   41%  2390   18% 
  23 Jazz 501             2307   31   31   360   43%  2364   23% 
  24 Ifrit m1.5           2299   35   35   300   55%  2261   20% 
  25 KMT Chess 1.2.1      2246   36   36   300   46%  2279   18% 
  26 Beowulf 2.4          2234   35   36   300   45%  2281   20% 
  27 Faile 1.4            1966   47   52   259   11%  2318   13% 

Conditions
* Open Source and Portable engines only: no closed source, commercial, or windows only programs.
* Copyleft: Ideally licensed under the GNU GPL, or with copyright restrictions that are not excessive.
* 1min+1sec/move, 64 MB Hash, 1 Thread, 64-bit, Ponder off, no EGTB.
* 8 book moves: neither to little, nor too much. Allows engines to develop their own plan, while offering a large enough number of opening positions.
* Bayeselo, offset Fruit 2.1 = 2700 elo.

[lucasart]

Rodent 0.12 finished. It seems to be as strong as 0.11 after all, and even a tiny bit more, just goes to show how elo depends a lot on a choice of opponents

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws 
   1 Critter 1.4          3246   32   31   400   76%  3027   30% 
   2 IvanHoe 999946h      3185   31   30   400   67%  3042   33% 
   3 Stockfish 2.2.2      3180   29   29   450   68%  3030   29% 
   4 Protector 1.4        2928   28   28   500   49%  2937   23% 
   5 Umko 1.2             2871   26   25   600   53%  2856   26% 
   6 Toga 1.4.1           2856   24   24   700   57%  2808   24% 
   7 Glaurung 2.2         2828   29   29   450   39%  2929   28% 
   8 Daydreamer 1.75      2747   23   23   650   60%  2675   30% 
   9 Fruit 2.1            2700   22   22   700   51%  2697   25% 
  10 Crafty 23.4          2690   25   25   550   47%  2710   24% 
  11 Cheng3 1.07          2654   25   24   550   54%  2624   25% 
  12 Rodent 0.12          2649   23   23   600   51%  2644   31% 
  13 GNU Chess 5.07.173b  2643   23   23   650   48%  2661   27% 
  14 Arasan 13.4          2643   24   24   600   49%  2653   24% 
  15 Scorpio 2.7          2630   21   22   750   45%  2671   27% 
  16 Pepito 1.59          2586   23   23   650   50%  2584   23% 
  17 DoubleCheck 2.6      2543   25   25   550   48%  2559   20% 
  18 Sloppy 0.2.2         2542   22   22   750   40%  2609   24% 
  19 EXchess 6.10         2525   25   25   600   56%  2468   23% 
  20 Greko 9.0            2485   24   24   650   38%  2578   24% 
  21 Pawny 0.3.1          2476   28   28   500   56%  2417   16% 
  22 Sungorus 1.4         2313   27   27   549   41%  2378   18% 
  23 Jazz 501             2295   31   32   360   43%  2352   23% 
  24 Ifrit m1.5           2287   35   35   300   55%  2249   20% 
  25 KMT Chess 1.2.1      2233   36   36   300   46%  2266   18% 
  26 Beowulf 2.4          2222   35   36   300   45%  2268   20% 
  27 Faile 1.4            1953   47   52   259   11%  2306   13% 

[lucasart]

Faile 1.4 testing finished

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws 
   1 Critter 1.4          3246   32   31   400   76%  3027   30% 
   2 IvanHoe 999946h      3185   31   30   400   67%  3042   33% 
   3 Stockfish 2.2.2      3180   29   29   450   68%  3030   29% 
   4 Protector 1.4        2928   28   28   500   49%  2937   23% 
   5 Umko 1.2             2871   26   25   600   53%  2856   26% 
   6 Toga 1.4.1           2856   24   24   700   57%  2808   24% 
   7 Glaurung 2.2         2828   29   29   450   39%  2929   28% 
   8 Daydreamer 1.75      2747   23   23   650   60%  2675   30% 
   9 Fruit 2.1            2700   22   22   700   51%  2697   25% 
  10 Crafty 23.4          2690   25   25   550   47%  2710   24% 
  11 Cheng3 1.07          2654   25   24   550   54%  2624   25% 
  12 Rodent 0.12          2649   23   23   600   51%  2644   31% 
  13 GNU Chess 5.07.173b  2643   23   23   650   48%  2661   27% 
  14 Arasan 13.4          2643   24   24   600   49%  2653   24% 
  15 Scorpio 2.7          2630   21   22   750   45%  2671   27% 
  16 Pepito 1.59          2586   23   23   650   50%  2584   23% 
  17 DoubleCheck 2.6      2543   25   25   550   48%  2559   20% 
  18 Sloppy 0.2.2         2542   22   22   750   40%  2609   24% 
  19 EXchess 6.10         2525   25   25   600   56%  2468   23% 
  20 Greko 9.0            2485   24   24   650   38%  2578   24% 
  21 Pawny 0.3.1          2476   28   28   500   56%  2418   16% 
  22 Sungorus 1.4         2314   27   27   550   41%  2378   18% 
  23 Jazz 501             2292   30   31   400   47%  2313   23% 
  24 Ifrit m1.5           2288   35   35   300   55%  2250   20% 
  25 KMT Chess 1.2.1      2234   36   36   300   46%  2267   18% 
  26 Beowulf 2.4          2223   35   36   300   45%  2269   20% 
  27 Faile 1.4            1960   43   47   300   12%  2305   13% 

[PK]

First of all, thanks for the test! I've got a couple of questions:

1) has Rodent been playing against slightly stronger selection of opponents?
2) am I correct to assume that it lost some ground against weaker opponents?

I wouldn't be surprised by that, as v. 0.12 reaches much lower depths, due to restrictions imposed on LMR.

[lucasart]

First of all, thanks for the test! I've got a couple of questions:

1) has Rodent been playing against slightly stronger selection of opponents?
2) am I correct to assume that it lost some ground against weaker opponents?

I wouldn't be surprised by that, as v. 0.12 reaches much lower depths, due to restrictions imposed on LMR.

Rodent 0.12 played 50 games against each of the following opponents (in this exact order, and w/o restarting Rodent for the total 600 games = small hash table benefit there)

Code: Select all

   7 Glaurung 2.2         2828   29   29   450   39%  2929   28% 
   8 Daydreamer 1.75      2747   23   23   650   60%  2675   30% 
   9 Fruit 2.1            2700   22   22   700   51%  2697   25% 
  10 Crafty 23.4          2690   25   25   550   47%  2710   24% 
  11 Cheng3 1.07          2654   25   24   550   54%  2624   25% 
  13 GNU Chess 5.07.173b  2643   23   23   650   48%  2661   27% 
  14 Arasan 13.4          2643   24   24   600   49%  2653   24% 
  15 Scorpio 2.7          2630   21   22   750   45%  2671   27% 
  16 Pepito 1.59          2586   23   23   650   50%  2584   23% 
  17 DoubleCheck 2.6      2543   25   25   550   48%  2559   20% 
  18 Sloppy 0.2.2         2542   22   22   750   40%  2609   24% 
  19 EXchess 6.10         2525   25   25   600   56%  2468   23% 

In fact it performed relatively better against weaker engines than stronger ones.

In the end, Rodent 0.12 did show an improvement compared to 0.11, but not a very big one (15-20 elo).

It's a very good performance, now only 51 elo behind Fruit 2.1 (which is the benchmark of my list calibrated with Fruit 2.1 = 2700 elo).

PS: I used the 64 bit Linux compile from Jim Ablett.