Open letter to chess programmers

[Don]

The fingerprint method should be verified by applying it to programs that are KNOWN to be related and on other ones that are KNOWN to be unrelated.

That has already been done a million times and graphs have been posted on this forum showing very high correlations with clones and little correlation with programs known to be different.

I can already suggest a means of cheating: Clone a program and run the clone on a very fast or very slow computer. That makes the 1 second constraint meaningless.

Actually, the fingerprint detection is surprisingly robust with respect to depth of search. Running it more slowly will only make a little difference and probably not enough to fool the fingerprint.

This point is not generally understood (or perhaps just not believed) but how programs choose moves is pretty idiosyncratic to each program and related very much to depth. In most positions your program is probably going to play the same move on 10 ply search as it will on a 20 ply search and it wont' necessarily be the same move that some other program will play.

You can also slow the program down artificially by software means: What arcade game programmer hasn't written delay loops? That can be the default, and only you know how to disable the secret delay loops...

[Rebel]

The fingerprint method should be verified by applying it to programs that are KNOWN to be related and on other ones that are KNOWN to be unrelated.

I can already suggest a means of cheating: Clone a program and run the clone on a very fast or very slow computer. That makes the 1 second constraint meaningless.

You can also slow the program down artificially by software means: What arcade game programmer hasn't written delay loops? That can be the default, and only you know how to disable the secret delay loops...

As Miguel and Don already stated lots of research has been done, 2 good places to start with are:

http://www.top-5000.nl/clone.htm
http://www.top-5000.nl/simex.htm

[wgarvin]

Open Source is great if a community works on a common goal, like creating the best operating system in the world.

In a competitive environment open source hurts more than it helps. This is not about giving mankind a strong chess engine, it is a competition and here demanding only open source to participate is not a very smart idea.

For someone like me that entered the scene only a few years ago it looks like the whole trouble started when Fruit became Open Source. I don't think the trouble was intend but the road to hell is paved with good intentions.

I think the idea idea of Richard and Marcel is good and worth a try

Thomas...

Not everyone in the community has the same motives, so you can't say open source is categorically bad for computer chess. Some people like the glory, others just want to share their work product with the community and hopefully advance the state of the art (or at least, enrich the scene with the variety of their implementation techniques).

Yes the clones and the cloners are a pox on the community, but don't blame this on the selfless programmers who generously shared their source with the community. Rather, place the blame where it belongs: on those individuals with weak morals, who clone and exploit the hard work of others for their own selfish gain. Blame the cloners, not their victims.

[Edit: I was responding mostly to the bolded portion above. Maybe requiring open source for competitions would be divisive/problematic. I was just irked by the suggestion that the people giving away their source code for free were the cause of the problems.. my view is that it is the unprincipled use of that code by others that actually causes the problem.]

[laoliveirajr]

.
Viva a Transparência !!!

Next weekend the fingerprints of the last 4 versios of the Capivara chess engine will be published at download site.

To the next versions will be published fingerprints too.

Code: Select all

Positions  8238 {CapivaraLK009a01a}	-------- simex was run with no parameters ...			
{GreKo 6.5        }	45.90	{Cyrano 06b17     }	42.17	{Umko 0.7         }	40.28
{Rotor 0.4        }	45.56	{Djinn 0.925x     }	42.16	{Hannibal 1.0a    }	40.26
{Cheng 1.03       }	45.48	{Nemo 1.0 beta    }	42.16	{nanoSzachy 3.5   }	40.14
{RedQueen 0.7.5   }	44.50	{The King 3.50    }	42.11	{Booot 5.1.0      }	40.12
{Spike 0.8a       }	44.45	{Leila 0.53h      }	42.10	{Aristarch 4.51   }	40.08
{Glass 1.5        }	44.42	{Loop 2007        }	41.87	{Cerebro 3.03d    }	40.08
{Bobcat 2.75      }	43.81	{GarboChess 3     }	41.85	{Spark-0.3        }	40.01
{Petir 4.39       }	43.75	{Doch 09.980      }	41.82	{Hiarcs 12        }	40.00
{Hamsters 0.71    }	43.63	{Movei 0.08.438   }	41.79	{Patzer 3.80      }	39.92
{Hussar 0.4       }	43.57	{Ares 1.004       }	41.77	{Tornado 3.42a    }	39.91
{N2 0.4           }	43.57	{CuckooChess 1.12 }	41.43	{Houdini 1.00     }	39.90
{Godel 2.05       }	43.54	{Thinker 4.7a     }	41.43	{SpiderChess      }	39.77
{Knight Dreamer   }	43.52	{Little Goliath   }	41.37	{Cyclone 2.1      }	39.74
{LambChop 10.99   }	43.49	{Snitch 1.6.2     }	41.37	{Shredder 11      }	39.63
{Ktulu 8          }	43.47	{Stockfish 1.01   }	41.37	{Pupsi2 0.08      }	39.61
{Naum 1.91        }	43.45	{Fruit 2.1        }	41.36	{RobboLito 0.09   }	39.54
{Dragon 4.6       }	43.38	{Atlas 3.14b      }	41.25	{Glaurung 1.01    }	39.37
{Pharaon 3.51     }	43.38	{ET Chess 130108  }	41.16	{Comet B68        }	39.22
{Pepito 1.59      }	43.04	{Gromit 3.8.2     }	41.16	{Naum 4.2         }	39.21
{Naraku 1.1       }	43.02	{Frenzee 3.5.19   }	41.15	{Tankist 1.1      }	39.15
{Green Light Chess}	43.01	{Thinker 5.4d     }	41.05	{Trace 1.37a      }	39.12
{SlowChess 2.96   }	42.98	{King of Kings    }	41.03	{Homer 2.01       }	39.10
{Cheese 1.3       }	42.97	{Jonny 2.83       }	41.02	{IvanHoe 999946b  }	39.10
{Diablo 0.51      }	42.95	{Ufim 6.00        }	40.91	{TwistedLogic 2008}	39.08
{Alaric 707       }	42.89	{Chess Tiger 2007 }	40.90	{Gandalf 6.0      }	38.93
{Colossus 2006e   }	42.89	{Quark 2.35       }	40.84	{Fire 1.01        }	38.89
{List 5.12        }	42.89	{SmarThink 1.20   }	40.76	{Deep Saros 0.9   }	38.77
{AnMon 5.75       }	42.86	{Protector 1.27   }	40.73	{Quazar 0.3a      }	38.67
{Tao 5.6          }	42.86	{GNU Chess 5.07   }	40.70	{Sjeng WC 2008    }	38.67
{Amy              }	42.67	{Strelka 1.8      }	40.63	{Bison 9.11       }	38.50
{Yace 0.99.87     }	42.61	{Gaviota 0.84     }	40.60	{Komodo 3         }	38.31
{Onno 1.04        }	42.51	{Bright-0.1d      }	40.59	{MishasMauler03   }	38.26
{Deuterium 090    }	42.49	{Arasan 13.3      }	40.58	{The Baron 2.23   }	38.26
{Thor's Hammer    }	42.47	{Philou 3.14.1    }	40.54	{Zappa Mexico II  }	37.99
{WildCat 60903    }	42.41	{Ruffian          }	40.48	{Critter 1.2      }	37.93
{Hermann 2.4      }	42.40	{DanaSah 4.60     }	40.47	{Vitruvius 0.83b  }	37.78
{Gull 0.6         }	42.35	{Rybka 1.0 Beta   }	40.43	{Armageddon 1.0   }	37.73
{Amyan 1.72       }	42.29	{Alfil 11         }	40.37	{Junior 10.1      }	35.53
{Crafty 23.4      }	42.28	{Dirty 25AUG2011  }	40.35	{CapivaraLK009a01a}	-----
{Viper 0.1        }	42.27	{ProDeo 1.74      }	40.28		

[asanjuan]

The fingerprint method should be verified by applying it to programs that are KNOWN to be related and on other ones that are KNOWN to be unrelated.

That has already been done a million times and graphs have been posted on this forum showing very high correlations with clones and little correlation with programs known to be different.

I can already suggest a means of cheating: Clone a program and run the clone on a very fast or very slow computer. That makes the 1 second constraint meaningless.

Actually, the fingerprint detection is surprisingly robust with respect to depth of search. Running it more slowly will only make a little difference and probably not enough to fool the fingerprint.

This point is not generally understood (or perhaps just not believed) but how programs choose moves is pretty idiosyncratic to each program and related very much to depth. In most positions your program is probably going to play the same move on 10 ply search as it will on a 20 ply search and it wont' necessarily be the same move that some other program will play.

You can also slow the program down artificially by software means: What arcade game programmer hasn't written delay loops? That can be the default, and only you know how to disable the secret delay loops...

When I think about this test, one thing that cames to my mind is the concept of random output when doing a time based search.

We all know that in every game the same engine can produce a different best move for the same position, just because when time is involved, the search result is non deterministic.

I guess what would be the percentage of coincidence if we execute the fingerprint N times with the same engine.

Can this test give us a (somehow) error margin? Has anyone tried this test?

[Laskos]

The fingerprint method should be verified by applying it to programs that are KNOWN to be related and on other ones that are KNOWN to be unrelated.

That has already been done a million times and graphs have been posted on this forum showing very high correlations with clones and little correlation with programs known to be different.

I can already suggest a means of cheating: Clone a program and run the clone on a very fast or very slow computer. That makes the 1 second constraint meaningless.

Actually, the fingerprint detection is surprisingly robust with respect to depth of search. Running it more slowly will only make a little difference and probably not enough to fool the fingerprint.

This point is not generally understood (or perhaps just not believed) but how programs choose moves is pretty idiosyncratic to each program and related very much to depth. In most positions your program is probably going to play the same move on 10 ply search as it will on a 20 ply search and it wont' necessarily be the same move that some other program will play.

You can also slow the program down artificially by software means: What arcade game programmer hasn't written delay loops? That can be the default, and only you know how to disable the secret delay loops...

When I think about this test, one thing that cames to my mind is the concept of random output when doing a time based search.

We all know that in every game the same engine can produce a different best move for the same position, just because when time is involved, the search result is non deterministic.

I guess what would be the percentage of coincidence if we execute the fingerprint N times with the same engine.

Can this test give us a (somehow) error margin? Has anyone tried this test?

with ~8,000 positions, an error of ~ 0.5% 1 standard deviation, if I remember well.

[asanjuan]

with ~8,000 positions, an error of ~ 0.5% 1 standard deviation, if I remember well.

wow!

[Adam Hair]

I think that the SD is even lower for most engines (excluding Rybka 3 and related engines).

[laoliveirajr]

Which the average time to run the program CSVN Fingerprinting test tool v1.0 do a full run?

In my Core2Quad Q6600 2400 is already running 24 hours ago ...

Already on the second day with the same screen:

Code: Select all

CSVN Fingerprinting test tool v1.0
----------------------------------

What type of engine is used? (W/U) : u
What is the name of the engine executable? : CapivaraLK009a01a.exe
done.
Engine search: 1/10000 |

[bob]

Which the average time to run the program CSVN Fingerprinting test tool v1.0 do a full run?

In my Core2Quad Q6600 2400 is already running 24 hours ago ...

Already on the second day with the same screen:

Code: Select all

CSVN Fingerprinting test tool v1.0
----------------------------------

What type of engine is used? (W/U) : u
What is the name of the engine executable? : CapivaraLK009a01a.exe
done.
Engine search: 1/10000 |

It runs 10K positions @ 1 second per position, so 10K seconds. If you have a dual-core box, it will run two at a time and cut this to 5 seconds, which is what happened on my 2ghz I7 dual-core macbook.

BTW, when I downloaded the thing, all I got was the Perl script. I had to go back and download the eps file separately. It then worked just fine on my mac...