Houdini 2.0b T4 settings

[Houdini]

It all depends what precision you're looking for. For a rating list that aims at a 20 Elo precision, 1000 games is fine. For fine-tuning an engine where you expect a change of the order of 5 Elo (and that is an optimistic case!), you need a lot more games.

The error levels decrease with the square root of the number of games, to reduce the error level from 20 points to 5 points you need to play 16 times more games (16,000 games).

[Tennison]

Test 2 : 3 min games

Book : HS7moves+
TB5 : ON
GUI : Fritz 11
Hash : 64MB

1 Houdini 2.0b T4 +13/-11/=16 52.50% 21.0/40
2 Houdini 2.0b +11/-13/=16 47.50% 19.0/40

I will make another 3min test during the night.

[Sedat Canbaz]

It all depends what precision you're looking for. For a rating list that aims at a 20 Elo precision, 1000 games is fine. For fine-tuning an engine where you expect a change of the order of 5 Elo (and that is an optimistic case!), you need a lot more games.

The error levels decrease with the square root of the number of games, to reduce the error level from 20 points to 5 points you need to play 16 times more games (16,000 games).

I have almost same view as you...

But however, is there any proof (rating list with popular time control) e.g after 16.000 games that the any Engine's Elo reduces the error level from 20 points to 5 points ???

Forget 30.000,forget 15.000,forget 8.000 games even there is no such rating based on 5.000 games per player so far (i mean with popular time control)

And I strongly believe (with very well-optimized neutral book) the number between 1.000 and 2.000 games are enough data to show the real strenght of the engines

BTW,I have no patience to see a such hero:
-Who will create 5.000 Elo Chess Engine (on i7 980X @4.0GHz)
or
-Who will create a rating list (with popular time control), based on 15.000 games per player


Best Regards,
Sedat

[gleperlier]

But I think it's a pretty good beginning. Better than if it scores only 40%

Ben, some good improvements start off with 45% after 300 games. Some bad ideas score 55% after 300 games. The whole point being that with so few games the random variations are 5 to 10 times larger than the expected strength difference between the two versions.

Playing 8000 games is not beyond reach of standard hardware. At 30 seconds/game you can play over 10,000 games/day using 4 cores.

30 secondes per game, it's ... it's... not chess....
30 secondes per game with ponder off....

[geots]

But I think it's a pretty good beginning. Better than if it scores only 40%

Ben, some good improvements start off with 45% after 300 games. Some bad ideas score 55% after 300 games. The whole point being that with so few games the random variations are 5 to 10 times larger than the expected strength difference between the two versions.

Playing 8000 games is not beyond reach of standard hardware. At 30 seconds/game you can play over 10,000 games/day using 4 cores.

Robert, I wouldn't advise doing that on a 4 core system.


george

[Houdini]

30 secondes per game, it's ... it's... not chess....
30 secondes per game with ponder off....

I fully agree.
We're not talking about chess, we're talking about tuning engines.

Robert

[geots]

It all depends what precision you're looking for. For a rating list that aims at a 20 Elo precision, 1000 games is fine. For fine-tuning an engine where you expect a change of the order of 5 Elo (and that is an optimistic case!), you need a lot more games.

The error levels decrease with the square root of the number of games, to reduce the error level from 20 points to 5 points you need to play 16 times more games (16,000 games).

Impossible to argue with your knowledge and brilliance. You're one of the 2 best I have ever seen. But I just can't imagine anyone gaining anything useful from 30" games.


gts

[Sedat Canbaz]

It all depends what precision you're looking for. For a rating list that aims at a 20 Elo precision, 1000 games is fine. For fine-tuning an engine where you expect a change of the order of 5 Elo (and that is an optimistic case!), you need a lot more games.

The error levels decrease with the square root of the number of games, to reduce the error level from 20 points to 5 points you need to play 16 times more games (16,000 games).

Impossible to argue with your knowledge and brilliance. You're one of the 2 best I have ever seen. But I just can't imagine anyone gaining anything useful from 30" games.


gts

+1

[Houdini]

As said above, we all agree. These games are not intended to produce valuable chess, their purpose is to help fine-tuning engine parameters.
It's part of the core business of every engine developer.

[Sedat Canbaz]

As said above, we all agree. These games are not intended to produce valuable chess, their purpose is to help fine-tuning engine parameters.
It's part of the core business of every engine developer.

Dear Robert

Now all its clear...

In other words, its a engine developer issue

I thought that the rating lists should be based on ultra fast time controls


Greetings,
Sedat