Time-Odds tournament

[hgm]

I am currently trying out Uri's idea of including handicapped versions of strong engines in a tournament, in order to get a more reliable determanation of the rating of the weak engines. In stead of using limited depth (as Uri suggested) I am using time odds to reduce the engine strength. I consider the problems introduced by limiting depth, namely nearly instantaneous but completely idiotic play in the end-game, too big a disadvantage of fixed-depth testing. It would have been better to use fixed number of nodes, but most WB engines don't support that. So using time odds seems the best option, especially since I am playing all games on the same hardware, so that nodes and time are strictly proportional anyway (for a given engine).

So what I finally do is play a tournament at a nominal time control of 36 min (sudden death). Some participants are handicapped by a factor 3, 9, 24 or 54, so they will get only 12:00, 4:00, 1:30 or 0:40 when they play. (And their opponent gets whatever his handicap prescribes.) The strongest two engines thus play in 5 versions, the normal one plus 4 handicapped copies. The slightly weaker engines only play with 4 or 3 incarnations (leaving out the largest handicaps), while the weak ones only participate with the version that plays the nominal time.

For more details, see http://home.hccnet.nl/h.g.muller/battlehome.html

[Uri Blass]

I am currently trying out Uri's idea of including handicapped versions of strong engines in a tournament, in order to get a more reliable determanation of the rating of the weak engines. In stead of using limited depth (as Uri suggested) I am using time odds to reduce the engine strength. I consider the problems introduced by limiting depth, namely nearly instantaneous but completely idiotic play in the end-game, too big a disadvantage of fixed-depth testing. It would have been better to use fixed number of nodes, but most WB engines don't support that. So using time odds seems the best option, especially since I am playing all games on the same hardware, so that nodes and time are strictly proportional anyway (for a given engine).

So what I finally do is play a tournament at a nominal time control of 36 min (sudden death). Some participants are handicapped by a factor 3, 9, 24 or 54, so they will get only 12:00, 4:00, 1:30 or 0:40 when they play. (And their opponent gets whatever his handicap prescribes.) The strongest two engines thus play in 5 versions, the normal one plus 4 handicapped copies. The slightly weaker engines only play with 4 or 3 incarnations (leaving out the largest handicaps), while the weak ones only participate with the version that plays the nominal time.

For more details, see http://home.hccnet.nl/h.g.muller/battlehome.html

Some comments:
1)The fact that Joker is leading suggest that the engines that are tested are weak because Joker is not strong in normal chess and I guess that you do not use special tricks in 10*8 that you do not use in normal chess.


2)Note that I played time odd match with movei
The problem with this idea is that it is not good for the very weak level because engines can lose on time.

Uri

[GenoM]

Some comments:
1)The fact that Joker is leading suggest that the engines that are tested are weak because Joker is not strong in normal chess and I guess that you do not use special tricks in 10*8 that you do not use in normal chess.<...>
Uri

Uri,

this comment has to be written in the golden book of CCC as the example of the brilliant logic thinking.

Regards,
Geno

[hgm]

Definitely 10x8 Chess laggs a lot behind normal Chess in terms of engine development. Joker80 is indeed identical to Joker in search algorithm; its eval only differs in that the centralization points (a sort of piece-square table shared by all light pieces) now come from a 10x8 table in stead of an 8x8 table, with two central files inserted with slightly larger bonuses. And the Piece values are slightly tuned (lower R and N value), while of course new piece values for Archbishop and Chancellor had to be intoduced. Apart from the latter, I think the changes are so minor that one can safely say that Joker80 in 10x8 Chess is about as strong as normal Joker in 8x8 Chess, so that there is room for a ~600 Elo improvement.

I think many of the other engines are weaker than Joker because they use wrong piece values. Unlike in normal Chess, there is no consensus about the piece value of A and C. Even the values Jokr80 is using now are tentative; I am stil working on more accurate value determination.

I agree that it is probably not possible to weaken one strong engine so much that it becomes as weak as the weakest unhandicapped engine. But I don't see that as a major problem. Not only the very strongest engines are comparatively free of major bug. For instance, TSCP Gothic is some 325 Elo points weaker than Joker80, but is a very stable and reliable engine. It is only weak because its evaluation completely sucks. So if Joker80 at 40 sec/game is not weak enough to meaningfully test BigLion80 at 36 min/game, then TSCP-G at 40 sec/game might be.

The main purpose of time handicapping in this case is to provide a larger variety of opponents at any point of the rating scale. This is sorely needed, as there are only very few 10x8 WB engines, and they are spread out over an enormous rating range. TSCP-G at 1:30 beat BigLion at 36:00 by 2-0!

This tournament is just a preliminary test, to get an impression how large the impact of time odds is, and what handicap factors are needed and can be achieved in practice.

[smrf]

Thank you for this new interesting event! I hope, that the handicapped engines will got communicated not only their reduced time frames, but also the opponents time frames reduced to a same way divided amount, even when those engines will use a multiple of that.

[hgm]

If I understand you correctly, this is indeed exactly what I do. Each engine i={1,2} can have its own time handicap factor F_i. Then I divide each time allotment on the WinBoard clock (base time or increment) for that engine by F_i each time the engine receives extra time, so that the audience is aware of the fact that the engine plays with shorter time. So I can send the engine the time that is truly on the clock for its own time. But for the opponent's time, I first multiply the time on its clock by F_opponent to translate it back to unhandicapped time units, and then divide it by F_i before it is sent to engine i as its opponent's time.

So it is really like the engines are living in different universes, where time runs at a different speed. Only if they are pondering they would notice that they are not simply running on an F_i times slower computer.

[smrf]

Dear Uri, 10x8 chess could not be compared that easy to traditional chess. 10x8 engines e.g. have to handle a 25% higher branching factor.

But if you are convinced of existing 10x8 engines being weak, it would be a better argument to enter the 10x8 scene by placing another engine written by yourself. It would be really interesting!

[Ovyron]

But if you are convinced of existing 10x8 engines being weak, it would be a better argument to enter the 10x8 scene by placing another engine written by yourself. It would be really interesting!

How hard is it to convert a 8x8 engine to play 10x8?

[Tony Thomas]

But if you are convinced of existing 10x8 engines being weak, it would be a better argument to enter the 10x8 scene by placing another engine written by yourself. It would be really interesting!

How hard is it to convert a 8x8 engine to play 10x8?

It cant be any harder than implementing MP support...

[hgm]

How hard is it to convert a 8x8 engine to play 10x8?

It depends on how the engine represents the board. For mailbox it is comparatively trivial: just change the size of the arrays, and the square numbering. For Joker it took me a week to do the conversion. This is comparatively long, as Joker used the 0x88 'holes' in sthe square numbers for other purposes in the hash move, so I had to use all kind of tricks to design a new move encoding that did not require extra space in my move lists and in the hash table.

And I had to add under-promotions; In 10x8 Chess promotions to A and C are too important to leave them out. (OTOH, promotions to R, B, N are even more unimportant, as the A & C alternatives exist. So Joker80 now only does promotions to Q, C and A in its search.)

For bitboards it would be much harder, as most bitboard algorithms derive their efficiency from the board representation residing in an atomic data-type. So you can easily do the conversion by defining your own 80-bit integer class, but what you get is likely to be incomparibly slow to the corresponding 8x8 engine.