fixed nodes testing

[hgm]

I don't understand how any of what you are saying has any relation to my post at all. Except that it suggests you have a misconception on what the WB nps command does, and how the GUI uses it.

[Desperado]

I don't understand how any of what you are saying has any relation to my post at all. Except that it suggests you have a misconception on what the WB nps command does, and how the GUI uses it.

Possible that i misunderstood. And indeed i really dont know
how WB use the nps command exactly.

Ok, i have read it again. And indeed i read a bit sloppily (mixed up
the units min,nodes...)

So a question is left. How are the average 1500 nodes/move finally controlled ?

I mean we both talking of averageNodes/move finally. But how
does the interrupter work ? timeBased (so i understood...) ? nodecountBased ?

if it is timebased, my last post stands. i would appreciate a short
introduction what nps-command is really doing otherwise.(thx)

Michael

[hgm]

The nps command switches the engine to node-based time control. But it otherwise does not alter the algorithm of the time control. If the engine wants to read the time, in nps mode it gets nodeCount/NPS in stead of the wall-clock time in seconds (where NPS is the parameter specified in the nps command). The engine then compares this 'virtual time' to the time left on the GUI clock as it would usually do.

The GUI decrements the clock by nodeCount/NPS in this mode, with nodeCount the number of nodes reported by the engine in its last thinking output, so that the clock also counts 'virtual seconds' rather than wall-clock time.

[Don]

Hello Don,

because i am working mainly on my tester currently, this is a _hot_ topic
for me. I ignored nodecount testing so far. But thinking more closely about
it one could build a _nearly_ 1:1 behaviour compared to time based games.

Why must it be an exact node count ?

A normal time based mechanism works basically like
"currentTime >= timeLimit" which can be translated if you want
into "currentNodeCount >= nodeCountLimit".

You ask, "why must it be an exact node count?" It does not have to be so, but we felt it was best for several reasons which I'll explain here.

1. We already have time control levels.

2. It adds complexity, because the user interface does not know about node accounting.

3. We prefer to reduce the things we have to account for, not increase them.

4. UCI defines the behavior, we don't want to make our own gratuitous changes.


Point 3 is important because I don't want to account for this additional variable - the whole modus operandi for me of fixed node testing is to remove as much variability as possible. In fact I suspect time control as the reason I will get different results (with statistical significance) at two different but very similar time controls.

Don't forget that UCI is essentially a stateless protocol. To implement a node level that acts like a time control level would require that you either break this very nice abstraction that UCI provides, or else the user interface would have to be engineered to track nodes and report them to the engine in a stateless way so that the engine is not required to track state.

Point 4 is important to me, I hate it when people decide to implement their own variations of accepted standards. The workaround if you want something like you suggest is to invent a new command, which UCI seems to permit and allows for:

Code: Select all

* if the engine or the GUI receives an unknown command or token it should just ignore it and try to
  parse the rest of the string in this line.
  Examples: "joho debug on\n" should switch the debug mode on given that joho is not defined,
            "debug joho on\n" will be undefined however.

So instead of "go nodes x" you could have "go avenodes x" where avenodes is a command that gives permission to budget the nodes. But then you have the complexity of deciding whether the interface needs to manage this for you or whether the engine now needs to start keeping it's own state.

The option to let an engine finish iterations is very important, if it likes
to do so also in time based games. This can have significantly influence on
the strength.

So maybe it only would be necessary to give engines an option of
_nodecount resolution_ to check for a stop.

I started to make this adaptive - the idea being that we have relatively course resolution but when the nodes remaining is less than the next poll interval the program could adjust the counter used for polling so that it would hit perfectly. But it turned out that I can just test it after each node (instead of when polling for user input) without a measurable speed hit.

In an more advanced nodcount testing frame, time management
arguements can _also_ be translated very easy. As example
in a fail low situation raise the nodecount limit by an amount of...
(instead the timelimit, self explaining i think).

Yes, but remember in UCI the interface sends this information and no interface that I know of knows how to pretend that nodes is time and do the accounting. Have you ever heard of a 40 million nodes in 2 minute time control? No UCI support so you must build it into your tester and also provide a way for the engine to report precisely how many nodes were consumed to the interface and also node forfeits would have be become a reality.

so, just some ideas...

Michael

edit:

(But it can also be more important first, to get a standard to count nodes...)

[Desperado]

oh, really, very interesting approach. didnt expect sth. like that.

That makes it necessary to think about some points again like:

* desired (virtual) time limit, is the first i have in mind
* ...
* ...

yeah, a lot to think now about the differences. Especially because
the "tester" must support such an "nps-option". The proposal i
had in mind, a tester like LittleBlitzer would only have to send
"infinite", and if the engine option is online, it sends the move when
it is done....(that is of course not a problem for selfmade testers,
and may be supported by a _lot_ of uci engines)

Ok, little break for me.

Michael

[hgm]

Yes, but remember in UCI the interface sends this information and no interface that I know of knows how to pretend that nodes is time and do the accounting. Have you ever heard of a 40 million nodes in 2 minute time control? No UCI support so you must build it into your tester and also provide a way for the engine to report precisely how many nodes were consumed to the interface and also node forfeits would have be become a reality.

Well, that is what you get for using an inferior protocol. WinBoard / XBoard would support all that, throught the options -firstNPS, -secondNPS.

[Don]

Yes, but remember in UCI the interface sends this information and no interface that I know of knows how to pretend that nodes is time and do the accounting. Have you ever heard of a 40 million nodes in 2 minute time control? No UCI support so you must build it into your tester and also provide a way for the engine to report precisely how many nodes were consumed to the interface and also node forfeits would have be become a reality.

Well, that is what you get for using an inferior protocol. WinBoard / XBoard would support all that, throught the options -firstNPS, -secondNPS.

Are you trying to pick a fight

[hgm]

Especially because
the "tester" must support such an "nps-option". The proposal i
had in mind, a tester like LittleBlitzer would only have to send
"infinite", and if the engine option is online, it sends the move when
it is done....(that is of course not a problem for selfmade testers,
and may be supported by a _lot_ of uci engines)

I guess it cannot be avoided to have something in the tester. But the only modification that was required in WinBoard to support this was to use engine-reported node count for decrementing the clock (suppressing the normal decrementing).

In UCI the command setoption name nps value NNNNN could be used as an analog to theWB nps command, switching the engine to nps mode. Those that value statelessness, can make it such that the engine would automatically reset this option after a search, so that the GUI would have to re-send it before every go command.

[Desperado]

Hello Don,

because i am working mainly on my tester currently, this is a _hot_ topic
for me. I ignored nodecount testing so far. But thinking more closely about
it one could build a _nearly_ 1:1 behaviour compared to time based games.

Why must it be an exact node count ?

A normal time based mechanism works basically like
"currentTime >= timeLimit" which can be translated if you want
into "currentNodeCount >= nodeCountLimit".

You ask, "why must it be an exact node count?" It does not have to be so, but we felt it was best for several reasons which I'll explain here.

1. We already have time control levels.

2. It adds complexity, because the user interface does not know about node accounting.

3. We prefer to reduce the things we have to account for, not increase them.

4. UCI defines the behavior, we don't want to make our own gratuitous changes.


Point 3 is important because I don't want to account for this additional variable - the whole modus operandi for me of fixed node testing is to remove as much variability as possible. In fact I suspect time control as the reason I will get different results (with statistical significance) at two different but very similar time controls.

Don't forget that UCI is essentially a stateless protocol. To implement a node level that acts like a time control level would require that you either break this very nice abstraction that UCI provides, or else the user interface would have to be engineered to track nodes and report them to the engine in a stateless way so that the engine is not required to track state.

Point 4 is important to me, I hate it when people decide to implement their own variations of accepted standards. The workaround if you want something like you suggest is to invent a new command, which UCI seems to permit and allows for:

Code: Select all

* if the engine or the GUI receives an unknown command or token it should just ignore it and try to
  parse the rest of the string in this line.
  Examples: "joho debug on\n" should switch the debug mode on given that joho is not defined,
            "debug joho on\n" will be undefined however.

So instead of "go nodes x" you could have "go avenodes x" where avenodes is a command that gives permission to budget the nodes. But then you have the complexity of deciding whether the interface needs to manage this for you or whether the engine now needs to start keeping it's own state.

The option to let an engine finish iterations is very important, if it likes
to do so also in time based games. This can have significantly influence on
the strength.

So maybe it only would be necessary to give engines an option of
_nodecount resolution_ to check for a stop.

I started to make this adaptive - the idea being that we have relatively course resolution but when the nodes remaining is less than the next poll interval the program could adjust the counter used for polling so that it would hit perfectly. But it turned out that I can just test it after each node (instead of when polling for user input) without a measurable speed hit.

In an more advanced nodcount testing frame, time management
arguements can _also_ be translated very easy. As example
in a fail low situation raise the nodecount limit by an amount of...
(instead the timelimit, self explaining i think).

Yes, but remember in UCI the interface sends this information and no interface that I know of knows how to pretend that nodes is time and do the accounting. Have you ever heard of a 40 million nodes in 2 minute time control? No UCI support so you must build it into your tester and also provide a way for the engine to report precisely how many nodes were consumed to the interface and also node forfeits would have be become a reality.

so, just some ideas...

Michael

edit:

(But it can also be more important first, to get a standard to count nodes...)

Just some comments, before i leave...

* uci will not be broken.
- i was talking of _options_ like hash is an option every
uci engine supports (not a new unsupported command)

- the tester (gui) only has to send the infinite command. Then
we will end up in gray area like what to do when your iteration limit
is reached already. Practically _all_ engines are sending the bestmove
in infinity mode, when search is done. So you can easily send the
bestmove at each iteration, nodecount , timebased or whatever
controlled mode when receiving the infinite command.

- so because there are no _new standards_ and you read about the
_improved_ idea i was describing in the next post, you will easily
see that there cannot be any polling problems at all. Everything
depends on the support of the engine developer how he will implement
nodecount handling (like you already asked for exact-nodecount-handling).
If the develpers who are interested in such
_options_ they will implement it. Uci protocol is kept untouched.

cheers

Michael

[hgm]

I don't think there is any need to specify a nodeCount resolution. Unlike reading the time, testing the number of nodes is extremely fast. So it should simply compare the node count to the node limits (converted from the time limits in msec by multiplying the latter by the NPS parameter divided by 1000) in every node.

I don't see the point of sending go infinite in stead of sending movestogo, wtime and btime. It would require a modification in the tester client anyway to send go infinite in a mode where it expects to get a bestmove back, so you might as well change the way the clock is decremented, and then send that time in a normalgo command.