pulsar and analysis mode

[adams161]

After watching pulsar get clobbered in 4 games in a row by the opponent exploiting pulsars lazy king safety in blitz, i took a routine i wrote for crazyhouse and turned it on for regular chess. I thought the idea would work, counting the pieces that have proximity to pulsars king, but i didnt want to just use the same multiplier ( its some number calculated by number of pieces times a multiplier) as crazyhouse.

I plugged the games into pulsar in analysis mode and found a number that would work. On a variety of what i termed critical moves, were it needed to perceive the threat, it saw the threat and consistently played a better move.

First in one particular game, it almost walked into a mate, i wanted to know why it did it, was hit hashing, was it null move or something else. with analysis mode i could disable things like hashing or null move and see what changes teh evaluate on that move. I determined it was null move that caused the problem. A check estension fixed it. But with just a check extension and without my king safety it sill found it to late. I guess the moves leading up to that line were to quiet with both a later in search check extension, and the new king safety it consistently found the right move early. It also found a better move on the previous move to help not even create that situation. that move was found as early as a ply 4 search or ply 3.

Now that i've got analysis mode i'm enthralled with how simple it makes it to debug games. I can modify the engine a dozen times, every 3 minutes trying changes or undoing them, and load the game in to analysis mode and consistently see if it find a better move and how soon it finds it.

I watched pulsar play a game this morning and they guy left his king side vulnerble, after he tried attackign pulsar on that very king side aned pulsar now had the sense to castle queen side, and pulsar would later rip into his king side and get checkmate. Again i'm going to have to watch that the new evaluate scores are even handed enough. there is always a danger it will be overly agressive in king attacks and lose endgames do to a faulty sense of risk and speculation. i think my values are ok. Its playing on icc but as it goes for computer accounts its slow to get games. maybe one or two an hour. but i'm commited to leaving it on for a long time and hope to get a good testing result.

ps, hg muller it played an excellent king pawn endgame last night and i think your hashing suggestion to rewrite my replace hash value on table code is paying off in endgames. staltemate issue next.

Mike

[vladstamate]

Its playing on icc but as it goes for computer accounts its slow to get games. maybe one or two an hour.

My engine, Plisk, plays on FICS and I get about 10 games an hour. This is on a registered computer account.

Vlad.

[adams161]

there is probably a lot of variables involved. time of day its playing, rating range it will play, time control. pulsar is basicly using assesswin > 0, by not playing people more than say 600 points lower rated. I have logged it onto fics and its also been slow to get games. Another big factor is presence. if you leave it on day after day it gains a following. pulsar hasnt been playing blitz in some time. like any other business you get regulars if your consistent. thats why i'm trying to just leave it on now for say 72 hours if my connectoin holds. or relog it in asap if it doesnt. and number of games is picking up, its been on 7 hours. last hour its had maybe 4 games.

Mike

[sje]

Is your program now accepting rated match requests on ICC? That might help it get more games.

[adams161]

i prefer not to play computers unless i'm present and they are author written. its rating may be a little higher playing humans by seeks but at least its consistent so i can tell if its going up or down when i make changes. when i was programming crazyhouse i had people asking me to play this time control or that time control but 99% of the games were 3 0 crazyhouse seeks and i knew it was consistent from month to month.

Mike

[brianr]

but at least its consistent so i can tell if its going up or down when i make changes

You should know that this approach (ICC games) to determine if changes are good or not is seriously flawed.
I suggest searching for some of the testing threads here to get a appreciation for the complex, exacting, and statistical issues involved.

[bob]

but at least its consistent so i can tell if its going up or down when i make changes

You should know that this approach (ICC games) to determine if changes are good or not is seriously flawed.
I suggest searching for some of the testing threads here to get a appreciation for the complex, exacting, and statistical issues involved.

I've been playing on ICC since 1995. It is a great way to do several things:

(1) test robustness of your code. This lets you have confidence that your program doesn't have serious bugs that will cause it to crash in a tournament. Just watch for time forfeits which indicate serious problems usually.

(2) test against a variety of opponents. If you go over lost games, you can develop a list of weaknesses that need work.

You can _not_ pay attention to your rating. It is way too volatile, and you can never tell what/who you are playing against. Weak humans with low ratings still use computer programs when playing other programs. Many programs list false hardware. Some change hardware frequently, going from fast to slow to fast again, making their strength vary significantly...

You need to test under controlled circumstances to determine if an improvement is good or bad... Not in an extremely noisy and volatile environment like ICC...

[adams161]

i'm sure my testing can be better, I do a lot of engine vs engine matches as well. I still think icc testing has value for two reasons, in addition to what you listed, because yes i agree icc is good for engine stablity testing as you listed, but i also think its good for code testing. For one if i screw up the code, the rating always goes down. I may lie to myself and say its having a bad day, or its that opening, or it has trouble in that endgame, but invariably if i've screwed the code up the rating starts dropping.

the other reason i trust icc games is i'm fairly aware of who i'm playing and what kinds of games i'm playing. I"m only marginally interested in people who play 1 or 2 games. Also i have less interest in lower rated players and if pulsars getting a stream of low rated players and adding 1 2 or 3 points i am aware of that. Players around pulsar's rating who play multiple games are my number one group of interest. When i was programmign crazyhouse you were really programming to beat the guys on the best list or the guys who were under the best list but close to being on. Many of them would play pulsar and many of them would play 10 or 20 games in a session. You dont see as much of that in blitz, more tourists, people in for one or two games, but i guess all i'm saying is i'm aware of the nature of the games its getting. Also i look at the range it plays in not the actual number.

Playing computers when im not around concerns me for several reasons. I dont trust computer ratings in particular. I let pulsar play unrated in its formula against anyone and some computers play. I finger some of these guys and they are lower rated than pulsar and running programs like shredder and have oddly enough, a 1-minute , 15-minute and 5-minute ratings which only humans can play so this account was not allways a C. Just what i need for me to leave it unatended for 12 hours and some guy who got busted for computer cheating and now has rybka running and a 2300 blitz rating and he soaks up 300 points from pulsars rating. I"m not uptight about holding pulsars rating up but it means for the immediate time the type of games pulsar is going to get after that etc are not going to be the same and frankly its a setback. Even tinker which is a good program, i played it two bullet games the other day, it won both, but pulsar was in the 2400s in bullet and tinker in teh 2100s, so pulsar lost like a bunch of points. Me i didnt really care so much, it was just two games, but lets face it, tinker is not a 2100 bullet program i dont think. someone eeked it.

So frankly at this point i'll leave its formula unrated, when i'm not around anyone can run any kind of game test I suppose, and a number of computers have played it. but i'm just going to play rated under my 2 controlled seeks right now.

And plenty of engine vs engine matches against different versions.

Mike