Fantacising about a testing cluster!

[Michael Sherwin]

Hi Denis,

Fasinating stuff, however, it is either more than I need or does not do what I want and I am not sure which.

This is what I am asking for:

Ten boxes without monitors or keyboards in one room connected to a main computer in another room so that if I click on a boxs icon that is in the other room it can be controled from the main computer just enough so that the box can be turned on or off and a beta can be transfered to that box and a gauntlet started or stoped. No parallel computing needed!

Mike

[mathmoi]

I like to test at 60/60 time controls. 60 minutes on the clock, 60 seconds per move added. That turns into 4-6 hours per game. or 6-8 games per day. 30 days isn't enough games to determine if version X is better than version X-1 at that speed. It takes more than a hundred games or two with two opponents that are fairly close in rating.

Hi,

At a rate 6-8 games per day you'll get to 210 games in 30 days, so 30 days should be enough to determine wich oponent is better.

[bob]

I like to test at 60/60 time controls. 60 minutes on the clock, 60 seconds per move added. That turns into 4-6 hours per game. or 6-8 games per day. 30 days isn't enough games to determine if version X is better than version X-1 at that speed. It takes more than a hundred games or two with two opponents that are fairly close in rating.

Hi,

At a rate 6-8 games per day you'll get to 210 games in 30 days, so 30 days should be enough to determine wich oponent is better.

If the two versions are close, 200 games is nowhere near enough to say which version is better. I'm working on something for the JICGA right now that shows some interesting results based on this very topic. I can produce some 200+ game matches where version A wins one, version B wins the other. Both versions playing 80 games against each of 3 opponents, using 40 starting positions, each played once with each color. 240 games total. Not always enough. In fact, rarely ever enough unless something is broken to make one version clearly worse.

[mathmoi]

I like to test at 60/60 time controls. 60 minutes on the clock, 60 seconds per move added. That turns into 4-6 hours per game. or 6-8 games per day. 30 days isn't enough games to determine if version X is better than version X-1 at that speed. It takes more than a hundred games or two with two opponents that are fairly close in rating.

Hi,

At a rate 6-8 games per day you'll get to 210 games in 30 days, so 30 days should be enough to determine wich oponent is better.

If the two versions are close, 200 games is nowhere near enough to say which version is better. I'm working on something for the JICGA right now that shows some interesting results based on this very topic. I can produce some 200+ game matches where version A wins one, version B wins the other. Both versions playing 80 games against each of 3 opponents, using 40 starting positions, each played once with each color. 240 games total. Not always enough. In fact, rarely ever enough unless something is broken to make one version clearly worse.

Hi,

From your previous post I understood that you tough something between 100 and 200 games was enough. I probably misunderstood what you meant.

Out of curiosity, how many games (and wich kind of results) do you think are needed to assert that version N+1 is an improvment over version N?

[krazyken]

You can get cheap windows 2000 boxes, PIII 1Ghz, for $80.

[Michael Sherwin]

You can get cheap windows 2000 boxes, PIII 1Ghz, for $80.

I don't remember any PIII boxes with 256 MB ram each. Can they hold that much? But, I guess that you are just making a point about price! Also older opreating systems (and maybe newer ones too) would not work with out a keyboard or monitor. I could buy all the pieces that I think would be required, however, I doubt that I would have a working system when it was put together. How would I find out how to make it all work and exactly what I would need to buy. A book maybe? I doubt if Dell and Gateway would work with or care about antiquated equipment that they no longer deal with.

[sje]

Some older systems need a keyboard to boot, but they may not need a keyboard to run once booted. Sometimes the BIOS can be configured to skip the boot time keyboard scan.

There are keyboard plug stubs available that mimic a keyboard. Also, older keyboards (PS/2 style) are quite cheap.

Most systems will work without a display.

The choice of operating system can affect the above. I recommend Linux to save time and headache.

[krazyken]

You can get cheap windows 2000 boxes, PIII 1Ghz, for $80.

I don't remember any PIII boxes with 256 MB ram each. Can they hold that much?

Sure they can hold that much they usually use PC133 RAM and can usually hold 512-1024MB

But, I guess that you are just making a point about price! Also older opreating systems (and maybe newer ones too) would not work with out a keyboard or monitor.

Not entirely true. You could set up the boxes with a KVM so you can switch one Keyboard/Monitor/Mouse between all the boxes. Probably will cost about $250 extra. If you have USB Keyboard and mouse (PS2 will work as well, but not officially), you just need to have the video plugged in at boot time, and can hot swap the cables if needed. Tweaking settings in BIOS may be needed. Also possible you can install VNC on all the machines and control them via network.

I could buy all the pieces that I think would be required, however, I doubt that I would have a working system when it was put together. How would I find out how to make it all work and exactly what I would need to buy. A book maybe? I doubt if Dell and Gateway would work with or care about antiquated equipment that they no longer deal with.

If you are not versed in computer hardware and are not comfortable in doing things like popping it open to install RAM, your best avenue of support is the "buy the local hardware guru a pizza" trick. Although there are oodles of books for upgrading and maintaining hardware. Just look around for used computers, there are bunches of them out there, which frequently need nothing more than a RAM upgrade to get them to do what you want.

[Tony]

Some older systems need a keyboard to boot, but they may not need a keyboard to run once booted. ....

Yeah, I remember that one. "Keyboard not present. Press F1 to continue"



Tony

[bob]

I like to test at 60/60 time controls. 60 minutes on the clock, 60 seconds per move added. That turns into 4-6 hours per game. or 6-8 games per day. 30 days isn't enough games to determine if version X is better than version X-1 at that speed. It takes more than a hundred games or two with two opponents that are fairly close in rating.

Hi,

At a rate 6-8 games per day you'll get to 210 games in 30 days, so 30 days should be enough to determine wich oponent is better.

If the two versions are close, 200 games is nowhere near enough to say which version is better. I'm working on something for the JICGA right now that shows some interesting results based on this very topic. I can produce some 200+ game matches where version A wins one, version B wins the other. Both versions playing 80 games against each of 3 opponents, using 40 starting positions, each played once with each color. 240 games total. Not always enough. In fact, rarely ever enough unless something is broken to make one version clearly worse.

Hi,

From your previous post I understood that you tough something between 100 and 200 games was enough. I probably misunderstood what you meant.

Out of curiosity, how many games (and wich kind of results) do you think are needed to assert that version N+1 is an improvment over version N?

Give me a while to find that answer. I am working on it right now, in fact. What I am doing is playing 2-game matches using 40 starting positions for 80 games per match. I am playing 80 such matches for each opponent. Then 40 matches but with 4 games per opponent per position per match. Then 20 matches with 8 games per match, etc.

My intent is to determine how many games it takes to produce a stable result. I can certainly tell you that 80 games and 160 games are nowhere near enough. But an exact answer will take a while.

80 matchs X 2 games X 40 positions X 3 opponents = 19, 200 games.

Fortunately I can play 256 at a time which takes about as long as it would to play 75 games one at a time, which is not too bad. Obviously my hope is that I won't need 19200 games to decide "good" or "bad". But I absolutely need more than 80-160. So I am interested in finding a statistically valid minimum number of games needed...

More once I have got the data...