Engine testing hardware.

[konsolas]

I'm at the point where the changes I make to my chess engine's code don't always seem to improve its playing strength. The general consensus that I am aware of is that I need to test improvements made to my engine in self play.

I've been using cutechess-cli in rounds of 500 6+0.1 matches, which not only lock up my non-groundbreaking computer for a few hours, but also are low in number, meaning I receive little certainty of a reward for locking up my PC.

I'd like to set up some form of dedicated hardware to run this testing. I've heard that people are using a variety of things such as raspberry pi clusters, ODROIDS (whatever they are) and some people are apparently renting online servers.

Can anyone recommend any hardware they use to run engine tests?

[cdani]

I use some pcs with AMD-FX 8350. You pay some more for electricity but they are cheaper, I paid like 500€ for each one, and with 8 cores you can run a lot of games. I use other computers but this is my base.

[konsolas]

I use some pcs with AMD-FX 8350. You pay some more for electricity but they are cheaper, I paid like 500€ for each one, and with 8 cores you can run a lot of games. I use other computers but this is my base.

Thanks,

Roughly how many do you mean by 'some'?

[jdart]

Stacking up a bunch of quad-core Raspberry Pis or something is probably the cheapest way to go. But you'd need a lot of them for serious testing.

I have several dual-socket server systems. One is an AMD box I built myself - it is 24 cores. The others I picked up on eBay. Xeon workstations are frequently leased and when they go off-lease you can pick them up for a fraction of the original price. But they are still not exactly cheap. I think the cheapest one I bought was about $700 (2 x 4 cores). But as time goes on you get better machines as a new generation of CPU comes off lease.

Leasing cloud instances is expensive. You can get a cheap VM from Amazon for example, but it is not suitable for running something at 100% CPU 24/7. However, I have a couple of times leased short-term a really big dedicated 64-core (4x16) AMD box. That was about $500 a month. Sounds pricey but that is at least a $6000 machine to buy.


--Jon

[cdani]

Roughly how many do you mean by 'some'?

There are ppl that do a not bad engine with just for example one 4 core computer. Ok, this is a bit short but specially in the beginnings most improvements are big enough to be detectable statistically with not that much games. When you arrive to do improvements that you expect for example to be less than 10-15 elo, you start needing more computer power or more patience But you will need a lot of months to arrive here if you start from 0.

Because Andscacs is already pretty strong, y need something more meaningful. Now I use three of the ones I told, and one I7-5820K. And sometimes I rent a server to speed up development like now that is playing at TCEC.

[ymatioun]

i run 8000 games overnight on my only computer. This way it does not interfere with anything. I just back into time limits that will keep all cores busy for 8-9 hours.

[sedicla]

I just bought a W10 mini pc windows. About U$80 from amazon.
I am testing it now. If it goes well i will probably buy another 1 or 2.
I like because it comes with windows 10. I use visual studio to develop. So I just insert a pen drive and start cutechess. In my case it is very convenient.

Does anyone has any experience with W10 mini pc?

[sedicla]

Roughly how many do you mean by 'some'?

There are ppl that do a not bad engine with just for example one 4 core computer. Ok, this is a bit short but specially in the beginnings most improvements are big enough to be detectable statistically with not that much games. When you arrive to do improvements that you expect for example to be less than 10-15 elo, you start needing more computer power or more patience But you will need a lot of months to arrive here if you start from 0.

Because Andscacs is already pretty strong, y need something more meaningful. Now I use three of the ones I told, and one I7-5820K. And sometimes I rent a server to speed up development like now that is playing at TCEC.

That's true, in the beginning improvements are noticeable with fewer games. Now for my engine I am testing changes with 12k games minimum. Time control 10s+0.1. I will try to get more computer power to test with more games.

[smatovic]

...my engine is still pretty weak, so i use to test in this order:

- kaufmann test
- STS 1-15
- 3*500 games on a quad-core PC over night with tc 40 moves per minute

I guess professional testing ends up in an consideration of hardware where power consumption can be a bigger monetary issue than investment costs...

--
Srdja

[tttony]

I just bought a W10 mini pc windows. About U$80 from amazon.
I am testing it now. If it goes well i will probably buy another 1 or 2.
I like because it comes with windows 10. I use visual studio to develop. So I just insert a pen drive and start cutechess. In my case it is very convenient.

Does anyone has any experience with W10 mini pc?

This one? --> https://www.amazon.com/HNCSMILE-Compute ... B00REGG6QU

@smatovic hey, your engine is for CPU or GPU? I remember testing your engine for GPU

@konsolas you can buy a cheap intel quad-core or amd quad-core that are cheapest