cutechess-cli in python

[lucasart]

I ran some tests, to measure the overhead of zinc compared to cutechess-cli.

First, I define a benchmark that is completely fair, magnifies the measure, and is depolluted from system noise as much as possible:

• SF vs. SF, Hash=16, Threads=1, 5ply EPD openings
• same adjudication rules (Draw: movenumber=40, movecount=8, score=20; Resign: movecount=3, score=500).
• 1000 games, concurrency 7
• no I/O to magnify the sensitivity of the measure: redirect stdout to /dev/null, and do not write a PGN

Now, I play games at fixed depth, increasing the depth:

• depth=1: cutechess-cli: 9.41", zinc: 54.9". overhead: 45.5"
• depth=2: cutechess-cli: 11.6", zinc: 62.3". overhead: 50.7"
• depth=3: cutechess-cli: 12.6", zinc: 60.8". overhead: 48.2"
• depth=4: cutechess-cli: 15.5", zinc: 63.8". overhead: 48.3"
• depth=5: cutechess-cli: 20.8", zinc: 69.2". overhead: 48.4"
• depth=6: cutechess-cli: 31.1", zinc: 79.5". overhead: 48.4"
• depth=7: cutechess-cli: 51.7", zinc: 101". overhead: 49.3"
• depth=8: cutechess-cli: 89", zinc: 141". overhead: 52"
• depth=9: cutechess-cli: 163", zinc: 214". overhead: 51"

So the average overhead is 5ms/game. To put that into perspective, we use STC: 10"+0.1" and LTC: 60"+0.6" in SF, making the overhead only 0.016% for STC, and 0.0026% for LTC.

I'm sure I could work on optimizing this, and find plenty of low handing watermelons. Eventually, I would reach a min overhead imposed by the choice of Python vs. C++. But 5ms per game is totally negligible in practice.

[lucasart]

Correction: 50ms/game.

Still negligible in SF testing conditions, but a bit more than I would like.

Let's hunt down these watermelons

[Michel]

I recommend you learn a bit more about Python before crapping on it.

Still unable to use civilized language?

CPython uses garbage collection by default https://docs.python.org/2/library/gc.html

You can disable it if you are sure that there are no cyclical structures in your code or in the packages you import. That's a risky assumption. There is also a possibility to invoke the garbage collector on demand.

[mar]

I recommend you learn a bit more about Python before crapping on it.

Python uses hybrid ref counting + GC to detect cycles.
Any JITter (pypy) will beat any interpreter (python). Try to worst-case bubble-sort 8k elements and see for yourself which one is faster.

[ilari]

Nice progress!

I can see the value in this project - the Stockfish testing framework already depends on Python, so this script would be easier to deploy (especially for multiple platforms) than cutechess-cli + QtCore. And if the testing framework needs some changes in the testing tool, they can be made quickly and easily.

[AndrewGrant]

Exactly. I am sure many of us have python installed, but not so many with QT.

[lucasart]

Added Chess960 and PolyGlot opening book (weighted random selection up to book depth).

Added Tournaments: only Gauntlet for now, RR coming soon.
Added PGN output.

Updated gap-analysis compared to cutechess-cli:

• CLI: won't implement. just change the parmaters and run the script with no arguments.
• JSON: won't implement (pointless w/o a CLI).
• Variants: Chess and Chess960 only. Won't implement other variants.
• Gaviota TB: won't implement. may implement TB adjudication later, but using Syzygy instead.
• Tournaments: Gauntlet implemented. RR will follow soon.
• SPRT: not implemented. coming soon.
• openings: EPD and PolyGlot only. Won't support PGN openings (useless/misguided feature IMO).
• recover: not implemented.
• repeat: the only behaviour available. I don't see any valid use case for repeat=false.
• restart: won't implement. Useless IMO.
• proto: only UCI. won't support CECP.
• timemargin: not implemented.
• ponder: not implemented.

As usual, code is on my github page: github.com/lucasart/zinc

[lucasart]

Added Round Robin tournaments and Syzygy TB adjudication.

[nionita]

Hi Lucas,

I wanted to use your zinc to do some parameter optimisations, but I saw it depends on a python package called chess.

Which package is this? On github I found ~1000 repos only written in python...

[ZirconiumX]

This one, Nicu.