Many authors are using OpenBench (https://github.com/AndyGrant/OpenBench), either in collaboration with others, or on their own, sometimes in public sometimes in private. I've made significant changes to improve OpenBench over the past few months, and I'm accouncing it here in the hopes that everyone using the tool sees it. I'm hoping to get everyone using the tool to update, so that I don't have to provide any technical support for older versions. Also the changes are cool.
Some but not all of the changes, in no particular order:
* Improved the way that engines are configured, so you can now drag-and-drop engines between frameworks in json form. Same for opening books.
* OpenBench now works with closed-source engines, as long as the engine is still on github even though marked as private. This is done by using github personal-access tokens, which have only read rights to workflow outputs, meaning that no code spilling can occur. Github CI workers build binaries on branch pushes, which are then used by workers. There is a wiki entry explaining the process in more detail.
* OpenBench now lets you run tests between different engines. I can now queue up a test between Ethereal and Berserk if I liked. This is pretty nifty for comparing engines, especially at different time controls, or with different thread counts. Workers will scale each engine individually to match your reference.
* OpenBench now supports fixed nodes, fixed movetime, and fixed depth testing. As well as having always supported cyclical time, and fischer time, and sudden death. You can also use different time controls for the dev and base engines, meaning you can test time odds between engines, or internally to one engine. Pondering support still does not exist.
* OpenBench has always supprted FRC and DFRC, but I like to remind people of that so here it is.
* Workers on OpenBench now report even more information to the server, including CPU brand, python version, worker version, lists of CPU flags, lists of compilers, and lists of private tokens held. This is pretty helpful for managing things.
* The entire CSS for the website has been rewritten. The new version is about 1000x better than the current, and also about 10x better than the Luecx rewrite that some people are using. Luecx and I wrote this together, painfully, while streaming to each other on Discord for hours at a time.
* Improved the algorithm that divides up workers into workloads, to more evenly distribute the workers, and also save time by repeating the same workload if possible.
* Updated Django to the latest version, 4.2.1, and made use of some newer python features.
* Add copy buttons to get test results in one click instead of having to do it manually.
* Allow spawning multiple cutechess copies at once, to handle multi-socket machines, to avoid cutechess' high concurrency issues.
* Add options to make the framework private unless logged in and enabled
* Add options to disable manual registration (prevent bots from bothering you)
* Allow custom, per-test adjudication settings. Or disable them completely with "None".
* Allow cleanly killing the client, mid task, by creating openbench.exit in the Client directory
* Allow users to have multiple repos saved for engines, so that you can seemlessly work on multiple engines at a time.
The changes are not yet live at chess.grantnet.us, but they will be at some point.
Anyone migrating their database will run into some trouble, so I've created a thread about it in the openbench discord, that goes through step by step the process. Feel free to come join if you need anything ( https://discord.gg/9MVg7fBTpM )
And also feel free to open issues on the repository if you encounter anything, or if there are features you are looking to have added. There is a short to medium term pipeline in the works, including: Support for tuning with TBPSA, support for data generation for use in NNUE, support for uploading PGNs back to the server in some format (and tools to decompress them). And probably some more.
Release of a greatly updated OpenBench
Moderator: Ras
-
AndrewGrant
- Posts: 1960
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant