n4k3dw4ff13s wrote: ↑Thu Feb 02, 2023 5:50 pm
Hello! I'm trying to create an opening book and I noticed something about how the program functions. It seems to load as many games from the .pgn into memory as possible (for me that's 12gb of games in memory) and it keeps 12gb of games in memory throughout the whole writing process. I wonder why this process isn't done in chunks. I noticed for an enterprise proprietary software like chessbase, they do this process in batches. I supposed they were thinking of everyday machines in mind. My .pgn file of 17.3 gb gets loaded into memory in chunks of 1.5-2.5gb at a time and only once the program writes those games to the disk (at 70-100mb/sec) does it load the next chunk.
It seems the games are skimmed in some manner at the start before this process begins allowing for this smooth clunking loading writing process.
I'm not sure about the speed differences in one loading method over the other, BUT I will say the chessbase opening book creation seems to have had user interaction in mind. I get the sense of progress when my fans turn on during the loading to memory phase and the disk writing phase and they quiet down as the next chunk is accessed and then back on again when the process repeats.
It's kinda not satisfying that everything is quiet as I wait for the final 38% percent of the progress bar to finish up during the disk writing phase.
I'd want to hear about why the process isn't done in chunks (not a critique but rather a curiosity) and if the structure of the .obs opening book is just better to be written to in this manner.
Actually, BSG processes PGN in chunks. It reads PGN block by block and then parses game by game. Later multi threads are applied to process games and build the opening tree. Processed games are dumped immediately to save space. I believe that makes BSG one of the fastest programs to build opening books.
(If you or someone needs/wants to look at that code, just look into OCGDB open source since BSG uses that code too for creating opening books).
Perhaps that is the problem with updating the progress bar. I will check it. Thanks.
Maybe I need to restart my computer and get a fresh install of Banksia. I tried to make a oba.db3 database and it processed my meeger 1.0gb file at 215 games a second. I have a cofeelake core 12 thread Intel processor, and 16 gb of ram on windows 10. Is that speed normal?
Also, I wasn't able to make a opening book for my 18gb file, so I broke it up into 18 1 gb files. If I make databases for all 18 .pgns, how do. I combine the databases?
You were right! I was not including all plies for every game in the creating of the .ctg files(chessbase opening books.) When I did so, the speed is probably similar to what we get in banksia of 215 games a second. However, I still do think the progress bar is bugged. Going back to the visual appeal of things, maybe it'd be better if rather than reporting games/second you include total games written to file or include both.
Thank you Pham!
Just installed BanksiaGui 0.58 Rc1 on Mac mini M1 and Mac mini 2010 (!) and they both work fine.
Started with them 2 tournaments between top chess engines to verify that all is ok. Now I'll try also online games feature.
Hello Pham,
did you deactivate my account in banksia forum?
(I cannot answer)
Someone did it
I'm moderator in all threads, my account has only been deleted in the Engines thread. What shoud that? Was that you or was it AlexChess?
My entire thread on ProteusSF has been deleted. Before the entire thread was deleted, one of my postings was manipulated yesterday with incorrect content that I had not written.
I'm just trying BanksiaGui Lichess client 0.58 Rc1 and it works fine even on Mac mini 2010 300 kN/s!!! On Mac M1 I can finally use all 8 CPUs 8 MN/s doubling ProteusSF speed compared to Playchess.com and Infinitychess.com availables only for Windows! Now I must invite all friends with top engines to use this BanksiaGui feature, since the BOTs online there aren't known and of average strengh.
AlexChess wrote: ↑Mon Mar 06, 2023 11:44 am
Hi Pham!
I'm just trying BanksiaGui Lichess client 0.58 Rc1 and it works fine even on Mac mini 2010 300 kN/s!!! On Mac M1 I can finally use all 8 CPUs 8 MN/s doubling ProteusSF speed compared to Playchess.com and Infinitychess.com availables only for Windows! Now I must invite all friends with top engines to use this BanksiaGui feature, since the BOTs online there aren't known and of average strengh.
did you deactivate my account in banksia forum? ...and deleted my engine thread?
What is the upgrade process for Banksia on Windows ? If I downloaded Version 0.58 (RC1), how would I move to the release version and retain all settings etc?
Mac mini M1 8GB-256GB, Windows 11 & Ubuntu ARM64.
