Odd Gunnar Malin wrote: ↑Sun Jan 09, 2022 12:10 pm
I didn't see how to get the database itself to filter out to only show the game at this point in the tree but I'm sure there must be.
Well, it was just to click on the correct icon. I have still to find a method to look at this game without losing the game I'm analysing.
I shouldn't use 4k screen with my eye sight. All previuse views are in a tab between the toolbar and the board/game
KLc wrote: ↑Sat Jan 08, 2022 9:03 pm
Ok, I downloaded the latest Lichess database (Nov 2021). It's a 24GB bzipped file that extracts to a 187GB PGN database containing ~88 million games. I tried opening this in HCE and it took several minutes for the first segment of the progress bar to appear. I didn't have the patience but I'm sure (because of what comes next) that it'll work and I guess it would take several hours to open. Since I'm not interested in bullet and blitz games anyways, I used pgn-extract to extract the rapid games, and along the fly I removed all comments (with evaluations and clock values) and variations.
Many thanks for the information.
It would also be interesting to know if comments and variations affect the performance (are they always stored in RAM too?).
If that's not too much to ask, could you try to extract the same 11 milion games, unchanged?
carldaman wrote: ↑Sun Jan 09, 2022 9:23 am
They claim HCE also works with Chessbase database formats such as cbh and ctg trees. Has anyone tried this?
Also, can anyone post a nice win by Hiarcs 15 against any decent engine (even 2600 Elo or higher)?
I'd like to see a demonstration of how it plays the game.
This thread has been rather low on real content so far.
Ozymandias wrote: ↑Sun Jan 09, 2022 10:03 am
You're completely out of line, and I hope mods will take notice. Those are questions posted on a specific thread about a specific product, opened by someone working there. If they don't consider this to be an effective way of communication, let them say it and proclaim this to be merely an announcement thread. (This doesn't seem to be the case).
No rant, no wildness, no randomness, no disrespect, (can't say the same for you).
+1
At this point, I would also ask if he is associated with the Hiarcs team in any way.
His posts read like commercial blurbs.
Whatever the case, as someone already pointed out, information has been sparse. I replied the very first post with a question about how it compared with Scid, but I guess the answer mustn't be favorable to HCE. For example, this is what I've been able to piece together:
I downloaded the latest Lichess database (Nov 2021). It's a 24GB bzipped file that extracts to a 187GB PGN database containing ~88 million games. I tried opening this in HCE and it took several minutes for the first segment of the progress bar to appear. I didn't have the patience but I'm sure (because of what comes next) that it'll work and I guess it would take several hours to open.
On the other hand, the experimental Scid5 is reported to be faster, although the developers don't think it's ready for release yet:
The downloaded file (lichess_db_standard_rated_2021-11.pgn.bz2) is 22.1GB and the decompressed PGN is 187GB.
It contains over 87 million games.
Converting the database to the new rocksdb format took 34 minutes.
So I have taken the December 2021 Lichess file which decompresses to about 221GB of PGN.
HCE running on a 16GB RAM Macbook Air M1 is currently converting this and Activity Monitor reports the read speed to be a fairly steady 12MB/second, so this is going to take about 5 hours.
I will report back on the size and performance in due course.
EDIT: read speed drops dramatically if the app is put into the background, you have to leave it as the foreground app in this process.
The strength of the pair of bishops and the bishop in the end zone seems to be a bit underestimated by that version. Perhaps having the ability to tweak this a bit in the valuation settings, the knowledge network I assume does not.
[pgn][Event "Test H15"]
[Site "?"]
[Date "2022.01.10"]
[Round "2"]
[White "rofChade 2.3 BMI"]
[Black "Deep HIARCS 15.0"]
[Result "1-0"]
[ECO "B18"]
[Annotator "0.07;0.00"]
[PlyCount "229"]
[TimeControl "900+10"]
George Sobala wrote: ↑Mon Jan 10, 2022 12:51 pm
So I have taken the December 2021 Lichess file which decompresses to about 221GB of PGN.
HCE running on a 16GB RAM Macbook Air M1 is currently converting this and Activity Monitor reports the read speed to be a fairly steady 12MB/second, so this is going to take about 5 hours.
I will report back on the size and performance in due course.
EDIT: read speed drops dramatically if the app is put into the background, you have to leave it as the foreground app in this process.
Question: Beyond perhaps just 'testing the limits' of HCE (and everything will have a limit)....what earthly good does it do to deal with all that junk: 221 GB of PGN...87 Million games, etc? It's largely rubbish games. It's like...accounting for all the grains of sand on the beach for...'some' reason.
George Sobala wrote: ↑Mon Jan 10, 2022 12:51 pm
So I have taken the December 2021 Lichess file which decompresses to about 221GB of PGN.
HCE running on a 16GB RAM Macbook Air M1 is currently converting this and Activity Monitor reports the read speed to be a fairly steady 12MB/second, so this is going to take about 5 hours.
I will report back on the size and performance in due course.
EDIT: read speed drops dramatically if the app is put into the background, you have to leave it as the foreground app in this process.
Question: Beyond perhaps just 'testing the limits' of HCE (and everything will have a limit)....what earthly good does it do to deal with all that junk: 221 GB of PGN...87 Million games, etc? It's largely rubbish games. It's like...accounting for all the grains of sand on the beach for...'some' reason.
George Sobala wrote: ↑Mon Jan 10, 2022 12:51 pm
So I have taken the December 2021 Lichess file which decompresses to about 221GB of PGN.
HCE running on a 16GB RAM Macbook Air M1 is currently converting this and Activity Monitor reports the read speed to be a fairly steady 12MB/second, so this is going to take about 5 hours.
I will report back on the size and performance in due course.
EDIT: read speed drops dramatically if the app is put into the background, you have to leave it as the foreground app in this process.
Question: Beyond perhaps just 'testing the limits' of HCE (and everything will have a limit)....what earthly good does it do to deal with all that junk: 221 GB of PGN...87 Million games, etc? It's largely rubbish games. It's like...accounting for all the grains of sand on the beach for...'some' reason.
To me, none whatsoever. But someone asked, so I thought I would find out ....
The limits were reached. After a couple of hours or so, the conversion process had clearly exhausted available RAM and swapping started. At the 3.5 hour mark, the blue line was still less than halfway, and read / write speeds were around the 400MB/sec mark - clearly swap related. I gave up the experiment.
EDIT: I estimate from my log files that in the swap phase, the output file was being written at about 1.7MB/sec, so perhaps 15-20 hours more of this would have completed the task.
