Database snapshot

[Ovyron]

But let me tell you I once used IDeA using depth 30 and using spv = 6 on every root node and on every principal moves and running it automated and minimaxing later

Let's be honest here: IDEA's algorithm is crap. I lost ALL the games, 100% of the correspondence games where IDEA was involved. The way it does things is nonsensical and it's almost like it wants to generate evaluation holes where a strong centaur can play a move that IDEA missed and throw all the analysis done to the wastebin.

IDEA spends most of its time analyzing and extending irrelevant variations, and a good portion of what is left finding alternative moves for those irrelevant lines. It pays so few attention to the mainline because most of the time it'll not find it.

You can't really compare ChessDB's AI with IDEA, despite the low depth I've checked variations where CDB is able to outperform Stockfish Depth 36, and this is after one has to go to the leaf nodes that have nothing and do a "5 second request", just because reaching D36 is too slow so by the time it's done CDB will have built a tree.

What you would see in this CDB tree is that the main move of the main line has some 7 score (which corresponds with 0.07) while D36 shows some absurd 0.40, but if you play the variations against each other, some will regret its choices, and you'll have to go back and try something else. Eventually CDB's eval might have dropped to 14 (0.14) but D36's eval would have dropped to 0.28, enough to the root to have been refuted, because white had many alternatives before that were >0.28, and doing it like this can be much faster than having to play D36's moves from the black side, because it takes a while, CDB's tree moves are just there instantly.

And this is with CDB doing its thing while the line you request gets a fragment of the resources! Imagine if you could focus all of CDB's resources in the line you're interested it, it'd be insane, and surely, if you gave the same time and resources to CDB and to IDEA and they played a game, IDEA would lose every time, and it wouldn't matter what depth IDEA used, while CDB has already been optimized to D22 because other depths don't perform as well.

I think the main advantage strong centaurs have over such automatic methods is knowing when to use MultiPV and when stick to SinglePV, when to go wide and when to go deep. 30% of the time CDB has the main move wrong (initially) and on a portion of that the best move will not be in its top 8 (and it won't find it unless you play it), that's when I can outperform it, but of course I had already done analysis by myself to find the best move, and that took a while...

[Master Om]

Let's be honest here: IDEA's algorithm is crap. I lost ALL the games, 100% of the correspondence games where IDEA was involved.

I had mixed results. Drew more won many and lost few.

The way it does things is nonsensical and it's almost like it wants to generate evaluation holes where a strong centaur can play a move that IDEA missed and throw all the analysis done to the wastebin.

I somewhat agree on this. Lost games with those.

IDEA spends most of its time analyzing and extending irrelevant variations, and a good portion of what is left finding alternative moves for those irrelevant lines. It pays so few attention to the mainline because most of the time it'll not find it.

Again agree on this. Hence Human interaction is must in idea and the eval window must be relevant one so that the lines u need is analysed. Hence making moves green and red is very much necessary which is time consuming.

You can't really compare ChessDB's AI with IDEA, despite the low depth I've checked variations where CDB is able to outperform Stockfish Depth 36, and this is after one has to go to the leaf nodes that have nothing and do a "5 second request", just because reaching D36 is too slow so by the time it's done CDB will have built a tree.

Whats ChessDB ? never used it. Any reference ?

I think the main advantage strong centaurs have over such automatic methods is knowing when to use MultiPV and when stick to SinglePV, when to go wide and when to go deep.

Agree on this. Depends on experience.

Therefore I have found a way to use IDeA on my way rather than its way. Its automated way is good for opening anlysis as its not a game play and u can steer to as many variations as u can and depth 22 of yours works fine as sum of small is greater than the whole. But its useless in endgames and intricate middlegames. I have lost most games with idea where I ran it automated and always gets pissed off when i see opponents move not in list of 8-10 opening choices and later it finds that opponents choice was best move. Then game is Over then.
Hence I now use a mix of IDeA and IA where i have my formula of creating the opening tree.

[Ovyron]

Whats ChessDB ? never used it. Any reference ?

Huh... you don't know ChessDB? That explains your misunderstandings with noobpwnftw, as he was talking about it. This thread is about it and how is it being constructed (ChessDB is using Depth 22 and our messages have referred as to why...)

Here's the link: https://www.chessdb.cn/queryc_en/

It probably already has surpassed your own opening database in several lines (mainly, those you have never played) and expands the lines much more intelligently than IDEA. The main benefit is that it doesn't use local resources, so you can do whatever you want while it does its thing (it's free as in beer and free because you don't waste time using it, unlike firing up a weak chess engine to ask what it thinks about positions), and it displays results immediately (unlike having an engine reach high depth.)

Its weaknesses are near the leaf nodes (where the accurate eval comes from the law of averages) and... in intricate middle games (where it may not check some variations because it thinks it already knows what's going on) you might need to force the main line (you already know) yourself for it to expand it. And of course it'll refuse to do anything once certain material is reached on the board.

noobpwnftw claims that analyzing positions manually is a waste of time because you can have an AI like ChessDB doing interactive analysis in a way similar to humans but much more efficiently (which becomes evident after I wake up and ChessDB has already refuted an entire tree while... well, I can't analyze from the dream world), but I guess we'll never know because he'd not use it to play a corr game.

Therefore I have found a way to use IDeA on my way rather than its way. Its automated way is good for opening anlysis

It might give that illusion, but the moves it misses in the middle game are still missed on the opening. At least, either it'll go very deep into an irrelevant line that it thinks it's the best or it'll go too shallow by analyzing too many irrelevant variations. The point not even human interaction saves it is when the tree gets big and you waste a lot of time minimaxing.

[Master Om]

Whats ChessDB ? never used it. Any reference ?

Huh... you don't know ChessDB? That explains your misunderstandings with noobpwnftw, as he was talking about it. This thread is about it and how is it being constructed (ChessDB is using Depth 22 and our messages have referred as to why...)

Here's the link: https://www.chessdb.cn/queryc_en/

It probably already has surpassed your own opening database in several lines (mainly, those you have never played) and expands the lines much more intelligently than IDEA. The main benefit is that it doesn't use local resources, so you can do whatever you want while it does its thing (it's free as in beer and free because you don't waste time using it, unlike firing up a weak chess engine to ask what it thinks about positions), and it displays results immediately (unlike having an engine reach high depth.)

Its weaknesses are near the leaf nodes (where the accurate eval comes from the law of averages) and... in intricate middle games (where it may not check some variations because it thinks it already knows what's going on) you might need to force the main line (you already know) yourself for it to expand it. And of course it'll refuse to do anything once certain material is reached on the board.

noobpwnftw claims that analyzing positions manually is a waste of time because you can have an AI like ChessDB doing interactive analysis in a way similar to humans but much more efficiently (which becomes evident after I wake up and ChessDB has already refuted an entire tree while... well, I can't analyze from the dream world), but I guess we'll never know because he'd not use it to play a corr game.

Therefore I have found a way to use IDeA on my way rather than its way. Its automated way is good for opening anlysis

It might give that illusion, but the moves it misses in the middle game are still missed on the opening. At least, either it'll go very deep into an irrelevant line that it thinks it's the best or it'll go too shallow by analyzing too many irrelevant variations. The point not even human interaction saves it is when the tree gets big and you waste a lot of time minimaxing.

I thought he released his personal daatabse and chessDB i thought this

It might give that illusion, but the moves it misses in the middle game are still missed on the opening.

No not illusion. I dont use its automated expansion one unless i feed the positions needs to be analysed. I in shallow positions analyse like a pyramid. But in opening positions i only analyse positions taken from eco first and then find novelties then. But the fact its expands useless variations is true. Thats why i had to find method to force it and i have a nice work around. Once I complete the opening phase I use IA to go further with IDeA to keep up the scores. Play feed analyse and minimax. The automated one is crap. I still have not found the settings which can force it to fins a good move. But Giving succuient time and depth has a better analysis than shallow depth 10-25 based analysis.

[Ovyron]

I thought he released his personal daatabse and chessDB i thought this

I think you missed this relevant thread. ChessDBCN is mainly useful in the opening, I have no idea about the quality of the ranked moves (as I've been able to nudge them) but some go beyond depth 70's quality. The closer to the leaves the less useful, but then you can request a position to be analyzed there, and backsolve refutations, so you can come the next day and see it already has significantly improved the quality of those leaf nodes.

It'd be interesting if you compared it to your own IDEA opening tree and reported back, at some point they're going to disagree so you can play their mainlines and one of them is going to fall, it'll be interesting to know who falls first.

I in shallow positions analyse like a pyramid.

I think a lot of resources are wasted here. If you matched 2 analysis methods, and gave them the same time and resources, but one was analyzing like a pyramid, checking every node at the root, and another would not go as wide at first (say, only check top 4 moves at the root, but treat other nodes you check as if they were the root) the pyramid one would lose badly, and the second wouldn't need higher depth than 22 to succeed.

It's like in the opening position, you only need to analyze 1.d4, 1.e4, 1.c4 and 1.nf3. If the opponent plays 1.g3 then you'll have nothing in the tree, but you'll already have a better position than if the opponent played your mainline for 1.nf3 for several moves. Checking 1.f4, 1.h4, 1.nc3 and the other root nodes in case the opponent plays them is a waste of time, it wouldn't be surprising if you lost IDEA games by setting it to analyze like a pyramid.

Here's a rule of thumb: apply the same analysis method to every node you visit. If you do something different at the root because it's the root, or you do something different depending on the distances from the nodes to the root, you're gonna miss some key move in a future position that would have shaped your tree completely differently, and you'd have found this key move if you analyzed that node like the root. And if analyzing every node like the root would become prohibitive because you'd only have time to analyze a few positions, then the analysis at the root is wasting time.

I think an analysis tree would look like this:



(except non-mainlines would become smaller)

Note that after a move is made (the branch above trunk) what remains looks the same but in a smaller scale. You wouldn't need to check the other root moves after your opponent played a move. If the best move on the third branching only appears after a move has been made then the analysis method wasted time in other things, because knowing the best move there (before a move was made) would have given the tree a different shape.

[noobpwnftw]

Well, here is the thing:

There is no magic here, I cut corners by essentially refuting the lines with shorter depth searches, it still takes a substantial amount of computations to back this up by resolving those shorter lines including multi PV to their exact scores.

On the other hand, I gave you the full tree structure and all its terminal node scores, so you should be able to run whatever algorithm as you see fit to move on from there.

Another thing is this data dump is probably the last time I'd like to provide in trivial RocksDB format. As the database scales I've moved on to some over-engineered DAFSA implementations to get better performance, I made this snapshot before conversion.

[Ovyron]

In my manual analysis I cut even more corners by using score ranges. Like, if some line is some 0.20-0.40 and the main line is 0.60 I just put that there and don't worry about the exact score, though I guess something like that wouldn't work in a DB where the user has already missed the 0.60 move.

Is this database snapshot the same as the one accessible from https://www.chessdb.cn/queryc_en/ ? Something I noticed was that it had reached and surpassed 9000000K positions, but every time it'd go up eventually it'd go back to sub 9000000K. But recently it has gone down to 5000000K as if almost half of the database was lost

[noobpwnftw]

s this database snapshot the same as the one accessible from https://www.chessdb.cn/queryc_en/ ? Something I noticed was that it had reached and surpassed 9000000K positions, but every time it'd go up eventually it'd go back to sub 9000000K. But recently it has gone down to 5000000K as if almost half of the database was lost

Yes it is the same. The numbers are off due to outstanding merge operations being counted multiple times.

[Master Om]

I thought he released his personal daatabse and chessDB i thought this

I think you missed this relevant thread. ChessDBCN is mainly useful in the opening, I have no idea about the quality of the ranked moves (as I've been able to nudge them) but some go beyond depth 70's quality. The closer to the leaves the less useful, but then you can request a position to be analyzed there, and backsolve refutations, so you can come the next day and see it already has significantly improved the quality of those leaf nodes.

It'd be interesting if you compared it to your own IDEA opening tree and reported back, at some point they're going to disagree so you can play their mainlines and one of them is going to fall, it'll be interesting to know who falls first.

I in shallow positions analyse like a pyramid.

I think a lot of resources are wasted here. If you matched 2 analysis methods, and gave them the same time and resources, but one was analyzing like a pyramid, checking every node at the root, and another would not go as wide at first (say, only check top 4 moves at the root, but treat other nodes you check as if they were the root) the pyramid one would lose badly, and the second wouldn't need higher depth than 22 to succeed.

It's like in the opening position, you only need to analyze 1.d4, 1.e4, 1.c4 and 1.nf3. If the opponent plays 1.g3 then you'll have nothing in the tree, but you'll already have a better position than if the opponent played your mainline for 1.nf3 for several moves. Checking 1.f4, 1.h4, 1.nc3 and the other root nodes in case the opponent plays them is a waste of time, it wouldn't be surprising if you lost IDEA games by setting it to analyze like a pyramid.

Here's a rule of thumb: apply the same analysis method to every node you visit. If you do something different at the root because it's the root, or you do something different depending on the distances from the nodes to the root, you're gonna miss some key move in a future position that would have shaped your tree completely differently, and you'd have found this key move if you analyzed that node like the root. And if analyzing every node like the root would become prohibitive because you'd only have time to analyze a few positions, then the analysis at the root is wasting time.

I think an analysis tree would look like this:



(except non-mainlines would become smaller)

Note that after a move is made (the branch above trunk) what remains looks the same but in a smaller scale. You wouldn't need to check the other root moves after your opponent played a move. If the best move on the third branching only appears after a move has been made then the analysis method wasted time in other things, because knowing the best move there (before a move was made) would have given the tree a different shape.

I think a lot of resources are wasted here.

No it isn't. First of all saving the data is one time process. 2ndly The analysis is based on lines i choose to play with all possible opponent replies and mine.

1.d4, 1.e4, 1.c4 and 1.nf3. If the opponent plays 1.g3 then you'll have nothing in the tree,

I dont need to use idea on those positions. I use it on exit of so called traditional mainlines. Upto that point i play it myself from memory.

I think an analysis tree would look like this:

u didn't get me. Mine is opposite of the image as that tree is fine for variations. I was talking about something else. i.e. finding the key move.

[Ovyron]

2ndly The analysis is based on lines i choose to play with all possible opponent replies and mine.

"All possible opponent replies" is the culprit here, because 90% of those moves will not be played, and if they're played you're already in good shape because the opponent missed (say) the top 4 moves and played a bad one. If you only analyze the top 4 and your opponent plays a different move, and it turns out it was better than those, then it's a problem with the method itself because you should have had that move in your top 4 in the first place.

Now I'm going to reveal something, I've always given funny names to my analysis methods, such as "Carousel" or "Power Drifting" and such. Most of them were garbage, but I didn't know until someone defeated them. One of those methods was "The Kitchen Sink", some of my opponents might recognize that one. Well, the kitchen sink was nothing but drawing an analysis tree like a pyramid, checking every root node for me and my opponent at the root, and ending with a tree that looks upside down, similar to what you're doing.

When I lost using it I discovered that I could have saved the game if I didn't waste so much time analyzing so many irrelevant lines. The key to analysis is to find what is the best opponent's move, and refute that. That's the only move you need to check. You check the others to make sure it's not one of them, but if you have to analyze more than 3 alternatives to make sure opponent's best isn't missed, your method to rank the top 4 moves on a position is flawed, and fixing it would make you save a lot of time, because on a position with 20 playable moves, you only need to check 4 lines and discard the other 16. 80% of time saved.