ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Compression of chess databases
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Daniel Shawul



Joined: 14 Mar 2006
Posts: 2186
Location: Ethiopia
PostPost subject: Re: Compression of chess databases    Posted: Thu Jul 19, 2012 4:21 am Reply to topic Reply with quote
phhnguyen wrote:
Thank you for replying.

I think we still have different views about database and methods of compressing. I may have some misunderstand some numbers and some details. Thus the best now I would like to know more, could you help me?

Sorry it is not about views as this is not a matter of opinion but facts. Thats why we need numbers.
Quote:

From what I have, I see that Richard's database size is reasonable for me. However, I don't understand how you got the number 2.467 MB. How did you create data? What is size of your raw data size before compressing? What name/kind of compressor did you use?

I just zipped the raw data that is made of an 8 bit/move. I get almost same raw data as Richard that is 4.4mb. He went on to use prediction and zipping to get a size of 1.9mb but I did not predict so it became 2.46 mb. So that is only 20% improvement for prediction. OTOH you used 6 bits/move to a size of 3.4mb which actually is (6/8) * 4.4mb so that makes sense. BUT your zip compression was so bad so it only brought you 10% to 3mb. This is what I have been saying all along. Infact this was so generous for you because there are only about 5 or so moves that can not be represented by 6bits/move in that database! So you only needed to patch up those 5 moves but still your compression ratio was so bad. With 4 bits/move there are far more number of moves that won't fit and the problem will worsen..

Note that I can further improve up on my compression ratio using a compressor that uses neural nets or something similar... But yours will be difficult to improve upon and it has already lost anyway after making a mess of the order. It may even be possible to compress the san moves (i.e 32bits/move+) as they are down to what you get i.e 3mb as those are already at 4mb anyway. It is not how many bits you spend on representing the move but the amount of disorder in it that matters most.

Note that you are doing entropy coder first which is the problem. Applying LZ over that data will not bring much. In any compression that is usually the _last_ step of compression e.g RLE + huffman , or LZ + huffman. Not the other way round as you did it i.e variable_length + LZ.
Quote:

More details is the best. I would like to try myself to make my comparison and conclusion before sticking long time to a method.

Thanks.

_________________
https://sites.google.com/site/dshawul/
https://github.com/dshawul
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger
Display posts from previous:   
Subject Author Date/Time
Compression of chess databases Edmund Moshammer Mon Jul 09, 2012 8:28 am
      Re: Compression of chess databases Richard Vida Mon Jul 09, 2012 11:14 am
            Re: Compression of chess databases Richard Vida Mon Jul 09, 2012 11:36 am
                  Re: Compression of chess databases Edmund Moshammer Mon Jul 09, 2012 11:48 am
                        Re: Compression of chess databases Richard Vida Mon Jul 09, 2012 11:58 am
                              Re: Compression of chess databases Edmund Moshammer Mon Jul 09, 2012 1:03 pm
                  Re: Compression of chess databases Daniel Shawul Mon Jul 09, 2012 3:06 pm
                        Re: Compression of chess databases Daniel Shawul Mon Jul 09, 2012 3:27 pm
                        Re: Compression of chess databases Richard Vida Mon Jul 09, 2012 5:34 pm
                              Re: Compression of chess databases Daniel Shawul Mon Jul 09, 2012 6:23 pm
                                    Re: Compression of chess databases Gary Mon Jul 09, 2012 8:39 pm
                                          Re: Compression of chess databases Daniel Shawul Mon Jul 09, 2012 9:01 pm
                                                Re: Compression of chess databases Gary Mon Jul 09, 2012 9:17 pm
                                                      Re: Compression of chess databases Daniel Shawul Mon Jul 09, 2012 9:29 pm
                  Re: Compression of chess databases Nguyen Pham Sun Jul 15, 2012 8:05 am
                        Re: Compression of chess databases J. Wesley Cleveland Sun Jul 15, 2012 6:02 pm
                              Re: Compression of chess databases Nguyen Pham Sun Jul 15, 2012 11:15 pm
                                    Re: Compression of chess databases Ronald de Man Mon Jul 16, 2012 1:01 am
                        Re: Compression of chess databases Richard Vida Mon Jul 16, 2012 12:31 pm
                              Re: Compression of chess databases Nguyen Pham Mon Jul 16, 2012 2:57 pm
                                    Re: Compression of chess databases Álvaro Begué Mon Jul 16, 2012 6:01 pm
                                          Re: Compression of chess databases Nguyen Pham Tue Jul 17, 2012 12:59 am
                                                Re: Compression of chess databases Kevin Hearn Tue Jul 17, 2012 2:39 am
                                                Re: Compression of chess databases Ronald de Man Tue Jul 17, 2012 9:29 pm
                                                      Re: Compression of chess databases Daniel Shawul Tue Jul 17, 2012 10:12 pm
                                    Re: Compression of chess databases J. Wesley Cleveland Tue Jul 17, 2012 5:20 am
                                          Re: Compression of chess databases Nguyen Pham Tue Jul 17, 2012 7:19 am
                                    Re: Compression of chess databases Richard Vida Tue Jul 17, 2012 9:04 pm
                                          Re: Compression of chess databases Richard Vida Tue Jul 17, 2012 11:35 pm
                                                Re: Compression of chess databases Daniel Shawul Tue Jul 17, 2012 11:59 pm
                                                      Re: Compression of chess databases Ronald de Man Wed Jul 18, 2012 12:55 am
                                                            Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 1:35 am
                                                Re: Compression of chess databases Nguyen Pham Wed Jul 18, 2012 5:23 am
                                                      Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 9:35 am
                                                            Re: Compression of chess databases Nguyen Pham Wed Jul 18, 2012 10:07 am
                                                                  Re: Compression of chess databases Uri Blass Wed Jul 18, 2012 10:31 am
                                                                        Re: Compression of chess databases Nguyen Pham Wed Jul 18, 2012 10:48 am
                                                                              Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 11:00 am
                                                                  Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 11:07 am
                                                                        Re: Compression of chess databases Nguyen Pham Wed Jul 18, 2012 11:47 am
                                                                              Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 12:41 pm
                                                                                    Re: Compression of chess databases Nguyen Pham Wed Jul 18, 2012 1:20 pm
                                                                                          Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 3:01 pm
                                                                                          Re: Compression of chess databases Daniel Shawul Wed Jul 18, 2012 3:40 pm
                                                                                          Re: Compression of chess databases Nguyen Pham Thu Jul 19, 2012 1:45 am
                                                                                          Re: Compression of chess databases Daniel Shawul Thu Jul 19, 2012 2:32 am
                                                                                          Re: Compression of chess databases Nguyen Pham Thu Jul 19, 2012 3:19 am
                                                                                          Re: Compression of chess databases Daniel Shawul Thu Jul 19, 2012 4:21 am
                                                                                          Re: Compression of chess databases Nguyen Pham Thu Jul 19, 2012 5:33 am
                                                                                          Re: Compression of chess databases Daniel Shawul Thu Jul 19, 2012 6:53 am
                                                                                          Re: Compression of chess databases Daniel Shawul Thu Jul 19, 2012 4:49 am
                                    Re: Compression of chess databases Daniel Shawul Tue Jul 17, 2012 9:25 pm
      Re: Compression of chess databases Vincent Diepeveen Wed Jul 11, 2012 12:14 am
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads