Scid vs PC CQL 5.2 feature

[stevenaaus]

Maybe some people here use CQL ? Lionel has written a CQL search feature for Scid vs PC. It's all in subversion, and here's some windows betas. Currently sort feature is unimplemented, but it may be included by 4.19 in a month or two.

32bit: https://sourceforge.net/projects/scidvs ... e/download

64 bit: https://sourceforge.net/projects/scidvs ... e/download

We have some unresolved CC warnings too. Maybe someone will look at them ?? We've moved to native windows gcc builds instead of mingwx from Linux.

http://scidvspc.sourceforge.net/4.19-r2815-test/

Lionel has been implementing the CQL (Chess Query Language) search feature some more - CQL 5.2 is now gold - and Oz has made up some windows betas.

32bit: https://sourceforge.net/projects/scidvs ... e/download

64 bit: https://sourceforge.net/projects/scidvs ... e/download

This feature is pretty heavy to use (there are a few examples in the help file), but people may also be interested in the other minor changes included (upto subversion rev 2815).
There are not many of these. A new 'Ascii' chess piece set and green colour scheme
and a 'default/autload databases' feature.

We are using a new windows build system, and hopefully there's no regressions.
Maybe our Windows big DB speed loading has changed ? Sorry, but i havent tested it.

So thanks to Lionel, Oz and the CQL authors, who have been working with Lionel i think.

Below is a changelog since 4.18.1 (though this beta still installs as 4.18)
Maybe in a month or two, 4.19 will be released.

cheers, Steve

CHANGES
Fix up the days of the months in the calender widget.

Fics: Change censor button a little, and Control-Censor censors person chatting to you.
Fix up window placement on milti display desktops.

Input engine fixes (DGT board). Widget now resizes ok, and bitmaps display properly on OS X. Thanks to Davy De Roeck for debugging it on OS X. Destroy signal handled a little differently, and maybe a minor memory leak fix too (unrelated, in bitmaps.tcl).

New search feature. General search can now match any (non-hidden) pre-game or final move comment.

Export Scid Flags feature.

Opening table filter operations didn't update the gamelist. (Previously stats::Refresh did a gamelist refresh).

Some tree tweaks/doco updates. Mostly non-functional, except the Fast+Slow mode now doesn't break the win/lose graph.

Streamline widget refresh a little

Bugfix: Best games 'Reset Column titles' was broke.

Re games loading at a certain position -= there is a conflict between the tree window and search results.
What position do we load at ? Now we prioritize the search result, and if no search result, load ply of the tree/current position.
Small associared speed bump.

CQl 5.2 search feature (Search ->CQL). Thanks Lionel and CQL team.

OS X has -O3 build instead of (mistakenly) no optimisation.

Tweak progress bar update frequencies (base export, cql search, game copy).

New Windows build system from Oz. (todo - test speed)

New Ascii pieces (Thanks HGM), and Green colour scheme (replaces red).


Big speed up for (bulk) Strip Comments/Variations feature. Also make the 'Stop' button work.

fix: sc_game startPos segfaults if !HasNonStandardStart

Clean up scripts a little, add a new script sc_filter_pgn

New 'Default Databases' feature (Options->Save Bases as Default)

Fix up showing the first gamelist/game in searches.

Some minor gamelist fixes when emptying clipbase/closing bases. (resets clipbase to read-write, and reset title arrows)

FICS: Offers Graph disabled while playing a game

Extend game load key bindings to gamelist treeview widget.

[BogStandard]

I have tried SCID CQL 5.2 and it is slightly faster than using Aquarium CQL 3.1 and much faster than using it from the command line.

For my test CQL script the results are 1k5 matches from 6M2 games.

Running the versions simultaneously, my timings are (manually done so 0 to +12sec error):

command line v5.1 24:15
command line v5.2 21:35
command line v3.1 16:52
Aquarium2017 v3.1 10:09
SCID4.19 v5.2 09:42

I have also created a batch file that runs from the command line creating multiple instances of the CQL.exe which work on separate fractions of the pgn database that I have previously split it into 30 parts:

command line batch (30 fractions) v5.2 01:26

I have wondered whether it might be possible to apply a similar method in SCID. That is, create multiple instances (of the internal cql.exe) working on their own allocated batch of games within the SCID database and concatenating the results. The fractions of games would be cpu-hyperthread-count less 1 or 2.

I shall post my batch script after this post.

My test CQL v3.1:
;;
;;Find Imprisoned Rook
;;
(match
:pgn HUGE3.1.pgn
:output Imprisoned_Rook180331v3.1.pgn
(position
Ra1 bb1 pa2
:flipcolor
:fliphorizontal
:flipvertical
);end position
); end match

My test CQL v5.1 and v5.2:
;
;Find Imprisoned Rook
;Rook is in a corner held by bishop and pawn
;
cql (
input HUGE5.2.pgn
output Imprisoned_Rook180331v5.2.pgn
)
flipcolor
fliphorizontal
flipvertical
{Ra1 bb1 pa2}

[BogStandard]

My batch script for CQL, copy all after this colon:

@echo off
REM CQL__BatchAnalyse.cmd
REM If you click this file in explorer it WILL run.
REM To amend it 'drag' or 'Open' it in your text editor

setlocal
REM Use Local variables that are destroyed afterwards.

set cqlfn="CQL_Imprisoned_Rook_5.2"
REM cqlfn is the filename of your CQL script file (without the .cql extension).
REM Change this, each time, to your CQL script filename.

set cqlsf="Huge"
REM cqlsf is the name of the pgn game source files that will be searched for CQL matches.
REM We only want the first part of the filename common to all the pgn files so we can just loop through them.
REM You should split your 'huge' pgn database into, at most, one less than the number of your CPU cores.
REM Having 2 Hyperthreading 8 core CPUs giving 32 cores and I have previously split hugebase.pgn into
REM 30 parts (Huge30-00.pgn to Huge30-29.pgn (using gameSplitter.exe)).

FOR %%X in ("%cqlsf%*.pgn") DO START cql.exe "--input" %%~nxX "--output" %cqlfn%%%~nxX %cqlfn%.cql
REM Now loop through the pgn files whose names begin with cqlsf [ FOR %%X in ("%cqlsf%*.pgn") DO ]
REM forcing a new parallel instance of cql.exe for each pgn file [START cql.exe ]
REM to read the input file [ "--input" %%~nxX ] and write the output file [ "--output" %cqlfn%%%~nxX ]
REM processing each game according to the cql instructions [ %cqlfn%.cql ]
REM The output filenames are a concatenation of the CQL script filename and the pgn filename.
REM NOTE: the CQL file must still contain an input file reference cql(input some.pgn) even though it
REM will be ignored because we are specifying the --input and --output files in our batch-produced command line.

pause
REM Wait for the user to press a key after they have seen all the command windows have closed.

type %cqlfn%%cqlsf%* > %cqlfn%.pgn
REM Now concatenate all the output game files (which will be zero bytes if there were no matches)
REM giving us one pgn file of all the results, named after the CQL script filename.

endlocal
REM Tidy up our script's use of memory.

pause
REM When the combined result file is created wait for a key press. This pause can be left out.

REM Summarising: For each input pgn you now have a new output pgn of extracted matching games
REM plus the combined output pgn. Move and delete the new files as required.
REM
REM
REM If your script doesn't run or the error is "The system cannot find the file specified." check:
REM you have given the correct CQL filename above:- set cqlfn=
REM the CQL script is located in the same folder as this script:- CQL__BatchAnalyse.cmd
REM along with the input pgn files
REM and the cql.exe.
REM Check the first CQL script command is like:- cql(input some.pgn)
REM
REM If it still doesn't run, run your cql script from the command line in your cql.exe folder:- cql.exe myscript.cql
REM You will then get to see the error reports the program gives when a script fails, which
REM are otherwise suppressed when cql.exe is called from this batch script.
REM