cutechess-cli suggestion (to Ilari)

[Michel]

I consulted some more recent books on sequential testing. It seems that LOS based sequential testing (which started this thread) is used in cases where there is a difference in social cost associated with H0 being true versus H1 being true.

A typical example is given by comparing two treatments A and B for a disease. H0 says that A and B are equivalent whereas H1 says the opposite. If A and B are equivalent you can essentially continue testing forever (just make the doctors prescribe both treatments). However if one treatment is really better you want to detect this as quickly as possible so that afterwards everyone can get the better treatment.

It seems that LOS based sequential tests can be analyzed exactly. However to convert the "exact" results into actual numbers you need numerical computations. The type of numerical computations you need are part of standard numerical packages.

(Needless to say that the LOS at which you accept H1 should be higher for a sequential test than for an equivalent fixed length test.)

[Michel]

BTW "LOS" seems to be used only in computer chess. The official terminology for LOS
based sequential testing is "Repeated significance testing".