Interesting article about strategy to defeat NN Go programs

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

jdart
Posts: 4368
Joined: Fri Mar 10, 2006 5:23 am
Location: http://www.arasanchess.org

Interesting article about strategy to defeat NN Go programs

Post by jdart »

https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.
dkappe
Posts: 1632
Joined: Tue Aug 21, 2018 7:52 pm
Full name: Dietrich Kappe

Re: Interesting article about strategy to defeat NN Go programs

Post by dkappe »

jdart wrote: Thu Nov 10, 2022 4:52 pm https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.
Very interesting. I hope the leela project gives this a try to uncover some of its weaknesses.
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
lkaufman
Posts: 5981
Joined: Sun Jan 10, 2010 6:15 am
Location: Maryland USA

Re: Interesting article about strategy to defeat NN Go programs

Post by lkaufman »

jdart wrote: Thu Nov 10, 2022 4:52 pm https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.
Well, it sounded interesting from the description, but when I read it I see that it basically amounts to the program not knowing the rules; it is ridiculous to "pass" in GO if there are useful moves remaining and the game is not yet won. The program just needs to be told to determine what the score would be if the game ended now, and not to pass if it's not 100% won. It's the same as the chess programs that would troll with queen and rook vs. bare king by waiting until move 49 of the 50 move rule to mate (as many NNs used to do), but without even knowing the 50 move rule. So nothing general to learn here, just they need to fix that bug in the program. In other words, an estimated 99.999% chance of winning is still not as good as an actual win, and obviously much worse than a loss, which is what simple counting would determine. Probably the problem is that the rules are a big vague for how to count when a game is terminated too early. But this is an issue for the rules of Go, not a programing issue.
Komodo rules!
MonteCarlo
Posts: 188
Joined: Sun Dec 25, 2016 4:59 pm

Re: Interesting article about strategy to defeat NN Go programs

Post by MonteCarlo »

It's also less interesting for another reason, as they only convincingly beat KataGo when it used no search at all.

They had a slight plus when it was allowed to search small numbers of nodes, but past 100 nodes they lost virtually every game, it seems.

It's not so surprising that one can exploit either a pure policy network or so little search that the choice of move is dominated by the policy network.

Cheers!
User avatar
Eelco de Groot
Posts: 4595
Joined: Sun Mar 12, 2006 2:40 am
Full name:   

Re: Interesting article about strategy to defeat NN Go programs

Post by Eelco de Groot »

Sory if too off topic to the thread, but a bit excited about the latest KataGo neural network being tested at https://katagotraining.org/

It has only played 156 Rating games, as I write this 172, so the 95% confidence interval is huge, but it started out really well;


Image
Network is kata1-b18c384nbt-s8738519040-d3984211304

After 172 games the rating had dropped again though, to 13607.4 +/- 58.0 but that is still 7 points better than the best Net so far. I can't really judge how it plays as I only started to learn the rules a bit, I can see it is good at creating 'eyes' but that's about it :)
(I watched this game: https://katagotraining.org/sgfplayer/ra ... s/1172965/ and after 17 moves or so White had already three of those eyes, as far as I could see, Black still trying to surround the territory that was unconquerable at top left of the board, with two eyes in it. But I don't know how winning it was at that point, it just looked very good there.)
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
User avatar
Eelco de Groot
Posts: 4595
Joined: Sun Mar 12, 2006 2:40 am
Full name:   

Re: Interesting article about strategy to defeat NN Go programs

Post by Eelco de Groot »

That network (...211304) in the end did not turn out so well, it seems they always start higher, but come down later because of the way they choose opponents maybe... But at least now, almost a month later, I think there was some small progress, KatGo may cross the 13,620 Elo level!

Okay I stop posting about KataGo progress for now, just this one update:

Image
Network is kata1-b18c384nbt-s9040915712-d4064991045
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
User avatar
Eelco de Groot
Posts: 4595
Joined: Sun Mar 12, 2006 2:40 am
Full name:   

Re: Interesting article about strategy to defeat NN Go programs

Post by Eelco de Groot »

There is a new Network for Kata! With new "dimensions" as some use the term in here (b28c512). Only 35 games played as I write this but I am very excited :mrgreen:

Image
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
Uri Blass
Posts: 10424
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Interesting article about strategy to defeat NN Go programs

Post by Uri Blass »

MonteCarlo wrote: Fri Nov 11, 2022 4:42 am It's also less interesting for another reason, as they only convincingly beat KataGo when it used no search at all.

They had a slight plus when it was allowed to search small numbers of nodes, but past 100 nodes they lost virtually every game, it seems.

It's not so surprising that one can exploit either a pure policy network or so little search that the choice of move is dominated by the policy network.

Cheers!
It is interesting if there is a strategy that weak chess players can use to beat lc0 in chess when lc0 is allowed to search one node per move or very small number of nodes and not repeating existing game.
User avatar
Eelco de Groot
Posts: 4595
Joined: Sun Mar 12, 2006 2:40 am
Full name:   

Re: Interesting article about strategy to defeat NN Go programs

Post by Eelco de Groot »

The new "2003" net is really making mincemeat of the competition in this 'equal visits' test. Out of the last 100 games I count just five losses and five draws. Rest wins. Haven't checked any games yet. But that is like running equal number of nodes test in chess, with a very slow but big Lc0 like net.
2024-05-02 - First "b28" neural net uploaded! This new larger neural net runs more slowly, similar to the old 60-block nets, but should be much stronger. From tests on discord, under normal usage this series of nets should be stronger than the b18 nets given equal total time, despite the slower speed, and be by far the strongest nets so far on a per-visit basis. We've also upgraded contributing to require KataGo v1.14.0 or later to support this net.
It does not run in my KataGo v1.14.0 GUI but that is maybe because I can't run a GPU on this computer, I have to use the EigenAVX2 version in it that is just 1.13.0 .
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan