Position Causes Stockfish and Komodo To Crash

[Dann Corbit]

Someone recently wasted their time writing up 20 paragraphs showing how they could "exploit" Stockfish into crashing....

If you send a chess engine garbage, you should expect garbage. If you don't know it is garbage, then maybe a GUI should be doing it for you.

The above has been affirmed many times in Stockfish PRs, where users come saying they have found a "bug"

Is that how you would write code for a customer? Unexpected input causes a program crash?
If a programmer is too lazy even to think hard, they can still use try catch (yes, I know, a 1% performance penalty).
It boggles my mind how chess programmers will spend 10,000 hours writing their programs but spend zero minutes checking the input for correctness.

Not doing more than what the spec requires is the right form of laziness. A customer should not need to pay for fancy and unnecessary stuff going beyond the spec.

The UCI spec requires the GUI to give the engine sane inputs. Therefore the engine should expect sane inputs.

(Admittedly the UCI spec is very mistaken where it requires an engine to ignore unrecognised tokens but to do its best to process the rest of the input line. I'm not convinced that many engines will "correctly" process "gobbledygook go infinite".)

On a job interview, if you suggest that you do not validate inputs when you write programs, and will simply let the program core dump to resolve such things because inputs should never have errors in them, I think it will speed up the interview nicely.

[syzygy]

Des your engine catch all such cases? Is your GUI thread aware of all the possible invalid input sequences that a malicious GUI could send it? It seems a lot of work to get 100% right.

most likely not, it's hard to cover 100% of the cases and all code paths. but at least I try to handle the obvious ones.

But then you accept that each engine will crash or freeze on a different set of input sequences, and the GUI author will still have to make sure to send only valid input to the engine.

Of couse there is no need to crash on illegal input, but I don't much point in doing extra work to avoid crashes without ever being able to be complete.

It is the GUI author's responsibility to give the UCI engine valid input.

I agree, but some people run engines from the command line. I certainly use command line when debugging my engine.

Me too, but when I mistype while debugging it doesn't matter to me whether the engine aborts, crashes or freezes.

[syzygy]

Someone recently wasted their time writing up 20 paragraphs showing how they could "exploit" Stockfish into crashing....

If you send a chess engine garbage, you should expect garbage. If you don't know it is garbage, then maybe a GUI should be doing it for you.

The above has been affirmed many times in Stockfish PRs, where users come saying they have found a "bug"

Is that how you would write code for a customer? Unexpected input causes a program crash?
If a programmer is too lazy even to think hard, they can still use try catch (yes, I know, a 1% performance penalty).
It boggles my mind how chess programmers will spend 10,000 hours writing their programs but spend zero minutes checking the input for correctness.

Not doing more than what the spec requires is the right form of laziness. A customer should not need to pay for fancy and unnecessary stuff going beyond the spec.

The UCI spec requires the GUI to give the engine sane inputs. Therefore the engine should expect sane inputs.

(Admittedly the UCI spec is very mistaken where it requires an engine to ignore unrecognised tokens but to do its best to process the rest of the input line. I'm not convinced that many engines will "correctly" process "gobbledygook go infinite".)

On a job interview, if you suggest that you do not validate inputs when you write programs, and will simply let the program core dump to resolve such things because inputs should never have errors in them, I think it will speed up the interview nicely.

You almost always have to verify user inputs. You always have to verify unfiltered input from the internet. You don't need to verify input from the GUI if the spec says the responsbility for verifying the input is with the GUI. You should be able to read and understand and act in accordance with the spec. A C library may crash on a double free. A CPU may trap on a division by zero. A company has no use for people who rewrite the spec while coding.

[Dann Corbit]

The GUI writers will tell you it is the responsibility of the engine or the user to detect bad data.
FEN/EPD positions often come from a file. Typically, when running stockfish, I have a bunch of machines all reading positions from a file using the chess engine itself.
There is no GUI involved anywhere in this picture.
As I have said else-thread, I already have tools to fix bad positions.
I am just reminding the chess programmers of their obvious responsibility.

"It's the other guy's fault for not checking." is the lamest of lame excuses, wallowing in a deep, sweaty pool of lameness.

If you expect perfect data, and you know that the data was created by humans, then how smart is that?

[syzygy]

The GUI writers will tell you it is the responsibility of the engine or the user to detect bad data.

Complain to them.

"It's the other guy's fault for not checking." is the lamest of lame excuses, wallowing in a deep, sweaty pool of lameness.

Any organisation of people needs to have rules that say where the responsibilities lie.

I look forward to reading your bug reports to the Glibc mailing list about crashing on double frees.

[Ras]

The GUI will likely get terribly confused if it does not crash.

If the GUI can't deal with "info string" or "bestmove 0000", it isn't a UCI compliant GUI in the first place.

Does your engine also refuse to crash or freeze if you change the TT size or the number of search threads halfway through a search?

No crash here because that will be buffered until after search ends. If the buffer overflows, it will be overwritten - which is OK because the GUI shouldn't be sending such commands during search in the first place.

Unless we're dealing with defective hardware, any engine crash is a bug that needs to be fixed, that's the kind of robustness that I'm striving for.

[smcracraft]

Leela crashes too.

[RubiChess]

What do you think about this kind of position?

KQQQQQQQ/QQQQQQQQ/QQQQQQQQ/QQQqqqqq/QQQqqqqq/QQQqqqqq/QQQqqqqq/QQQqqqqk b - - 1 1

GUIs might (and do) accept this as a "puzzle" position although it is not legal in terms of reachable from the start position. Which is totally okay in my opinion.
Stockfish crashes on it.

The recent removal of piece lists might have fixed that (it seems to work for me).

True. Current master doesn't crash. I must admit that the crash happened with a Stockfish version three months old.

[RubiChess]

Sure, you can have your opinion. But at the end of the day Marco Costalba decides

I guess you are wrong.

[Michel]

There is a universal consensus, outside computer chess apparently, that a program should always handle illegal input gracefully. Would you want ld to crash if it does not recognize a file format?

There is another point good which has already been mentioned. Since the concept of a legal chess position is not well defined (can we have 10 queens? 20? black and white pawns all on the 7th rank?) it seems better to let the engine decide if it can handle a fen and return an error (info string) if this is not so.