Android behavior change: no more W^X violations

[Pi4Chess]

Yes, I saw your one star rating on Google Play. Thank you for that...

For many reasons, I prefer not to ship engine binaries in Chess for Android. However, luckily several engine packages are available on Google Play, which work really well with Chess for Android. See, for example, Karl's stockfish collection.

This package of engines from the same Karl Schreiner is nice also : https://play.google.com/store/apps/deta ... ss.engines

[abik]

This package of engines from the same Karl Schreiner is nice also : https://play.google.com/store/apps/deta ... ss.engines

It sure is, and works well too. In fact, I was already running an engine tournament with that set, just to show that it still works on Android 10!
I will posts results shortly....

[abik]

To avoid this thread from becoming too depressing, I wanted to include something fun, just focusing on what still works in our Android ecosystem! So, I ran a quick tournament between all engines in the packages of Karl, Javiolo, and myself, using the latest Chess for Android running on Android 10, with 10 seconds for the full game, 10 random games. The results are below. The last position of brainfish64pack is due to a glitch in the engine (starting from an opening line it seems).

Code: Select all

1   SugaR_XPrO 20180424 64      137.5/150
2   Cfish 20180418 64           121.0/150
3   Aristides 20171222 64       119.5/150
4   CiChess 20171107 64         112.0/150
5   Nayeem 20170115 64          108.5/150
6   CorChess 28041820180425 64  106.5/150
7   McBrain 20180210 64         103.5/150
8   Laser 1.8 beta               89.0/150
9   Andscacs 0.921               78.0/150
10  Toga II 4.01                 60.5/150
11  Rodent IV 0.32               51.0/150
12  Gambit Fruit 2.2 beta 4bx    48.5/150
13  OpenTal 1.1                  33.0/150
14  BikJump v2.5 (64-bit)        20.0/150
15  Chess for Android            11.5/150
16  brainfish64pack               0.0/150

[lucasart]

I internally proposed adding an explicit W^X permission to Android so that users can opt-in for apps that they trust (like Chess for Android ). Let's see how that goes, but I am not sure how serious such a proposal will be taken, or if that even addresses all security concerns.

Yes, that makes perfect sense.

I mean, when you look at how many permission the average Play Store app is asking for, it's ridiculous! Wanna play Tetris ? Sure! Obviously, Tetris will need access to all your cameras, microphone, accelerometer, etc. Oh, and of course, your picture gallery, your calendar, your phone directory, your GPS location, and your credit card history. Funny how security was never a concern there

In comparison, a Chess GUI asking the user if it can execute programs. Seems a reasonable thing to ask...

[Ugochukwu01]

To avoid this thread from becoming too depressing, I wanted to include something fun, just focusing on what still works in our Android ecosystem! So, I ran a quick tournament between all engines in the packages of Karl, Javiolo, and myself, using the latest Chess for Android running on Android 10, with 10 seconds for the full game, 10 random games. The results are below. The last position of brainfish64pack is due to a glitch in the engine (starting from an opening line it seems).

Code: Select all

1   SugaR_XPrO 20180424 64      137.5/150
2   Cfish 20180418 64           121.0/150
3   Aristides 20171222 64       119.5/150
4   CiChess 20171107 64         112.0/150
5   Nayeem 20170115 64          108.5/150
6   CorChess 28041820180425 64  106.5/150
7   McBrain 20180210 64         103.5/150
8   Laser 1.8 beta               89.0/150
9   Andscacs 0.921               78.0/150
10  Toga II 4.01                 60.5/150
11  Rodent IV 0.32               51.0/150
12  Gambit Fruit 2.2 beta 4bx    48.5/150
13  OpenTal 1.1                  33.0/150
14  BikJump v2.5 (64-bit)        20.0/150
15  Chess for Android            11.5/150
16  brainfish64pack               0.0/150

But this engines are old... So sad, google is now a wolf in sheep's clothing

[c4akarl]

I continue to investigate the issue, but in the meantime, did other chess app developers for Android encounter this issue already?

I'm just a Java developer and I use the engines (*.so) in my apps "Chess for All", "Chess Engines OEX" and "Stockfish Engines OEX". Bernhard C. März (user: shogi4fun) supports me in compiling some engines. Below is my experience with the SDK >= 29 problems.

Cannot run program "/data/user/0/com.package.name/files/engine.exe": error=13, Permission denied #343 at targetSdkVersion 29

This error is incomprehensible. This path is the external app directory. There shouldn't be any r/w problems here. Does the error also come if you set "targetSdkVersion 29" in build.gradle (module)?

As far as I know it is not possible to implement all functionality currently in DroidFish using targetSdkVersion 29, so the only option is to use targetSdkVersion 28 and don't distribute the app in the play store.

I have already set all my chess apps to "targetSdkVersion 29". In Chess for All I also activated this option in the manifest:
android:requestLegacyExternalStorage = "true"

This option is ignored from SDK version >= 30. So I've postponed the problem by one version. From API >= 30, it is no longer possible to set file activities in the external storage area (sd card) without directory permission. Access to the external app directory (getExternalFilesDir()) is allowed without permissions.

Is the OEX standard the way to go or even this is impacted ?

Gerhard Kalab did a good job with the OEX tool (com.kalab.chess.enginesupport). The engines are stored in a safe area and if the engine does not access the file system, there should be no problem with SDK >= 29.

I only use OEX in my apps to provide the engines. Engines that do not require file access run without problems. In order to get the file access problem under control, I have already tried a few things with my engine specialist Berhard C. März, here is the result:

Only an embedded variant (E.g. Stockfish Engines OEX app, engine "Stockfish 20200926 NNUE") actually works without problems.

A solution to provide additional files by copying them in the OEX app, the engine works (E.g. Chess Engines OEX app, Engine Rodent IV). For the engine developer, however, the effort is very high. Separate directory structures must be defined for each OEX app. An additional problem is that if the user does not open the app after installing the OEX app, the files are not copied.

Another solution is to give the user the option of defining the path/file in the GUI app via the UCI options (String). I realized this in the Chess for All app, UCI options (see info button). However, the effort is very high for both the user and the developer.

I think that these file delivery problems in OEX (Engine / OEX-App / GUI-App) can be solved.

The biggest problem for me with these new file accesses in SDK >= 29 is the fact that you no longer have direct access to the important "File" and "RandomAccessFile" classes. The files must be in the external storage area (persistent) and access to the path/file is essential. It is not possible with a file manager plus directory permission to get the files in the "File" format. All attempts failed.

The only possibility that I found in a documentation is the new permission MANAGE_EXTERNAL_STORAGE. However, whether Google will release this for chess apps is more than questionable. Here is the link:
https://developer.android.com/training/ ... -all-files

How should a chess app work without saving, reading and changing data? A chess app without PGN processing?
Does anyone have a solution to this problem?

Sources:
https://github.com/c4akarl

[shogi4fun]

Only an embedded variant (E.g. Stockfish Engines OEX app, engine "Stockfish 20200926 NNUE") actually works without problems.

and I like to add this:
there are different technics of embed files.
SF is doing it in an fantastic way (I didn't went deep into it and won't do it ), somehow able using it direct inside the code.

lc0 has also this feature, but doing it in a much simple way (this I completely understand ), getting the data by file access, which can cause some trouble.
So we detected lot of problems on armeabi-v7a hardware, while it seems working on arm64-v8a, at least till now. Most likely also depending on the android version.

But this engines are old... So sad, google is now a wolf in sheep's clothing

"newest" is not the only thing that matters - old engines have charm and there own character, different styles - compared to all the SF-clones today.
Also they can beat most humans.

Google is doing a lot of bad things (see this thread ), but not relating engines.
(Still it's by the USER, exactly by YOU to publish as many engines as possible, new ones, old ones, strong ones, beginner friendly ones, fantastic style playing ones, ...)

Many thanks to Karl, for one of only a few persons, publishing engines on google play store, even all for free.

[abik]

This error is incomprehensible.

Cryptic as it may be, this is the direct result of the W^X change, which blocks exec() on application data files starting at API 29. This is why formats like OEX and chessbase compatible format (when done right!) still work, all these formats ship the binaries natively pre-installed.

Speaking of the Chessbase compatible format, I am curious if this format is still used (other than by my own engine package). Does anyone have commercial Chessbase apps installed on their device, and sees the engines (like Fritz) in the import feature of Chess for Android?

The biggest problem for me with these new file accesses in SDK >= 29 ....

Yes, you bring up an on one hand unrelated (to W^X) but on the other hand related (to API 29) and very important other upcoming change. This is going to be challenging....

[Ras]

there are different technics of embed files.
SF is doing it in an fantastic way (I didn't went deep into it and won't do it ), somehow able using it direct inside the code.

You can compile binary data right into the executable. There's even the Linux tool "xxd" which can convert any binary data file into a C source file.

My engine does something similar (though with a custom tool) with its opening book so that it doesn't need file system access for that. The drawback is of course that updating the data requires recompiling the engine, that's why this technique isn't more widespread.

[abik]

Perhaps I should start a new thread, but this is tangibly related to the old topic: the now pending mandatory move to API30 forced by Google Play (exactly one year after the mandatory move to API29 that prompted my initial post of this thread) is no picnic either.

Importing engines from SD card was made impossible by API29, and API30 requires some minor tweaks to keep importing from an engine package working (nothing major but still something you have to change in GUIs that want to keep supporting this). More serious are the new restrictions on requesting file access. My recent update was rejected because loading/saving PGN games was not considered a "core" functionality of a chess app (sic! sic! sic!). I had requested general file access permissions, since users can enable and disable these permissions anyway if they don't trust an app. But this was considered too much of a sledgehammer, so I will have to investigate the more restricted permission model (which at first glance, seems to focus on video, audio, and photos, not so much on PGN files). As much as I love my company, I am less enthusiastic about the direction Google Play is taking. This is no fun for developers. Anyway, other programmers, please share your experience. I will also post updates here as I make progress in PGN support for API30 onwards.