Android behavior change: no more W^X violations

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
abik
Posts: 770
Joined: Fri Dec 01, 2006 9:46 pm
Location: Mountain View, CA, USA
Full name: Aart Bik
Contact:

Android behavior change: no more W^X violations

Post by abik » Sat Dec 05, 2020 3:03 am

Argh!

This Android 10 behavior change is likely to drastically change the chess eco system on Android. In a nutshell, this no longer allows installing and running arbitrary engine binaries into a chess app. Pre-packaged binaries may still run, but it seems the good times of just putting arm binaries on SD card and install and import them in Chess for Android are over.

This impacts all new apps that target API 29. Furthermore, Google Play requires all new apps and app updates to target API level 29 or higher. So, no way around this restriction through the regular distribution channels.

I continue to investigate the issue, but in the meantime, did other chess app developers for Android encounter this issue already?

User avatar
MikeB
Posts: 4840
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Android behavior change: no more W^X violations

Post by MikeB » Sat Dec 05, 2020 4:01 am

abik wrote:
Sat Dec 05, 2020 3:03 am
Argh!

This Android 10 behavior change is likely to drastically change the chess eco system on Android. In a nutshell, this no longer allows installing and running arbitrary engine binaries into a chess app. Pre-packaged binaries may still run, but it seems the good times of just putting arm binaries on SD card and install and import them in Chess for Android are over.

This impacts all new apps that target API 29. Furthermore, Google Play requires all new apps and app updates to target API level 29 or higher. So, no way around this restriction through the regular distribution channels.

I continue to investigate the issue, but in the meantime, did other chess app developers for Android encounter this issue already?
Awful, no need for that... I was aware of that shortly after starting working on Black Diamond( Droidfish clone) ,and made the simple decision not to even bother with the playstore. Also time to turn off automatic OS updates
Image

User avatar
lucasart
Posts: 3218
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: Android behavior change: no more W^X violations

Post by lucasart » Sat Dec 05, 2020 7:33 am

abik wrote:
Sat Dec 05, 2020 3:03 am
Argh!

This Android 10 behavior change is likely to drastically change the chess eco system on Android. In a nutshell, this no longer allows installing and running arbitrary engine binaries into a chess app. Pre-packaged binaries may still run, but it seems the good times of just putting arm binaries on SD card and install and import them in Chess for Android are over.

This impacts all new apps that target API 29. Furthermore, Google Play requires all new apps and app updates to target API level 29 or higher. So, no way around this restriction through the regular distribution channels.

I continue to investigate the issue, but in the meantime, did other chess app developers for Android encounter this issue already?
This is a complete disaster. If I understand correctly, this "protection" is enforced by the operating system, not even Play Store. So you can't bypass it by using a Play Store replacement like Aptoid of Fdroid.

Next step, they will disable the possibility to download and install APK files, so you won't have any other choice than to use Play Store. Already they've put this choice out of reach for 99% non-technical users, by blocking by default APK installation, and hiding the option deep in the menus. Soon it will be impossible, and Play Store will become the evil twin of Apple's app store.

The whole point of Android to exist in the first place (when it was the underdog compared to Apple) was to be the opposite of Apple with their abusive monopoly. Sadly, the inevitable is slowly happening. Google is becoming more and more like the enemy... :shock:

It's brilliant how all this is done with the excuse of protecting the consumer. You know, we do this to protect you from all these "hackers" out there to get you. For the 99.9% non-technical users, this sounds credible enough...
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

User avatar
abik
Posts: 770
Joined: Fri Dec 01, 2006 9:46 pm
Location: Mountain View, CA, USA
Full name: Aart Bik
Contact:

Re: Android behavior change: no more W^X violations

Post by abik » Sat Dec 05, 2020 8:12 am

lucasart wrote:
Sat Dec 05, 2020 7:33 am
If I understand correctly, this "protection" is enforced by the operating system, not even Play Store.
That is correct. The protection is done by Android. But Google Play enforces that new apps or updates to existing apps target API 29. I just made an update with some unrelated changes and bumped the target. And now I am being flooded with complaints from angry users who can no longer install engines. I also cannot revert back to the previous version.... :-(

Pre-packaged engine binaries still work, so perhaps that is the only path forward. As I research this more, I become less optimistic that the install from SD will work again with normal distribution (viz. without sideloading or rooting).

But stay tuned though as I ask around some more.
Last edited by abik on Sat Dec 05, 2020 8:15 am, edited 1 time in total.

mar
Posts: 2494
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Android behavior change: no more W^X violations

Post by mar » Sat Dec 05, 2020 8:15 am

Sounds like what Apple has been doing for a very long time. Welcome to golden cage, droid users :)

Next step is to prevent allocating executable pages and custom JITting.

Apps are sandboxed, so this is not at all about security, of course.
Martin Sedlak

User avatar
abik
Posts: 770
Joined: Fri Dec 01, 2006 9:46 pm
Location: Mountain View, CA, USA
Full name: Aart Bik
Contact:

Re: Android behavior change: no more W^X violations

Post by abik » Sat Dec 05, 2020 8:28 am

mar wrote:
Sat Dec 05, 2020 8:15 am
Sounds like what Apple has been doing for a very long time. Welcome to golden cage, droid users :)
When I started back in 2010 as Android Pioneer looking into importing UCI engines into my chess GUI, lots of folks told me that could not be done pointing at the Apple model. So I was ecstatic when I actually found a way to do this by installing from SD card (the mechanism of prepackaging with OEX or Chessbase compatible format followed shortly after that).

So I would be extremely sad if 10 years later I have to say bye bye to this cool feature again..

mar
Posts: 2494
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Android behavior change: no more W^X violations

Post by mar » Sat Dec 05, 2020 8:40 am

abik wrote:
Sat Dec 05, 2020 8:28 am
When I started back in 2010 as Android Pioneer looking into importing UCI engines into my chess GUI, lots of folks told me that could not be done pointing at the Apple model. So I was ecstatic when I actually found a way to do this by installing from SD card (the mechanism of prepackaging with OEX or Chessbase compatible format followed shortly after that).

So I would be extremely sad if 10 years later I have to say bye bye to this cool feature again..
I understand the frustration.
If there's no workaround for this then that would be very sad indeed.

I've spent some quality time with Chess for Android in the past,
thanks for this amazing program, you've made a lot of people happy in the past 10 years.
Martin Sedlak

User avatar
abik
Posts: 770
Joined: Fri Dec 01, 2006 9:46 pm
Location: Mountain View, CA, USA
Full name: Aart Bik
Contact:

Re: Android behavior change: no more W^X violations

Post by abik » Sat Dec 05, 2020 8:59 am

mar wrote:
Sat Dec 05, 2020 8:40 am
I've spent some quality time with Chess for Android in the past,
thanks for this amazing program, you've made a lot of people happy in the past 10 years.
Thanks for your kind words!

Note that we had a small security-related crisis before with the introduction of Lollipop, but that one was luckily easily resolved by compiling the engines differenty. Now I am afraid we will be less lucky...

User avatar
lucasart
Posts: 3218
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: Android behavior change: no more W^X violations

Post by lucasart » Sat Dec 05, 2020 9:36 am

abik wrote:
Sat Dec 05, 2020 8:59 am
mar wrote:
Sat Dec 05, 2020 8:40 am
I've spent some quality time with Chess for Android in the past,
thanks for this amazing program, you've made a lot of people happy in the past 10 years.
Thanks for your kind words!

Note that we had a small security-related crisis before with the introduction of Lollipop, but that one was luckily easily resolved by compiling the engines differenty. Now I am afraid we will be less lucky...
So if the exec() syscall is now forbidden, what is the alternative that Google proposes ?
Should we completely review the UCI model of pipe communication between processes, and instead have engines be compiled as dynamic libraries ? Or will that also be banned for security reasons ?
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Ras
Posts: 1947
Joined: Tue Aug 30, 2016 6:19 pm
Full name: Rasmus Althoff
Contact:

Re: Android behavior change: no more W^X violations

Post by Ras » Sat Dec 05, 2020 12:34 pm

abik wrote:
Sat Dec 05, 2020 3:03 am
I continue to investigate the issue, but in the meantime, did other chess app developers for Android encounter this issue already?
Yes. From Peter Osterlund's Droidfish repo referring to that problem:
As far as I know it is not possible to implement all functionality currently in DroidFish using targetSdkVersion 29, so the only option is to use targetSdkVersion 28 and don't distribute the app in the play store.
Source: https://github.com/peterosterlund2/droidfish/issues/59

Solution: don't use API 29 or higher. Avoid Google's play store and use either the F-Droid store or sideloading APKs.
Rasmus Althoff
https://www.ct800.net

Post Reply