Compiling for Android with Android NDK and ndk-build

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Archimedes
Posts: 135
Joined: Tue Mar 05, 2019 3:43 pm
Full name: Archimedes

Compiling for Android with Android NDK and ndk-build

Post by Archimedes »

Should be public. :wink:

When using Android NDK, there are three ways to produce android builds: ndk-build, cmake or standalone toolchains.

The following (very short) description will show you, how it works with (the command line tool) ndk-build. I'm using ndk-build, because i'm not delivering a software (app) for different environments, i'm only interested in compiling a source code (c or c++) to android. No more, no less.

But before we can start, we need the Android NDK. It can be downloaded from https://developer.android.com/ndk/downloads. Choose the right version for you. Unzip the zip archive to a directory of your choice and put the root directory of your android-ndk directory to your search path (the command line tool ndk-build should start from the command line in every directory). In Windows you enter the path to the system or user path, in Linux you may add the following lines at the end of the file „.bashrc“ in your home directory (assuming you unzip the content of the zip archiv to usr/lib/android-ndk), so the path is active, when you start the terminal.

Code: Select all

export PATH=$PATH:/usr/lib/android-ndk
OK, now we are ready to go. The following examples works in Windows and in Linux.

Let us begin with an current project, which is easy to compile: Demolito. :D

1. Download the sources from https://github.com/lucasart/Demolito. If you donwload the sources as zip archive, unzip the files to a directory of your choice (e. g. Demolito).

2. When you go in to the root directory of Demolito, you will see a src directory. Create a directory named jni in the root directory of Demolito and put the following file in to that directory.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Demolito
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += -DVERSION=\"2020-10-10\"
include $(BUILD_EXECUTABLE)
As you can see, you need only one file, named Android.mk, and a few lines of code to make Android builds with ndk-build from Android NDK. All lines above are necessary. For further information, what they means, please inform you at https://developer.android.com/ndk/.

For us, line 3, 4 and 5 are necessary.

Line 3: Here you define a name for your builds (executables). In our case we choose the name Demolito.
Line 4: Here are all source files listed, which have to be compiled. In our case we want to compile all *.c files in the src directory.
Line 5: Here you can define additional compiler flags for the compiling process. In your case we only need the parameter „VERSION“, which is used in the source code of Demolito for showing the correct version number. No more parameter needed. Android NDK, i mean ndk-build, will do the rest (in most cases, ndk-build choose the right parameter for you.)

3. Start the command line and change to the jni directory and call ndk-built. The compiling process is starting and when there is no error, the executables will be found in the libs directory.

4. Ready.

OK, this is a very basic example, which works with the default settings of Android NDK.

Normally, we have to add a few more compiler settings. Let us describe some of them with the Demolito example.

When we take a look at the makefile in the src directory, we can see, that the compiler language of the project is gnu11. Normally ndk-build find this out automatically for you, but in some cases, this won‘t work. So we add this information to the Android.mk file, so the compiler knows, in what compiler language it should compile the project.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Demolito
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += -std=gnu11 -DVERSION=\"2020-10-10\"
include $(BUILD_EXECUTABLE)
Also, when we want some optimizations regarding runtime performance, we can add additional code optimizations with the -O3 compiler flag (default with ndk-build is -O2). This will give us a very small speed up. If you want a bigger speed up, you can use link time optimization with the compiler and linking flag -flto. To make the story short, here the resulted file.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Demolito
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += -std=gnu11 -O3 -flto -DVERSION=\"2020-10-10\"
LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
With this configuration you will receive compatible and fast compiles of Demolito for Android devices. Further optimizations can be don with profile guide optimizations (pgo) and so on. But be carefull with additional (optimized) compiler flags. :wink:

For completeness, i‘m using the following Android.mk file for compiling Demolito.

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Demolito
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += -std=gnu11 -DNDEBUG -O3 -flto -Wfatal-errors -Wall -Wextra -Wshadow -DVERSION=\"2020-10-10\"
LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
The -W flags will show you only information during the build process. Normally they only necessary for the developer, so they can see, what code they could optimize and so on.

With ndk-build you have not only the file Android.mk for setting compiling options, there is also a file named Application.mk which can be used to tell the compiler additional instructions.

For example, if you want, that your builds are compatible with the Android API level 16, you have to put the following line in to Application.mk (which should be also in the jni directory).

Application.mk:

Code: Select all

APP_PLATFORM := android-16
However, this setting is normally not necessary, because it‘s the default setting, when using ndk-build. :wink:

This example shows, all what you need for compiling to Android is, an installed Android NDK (root directory should be in the search path), two files, named Android.mk and Application.mk, with the correct settings, in the jni directory. The only thing you have to do is, calling ndk-build from the command line within the jni directory.

Keep in mind, this is only a very short and by far not complete description, but it shows the idea behind of cross compiling with ndk-build from Android NDK.
Archimedes
Posts: 135
Joined: Tue Mar 05, 2019 3:43 pm
Full name: Archimedes

Re: Compiling for Android with Android NDK and ndk-build

Post by Archimedes »

Here are some examples of Android.mk and Application.mk from another projects (not further optimized, to show you the idea).

Phalanx needs the following settings.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Phalanx
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += --std=gnu89 -DNDEBUG -O3 -flto -Wfatal-errors -Wall -Wextra -Wshadow
LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
Very similar to Demolito. Only the name of the executables and the compiler language has changed. A special version parameter, as used in the source code of Demolito, is not necessary.

Application.mk:

Code: Select all

APP_PLATFORM := android-16
Nothing special here.

Now let‘s see, how the files looks like for the Texel project, which is written in the c++ language.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Texel
LOCAL_C_INCLUDES := ../lib/texellib ../lib/texellib/gtb/compression ../lib/texellib/gtb/compression/lzma ../lib/texellib/gtb/sysport
LOCAL_SRC_FILES := $(wildcard ../app/texel/*.cpp ../lib/texellib/*.cpp ../lib/texellib/util/*.cpp ../lib/texellib/syzygy/*.cpp ../lib/texellib/gtb/*.c ../lib/texellib/gtb/sysport/*.c ../lib/texellib/gtb/compression/*.c ../lib/texellib/gtb/compression/lzma/*.c)
LOCAL_CFLAGS += -DNDEBUG -O3 -flto -fexceptions -Wfatal-errors -Wall -Wextra -Wshadow
LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
As you can see in line 5 (LOCAL_SRC_FILES), you have a lot of more locations, where the source files are located (the compiler must know them). And in line 4 (LOCAL_C_INCLUDES) you listed all the include directories. The compiler sometimes not find additional files (mostly header files) during compilation process. This is the right place to tell the compiler, where to find them. In line 6 (LOCAL_CFLAGS) you will find an new parameter -fexceptions. This is because the source code contains exceptions and the parameter tells the compiler to use them, otherwise it will bring up an error.

Application.mk:

Code: Select all

APP_PLATFORM := android-16
APP_STL := c++_static
Last line is needed for c++, it tells the compiler to include the required libc++ library as static library.
Last edited by Archimedes on Wed Dec 23, 2020 12:30 pm, edited 1 time in total.
Archimedes
Posts: 135
Joined: Tue Mar 05, 2019 3:43 pm
Full name: Archimedes

Re: Compiling for Android with Android NDK and ndk-build

Post by Archimedes »

In the meanwhile, i'm using always the same structure in the files Android.mk and Application.mk. The layout of the two files are always the same. So it's easier for me, when dealing with different chess engines.

As an example, see, how the files looks like for Demolito now.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Demolito
LOCAL_C_INCLUDES := 
LOCAL_SRC_FILES := $(wildcard ../src/*.c)
LOCAL_CFLAGS += -std=gnu11 -DNDEBUG -O3 -flto -fno-stack-protector -fomit-frame-pointer -Wfatal-errors -Wall -Wextra -Wshadow -DVERSION=\"2020-10-10\"

ifeq ($(TARGET_ARCH_ABI), arm64-v8a)
  LOCAL_CFLAGS += 
else
  ifeq ($(TARGET_ARCH_ABI), armeabi-v7a)
    LOCAL_CFLAGS += 
  else
    ifeq ($(TARGET_ARCH_ABI), x86)
      LOCAL_CFLAGS += -mssse3 -mfpmath=sse -m32
    else
      ifeq ($(TARGET_ARCH_ABI), x86_64)
        LOCAL_CFLAGS += -msse4.2 -mpopcnt -m64
      endif
    endif
  endif
endif

LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
As you can see, i'm using -O3, -flto, -fno-stack-protector and -fomit-frame-pointer as target independent compiler settings now and i have an extra if-then construction for dealing with different architecture compiler settings. The if-then construct you see, is the same for all chess engines. Some chess engines needs extra compiler definitions (e. g. -DUSE_NEON), some not (like Demolito).

Application.mk:

Code: Select all

APP_ABI := arm64-v8a armeabi-v7a x86 x86_64
APP_PLATFORM := android-21
APP_STL := 
Here is an example, how the files looks like with the Scorpio engine.

Android.mk:

Code: Select all

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := Scorpio
LOCAL_C_INCLUDES := 
LOCAL_SRC_FILES := $(wildcard ../src/*.cpp)
LOCAL_CFLAGS += -std=c++11 -DNDEBUG -O3 -flto -fno-stack-protector -fomit-frame-pointer -Wfatal-errors -Wall -Wextra -Wshadow -DPARALLEL -DUSE_SPINLOCK

ifeq ($(TARGET_ARCH_ABI), arm64-v8a)
  LOCAL_CFLAGS += -DARC_64BIT -DHAS_POPCNT
else
  ifeq ($(TARGET_ARCH_ABI), armeabi-v7a)
    LOCAL_CFLAGS += 
  else
    ifeq ($(TARGET_ARCH_ABI), x86)
      LOCAL_CFLAGS += -mssse3 -mfpmath=sse -m32 -DHAS_PREFETCH
    else
      ifeq ($(TARGET_ARCH_ABI), x86_64)
        LOCAL_CFLAGS += -msse4.2 -mpopcnt -m64 -DARC_64BIT -DHAS_POPCNT -DHAS_PREFETCH
      endif
    endif
  endif
endif

LOCAL_LDFLAGS += -flto
include $(BUILD_EXECUTABLE)
The compiler definitions -DPARALLEL and -DUSE_SPINLOCK are used for all architectures (arm64-v8a, armeabi-v7a, x86 and x86_64). Architecture dependant compiler definitions are in the if-then construction. For example, -DHAS_PREFETCH can only be used for the x86 and x86_64 architecture.

Application.mk:

Code: Select all

APP_ABI := arm64-v8a armeabi-v7a x86 x86_64
APP_PLATFORM := android-21
APP_STL := c++_static
Now, you have an idea, how you can compile with Android NDK and ndk-build for Android devices. It's just a way of many others. But when you get the trick, you can do it with other solutions too, as you are able to deal with compiler settings now.
Archimedes
Posts: 135
Joined: Tue Mar 05, 2019 3:43 pm
Full name: Archimedes

Re: Compiling for Android with Android NDK and ndk-build

Post by Archimedes »

I always include the used Android.mk and Application.mk files in the zip archive, when i do a compile for Android. So you can always see, what compiler settings are used for compiling.

Chess Engines for Android

Keep in mind, Android NDK is using the clang compiler and with it, you have to change the source code very often for proper compiling. But most of the time this can be done in a minimal invasive way.
Pi4Chess
Posts: 253
Joined: Mon Nov 16, 2020 12:13 pm
Full name: Manuel Rivera

Re: Compiling for Android with Android NDK and ndk-build

Post by Pi4Chess »

Nice tutorial Archimedes !

Now we need a tutorial for How to create an Android app to put engines with OEX standard onto Google play store.
Seems the only way to install engines with last android upgrades. Chess for Android now cannot install/uninstall engine binaries and only sees the OEX ones previously installed.

I tried to not install last Android upgrade but after some days delaying it the system auto-ugraded in the middle of a night...
User avatar
AlexChess
Posts: 1494
Joined: Sat Feb 06, 2021 8:06 am
Full name: Alex Morales

Re: Compiling for Android with Android NDK and ndk-build

Post by AlexChess »

Pi4Chess wrote: Wed Dec 23, 2020 2:27 pm Nice tutorial Archimedes !

Now we need a tutorial for How to create an Android app to put engines with OEX standard onto Google play store.
Seems the only way to install engines with last android upgrades. Chess for Android now cannot install/uninstall engine binaries and only sees the OEX ones previously installed.

I tried to not install last Android upgrade but after some days delaying it the system auto-ugraded in the middle of a night...
I agree with you!

Engines not compatible with Chess on Android 6.2.1 like Koivisto, Igel 3.0.10 and Scorpio 3 would probably work on Chess for All that (let engine-engine matches, too) like they work on Droidchess for analisys only. An OEX collection of all Archimedes's wonderful and instantly updated engines would work with all 3 IDEs :D

PS: I've disabled auto-update and now Chess on Android 6.2.1 doesn't install the new version overnight.
Chess engines and dedicated chess computers fan since 1981 :D Mac mini M1 8GB-256GB, Windows 11 & Ubuntu ARM64.
ProteusSF Dev Forum TROLLS KINDERGARTEN