rigacci.org

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Creating a Kivy app for Android 11 on Debian 12 Bookworm ======

**[[https://kivy.org/|Kivy]]** is a free and open source Python framework for developing mobile apps, with a single codebase, you will be able to deploy apps on **Windows**, **Linux**, **macOS**, **iOS** and **Android**. In this page I will focus on builing an Android app using a GNU/Linux workstation running Debian 12.

The Android target will be **Android 11 Red Velvet Cake**, this is an aspect that should not be overlooked because starting from Android 10 a series of new features have been introduced by Google that have made the life of Android developers much more complicated. With Android 11 the situation has become further exacerbated.

Many pages on the internet that explain how to compile an Android package on Linux are quite dated; they often refer to the old open source Java SDK - OpenJDK version 8 or 9 - while the current Debian stable distribution ships with **OpenJDK 17**. Here I tried to revamp all the processes using a freshly installed **Debian 12 Bookworm** workstation.

===== Preparing Debian 12 =====

Actually I used a KVM virtual machine to install Debian 12. I allocated a virtual host with 4 Gb of RAM, 2 CPU, 16 Gb of disk space and 1 Gb of swap. While the CPU and RAM specs are acceptable (I can compile my test app in less than one minute), consider to allocate **more hard disk space**. The bare **operating system** required about **4 Gb** of space, but the building environment required about **8 Gb** of space for a simple 30 kb Kivy script!

^ Minimal Requirements  ^^
^ CPU  | Intel i3 2.9 GHz, 2 Cores  |
^ RAM  | 4 Gb  |
^ Hard Disk  | 20 Gb  |

During the install I opted for a tex-only workstation (**no desktop environment**), but with the SSH server so I can login remotely. Once the installation was terminated, I added the following packages:

<code>
apt install git zip unzip openjdk-17-jdk python3-pip autoconf libtool pkg-config \
        zlib1g-dev libncurses5-dev libncursesw5-dev libtinfo5 cmake libffi-dev \
        libssl-dev python3-full lld
</code>

We opted to leave the overall system as clean as possibile, so all **the developing environmente** will be **installed in user space**, as explained in the following paragraphs.

==== Solving the pip3 install problem ====

Starting with version 12 Bookworm, Debian adheres to the **[[https://peps.python.org/pep-0668/|Python Enhancement Proposal 668]]** titled //Marking Python base environments as “externally managed”//, so if an unprivileged user tries to install a Python package in its user space $HOME directory, he gets the following error:

<code>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python
installation or OS distribution provider. You can override this, at
the risk of breaking your Python installation or OS, by passing
--break-system-packages.
hint: See PEP 668 for the detailed specification.
</code>

This means that the Python user space should be remain under the Debian package management system; **ading Python packages using pip3** (e.g. downloaded and installed by the user) **does not work any more**. This is a problem!

Python PEP668 suggests to use the new **//venv// method** to install Python libraries in user space, instead of the old plain **pip3** method, but this requires you to update all your habits and automation scripts. In particular there is the problem that some scripts downloaded and executed during the setup process will launch the command ''%%pip install --user ...%%'', but the ''%%--user%%'' option is not compatible with the //venv// method, you will get the error:

<code>
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
</code>

The packaging tool **[[https://github.com/kivy/python-for-android|python-for-android]]** is required in our building framework, but the installation scripts will probably launch ''pip install python-for-android'', thus failing in a default Debian 12 system.

After all the most promising solution seems to be to revert to the older Debian 11 behaviour, disabling the **externally managed** constraint of Python.

Someone suggests to solve the problem by deleting the file **/usr/lib/python3.11/EXTERNALLY-MANAGED**, but this is highly discouraged, because you will remove a system file installed by the package **libpython3.11-stdlib**. If e.g. the package will be upgraded, the file will be installed again. The preferred solution is to override the externally-managed flag by creating the user config file **$HOME/.config/pip/pip.conf** with the following option:

<file>
[global]
break-system-packages = true
</file>


==== Using the new venv install method ====

:!: **WARNING**: Till now we did not use this new preferred method, because the package **python-for-android** uses the old ''pip install'' method. See below about the //[[#installation_in_user_space|user method]]// install.

Instead of disabling the externally-managed-environment flag or instead of using the pip3 ''%%--user%%'' install method, it should be possible to use the venv pip3 install method. You need to install the **python3-venv** Debian package. This recipe did not work for me, but should be investigated further:

<code bash>
python3 -m venv $HOME/python-buildozer
# Activate the venv in this shell session:
source $HOME/python-buildozer/bin/activate
$HOME/python-buildozer/bin/pip3 install --upgrade buildozer
$HOME/python-buildozer/bin/pip3 install --upgrade Cython==0.29.33 virtualenv
</code>

Basically a //venv// is a Python environment which is separated from others you may eventually create and which is separated from the system-wide Python environment. You select in what venv you want to work by just sourcing the ''activate'' script at the prompt of your working shell: **''source python-buildozer/bin/activate''**. The script will change your PATH so if you launch e.g. ''python'' you will get the Python environment with the modules installed only in that venv.


==== Installation in user space ====

**NOTICE**: We used the //user method// to install the Python packages; this require to disable the Debian 12 Python //externally-managed-environment//. The use of the new //venv method// shuld be instead investigated. See the above paragraphs.

The entire building framework is composed of several pieces which we will install from direct internet download. This is mandatory because many of them are not included into the Debian stable distribution (or they are obsolete).

  * **[[https://buildozer.readthedocs.io/en/latest/|Buildozer]]** - Buildozer is a tool that aim to package mobiles application easily. It automates the entire build process, download the prerequisites like python-for-android, Android SDK, NDK, etc.
  * **[[https://python-for-android.readthedocs.io/en/latest/|python-for-android]]** - Let you package Python code into standalone android APKs.
  * **[[https://developer.android.com/ndk|Android NDK]]** - The Android NDK is a toolset that lets you implement parts of your app in native code, using languages such as C and C++.
  * **[[https://developer.android.com/tools|Android Commandline Tools for Linux]]** - It is a part of the Android SDK that can be downloaded and installed separately, it contains for example [[https://developer.android.com/tools/sdkmanager|sdkmanager]], [[https://developer.android.com/tools/apkanalyzer|apkanalyzer]], etc.

The firs installation step in user space is to install the **buildozer** Python package from the internet. We will use the **user scheme**, i.e. the package will be installed into the user space **$HOME/.local/lib/python3.11/site-packages**:

<code>
pip3 install --user --upgrade buildozer
</code>

The package will insall also some executable scripts into the **$HOME/.local/bin/** directory, you must add this into your PATH environment variable to have access to them (Debian should do this for you at login, if the directory exists).

Then you must install the **Cython** and **virtualenv** Python packages, again from the internet. **NOTICE**: //venv// and //virtualenv// are similar in functionality but differ in implementation; we required a specific version of the //Cython// module because the Buildozer reccomend that.

<code>
pip3 install --user --upgrade Cython==0.29.33 virtualenv
</code>

Also the Cython package will install some executables into **$HOME/.local/bin/**.

All the remaining components of the building framework (Python libraries, Android SDK, etc.) will be automatically downloaded by the Buildozer program, read the following paragraphs.

==== Prepare the Kivy project in user space ====

:!: **INFO**: This procedure is **required only once** when we start a new projec.

Now it is time to prepare the Kivy project directory for the build of the Android package. You must check the following:

  - Create a **subdirectory** where to store all the project files (Python code, images, fonts, json files, etc.).
  - The program entry point must be called **main.py** and reside into the root directory of the project.
  - You must install all the Python modules required by the program in your environment (//user// or //venv//).
  - Your building path **must not contain** the **icc** substring :-O, otherwise you will get the error **%%no such file or directory: 'strict'%%**

I created the directory **openolyimageshare** for my test app, then I execute the following:

<code>
cd openolyimageshare
buildozer init
</code>

The init process is rather time and space consuming. A good internet connection is required because it needs to download **[[https://github.com/kivy/python-for-android.git|python-for-android]]**, the **[[https://dl.google.com/android/repository/commandlinetools-linux-6514223_latest.zip|commandlinetools-linux]]**, the **[[https://dl.google.com/android/repository/android-ndk-r25b-linux.zip|Android NDK]]** and so on. My simple test app required the following space in the following directory:

  * **$HOME/.buildozer/** - About 2.6 Gb
  * **$HOME/openolyimageshare/.buildozer/** - About 3.5 Gb

During the **buildozer init** process you could run into a really sneaky problem caused by your working directory containing the **icc** sitring (I ran in this problem because my $HOME is /home/n**icc**olo/):

<code>
unknown argument: '-fp-model'
clang-14clang-14: : error: error: no such file or directory: 'strict'no such file or directory: 'strict'
</code>

Fortunately enough I found the **[[https://github.com/kivy/python-for-android/issues/2329|Kivy issue #2329]]**, which explain the problem. I renamed my HOME directory as a workaround.

==== Writing the buildozer.spec file ====

Now that the builing framework is prepared, it is time to edit the **buildozer.spec** file that you will find into the project directory. Here you put the information required to compile the Android package. Here I resume the sections that I modified:

<file>
[app]
title = Open Oly ImageShare
package.name = openolyimageshare
package.domain = org.rigacci
source.include_exts = py,png,jpg,kv,atlas,ttf,json
source.include_patterns = res/fonts/*,res/img/*,res/layout/*
version = 0.1
requirements = python3,kivy,requests
presplash.filename = %(source.dir)s/data/presplash.png
icon.filename = %(source.dir)s/data/icon.png
android.permissions = android.permission.INTERNET,
    android.permission.READ_EXTERNAL_STORAGE,
    android.permission.WRITE_EXTERNAL_STORAGE,
    android.permission.CAMERA
</file>

The **title** is the app name that will be shown by the Android system; it will be listed under the //Settings// => //Apps//, it will be used as the label below the app icon, etc.

The **package.domain** and **package.name** will be concatenated to form a globally unique identifier of your app. You should use an actual internet domain name that you own (reversing the order of the hierarchy, i.e. the Tope Level Domain must come first) and an unique name under your domain. Into the Android system your program will be installed under a directory named **%%[package.domain].[package.name]%%**, so that name should be all lowercase, with no spaces, no special chars, etc.

With **source.include_exts** and **source.include_patterns** you tell to Buildozer what files must be included into the package. Including a directory does not automatically include all the files within, you still must indicate the extensions.

In **requirements** you must declare which Python packages are required by your code. Chek all the ''import'' statements in your code and list only the ones which are not into the default library (the ones that you installed via Debian package or pip).

You can use **presplash.filename** and **icon.filename** to include two artwork in your app. The //presplash// will be displayed at startup, during the initialization of the environment (unfortunately it is rather time consuming). The //icon// is instead what you can imagine. Use PNG graphics at least 512 x 512 pixels, you can use transparency too.

In **android.permissions** you must list all the permissions that your app will require from the operating system. If you forget to declare something your app simply will not be able to do that operation. Beware that starting from Android 10 the access to the external storage (basically the space into the SD card or into the device memory) has undergone a drastic change, see the table below for a basic overview.

^ READ_EXTERNAL_STORAGE   | This was the long-established permission required by the apps to read the [[https://android.googlesource.com/platform/docs/source.android.com/+/android-4.2.1_r1.1/src/tech/storage/index.md|external storage]] (a permissionless filesystem residing into the SD card or into a dedicated partition of the device internal storage). If you want e.g. to read the pictures from the DCIM folder of an **Android 8** device, you must grant this permission. This is not longer true starting from **Android 10** (API level 29) which introduced the **scoped storage**, designed to protect app and user data and reduce file clutter; requesting ''READ_EXTERNAL_STORAGE'' in Android 10 actually means requesting access only to **photos and media**. Afterwards **Android 11** (API level 30) fixed several problems with that implementation; in Android 11 and above requesting ''READ_EXTERNAL_STORAGE'' does not give you any actual permission. Generally, in Android 11, an app cannot access the root directory of the SD card and the Download directory, it can access only its ASD (App Specific Directory).  |
^ WRITE_EXTERNAL_STORAGE  | The same as ''READ_EXTERNAL_STORAGE'', but for write permission.  |
^ CAMERA                  | Writing .jpg or similar media files under the Android shared folders **DCIM** or **Pictures** (or subfolders therein), is always permitted; no particular permission is required even in Android 11. Requesting the CAMERA permission allow to use the camera itself, the flash, etc.  |
^ MANAGE_EXTERNAL_STORAGE  | Starting from Android 11, this is the permission required to access all the files into the storage. The app should also provide an ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION intent. App requesting this permission may have trouble getting into the Google Play Store.  |

=== Target API vs Minimum API vs Compile SDK ===

Into the **buildozer.spec** you can define the following:

<file>
# (int) Target Android API, should be as high as possible.
#android.api = 31

# (int) Minimum API your APK / AAB will support.
#android.minapi = 21

# (int) Android SDK version to use
#android.sdk = 20
</file>

By setting a **target API** lower than your SDK can support, you can declare what will be the highest Android version that your app is designed to be compatible with. At the moment Buildozer will try to target API 31 (**Android 12**).

Declaring a **minimum API** you can tell what is the minimum Android version required by your app. We did not declared the android.minapi, so Buildozer choosed API 21 (**Android 5.0.2**).

The **compile SDK** is the environment you want to use to create the app, i.e. the SDK you downloaded from Google (which generally support the higher API available at the moment). This will affects what functions and constructs you can use in your program. If you do not specify a version, Buildozer should detect the highest SDK downloaded and use it. In our case only one SDK was downloaded, and it was SDK API 31.

==== Compiling the package (debug or release) ====

:!: **INFO**: This is **the only procedure** to be executed whenever you want to compile a new version; the **version number** must be updated into the **main.py** source code.

Enter the project directory and edit the **main.py** source code updating the definition of the **%%__version__%%** variable (the **buildozer.spec** will refer this value to create the package name). Then choose to make a debug build:

<code>
buildozer android debug
</code>

The package will be created into the **bin/** subdirectory, using a name like **packagename-version-arm64-v8a_armeabi-v7a-debug.apk**.

When you are ready to **publish** your package, you must create the //release// binary:

<code>
buildozer android release
</code>

In this case the package created into the **bin/** subdirectory will be named like **packagename-version-arm64-v8a_armeabi-v7a-release.aab**, which is not directly installable into the device.


===== Web Resources =====

  * **[[https://buildozer.readthedocs.io/en/latest/installation.html|Buildozer Installation]]**
  * **[[https://github.com/kivy/python-for-android|python-for-android]]**
  * **[[https://towardsdatascience.com/3-ways-to-convert-python-app-into-apk-77f4c9cd55af|3 Ways to Convert Python App into APK]]**
  * **[[https://kivy.org/doc/stable/guide/packaging-android.html|Programming Guide - Create a package for Android]]**


  * **[[https://peps.python.org/pep-0668/|Python Enhancement Proposal 668]]**
  * **[[https://medium.com/@kiena/troubleshooting-externally-managed-environment-error-in-debian-12-pip3-installation-439d62e5a970|Troubleshooting ‘Externally Managed Environment’ Error in Debian 12 Pip3 Installation]]**
  * **[[https://stackoverflow.com/questions/75602063/pip-install-r-requirements-txt-is-failing-this-environment-is-externally-mana|pip install -r requirements.txt is failing: "This environment is externally managed"]]**
  * **[[https://stackoverflow.com/questions/75608323/how-do-i-solve-error-externally-managed-environment-every-time-i-use-pip-3|How do I solve "error: externally-managed-environment" every time I use pip 3?]]**
  * **[[https://www.jeffgeerling.com/blog/2023/how-solve-error-externally-managed-environment-when-installing-pip3|How to solve "error: externally-managed-environment" when installing via pip3]]**

  * **[[https://github.com/kivy/python-for-android/issues/2329|-fp-model argument not found, directory strict not found]]**

  * **[[https://developer.android.com/about/versions/11/privacy/storage|Storage updates in Android 11]]**
  * **[[https://stackoverflow.com/questions/64221188/write-external-storage-when-targeting-android-10|WRITE_EXTERNAL_STORAGE when targeting Android 10]]**
  * **[[https://community.appinventor.mit.edu/t/how-to-access-non-media-media-files-on-android-11/54828|How to access non-media & media files on Android 11+]]**
  * **[[https://community.appinventor.mit.edu/t/asd-app-specific-directory-vs-private-folder/19154|ASD (app specific directory) vs Private folder]]**
  * **[[https://developer.android.com/training/data-storage/shared/media|Access media files from shared storage]]**