Creating a Streamlit Executable
Contents
Creating a Streamlit Executable#
Document overview#
This document details the process undergone to create an Electron application that embeds streamlit. The result is an easily shareable binary built for each of the major operating systems.
This document refers to the state of the PyMedPhys code base as at commit hash 836f272d092f294099bb51db05bab80d2bfcb628. All code links will be pointing to the code base at that commit hash.
Quick version#
To actually build and create the PyMedPhys Streamlit binary all you actually need to do is the following:
Install Poetry
Install PyOxidizer by installing all the project build dependencies:
poetry install -E build -E cli
Then run
poetry run pymedphys dev build --install
Which runs this under the hood.
The final built application for your current OS will be contained within
js/app/dist
.
If you would like to dig deeper into what the above is doing under the hood, or you would like to adapt PyMedPhys’ approach to your own Streamlit project, then read on.
Approach overview#
To create this binary build four key steps were undergone:
Create a completely system independent self contained Python environment
Set up the electron desktop application wrapper
Make the desktop application boot and connect to the streamlit running within the self contained Python environment.
Use electron builder to create an installer, run this within GitHub actions across the three major OSs.
Further detail into each of these steps along with the corresponding code is given below. Also, if you’d like help achieving any of this, some details around that are provided at the end of this document.
Creating a self contained PyMedPhys python bundle#
Justification#
To create the self contained Python environment PyOxidizer was utilised. This tool contains a range of embeddable Python distributions for each OS. There are a range of issues with a standard Python install, venv or otherwise, that make it hard to pick up the Python install off one computer onto another computer in a portable fashion. One such issue is that the standard Python install explicitly hardcodes the install location. Therefore, subsequent moving of the installation causes issues.
There are many tools that can create embedded Python installations. PyInstaller is one, however I have had issues where it gets flagged as the created program gets flagged as a virus on Windows. It also, in the name of trying to make a small install size, by default strips out a lot of dependencies. This “stripping out” makes the resulting binary have many weird edge case issues.
At the end of the day, the goal was to utilise a tool that creates an embedded Python distribution that is as close to “standard” as possible. PyOxidizer can be configured to achieve that, so that was utilised.
The pyoxidizer.bzl
configuration file#
The configuration file for building the binary can be found at pyoxidizer.bzl.
We’ll go into the key details that make this configuration file different here. Further details about the PyOxidizer configuration file can be found in the official docs.
PyOxidizer comes with a range of amazing tooling to create custom binaries. We however will be mostly turning all of this amazing tooling off for maximum compatibility with a standard Python installation. The key steps are the following.
Files mode#
Make resource handling be set to “files” mode (needed for NumPy at least) #L20
policy.set_resource_handling_mode("files")
File system relative resources with site-packages name#
Use filesystem-relative for the resource location, and use a filename of “site-packages” #L22-L24
policy.resources_location = "filesystem-relative:site-packages"
A specific fix for Streamlit is the need to use “site-packages” within the resource location (#L22-L27). This is due to Streamlit using that file directory location as an indicator that Streamlit is not running in developer mode.
Use standard importer#
Disable the PyOxidizer importer #L30-L31
python_config.filesystem_importer = True
python_config.oxidized_importer = False
Running the build#
Once that configuration file is created, and once PyOxidizer is installed then
running poetry run pyoxidizer build install
will create a pymedphys
binary
within a dist
directory within the repository root.
The use and setting up of Electron#
Justification#
Now that we have a self contained PyMedPhys distribution, arguably we could be done. Also, arguably, bringing in the heavy beast of Electron potentially complicates things significantly. However, it does have a few benefits. The primary benefits being that electron builder manages the creation of the installer as well as management of auto-update capacity, if we want to support that in the future.
PyOxidizer does also manage the creation of the installer. However, with my trialling of it in the past this installer was only able to run as administrator on Windows, which is likely a deal breaker in many scenarios.
Instead of using Electron we could manually handle the installer ourselves. But then we’d have to do that independently for each OS. By using electron, we can run the same set up on one OS, and have it automatically create installers on each target OS with little to no adjustments needed. Having a bulky executable created at the other side is a reasonable trade off in my opinion for significantly reduced maintenance burden on PyMedPhys.
Setting up Electron#
To set up the Electron code base I began with the following boilerplate code:
I significantly stripped it back so as to include as little as possible while still having success.
The resulting Electron application code utilised can be found at
js/app
.
To install all of the required dependencies run yarn install
within the
js/app
directory. You will need to have both Node
and Yarn 1.x
installed to achieve this.
The key components of the Electron code base#
Below is a short overview of the key components of the Electron application code base.
The package.json
file#
A resources directory called python
is selected and included within the
build:
"directories": {
"buildResources": "resources"
},
"extraResources": [
{
"from": "python",
"to": "python",
"filter": [
"**/*"
]
}
],
This directory doesn’t exist in the source tree. When the Electron app is being
built with poetry run pymedphys dev build
this python
directory is created
by running pyoxidizer
and then moving the resulting built distribution over
to js/app/python
:
PYOXIDIZER_BUILD = REPO_ROOT.joinpath("build")
PYOXIDIZER_DIST = PYOXIDIZER_BUILD.joinpath("dist")
ELECTRON_APP_DIR = REPO_ROOT.joinpath("js", "app")
PYTHON_APP_DESTINATION = ELECTRON_APP_DIR.joinpath("python")
...
subprocess.check_call(
["poetry", "run", "pyoxidizer", "build", "install"], cwd=REPO_ROOT
)
shutil.move(PYOXIDIZER_DIST, PYTHON_APP_DESTINATION)
The main.ts
file#
Within main.ts
the streamlit GUI itself is booted from within the above
defined python
resources directory:
appStreamlitServer = spawn("./pymedphys", ["gui", "--electron"], {
cwd: path.join(process.resourcesPath, "python"),
});
And the entire Electron application is really just a light wrapper that opens up the Streamlit hosted URL:
streamlitPortDelegate.promise.then((port) => {
const pymedphysAppUrl = url.format({
pathname: `localhost:${port}`,
protocol: "http:",
slashes: true,
});
mainWindow.loadURL(pymedphysAppUrl);
});
So that the Electron app doesn’t open the Streamlit webpage before the server is running, it waits for the Streamlit CLI to print out that the server is ready:
appStreamlitServer.stdout.once("data", (data) => {
const stdoutJson = JSON.parse(`${data}`);
const port: string = stdoutJson["port"];
streamlitPortDelegate.resolve(port);
});
Booting the streamlit app and syncing the port with Electron#
When the user starts the application, first the Electron main.ts
script
starts. This
spawns the Streamlit GUI.
Then, the server waits for a given promise to resolve called streamlitPortDelegate
.
This promise only resolves when the Streamlit CLI prints out a
specific JSON string within the CLI.
However, by default, Streamlit doesn’t print the port that it is serving on to the CLI in such a machine readable format. Instead, it uses a more human readable approach. As such, the Streamlit package was lightly “monkey patched” to print out its utilised port in this JSON format:
def _patch_streamlit_print_url():
_original_print_url = st.bootstrap._print_url
def _new_print_url(is_running_hello: bool) -> None:
port = int(st.config.get_option("browser.serverPort"))
sys.stdout.flush()
print(json.dumps({"port": port}))
sys.stdout.flush()
_original_print_url(is_running_hello)
st.bootstrap._print_url = _new_print_url
That way, when Streamlit is serving and ready, the first thing it does is print something like the following to the CLI:
{ "port": 8051 }
This then subsequently triggers the promise within Electron and loads up the Streamlit URL at the given port (same as copied in from above):
streamlitPortDelegate.promise.then((port) => {
const pymedphysAppUrl = url.format({
pathname: `localhost:${port}`,
protocol: "http:",
slashes: true,
});
mainWindow.loadURL(pymedphysAppUrl);
});
Building for all OSs within GitHub Actions#
Once the built Streamlit app was up and running on a given OS, then, given the cross platform nature of the tools utilised, all that was required to create cross platform installers was to run the binary build across the various platforms within the CI suite.
The actual build itself was scripted out within the PyMedPhys dev CLI. This was then utilised within the CI:
- name: Build Binary
if: matrix.task == 'build'
run: |
poetry run pymedphys dev build --install
To set up the CI, need to make sure that
Node, Yarn,
Python
and
Poetry
were all installed. Needed to also install PyOxidizer, which was included as
PyMedPhys build
dependency extras. So installation of PyOxidizer and other
CLI dependencies was achieved with poetry install -E build -E cli
.
Once this build was completed within the CI, the resulting artifacts needed to be uploaded. That was achieved with:
- uses: actions/upload-artifact@v3
if: matrix.task == 'build'
with:
name: PyMedPhysApp-${{ runner.os }}
path: |
js/app/dist/*.dmg
js/app/dist/*.exe
js/app/dist/*.snap
js/app/dist/*.AppImage
!js/app/dist/*unpacked/
Getting help#
If you’d like to get help with any of the above, or you’d like to use some of the above to make your own portable Streamlit executables and you’re running into trouble. Feel free to reach out over at the PyMedPhys discourse group https://pymedphys.discourse.group/.