Dylan McCall<p>This blog post has been floating around as a draft for several years. It eventually split off into a presentation at <a href="https://dylanmc.ca//-/blog/2022/08/10/guadec-2022/" rel="nofollow noopener" target="_blank">GUADEC 2022</a>, titled <em><a href="https://events.gnome.org/event/77/contributions/287/" rel="nofollow noopener" target="_blank">Offline learning with GNOME and Kolibri</a></em> (<a href="https://youtu.be/KRpzzI_FSoU?t=19430" rel="nofollow noopener" target="_blank">YouTube</a>). In that presentation, <a href="https://cosocial.ca/@manuq@mastodon.uy" rel="nofollow noopener" target="_blank">Manuel Quiñones</a> and I explained how Endless OS reaches a unique audience by providing Internet-optional learning resources, and we provided an overview of our work with Kolibri. This post goes into more detail about the technical implementation of the Kolibri desktop app for GNOME, and in particular how it integrates with Endless OS.</p><p><strong>Integrating a flatpak app with an immutable OS</strong></p><p>In <a href="https://www.endlessos.org/" rel="nofollow noopener" target="_blank">Endless OS</a>, way back with Endless OS 4 in 2021, we added <a href="https://learningequality.org/kolibri/" rel="nofollow noopener" target="_blank">Kolibri</a>, an app created by Learning Equality, as a new way to discover educational content. Kolibri has a rich library of video lessons, games, documents, e-books, and more; as well as tools for guided learning – both for classrooms, and for families learning at home. The curation means it is safe and comfortable to freely explore. And all of this works offline, with everything stored on your device.</p><p>Making this all come together was an interesting challenge, but looking back on it with <a href="https://www.endlessos.org/post/getting-started-with-endless-os-6" rel="nofollow noopener" target="_blank">Endless OS 6</a> alive and well, I can say that it worked out nicely.</p> <p><strong>The Kolibri app for GNOME</strong></p><p>Learning Equality designed Kolibri with offline, distributed learning in mind. While an organization can run a single large Kolibri instance that everyone reaches with a web browser, it is equally possible for a group of people to use many small instances of Kolibri, where those instances connect with each other intermittently to exchange information. The developers are deeply interested in <a href="https://en.wikipedia.org/wiki/Sneakernet" rel="nofollow noopener" target="_blank">sneaker net-style</a> use cases, and indeed Kolibri’s resilience has allowed it to thrive in <a href="https://learningequality.org/impact/our-impact/" rel="nofollow noopener" target="_blank">many challenging situations</a>.</p><p>Despite using Django and CherryPy at its heart, Kolibri often presents itself as a desktop app which expects to run on end user devices. Behind the scenes, the existing <a href="https://learningequality.org/kolibri/download/" rel="nofollow noopener" target="_blank">Windows and MacOS apps</a> each bundle a Kolibri server, running it in the background for as long as the desktop app is running.</p><p>We worked with Learning Equality to create a new <a href="https://flathub.org/apps/org.learningequality.Kolibri" rel="nofollow noopener" target="_blank">Kolibri app for GNOME</a>. It uses modern GTK with WebKitGTK to show Kolibri itself. It also includes a desktop search provider, so you can search for Kolibri content from anywhere.</p> <a href="https://flathub.org/apps/org.learningequality.Kolibri" rel="nofollow noopener" target="_blank"></a> <p>The Kolibri GNOME app is distributed as a flatpak, so its dependencies are neatly organized, it runs in a well-defined sandbox, and it is easy to install it from Flathub. For Endless OS, using flatpak means it is trivial to update Kolibri independent from Endless OS’s immutable base system.</p><p><strong>Kolibri Daemon</strong></p><p>But Endless OS doesn’t <em>just</em> include Kolibri. One of my favourite parts of Endless OS is it provides useful content out of the box, which is great for people with limited internet access. So in addition to Kolibri itself, we want a rich library of Kolibri <em>content</em> pre-installed. And with so much already there, ready to be used, we want it to be easy for people to search for that content right away and start Kolibri for the first time.</p> If we add more users, each with their own Kolibri content, we can imagine the size of that database becoming a problem. <p>This becomes both a technical challenge and a philosophical challenge. Normally, each desktop user has their own instance of Kolibri, with its own hidden directory full of content. Because it is a flatpak, it normally doesn’t see the rest of the system unless we explicitly give it permission to, and every time we do that we need to think carefully about what it means. Should we really grant a WebView the ability to read and write <code>/run/media</code>? We try to avoid it.</p><p>At the same time, we want a way to create new apps which use content from Kolibri, so that library of pre-installed content is visible up front, from the apps grid. But it would be<em> </em>expensive if each of these apps ran its own instance of Kolibri. And whatever solution we employ, we don’t want to diverge significantly from the Kolibri people are using outside of Endless OS.</p><p>To solve these problems, we split the code which starts and stops the Kolibri service into a separate component, <code>kolibri-daemon</code>. The desktop app (<code>kolibri-gnome</code>) and the search provider each communicate with <code>kolibri-daemon</code> using D-Bus.</p> The desktop app communicates through kolibri-daemon, instead of starting Kolibri itself. <p>This design is exactly what happens when you start the Kolibri app from Flathub. And with the components neatly separated, on Endless OS we add <a href="https://github.com/endlessm/eos-kolibri" rel="nofollow noopener" target="_blank">eos-kolibri</a>, which takes it a step further: it adds a <code>kolibri</code> system user and a service which runs <code>kolibri-daemon</code> on the D-Bus system bus. The resulting changes turn out to be straightforward, because D-Bus provides most of what we need for free.</p> Kolibri on Endless OS is almost the same, except kolibri-daemon is run by the Kolibri system user. <p>With this in place, every user on the system shares the same Kolibri content, and it is installed to a single well-known location: <code>/var/lib/kolibri</code>. <em>Now</em>, pre-installing Kolibri content is a problem we can solve at the system level, and in the <a href="https://support.endlessos.org/en/deployment/image-builder" rel="nofollow noopener" target="_blank">Endless OS image builder</a>. Independent from the app itself.</p><p><strong>Channel apps</strong></p><p>Now that we have solved the problem of Kolibri content being duplicated, we can come back to having <em>multiple apps</em> share the same Kolibri service. In Endless OS, we want users to easily see the content they have installed, and we do this by adding launchers to the apps grid.</p><p>First, we need to create those apps. If someone has installed content from a Kolibri channel like <em>TED-Ed Lessons</em> or <em>Blockly Games</em>, we want Kolibri to generate a launcher for that channel.</p><p>But remember, Kolibri on Endless OS is an unprivileged system service. It can’t talk to the <a href="https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.DynamicLauncher.html" rel="nofollow noopener" target="_blank">DynamicLauncher portal</a>. That belongs to the user’s session, and we want these launchers to be visible before a user ever starts Kolibri in their own session. Kolibri also can’t be creating files in <code>/usr/share/applications</code>. That would be far too much responsibility.</p><p>Instead, we add a <a href="https://github.com/endlessm/kolibri-app-desktop-xdg-plugin" rel="nofollow noopener" target="_blank">Kolibri plugin to generate desktop entries</a> for channels. The desktop entries refer to the Kolibri app using a custom URI scheme, a layer of indirection because Kolibri (potentially inside a flatpak) is unaware of how the host system launches it. The URI scheme provides enough information to start in a channel-specific app mode, instead of in its default configuration.</p><p>Finally, instead of placing the desktop entry files in one of the usual places, we place them in a well-known location inside Kolibri’s data directory. That way the channel apps are available, but not visible by default.</p><p>In Endless OS, the channel launchers end up in <code>/var/lib/kolibri/data/content/xdg</code>, so in <a href="https://github.com/endlessm/eos-kolibri/blob/master/data/user-environment-generators/61-eos-kolibri.sh.in" rel="nofollow noopener" target="_blank">our system configuration</a> we add that directory to <code><a href="https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html" rel="nofollow noopener" target="_blank">XDG_DATA_DIRS</a></code>. This turns out to be a good choice, because it is trivial to start <a href="https://github.com/endlessm/kolibri-app-desktop-xdg-plugin/commit/8741eabc6e43bb6cad73671b69f49aa3ea59db52" rel="nofollow noopener" target="_blank">generating search providers</a> for those apps, as well.</p> Kolibri channels along with other Education apps in Endless OS. <p><strong>Search providers</strong></p><p>To make sure people can find <em>everything</em> we’ve included in Endless OS, we add as many desktop search providers as we can think of, and we encourage users to explore them. The search bar in Endless OS is not just for apps.</p><p>That means we need a search provider for Kolibri. It’s a simple enough problem. We extended <a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/src/kolibri_daemon/application.py" rel="nofollow noopener" target="_blank"><code>kolibri-daemon</code>‘s D-Bus interface</a> with its own equivalents for <a href="https://developer.gnome.org/documentation/tutorials/search-provider.html" rel="nofollow noopener" target="_blank">the GNOME Shell search provider interface</a>. It is capable of reading directly from Kolibri’s database, so we can avoid starting an HTTP server. But we also want to avoid dealing with <code>kolibri-daemon</code> as much as possible. It is a Python process, heavy with web server stuff and complicated multiprocessing code. And, besides, the daemon could be connecting to the D-Bus system bus, and the shell only talks to search providers on the session bus. That’s why <a href="https://github.com/learningequality/kolibri-installer-gnome/tree/master/src/kolibri_gnome_search_provider" rel="nofollow noopener" target="_blank">the search provider <em>itself</em></a> is a separate proxy application, written in C.</p> Kolibri returning search results in GNOME Shell. <p>But in Endless OS, we don’t just need one search provider, either. We want one for each of those channel apps we generated. So, I mentioned that our Kolibri plugin generates a search provider to go with each desktop file. Of course, loading and searching through Kolibri’s sqlite database is already expensive <em>once</em>, so it would be absurd to do it for every channel that is installed. That’s a lot of processes!</p><p>Fortunately, those search providers are all the same D-Bus service, with a different object path for each Kolibri channel. That one D-Bus service receives a lot of identical search queries for a lot of different object paths, but at least the system is only starting <em>one</em> process for it all. In the search provider code, I added a bespoke <a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/src/kolibri_gnome_search_provider/kolibri-task-multiplexer.c" rel="nofollow noopener" target="_blank">task multiplexer</a>, which allows the service to run a single search in <code>kolibri-daemon</code> for a given query, then group the results and return them to different invocations from the shell.</p> Kolibri returning search results through several channel apps in Endless OS. <p>It is a complicated workaround, but it means search results appear in distinct buckets with meaningful names and icons. For our purpose in Endless OS, it was definitely worth the trouble.</p><p><strong>User accounts</strong></p><p>There was one last wrinkle here: Kolibri kept asking people to set it up, make a user account (with a password!), and sign in. It is, after all, a standalone learning app with a big complicated database that keeps track of learning progress and understands how to sync content between devices. But this isn’t a great experience if you’re just here to <a href="https://www.khanacademy.org/math/cc-eighth-grade-math/cc-8th-linear-equations-functions/8th-linear-functions-modeling/v/interpreting-features-of-linear-functions-example" rel="nofollow noopener" target="_blank">watch that lecture about cats</a>.</p><p>What we want is for Kolibri to <em>already know</em> who is accessing it. They’re already signed in as a desktop user. And most of the time, we want to blaze right through that initial “set up your device” step, or at least make it as smooth as possible.</p><p>To do that, we added an interface in <code>kolibri-daemon</code> so the desktop app can get an authentication token to use over HTTP. On the other side, <code>kolibri-daemon</code> privately communicates with Kolibri itself to verify an authentication token, and it communicates with <a href="https://www.freedesktop.org/wiki/Software/systemd/logind/" rel="nofollow noopener" target="_blank">logind</a> to build a profile for the authenticating user.</p><p>It was ugly at first, with a custom <a href="https://github.com/endlessm/kolibri-desktop-auth-plugin/" rel="nofollow noopener" target="_blank">kolibri-desktop-auth-plugin</a> which sat on top of Kolibri’s authentication system. But after some iteration, upstream Kolibri now has its own understanding of desktop users. On the surface, it uses Kolibri’s <a href="https://github.com/learningequality/kolibri/tree/develop/kolibri/plugins/app" rel="nofollow noopener" target="_blank">app interface plugin</a> for platform integration. With the newest version of Kolibri we have been able to solve authentication in a way that I am properly happy with.</p> <p>My favourite part of the feature has been seeing it come together with Kolibri’s first run wizard. Given a working authentication token, Kolibri knows to skip creating an initial user account, leaving only some simple questions about how the user is planning to use Kolibri; independently or connecting to an existing classroom.</p><p><strong>That’s it!</strong></p><p>It has been great to work on the Kolibri desktop app, and I expect to take some of the approaches and lessons here over to other projects. It is the first big new Python desktop app I have worked with, and it was interesting using some modern Python tools in tandem with the GNOME ways of doing things. The resulting codebase has some fun details:</p><ul><li>The source repository includes <a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/build-aux/flatpak/org.learningequality.Kolibri.Devel.json" rel="nofollow noopener" target="_blank">a Flatpak manifest</a>, so it builds and runs out of the box in <a href="https://apps.gnome.org/Builder/" rel="nofollow noopener" target="_blank">GNOME Builder</a>. As soon as that was working, I used Builder for everything.</li><li><a href="https://mesonbuild.com/" rel="nofollow noopener" target="_blank">Meson</a> is truly indispensable for this kind of thing. We’re sharing build configuration between a bunch of Python modules, all sorts of configuration and data files, <em>and</em> a pair of C projects – one of which is imported by a Python module using GObject introspection. This all works (in a mere 577 lines of meson.build, if you’re counting) because the build system is language-agnostic, and I love it for that. I know that isn’t a lot to ask, but the go-to for Python is decidedly not language-agnostic, and I do not love it.</li><li>We added <a href="https://pre-commit.com/" rel="nofollow noopener" target="_blank">pre-commit</a> to automatically clean up source files and run quick tests against them. It doesn’t actually require you have a Python codebase, but it is written in Python and I think people are afraid of how Pythony it looks? It’s really convenient, and it does a good job taking care of the usual nightmare of setting up a virtual environment to run all its tools. I often don’t bother with the actual git hook part, and instead I remember to run the thing manually, and we use the <a href="https://github.com/pre-commit/action" rel="nofollow noopener" target="_blank">pre-commit github action</a> to be sure.</li><li>At some point, I added Python type hinting to every part of the project. This tremendously improved the development experience with Builder, and it allowed me to add a <a href="https://mypy-lang.org/" rel="nofollow noopener" target="_blank">mypy</a> pre-commit hook to catch mistakes.</li><li>I got annoyed at the problem of needing to write release notes in the appdata file before knowing what the next release is called, so I devised <a href="https://github.com/learningequality/kolibri-installer-gnome?tab=readme-ov-file#managing-release-notes" rel="nofollow noopener" target="_blank">a fun scheme</a> where we add notes under <code>"{current_version}+next"</code>, and then <a href="https://callowayproject.github.io/bump-my-version/" rel="nofollow noopener" target="_blank">bump-my-version</a> (another tool that looks very Pythony but everyone should use it) knows to mark that release entry as released, setting the date and version appropriately. <a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/.bumpversion.toml#L14-L19" rel="nofollow noopener" target="_blank">I wish it didn’t involve regex</a>, but as a concept it has been nice to use. I was tempted to write a pre-commit hook which actually insists on an up to date “next release” entry in appdata, but I should find another project to try it with.</li><li>With that said, a better workflow probably involves <code>appstream-util news-to-appdata</code>.</li><li>Managing history in WebKit can be tricky because the <a href="https://webkitgtk.org/reference/webkitgtk/stable/class.BackForwardList.html" rel="nofollow noopener" target="_blank"><code>BackForwardList</code></a> is read-only. That was an issue with the Kolibri app because we (with our UI consisting almost entirely of a WebView) need to communicate about Kolibri’s state before its HTTP server is running. Kolibri upstream provides a static HTML loading screen for this purpose, which is fine, but now we have this file in our WebView’s back / forward list. I solved it by <a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/src/kolibri_gnome/kolibri_webview.py" rel="nofollow noopener" target="_blank">swapping between different WebViews</a>, and later showing one in a dialog just for Kolibri’s setup wizard. At first, that was all to keep the history stack organized, but at the same time I found it made the app feel a little less like a web browser in a trench coat. We can switch from the loading WebView to the real thing with a nice crossfade, and only when the UI is actually <em>for real</em> finished loading.</li><li>This whole project uses a lot of GObject throughout. At some point I finally read <a href="https://pygobject.gnome.org" rel="nofollow noopener" target="_blank">the pygobject manual</a> and found myself happily doing property binding, signals and async functions and all those good things from Python. It was a much better experience than earlier in the project’s life where there was a type of angry mishmash between vanilla Python and GObject. (The thing that really freed this up was when I moved a lot of D-Bus code over to a C helper library with <a href="https://mesonbuild.com/Gnome-module.html#gnomegdbus_codegen" rel="nofollow noopener" target="_blank">gdbus-codegen</a>, which allowed me to delete the equivalent duplicative Python code, and also introduced a bunch more GObject). It’s easy to see why GObject works best with a language that <em>doesn’t</em> carry its own big standard library, but I was happy with how productive I could be in Python once I started actively preferring GObject, especially with the various magic helpers provided by PyGObject. In a future starting-from-scratch project, I would be tempted to make that a rule when adding imports and writing new classes.</li><li>I made many commits here because I am obsessive about silly things, but this all works thanks to the genius and hard work of the folks at Learning Equality, as well as everyone at Endless, including <a href="https://dbnicholson.com/" rel="nofollow noopener" target="_blank">Dan Nicholson</a>, <a href="https://mastodon.social/@danigm" rel="nofollow noopener" target="_blank">Daniel Garcia Moreno</a>, <a href="https://feaneron.com/" rel="nofollow noopener" target="_blank">Georges Stavracas</a>, <a href="https://github.com/starnight" rel="nofollow noopener" target="_blank">Jian-Hong Pan</a>, <a href="https://mastodon.uy/@manuq" rel="nofollow noopener" target="_blank">Manuel Quiñones</a>, and <a href="https://mastodon.me.uk/@wjt" rel="nofollow noopener" target="_blank">Will Thompson</a>.</li></ul><p>I have to admit I got carried away with certain aspects of this. In the end there is a certain discontent to be had spending creative energy on what is, from many angles, a glorified web browser. It’s frustrating when the web stack leads us to treat an application as a black box behind an HTTP interface, which makes integration difficult: boot it up (in its own complex runtime environment which is <em>heroically</em> not a Docker container); wait until it is ready (<a href="https://github.com/learningequality/kolibri-installer-gnome/blob/master/src/kolibri_daemon/kolibri_http_process.py#L146-L190" rel="nofollow noopener" target="_blank">Kolibri is good at this</a>, but sometimes you’re just watching a file or polling some well-known port); authenticate; ask it (over HTTP) some trivial question that amounts to a single SQL command; return <code>None</code>. <em>But look at that nice framework we’re using!</em></p><p>At the same time, it isn’t lost on me that a software stack like Kolibri’s simply <em>is</em> a popular choice for a cross-platform app. It’s worth understanding how to work with it in a way that still does the best we can to be useful, efficient, and comfortable to use.</p> <p>Beyond all the tech stuff, I want to emphasize that Kolibri is an exceptionally cool project. I truly admire what Learning Equality are doing with it, and if you’re interested in offline-first content, data sovereignty, or just open source learning in general, I highly recommend checking it out – either <a href="https://flathub.org/apps/org.learningequality.Kolibri" rel="nofollow noopener" target="_blank">our app on Flathub</a>, or at <a href="https://learningequality.org/kolibri" rel="nofollow noopener" target="_blank">learningequality.org/kolibri</a>.</p><p><a href="https://dylanmc.ca//-/blog/2024/08/20/kolibri-and-endless-os/" class="" rel="nofollow noopener" target="_blank">https://dylanmc.ca//-/blog/2024/08/20/kolibri-and-endless-os/</a></p><p><a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://dylanmc.ca//-/blog/tag/endless-os/" target="_blank">#EndlessOS</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://dylanmc.ca//-/blog/tag/gnome/" target="_blank">#GNOME</a></p>