Optimized by Otto

Automated security validation: How 7,000+ tests shaped MariaDB's new AppArmor profile

Thu, 19 Mar 2026 00:00:00 +0000

Linux kernel security modules provide a good additional layer of security around individual programs by restricting what they are allowed to do, and at best block and detect zero-day security vulnerabilities as soon as anyone tries to exploit them, long before they are widely known and reported. However, the challenge is how to create these security profiles without accidentally also blocking legitimate actions. For MariaDB in Debian and Ubuntu, a new AppArmor profile was recently created by leveraging the extensive test suite with 7000+ tests, giving good confidence that AppArmor is unlikely to yield false positive alerts with it.

AppArmor is a Mandatory Access Control (MAC) system, meaning that each process controlled by AppArmor has a sort of an “allowlist” called profile that defines all capabilities and file paths a program can access. If a program tries to do something not covered by the rules in its AppArmor profile, the action will be denied on the Linux kernel level and a warning logged in the system journal. This additional security layer is valuable because even if a malicious user found a security vulnerability some day in the future, the AppArmor profile severely restricts the ability to exploit it and gain access to the operating system.

AppArmor was originally developed by Novell for use in SUSE Linux, but nowadays the main driver is Canonical and AppArmor is extensively used in Ubuntu and Debian, and many of their derivatives (e.g. Linux Mint, Pop!_OS, Zorin OS) and in Arch. AppArmor’s benefit compared to the main alternative SELinux (used mainly in the RedHat/Fedora ecosystem) is that AppArmor is easier to manage. AppArmor continues to be actively developed, with new major version 5.0 expected to arrive soon.

I also have some personal history contributing some notification handler scripts in Python and I also created the website that AppArmor.net still runs.

Regular review of denials in the system log required

Any system administrator using Debian/Ubuntu needs to know how to check for AppArmor denials. The point of using AppArmor is kind of moot if nobody is checking the denials. When AppArmor blocks an action, it logs the event to the system audit or kernel logs. Understanding these logs is crucial for troubleshooting custom configurations or identifying potential security incidents.

To view recent denials, check /var/log/audit/audit.log or run journalctl -ke --grep=apparmor.

A typical denial entry for MariaDB will look like this (split across multiple lines for legibility):

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open"
profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234
comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

How to interpret this output:

msg=audit(…): The audit timestamp and event serial number.
apparmor=“DENIED”: Indicates AppArmor blocked the action.
operation: The action being attempted (e.g., open, mknod, file_mmap, file_perm).
profile: The specific AppArmor profile that triggered the denial (in this case the /usr/sbin/mariadbd profile).
name: The file path or resource that was blocked. In the example above, a custom data path was denied access because it wasn’t defined in the profile’s allowed abstractions.
comm: The command name that triggered the denial (here mariadbd).
requested_mask / denied_mask: Shows the permissions requested (e.g., r for read, w for write).
pid: The process ID.
fsuid: The user ID of the process attempting the action.
ouid: The owner user ID of the target file.

If an action seems legit and should not be denied, the sysadmin needs to update the existing rules at /etc/apparmor.d/ or drop a local customization file in at /etc/apparmor.d/local/. If the denied action looks malicious, the sysadmin should start a security investigation and if needed report a suspected zero-day vulnerability to the upstream software vendor (e.g. Ubuntu customers to Canonical, or MariaDB customers to MariaDB).

AppArmor in MariaDB - not a novel thing, and not easy to implement well

Based on old bug reports, there was an AppArmor profile already back in 2011, but it was removed in MariaDB 5.1.56 due to backlash from users running into various issues. A new profile was created in 2015, but kept opt-in only due to the risk of side effects. It likely had very few users and saw minimal maintenance, getting only a handful of updates in the past 10 years.

The primary challenge in using mandatory access control systems with MariaDB lies in the sheer breadth of MariaDB’s operational footprint with diverse storage engines and plugins. Also the code base in MariaDB assumes that system calls to Linux always work – which they do under normal circumstances – and do not handle errors well if AppArmor suddenly denies a system call. MariaDB is also a large and complex piece of software to run and operate, and it can be very challenging for system administrators to root-cause that a misbehavior in their system was due to AppArmor blocking a single syscall.

Ironically, AppArmor is most beneficial exactly due to the same reasons for MariaDB. The larger and more complex a software is, the larger are the odds of a security vulnerability arising between the various components. And AppArmor profile helps reduce this complexity down to a single access list.

Over the years there has been users requesting to get the AppArmor profile back, such as Debian Bug#875890 since 2017. The need was raised recently again by the Ubuntu security team during the MariaDB Ubuntu ‘main’ inclusion review in 2025, which prompted a renewed effort by Debian/Ubuntu developers, mainly myself and Aquila Macedo, with upstream MariaDB assistance from Daniel Black.

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews

The key to creating a robust AppArmor profile is the ability to know in detail what is expected and normal behavior of the system. One could in theory read all of the source code in MariaDB, but with over two million lines, it is of course not feasible in practice. However, MariaDB does have a very extensive 7000+ test suite, and running it should trigger most code paths in MariaDB. Utilizing the test suite was key in creating the new AppArmor profile for MariaDB: we installed MariaDB on a Ubuntu system, enabled AppArmor in complain mode and iterated on the allowlist by running the full mariadb-test-run with all MariaDB plugins and features enabled until we had a comprehensive yet clean list of rules.

To be extra diligent, we also reworked the autopkgtest for MariaDB in Debian and Ubuntu CI systems to run with the AppArmor profile enabled and to print all AppArmor notices at the end of the run, making it easy to detect now and in the future if the MariaDB test suite triggers any AppArmor denials. If any test fails, the release would not get promoted further, protecting users from regressions.

While developing and triggering manual test runs we used the maximal achievable test suite with 7177 tests. The test is however so extensive it takes over two hours to run, and it also has some brittle tests, so the standard test run in Debian and Ubuntu autopkgtest is limited just to MariaDB’s main suite with about 1000 tests. Having some tests fail while testing the AppArmor profile was not a problem, because we didn’t need all the tests to pass – we merely needed them to run as many code paths as possible to see if they run any system calls not accounted for in the AppArmor profile.

Note that extending the profile was not just mechanical copying of log messages to the profile. For example, even though a couple of tests involve running the dash shell, we decided to not allow it, as it opens too much of a path for a potential exploit to access the operating system.

The result of this effort is a modernized, robust profile that is now production-ready. Those interested in the exact technical details can read the Debian Bug#1130272 and the Merge Request discussions at salsa.debian.org, which hosts the Debian packaging source code.

Now available in Debian unstable, soon Ubuntu – feedback welcome!

Even though the file is just 200 lines long, the work to craft it spanned several weeks. To minimize risk we also did a gradual rollout by releasing the first new profile version in complain mode, so AppArmor only logs would-be-denials without blocking anything. The AppArmor profile was switched to enforce mode only in the very latest MariaDB revision 1:11.8.6-4 in Debian, and a NEWS item issued to help increase user awareness of this change. It is also slated for the upcoming Ubuntu 26.04 “Resolute Raccoon” release next month, providing out-of-the-box hardening for the wider ecosystem.

While automated testing is extensive, it cannot simulate everything. Most notably various complicated replication topologies and all Galera setups are likely not covered. Thus, I am calling on the community to deploy this profile and monitor for any audit denials in the kernel logs. If you encounter unexpected behavior or legitimate denials, please submit a bug report via the Debian Bug Tracking System.

To ensure you are running the latest MariaDB version, run apt install --update --yes mariadb-server. To view the latest profile rules, run cat /etc/apparmor.d/mariadbd and to see if it is enforced review the output of aa-status. To quickly check if there were any AppArmor denials, simply run journalctl -k | grep -i apparmor | grep -i mariadb.

Systemd hardening also adopted as security features keep evolving

For those interested in MariaDB security hardening, note that also new systemd hardening options were rolled out in Debian/Ubuntu recently. Note that Debian and Ubuntu are mainly volunteer-driven open source developer communities, and if you find this topic interesting and you think you have the necessary skills, feel free to submit your improvement ideas as Merge Requests at salsa.debian.org/mariadb-team. If your improvement suggestions are not Debian/Ubuntu specific, please submit them directly to upstream at GitHub.com/MariaDB.

Stop using MySQL in 2026, it is not true open source

Sun, 11 Jan 2026 00:00:00 +0000

If you care about supporting open source software, and still use MySQL in 2026, you should switch to MariaDB like so many others have already done.

The number of git commits on github.com/mysql/mysql-server has been significantly declining in 2025. The screenshot below shows the state of git commits as of writing this in January 2026, and the picture should be alarming to anyone who cares about software being open source.

This is not surprising – Oracle should not be trusted as the steward for open source projects

When Oracle acquired Sun Microsystems and MySQL along with it back in 2009, the European Commission almost blocked the deal due to concerns that Oracle’s goal was just to stifle competition. The deal went through as Oracle made a commitment to keep MySQL going and not kill it, but (to nobody’s surprise) Oracle has not been a good steward of MySQL as an open source project and the community around it has been withering away for years now. All development is done behind closed doors. The publicly visible bug tracker is not the real one Oracle staff actually uses for MySQL development, and the few people who try to contribute to MySQL just see their Pull Requests and patch submissions marked as received with mostly no feedback and then those changes may or may not be in the next MySQL release, often rewritten, and with only Oracle staff in the git author/committer fields. The real author only gets a small mention in a blog post. When I was the engineering manager for the core team working on RDS MySQL and RDS MariaDB at Amazon Web Services, I oversaw my engineers’ contributions to both MySQL and MariaDB (the latter being a fork of MySQL by the original MySQL author, Michael Widenius). All the software developers in my org disliked submitting code to MySQL due to how bad the reception by Oracle was to their contributions.

MariaDB is the stark opposite with all development taking place in real-time on github.com/mariadb/server, anyone being able to submit a Pull Request and get a review, all bugs being openly discussed at jira.mariadb.org and so forth, just like one would expect from a true open source project. MySQL is open source only by license (GPL v2), but not as a project.

MySQL’s technical decline in recent years

Despite not being a good open source steward, Oracle should be given credit that it did keep the MySQL organization alive and allowed it to exist fairly independently and continue developing and releasing new MySQL versions well over a decade after the acquisition. I have no insight into how many customers they had, but I assume the MySQL business was fairly profitable and financially useful to Oracle, at least as long as it didn’t gain too many features to threaten Oracle’s own main database business.

I don’t know why, perhaps because too many talented people had left the organization, but it seems that from a technical point of view MySQL clearly started to deteriorate from 2022 onward.

When MySQL 8.0.29 was released with the default ALTER TABLE method switched to run in-place, it had a lot of corner cases that didn’t work, causing the database to crash and data to corrupt for many users. The issue wasn’t fully fixed until a year later in MySQL 8.0.32. To many users annoyance Oracle announced the 8.0 series as “evergreen” and introduced features and changes in the minor releases, instead of just doing bugfixes and security fixes like users historically had learnt to expect from these x.y.Z maintenance releases.

There was no new major MySQL version for six years. After MySQL 8.0 in 2018 it wasn’t until 2023 when MySQL 8.1 was released, and it was just a short-term preview release. The first actual new major release MySQL 8.4 LTS was released in 2024. Even though it was a new major release, many users got disappointed as it had barely any new features.

Many also reported degraded performance with newer MySQL versions, for example the benchmark by famous MySQL performance expert Mark Callaghan below shows that on write-heavy workloads MySQL 9.5 throughput is typically 15% less than in 8.0.

Due to newer MySQL versions deprecating many features, a lot of users also complained about significant struggles regarding both MySQL 5.7->8.0 and 8.0->8.4 upgrades. With few new features and heavy focus on code base cleanup and feature deprecation, it became obvious to many that Oracle had decided to just keep MySQL barely alive, and put all new relevant features (e.g. vector search) into Heatwave, Oracle’s closed-source and cloud-only service for MySQL customers.

As it was evident that Oracle isn’t investing in MySQL, Percona’s Peter Zaitsev wrote Is Oracle Finally Killing MySQL in June 2024. At this time MySQL’s popularity as ranked by DB-Engines had also started to tank hard, a trend that likely accelerates in 2026.

In September 2025 news reported that Oracle was reducing its workforce and that the MySQL staff was getting heavily reduced. Obviously this does not bode well for MySQL’s future, and Peter Zaitsev posted already in November stats showing that the latest MySQL maintenance release contained fewer bug fixes than before.

Open source is more than ideology: it has very real effects on software security and sovereignty

Some say they don’t care if MySQL is truly open source or not, or that they don’t care if it has a future in coming years, as long as it still works now. I am afraid people thinking so are taking a huge risk. The database is often the most critical part of a software application stack, and any flaw or problem in operations, let alone a security issue, will have immediate consequences, and “not caring” will eventually get people fired or sued.

In open source problems are discussed openly, and the bigger the problem, the more people and companies will contribute to fixing it. Open source as a development methodology is similar to the scientific method with free flow of ideas that are constantly contested and only the ones with the most compelling evidence win. Not being open means more obscurity, more risk and more “just trust us bro” attitude.

This open vs. closed is very visible for example in how Oracle handles security issues. We can see that in 2025 alone MySQL published 123 CVEs about security issues, while MariaDB had 8. There were 117 CVEs that only affected MySQL and not MariaDB in 2025. I haven’t read them all, but typically the CVEs hardly contain any real details. As an example, the most recent one CVE-2025-53067 states “Easily exploitable vulnerability allows high privileged attacker with network access via multiple protocols to compromise MySQL Server.” There is no information a security researcher or auditor could use to verify if any original issue actually existed, or if it was fixed, or if the fix was sufficient and fully mitigating the issue or not. MySQL users just have to take the word of Oracle that it is all good now. Handling security issues like this is in stark contrast to other open source projects, where all security issues and their code fixes are open for full scrutiny after the initial embargo is over and CVE made public.

There is also various forms of enshittification going on one would not see in a true open source project, and everything about MySQL as a software, documentation and website is pushing users to stop using the open source version and move to the closed MySQL versions, and in particular to Heatwave, which is not only closed-source but also results in Oracle fully controlling customer’s databases contents.

Of course, some could say this is how Oracle makes money and is able to provide a better product. But stories on Reddit and elsewhere suggest that what is going on is more like Oracle milking hard the last remaining MySQL customers who are forced to pay more and more for getting less and less.

There are options and migrating is easy, just do it

A large part of MySQL users switched to MariaDB already in the mid-2010s, in particular everyone who had cared deeply about their database software staying truly open source. That included large installations such as Wikipedia, and Linux distributions such as Fedora and Debian. Because it’s open source and there is no centralized machine collecting statistics, nobody knows what the exact market shares look like. There are however some application specific stats, such as that 57% of WordPress sites around the world run MariaDB, while the share for MySQL is 42%.

For anyone running a classic LAMP stack application such as WordPress, Drupal, Mediawiki, Nextcloud, or Magento, switching the old MySQL database to MariaDB is be straightforward. As MariaDB is a fork of MySQL and mostly backwards compatible with it, swapping out MySQL for MariaDB can be done without changing any of the existing connectors or database clients, as they will continue to work with MariaDB as if it was MySQL.

For those running custom applications and who have the freedom to make changes to how and what database is used, there are tens of mature and well-functioning open source databases to choose from, with PostgreSQL being the most popular general database. If your application was built from the start for MySQL, switching to PostgreSQL may however require a lot of work, and the MySQL/MariaDB architecture and storage engine InnoDB may still offer an edge in e.g. online services where high performance, scalability and solid replication features are of highest priority. For a quick and easy migration MariaDB is probably the best option.

Switching from MySQL to the Percona Server is also very easy, as it closely tracks all changes in MySQL and deviates from it only by a small number of improvements done by Percona. However, also precisely because of it being basically just a customized version of the MySQL Server, it’s not a viable long-term solution for those trying to fully ditch the dependency on Oracle.

There are also several open source databases that have no common ancestry with MySQL, but strive to be MySQL-compatible. Thus most apps built for MySQL can simply switch to using them without needing SQL statements to be rewritten. One such database is TiDB, which has been designed from scratch specifically for highly scalable and large systems, and is so good that even Amazon’s latest database solution DSQL was built borrowing many ideas from TiDB. However, TiDB only really shines with larger distributed setups, so for the vast majority of regular small- and mid-scale applications currently using MySQL, the most practical solution is probably to just switch to MariaDB, which on most Linux distributions can simply be installed by running apt/dnf/brew install mariadb-server.

Whatever you end up choosing, as long as it is not Oracle, you will be better off.

DEP-18: A proposal for Git-based collaboration in Debian

Sun, 30 Nov 2025 00:00:00 +0000

I am a huge fan of Git, as I have witnessed how it has made software development so much more productive compared to the pre-2010s era. I wish all Debian source code were in Git to reap the full benefits.

Git is not perfect, as it requires significant effort to learn properly, and the ecosystem is complex with even more things to learn ranging from cryptographic signatures and commit hooks to Git-assisted code review best practices, ‘forge’ websites, and CI systems.

Sure, there is still room to optimize its use, but Git certainly has proven itself and is now the industry standard. Thus, some readers might be surprised to learn that Debian development in 2025 is not actually based on Git. In Debian, the version control is done by the Debian archive itself. Each ‘commit’ is a new upload to the archive, and the ‘commit message’ is the debian/changelog entry. The ‘commit log’ is available at snapshots.debian.org.

In practice, most Debian Developers (people who have the credentials to upload to the Debian archive) do use Git and host their packaging source code on salsa.debian.org – the GitLab instance of Debian. This is, however, based on each DD’s personal preferences. The Debian project does not have any policy requiring that packages be hosted on salsa.debian.org or be in version control at all.

Is collaborative software development possible without git and version control software?

Debian, however, has some peculiarities that may be surprising to people who have grown accustomed to GitHub, GitLab or various company-internal code review systems.

In Debian:

The source code of the next upload is not public but resides only on the developer’s laptop.
Code contributions are plain patch files, based on the latest revision released in the Debian archive (where the unstable area is equivalent to the main development branch).
These patches are submitted by email to a bug tracker that does no validation or testing whatsoever.
Developers applying these patches typically have elaborate Mutt or Emacs setups to facilitate fetching patches from email.
There is no public staging area, no concept of rebasing patches or withdrawing a patch and replacing it with a better version.
The submitter won’t see any progress information until a notification email arrives after a new version has been uploaded to the Debian archive.

This system has served Debian for three decades. It is not broken, but using the package archive just feels… well, archaic.

There is a more efficient way, and indeed the majority of Debian packages have a metadata field Vcs-Git that advertises which version control repository the maintainer uses. However, newcomers to Debian are surprised to notice that not all packages are hosted on salsa.debian.org but at various random places with their own account and code submission systems, and there is nothing enforcing or even warning if the code there is out of sync with what was uploaded to Debian. Any Debian Developer can at any time upload a new package with whatever changes, bypassing the Git repository, even when the package advertised a Git repository. All PGP signed commits, Git tags and other information in the Git repository are just extras currently, as the Debian archive does not enforce or validate anything about them.

This also makes contributing to multiple packages in parallel hard. One can’t just go on salsa.debian.org and fork a bunch of repositories and submit Merge Requests. Currently, the only reliable way is to download source packages from Debian unstable, develop patches on top of them, and send the final version as a plain patch file by email to the Debian bug tracker. To my knowledge, no system exists to facilitate working with the patches in the bug tracker, such as rebasing patches 6 months later to detect if they or equivalent changes were applied or if sending refreshed versions is needed.

To newcomers in Debian, it is even more surprising that there are packages that are on salsa.debian.org but have the Merge Requests feature disabled. This is often because the maintainer does not want to receive notification emails about new Merge Requests, but rather just emails from bugs.debian.org. This may sound arrogant, but keep in mind that these developers put in the effort to set up their Mutt/Emacs workflow for the existing Debian process, and extending it to work with GitLab notifications is not trivial. There are also purists who want to do everything via the command-line (without having to open a browser, run JavaScript and maintain a live Internet connection), and tools like glab are not convenient enough for the full workflow.

Inefficient ways of working prevent Debian from flourishing

I would claim, based on my personal experiences from the past 10+ years as a Debian Developer, that the lack of high-quality and productive tooling is seriously harming Debian. The current methods of collaboration are cumbersome for aspiring contributors to learn and suboptimal to use for both new and seasoned contributors.

There are no exit interviews for contributors who left Debian, no comprehensive data on reasons to contribute or stop contributing, nor are there any metrics tracking how many people tried but failed to contribute to Debian. Some data points to support my concerns do exist:

The contributor database shows that the number of contributors is growing slower than Debian’s popularity.
Most packages are maintained by one person working alone (just pick any package at random and look at the upload history).

Debian should embrace git, but decision-making is slow

Debian is all about community and collaboration. One would assume that Debian prioritized above all making collaboration tools and processes simpler, faster and less error-prone, as it would help both current and future package maintainers. Yet, it isn’t so, due to some reasons unique to Debian.

There is no single company or entity running Debian, and it has managed to operate as a pure meritocracy and do-cracy for over 30 years. This is impressive and admirable. Unfortunately, some of the infrastructure and technical processes are also nearly 30 years old and very difficult to change for the same reason: the nature of Debian’s distributed decision-making process.

As a software developer and manager with 25+ years of experience, I strongly feel that developing software collaboratively using Git is a major step forward that Debian needs to take, in one form or another, and I hope to see other DDs voice their support if they agree.

Debian Enhancement Proposal 18

Following how consensus is achieved in Debian, I started drafting DEP-18 in 2024, and it is currently awaiting enough thumbs up at https://salsa.debian.org/dep-team/deps/-/merge_requests/21 to get into CANDIDATE status next.

In summary, the DEP-18 proposes that everyone keen on collaborating should:

Maintain Debian packaging sources in Git on Salsa.
Use Merge Requests to show your work and to get reviews.
Run Salsa CI before upload.

The principles above are not novel. According to stats at e.g. trends.debian.net, and UDD, ~93% of all Debian source packages are already hosted on salsa.debian.org. As of June 1st, 2025, only 1640 source packages remain that are not hosted on Salsa. The purpose of DEP-18 is to state in writing what Debian is currently doing for most packages, and thus express what among others new contributors should be learning and doing, so basic collaboration is smooth and free from structural obstacles.

Most packages are also already allowing Merge Requests and using Salsa CI, but there hasn’t been any written recommendation anywhere in Debian to do so. The Debian Policy (v.4.7.2) does not even mention the word “Salsa” a single time. The current process documentation on how to do non-maintainer uploads or salvaging packages are all based on uploading packages to the archive, without any consideration of using git-based collaboration such as posting a Merge Request first. Personally I feel posting a Merge Request would be a better approach, as it would invite collaborators to discuss and provide code reviews. If there are no responses, the submitter can proceed to merge, but compared to direct uploads to the Debian archive, the Merge Request practice at least tries to offer a time and place for discussions and reviews to happen.

It could very well be that in the future somebody comes up with a new packaging format that makes upstream source package management easier, or a monorepo with all packages, or some other future structures or processes. Having a DEP to state how to do things now does not prevent people from experimenting and innovating if they intentionally want to do that. The DEP is merely an expression of the minimal common denominators in the packaging workflow that maintainers and contributors should follow, unless they know better.

Transparency and collaboration

Among the DEP-18 recommendations is:

The recommended first step in contributing to a package is to use the built-in “Fork” feature on Salsa. This serves two purposes. Primarily, it allows any contributor to publish their Git branches and submit them as Merge Requests. Additionally, the mere existence of a list of “Forks” enables contributors to discover each other, and in rare cases when the original package is not accepting improvements, collaboration could arise among the contributors and potentially lead to permanent forks in the general meaning. Forking is a fundamental part of the dynamics in open source that helps drive quality and agreement. The ability to fork ultimately serves as the last line of defense of users’ rights. Git supports this by making both temporary and permanent forks easy to create and maintain.

Further, it states:

Debian packaging work should be reasonably transparent and public to allow contributors to participate. A maintainer should push their pending changes to Salsa at regular intervals, so that a potential contributor can discover if a particular change has already been made or a bug has been fixed in version control, and thus avoid duplicate work.

Debian maintainers should make reasonable efforts to publish planned changes as Merge Requests on Salsa and solicit feedback and reviews. While pushing changes directly on the main Git branch is the fastest workflow, second only to uploading all changes directly to Debian repositories, it is not an inclusive way to develop software. Even packages that are maintained by a single maintainer should at least occasionally publish Merge Requests to allow new contributors to step up and participate.

I think these are key aspects leading to transparency and true open source collaboration. Even though this talks about Salsa — which is based on GitLab — the concepts are universal and will work also on other forges, like Forgejo or GitHub. The point is that sharing work-in-progress on a real-time platform, with CI and other supporting features, empowers and motivates people to iterate on code collaboratively. As an example of an anti-pattern, Oracle MySQL publishes the source code for all their releases and is license-compliant, but as they don’t publish their Git commits in real-time, it does not feel like a real open source project. Non-Oracle employees are not motivated to participate as second-class developers who are kept in the dark. Debian should embrace git and sharing work in real-time, embodying a true open source spirit.

Note that the Debian Enhancement Proposals are not binding. Only the Debian Policy and Technical Committee decisions carry that weight. The nature of collaboration is voluntary anyway, so the DEP does not need to force anything on people who don’t want to use salsa.debian.org.

The DEP-18 is also not a guide for package maintainers. I have my own views and have written detailed guides in blog articles if you want to read more on, for example, how to do code reviews efficiently.

Within DEP-18, there is plenty of room to work in many different ways, and it does not try to force one single workflow. The goal here is to simply have agreed-upon minimal common denominators among those who are keen to collaborate using salsa.debian.org, not to dictate a complete code submission workflow.

Once we reach this, there will hopefully be less friction in the most basic and recurring collaboration tasks, giving DDs more energy to improve other processes or just invest in having more and newer packages for Debian users to enjoy.

Next steps

In addition to lengthy online discussions on mailing lists and DEP reviews, I also presented on this topic at DebConf 2025 in Brest, France. Unfortunately the recording is not yet up on Peertube.

The feedback has been overwhelmingly positive. However, there are a few loud and very negative voices that cannot be ignored. Maintaining a Linux distribution at the scale and complexity of Debian requires extraordinary talent and dedication, and people doing this kind of work often have strong views, most of the time for good reasons. We do not want to alienate existing key contributors with new processes, so maximum consensus is desirable.

We also need more data on what the 1000+ current Debian Developers view as a good process to avoid being skewed by a loud minority. If you are a current or aspiring Debian Developer, please add a thumbs up if you think I should continue with this effort (or a thumbs down if not) on the Merge Request that would make DEP-18 have candidate status.

There is also technical work to do. Increased Git use will obviously lead to growing adoption of the new tag2upload feature, which will need to get full git-buildpackage support so it can integrate into salsa.debian.org without turning off Debian packaging security features. The git-buildpackage tool itself also needs various improvements, such as making contributing to multiple different packages with various levels of diligence in debian/gbp.conf maintenance less error-prone.

Eventually, if it starts looking like all Debian packages might get hosted on salsa.debian.org, I would also start building a review.debian.org website to facilitate code review aspects that are unique to Debian, such as tracking Merge Requests across GitLab projects in ways GitLab can’t do, highlighting which submissions need review most urgently, feeding code reviews and approvals into the contributors.debian.org database for better attribution, and so forth.

Details on this vision will be in a later blog post, so subscribe to updates!

Zero-configuration TLS and password management best practices in MariaDB 11.8

Sun, 14 Sep 2025 00:00:00 +0000

Locking down database access is probably the single most important thing for a system administrator or software developer to prevent their application from leaking its data. As MariaDB 11.8 is the first long-term supported version with a few new key security features, let’s recap what the most important things are every DBA should know about MariaDB in 2025.

Back in the old days, MySQL administrators had a habit of running the clumsy mysql-secure-installation script, but it has long been obsolete. A modern MariaDB database server is already secure by default and locked down out of the box, and no such extra scripts are needed. On the contrary, the database administrator is expected to open up access to MariaDB according to the specific needs of each server. Therefore, it is important that the DBA can understand and correctly configure three things:

Separate application-specific users with granular permissions allowing only necessary access and no more.
Distributing and storing passwords and credentials securely
Ensuring all remote connections are properly encrypted

For holistic security, one should also consider proper auditing, logging, backups, regular security updates and more, but in this post we will focus only on the above aspects related to securing database access.

How encrypting database connections with TLS differs from web server HTTP(S)

Even though MariaDB (and other databases) use the same SSL/TLS protocol for encrypting remote connections as web servers and HTTPS, the way it is implemented is significantly different, and the different security assumptions are important for a database administrator to grasp.

Firstly, most HTTP requests to a web server are unauthenticated, meaning the web server serves public web pages and does not require users to log in. Traditionally, when a user logs in over a HTTP connection, the username and password were transmitted in plaintext as a HTTP POST request. Modern TLS, which was previously called SSL, does not change how HTTP works but simply encapsulates it. When using HTTPS, a web browser and a web server will start an encrypted TLS connection as the very first thing, and only once established, do they send HTTP requests and responses inside it. There are no passwords or other shared secrets needed to form the TLS connection. Instead, the web server relies on a trusted third party, a Certificate Authority (CA), to vet that the TLS certificate offered by the web server can be trusted by the web browser.

For a database server like MariaDB, the situation is quite different. All users need to first authenticate and log in to the server before getting being allowed to run any SQL and getting any data out of the server. The database server and client programs have built-in authentication methods, and passwords are not, and have never been, sent in plaintext. Over the years, MySQL and its successor, MariaDB, have had multiple password authentication methods: the original SHA-1-based hashing, later double SHA-1-based mysql_native_password, followed by sha256_password and caching_sha256_password in MySQL and ed25519 in MariaDB. The MariaDB.org blog post by Sergei Golubchik recaps the history of these well.

Even though most modern MariaDB installations should be using TLS to encrypt all remote connections in 2025, having the authentication method be as secure as possible still matters, because authentication is done before the TLS connection is fully established.

To further harden the authentication against man-in-the-middle attacks, a new password authentication method PARSEC was introduced in MariaDB 11.8, which builds upon the previous ed25519 public-key-based verification (similar to how modern SSH does), and also combines key derivation with PBKDF2 with hash functions (SHA512,SHA256) and a high iteration count.

At first it may seem like a disadvantage to not wrap all connections in a TLS tunnel like HTTPS does, but actually not having the authentication done in a MitM resistant way regardless of the connection encryption status allows a clever extra capability that is now available in MariaDB: as the database server and client already have a shared secret that is being used by the server to authenticate the user, it can also be used by the client to validate the server’s TLS certificate and no third parties like CAs or root certificates are needed. MariaDB 11.8 was the first LTS version to ship with this capability for zero-configuration TLS.

Note that the zero-configuration TLS also works on older password authentication methods and does not require users to have PARSEC enabled. As PARSEC is not yet the default authentication method in MariaDB, it is recommended to enable it in installations that use zero-configuration TLS encryption to maximize the security of the TLS certificate validation.

Why the ‘root’ user in MariaDB has no password and how it makes the database more secure

Relying on passwords for security is problematic, as there is always a risk that they could leak, and a malicious user could access the system using the leaked password. It is unfortunately far too common for database passwords to be stored in plaintext in configuration files that are accidentally committed into version control and published on GitHub and similar platforms. Every application or administrative password that exists should be tracked to ensure only people who need it know it, and rotated at regular intervals to ensure old employees etc won’t be able to use old passwords. This password management is complex and error-prone.

Replacing passwords with other authentication methods is always advisable when possible. On a database server, whoever installed the database by running e.g. apt install mariadb-server, and configured it with e.g. nano /etc/mysql/mariadb.cnf, already has full root access to the operating system, and asking them for a password to access the MariaDB database shell is moot, since they could circumvent any checks by directly accessing the files on the system anyway. Therefore, MariaDB, since version 10.4 stopped requiring the root user to enter a password when connecting locally, and instead checks using socket authentication whether the user is the operating-system root user or equivalent (e.g. running sudo). This is an elegant way to get rid of a password that was actually unnecessary to begin with. As there is no root password anymore, the risk of an external user accessing the database as root with a leaked password is fully eliminated.

Note that socket authentication only works for local connections on the same server. If you want to access a MariaDB server remotely as the root user, you would need to configure a password for it first. This is not generally recommended, as explained in the next section.

Create separate database users for normal use and keep ‘root’ for administrative use only

Out of the box a MariaDB installation is already secure by default, and only the local root user can connect to it. This account is intended for administrative use only, and for regular daily use you should create separate database users with access limited to the databases they need and the permissions required.

The most typical commands needed to create a new database for an app and a user the app can use to connect to the database would be the following:

sql

CREATE DATABASE app_db;
CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%';
FLUSH PRIVILEGES;

Alternatively, if you want to use the parsec authentication method, run this to create the user:

sql

CREATE OR REPLACE USER 'app_user'@'%'
 IDENTIFIED VIA parsec
 USING PASSWORD('your_secure_password');

Note that the plugin auth_parsec is not enabled by default. If you see the error message ERROR 1524 (HY000): Plugin 'parsec' is not loaded fix this by running INSTALL SONAME 'auth_parsec';.

In the CREATE USER statements, the @'%' means that the user is allowed to connect from any host. This needs to be defined, as MariaDB always checks permissions based on both the username and the remote IP address or hostname of the user, combined with the authentication method. Note that it is possible to have multiple user@remote combinations, and they can have different authentication methods. A user could, for example, be allowed to log in locally using the socket authentication and over the network using a password.

If you are running a custom application and you know exactly what permissions are sufficient for the database users, replace the ALL PRIVILEGES with a subset of privileges listed in the MariaDB documentation.

For new permissions to take effect, restart the database or run FLUSH PRIVILEGES.

Allow MariaDB to accept remote connections and enforce TLS

Using the above 'app_user'@'%' is not enough on its own to allow remote connections to MariaDB. The MariaDB server also needs to be configured to listen on a network interface to accept remote connections. As MariaDB is secure by default, it only accepts connections from localhost until the administrator updates its configuration. On a typical Debian/Ubuntu system, the recommended way is to drop a new custom config in e.g. /etc/mysql/mariadb.conf.d/99-server-customizations.cnf, with the contents:

[mariadbd]
# Listen for connections from anywhere
bind-address = 0.0.0.0
# Only allow TLS encrypted connections
require-secure-transport = on

For settings to take effect, restart the server with systemctl restart mariadb. After this, the server will accept connections on any network interface. If the system is using a firewall, the port 3306 would additionally need to be allow-listed.

To confirm that the settings took effect, run e.g. mariadb -e "SHOW VARIABLES LIKE 'bind_address';" , which should now show 0.0.0.0.

When allowing remote connections, it is important to also always define require-secure-transport = on to enforce that only TLS-encrypted connections are allowed. If the server is running MariaDB 11.8 and the clients are also MariaDB 11.8 or newer, no additional configuration is needed thanks to MariaDB automatically providing TLS certificates and appropriate certificate validation in recent versions.

On older long-term-supported versions of the MariaDB server one would have had to manually create the certificates and configure the ssl_key, ssl_cert and ssl_ca values on the server, and distribute the certificate to the clients as well, which was cumbersome, so good it is not required anymore. In MariaDB 11.8 the only additional related config that might still be worth setting is tls_version = TLSv1.3 to ensure only the latest TLS protocol version is used.

Finally, test connections to ensure they work and to confirm that TLS is used by running e.g.:

shell

mariadb --user=app_user --password=your_secure_password \
 --host=192.168.1.66 -e '\s'

The result should show something along:

--------------
mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64)
...
Current user: app_user@192.168.1.66
SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK
...

If running a Debian/Ubuntu system, see the bundled README with zcat /usr/share/doc/mariadb-server/README.Debian.gz to read more configuration tips.

Should TLS encryption be used also on internal networks?

If a database server and app are running on the same private network, the chances that the connection gets eavesdropped on or man-in-the-middle attacked by a malicious user are low. However, it is not zero, and if it happens, it can be difficult to detect or prove that it didn’t happen. The benefit of using end-to-end encryption is that both the database server and the client can validate the certificates and keys used, log it, and later have logs audited to prove that connections were indeed encrypted and show how they were encrypted.

If all the computers on an internal network already have centralized user account management and centralized log collection that includes all database sessions, reusing existing SSH connections, SOCKS proxies, dedicated HTTPS tunnels, point-to-point VPNs, or similar solutions might also be a practical option. Note that the zero-configuration TLS only works with password validation methods. This means that systems configured to use PAM or Kerberos/GSSAPI can’t use it, but again those systems are typically part of a centrally configured network anyway and are likely to have certificate authorities and key distribution or network encryption facilities already set up.

In a typical software app stack however, the simplest solution is often the best and I recommend DBAs use the end-to-end TLS encryption in MariaDB 11.8 in most cases.

Hopefully with these tips you can enjoy having your MariaDB deployments both simpler and more secure than before!

Going full-time as an open source developer

Wed, 16 Apr 2025 00:00:00 +0000

After careful consideration, I’ve decided to embark on a new chapter in my professional journey. I’ve left my position at AWS to dedicate at least the next six months to developing open source software and strengthening digital ecosystems. My focus will be on contributing to Linux distributions (primarily Debian) and other critical infrastructure components that our modern society depends on, but which may not receive adequate attention or resources.

The Evolution of Open Source

Open source won. Over the 25+ years I’ve been involved in the open source movement, I’ve witnessed its remarkable evolution. Today, Linux powers billions of devices — from tiny embedded systems and Android smartphones to massive cloud datacenters and even space stations. Examine any modern large-scale digital system, and you’ll discover it’s built upon thousands of open source projects.

I feel the priority for the open source movement should no longer be increasing adoption, but rather solving how to best maintain the vast ecosystem of software. This requires building robust institutions and processes to secure proper resourcing and ensure the collaborative development process remains efficient and leads to ever-increasing quality of software.

What is Special About Debian?

Debian, established in 1993 by Ian Murdock, stands as one of these institutions that has demonstrated exceptional resilience. There is no single authority, but instead a complex web of various stakeholders, each with their own goals and sources of funding. Every idea needs to be championed at length to a wide audience and implemented through a process of organic evolution.

Thanks to this approach, Debian has been consistently delivering production-quality, universally useful software for over three decades. Having been a Debian Developer for more than ten years, I’m well-positioned to contribute meaningfully to this community.

If your organization relies on Debian or its derivatives such as Ubuntu, and you’re interested in funding cyber infrastructure maintenance by sponsoring Debian work, please don’t hesitate to reach out. This could include package maintenance and version currency, improving automated upgrade testing, general quality assurance and supply chain security enhancements.

Best way to reach me is by e-mail otto at debian.org. You can also book a 15-minute chat with me for a quick introduction.

Grow or Die

My four-year tenure as a Software Development Manager at Amazon Web Services was very interesting. I’m grateful for my time at AWS and proud of my team’s accomplishments, particularly for creating an open source contribution process that got Amazon from zero to the largest external contributor to the MariaDB open source database.

During this time, I got to experience and witness a plethora of interesting things. I will surely share some of my key learnings in future blog posts. Unfortunately, the rate of progress in this mammoth 1.5 million employee organization was slowing down, and I didn’t feel I learned much new in the last few years. This realization, combined with the opportunity cost of not spending enough time on new cutting-edge technology, motivated me to take this leap.

Being a full-time open source developer may not be financially the most lucrative idea, but I think it is an excellent way to force myself to truly assess what is important on a global scale and what areas I want to contribute to.

Working fully on open source presents a fascinating duality: you’re not bound by any external resource or schedule limitations, and the progress you make is directly proportional to how much energy you decide to invest. Yet, you also depend on collaboration with people you might never meet and who are not financially incentivized to collaborate. This will undoubtedly expose me to all kinds of challenges. But what would be better in fostering holistic personal growth? I know that deep down in my DNA, I am not made to stay cozy or to do easy things. I need momentum.

OK, let’s get going 🙂

Debian Salsa CI in Google Summer of Code 2025

Tue, 25 Mar 2025 00:00:00 +0000

Are you a student aspiring to participate in the Google Summer of Code 2025? Would you like to improve the continuous integration pipeline used at salsa.debian.org, the Debian GitLab instance, to help improve the quality of tens of thousands of software packages in Debian?

This summer 2025, I and Emmanuel Arias will be participating as mentors in the GSoC program. We are available to mentor students who propose and develop improvements to the Salsa CI pipeline, as we are members of the Debian team that maintains it.

A post by Santiago Ruano Rincón in the GitLab blog explains what Salsa CI is and its short history since inception in 2018. At the time of the article in fall 2023 there were 9000+ source packages in Debian using Salsa CI. Now in 2025 there are over 27,000 source packages in Debian using it, and since summer 2024 some Ubuntu developers have started using it for enhanced quality assurance of packaging changes before uploading new package revisions to Ubuntu. Personally, I have been using Salsa CI since its inception, and contributing as a team member since 2019. See my blog post about GitLab CI for MariaDB in Debian for a description of an advanced and extensive use case.

Helping Salsa CI is a great way to make a global impact, as it will help avoid regressions and improve the quality of Debian packages. The benefits reach far beyond just Debian, as it will also help hundreds of Debian derivatives, such as Ubuntu, Linux Mint, Tails, Purism PureOS, Pop!_OS, Zorin OS, Raspberry Pi OS, a large portion of Docker containers, and even the Windows Subsystem for Linux.

Improving Salsa CI: more features, robustness, speed

While Salsa CI with contributions from 71 people is already quite mature and capable, there are many ideas floating around about how it could be further extended. For example, Salsa CI issue #147 describes various static analyzers and linters that may be generally useful. Issue #411 proposes using libfaketime to run autopkgtest on arbitrary future dates to test for failures caused by date assumptions, such as the Y2038 issue.

There are also ideas about making Salsa CI more robust and code easier to reuse by refactoring some of the yaml scripts into independent scripts in #230, which could make it easier to run Salsa CI locally as suggested in #169. There are also ideas about improving the Salsa CI’s own CI to avoid regressions from pipeline changes in #318.

The CI system is also better when it’s faster, and some speed improvement ideas have been noted in #412.

Improvements don’t have to be limited to changes in the pipeline itself. A useful project would also be to update more Debian packages to use Salsa CI, and ensure they adopt it in an optimal way as noted in #416. It would also be nice to have a dashboard with statistics about all public Salsa CI pipeline runs as suggested in #413.

These and more ideas can be found in the issue list by filtering for tags Newcomer, Nice-To-Have or Accepting MRs. A Google Summer of Code proposal does not have to be limited to these existing ideas. Participants are also welcome to propose completely novel ideas!

Good time to also learn Debian packaging

Anyone working with Debian team should also take the opportunity to learn Debian packaging, and contribute to the packaging or maintenance of 1-2 packages in parallel to improving the Salsa CI. All Salsa CI team members are also Debian Developers who can mentor and sponsor uploads to Debian.

Maintaining a few packages is a great way to eat your own cooking and experience Salsa CI from the user perspective, and likely to make you better at Salsa CI development.

Apply now!

The contributor applications opened yesterday on March 24, so to participate act now! If you are an eligible student and want to attend, head over to summerofcode.withgoogle.com to learn more.

There are over a thousand participating organizations, with Debian, GitLab and MariaDB being some examples. Within these organizations there may be multiple subteams and projects to choose from. The full list of participating Debian projects can be found in the Debian wiki.

If you are interested in GSoC for Salsa CI specifically, feel free to

Reach out to me and Emmanuel by email at otto@ and eamanu@ (debian.org).
Sign up at salsa.debian.org for an account (note it takes a few days due to manual vetting and approval process)
Read the project README, STRUCTURE and CONTRIBUTING to get a developer’s overview
Participate in issue discussions at https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/

Note that you don’t have to wait for GSoC to officially start to contribute. In fact, it may be useful to start immediately by submitting a Merge Request to do some small contribution, just to learn the process and to get more familiar with how everything works, and the team maintaining Salsa CI. Looking forward to seeing new contributors!

Update May 8th, 2025
The approved Google Summer of Code 2025 students have now been announced, and I am thrilled that 9 students got approved for Debian. I will be mentoring two students, Aquila and Aayush. You can follow their progress on the Salsa CI Team GitLab page.

10 habits to help becoming a Debian maintainer

Sun, 26 Jan 2025 00:00:00 +0000

Becoming a Debian maintainer is a journey that combines technical expertise, community collaboration, and continuous learning. In this post, I’ll share 10 key habits that will both help you navigate the complexities of Debian packaging without getting lost and enable you to contribute more effectively to one of the world’s largest open source projects.

1. Read and re-read the Debian Policy, the Developer’s Reference and the git-buildpackage manual

Anyone learning Debian packaging and aspiring to become a Debian maintainer is likely to wade through a lot of documentation, only to realize that much of it is outdated or sometimes outright incorrect.

Therefore, it is important to learn right from the start which sources are the most reliable and truly worth reading and re-reading. I recommend these documents, in order of importance:

The Debian Policy Manual: Describes the structure of the operating system, the package archive, and requirements for packages to be included in the Debian archive.
The Developer’s Reference: A collection of best practices and process descriptions Debian packagers are expected to follow while interacting with one another.
The git-buildpackage man pages: While the Policy focuses on the end result and is intentionally void of practical instructions on creating or maintaining Debian packages, the Developer’s Reference goes into greater detail. However, it too lacks step-by-step instructions. For the exact commands, consult the man pages of git-buildpackage and its subcommands (e.g., gbp clone, gbp import-orig, gbp pq, gbp dch, gbp push). See also my post on Debian source package Git branch and tags for easy to understand diagrams.

2. Make reading man pages a habit

In addition to the above, try to make a habit of checking out the man page of every new tool you use to ensure you are using it as intended.

The best place to read accurate and up-to-date documentation is manpages.debian.org. The manual pages are maintained alongside the tools by their developers, ensuring greater accuracy than any third-party documentation.

If you are using a tool in the way the tool author documented, you can be confident you are doing the right thing, even if it wasn’t explicitly mentioned in some third-party guide about Debian packaging best practices.

3. Read and write emails

While members of the Debian community have many channels of communication, the mailing lists are by far the most prominent. Asking questions on the appropriate list is a good way to get current advice from other people doing Debian packaging. Staying subscribed to lists of interest is also a good way to read about new developments as they happen.

Note that every post is public and archived permanently, so the discussions on the mailing lists also form a body of documentation that can later be searched and referred to.

Regularly writing short and well-structured emails on the mailing lists is great practice for improving technical communication skills — a useful ability in general. For Debian specifically, being active on mailing lists helps build a reputation that can later attract collaborators and supporters for more complex initiatives.

4. Create and use an OpenPGP key

Related to reputation and identity, OpenPGP keys play a central role in the Debian community. OpenPGP is used to various degrees to sign Git commits and tags, sign and encrypt email, and — most importantly — to sign Debian packages so their origin can be verified.

The process of becoming a Debian Maintainer and eventually a Debian Developer culminates in getting your OpenPGP key included in the Debian keyring, which is used to control who can upload packages into the Debian archive.

The earlier you create a key and start using it to gain reputation for that specific key that is used to sign your work, the better. Note that due to a recent schism in the OpenPGP standards working group, it is safest to create an OpenPGP key using GnuPG version 2.2.x (not 2.4.x), or using Sequoia-PGP.

5. Integrate Salsa CI in all work

One reason Debian remains popular, even 30 years after its inception, is due to its culture of maintaining high standards. For a newcomer, learning all the quality assurance tools such as Lintian, Piuparts, Adequate, various build variations, and reproducible builds may be overwhelming. However, these tasks are easier to manage thanks to Salsa CI, the continuous integration pipeline in Debian that runs tests on every commit at salsa.debian.org.

The earlier you activate Salsa CI in the package repository you are working on, the faster you will achieve high quality in your package with fewer missteps. You can also further customize a package-specific salsa-ci.yml to have more testing coverage.

6. Fork on Salsa and use draft Merge Requests to solicit feedback

All modern Debian packages are hosted on salsa.debian.org. If you want to make a change to any package, it is easy to fork, make an initial attempt at the change, and publish it as a draft Merge Request (MR) on Salsa to solicit feedback.

People might have surprising reasons to object to the change you propose, or they might need time to get used to the idea before agreeing to it. Also, some people might object to a vague idea out of suspicion but agree once they see the exact implementation. There may also be a surprising number of people supporting your idea, and if there is an MR, they have a place to show their support.

Don’t expect every Merge Request to be accepted. However, proposing an idea as running code in an MR is far more effective than raising the idea on a mailing list or in a bug report. Get into the habit of publishing plenty of merge requests to solicit feedback and drive discussions toward consensus.

7. Use git rebase frequently

Linear Git history is much easier to read. The ease of reading git log and git blame output is vital in Debian, where packages often have updates from multiple people spanning many years — even decades. Debian packagers likely spend more time than the average software developer reading Git history.

Make sure you master Git commands such as gitk --all, git citool --amend, git commit -a --fixup <commit id>, git rebase -i --autosquash <target branch>, git cherry-pick <commit id 1> <id 2> <id 3>, and git pull --rebase.

If rebasing is not done on your initiative, rest assured others will ask you to do it. Thus, if the commands above are familiar, rebasing will be quick and easy for you.

8. Reviews: give some, get some

In open source, the larger a project becomes, the more it attracts contributions, and the bottleneck for its growth isn’t how much code developers can create but how much code submissions can be properly reviewed.

At the time of writing, the main Salsa group “Debian” has over 800 open merge requests pending reviews and approvals. Feel free to read and comment on any merge request you find. You don’t have to be a subject matter expert to provide valuable feedback. Even if you don’t have specific feedback, your comment as another human acknowledging that you read the MR and found no issues is viewed positively by the author. Besides, if you spend enough time reviewing MRs in a specific domain, you will eventually become an expert in it. Code reviews are not just about providing feedback to the submitter; they are also great learning opportunities for the reviewer.

As a rule of thumb, you should review at least twice as many merge requests as you submit yourself.

9. Improve Debian by improving upstream

It is common that while packaging software for Debian, bugs are uncovered and patched in Debian. Do not forget to submit the fixes upstream, and add a Forwarded field to the file in debian/patches! As the person building and packaging something in Debian, you automatically become an authority on that software, and the upstream is likely glad to receive your improvements.

While submitting patches upstream is a bit of work initially, getting improvements merged upstream eventually saves time for everyone and makes packaging in Debian easier, as there will be fewer patches to maintain with each new upstream release.

10. Don’t hold any habits too firmly

Last but not least: Once people learn a specific way of working, they tend to stick to it for decades. Learning how to create and maintain Debian packages requires significant effort, and people tend to stop learning once they feel they’ve reached a sufficient level. This tendency to get stuck in a “local optimum” is understandable and natural, but try to resist it.

It is likely that better techniques will evolve over time, so stay humble and re-evaluate your beliefs and practices every few years.

Mastering these habits takes time, but each small step brings you closer to making a meaningful impact on Debian. By staying curious, collaborative, and adaptable, you can ensure your contributions stand the test of time — just like Debian itself. Good luck on your journey toward becoming a Debian Maintainer!

Advanced Git commands every senior software developer needs to know

Thu, 29 Feb 2024 00:00:00 +0000

Git is by far the most popular software version control system today, and every software developer surely knows the basics of how to make a Git commit. Given the popularity, it is surprising how many people don’t actually know the advanced commands. Mastering them might help you unlock a new level of productivity. Let’s dive in!

Avoid excess downloads with selective and shallow Git clone

When working with large Git repositories, it is not always desirable to clone the full repository as it would take too long to download. Instead you can execute the clone for example like this:

git clone --branch 11.5 --shallow-since=3m https://github.com/MariaDB/server.git mariadb-server

This will make a clone that only tracks branch 11.5 and no other branches. Additionally, this uses the shallow clone feature to fetch commit history only for the past 3 months instead of the entire history (which in this example would otherwise be 20+ years). You could also specify 3w or 1y to fetch three weeks or one year. After the initial clone, you can use git remote set-branches --add origin 10.11 to start tracking an additional branch, which will be downloaded on git fetch.

If you already have a git repository, and all you want to do is fetch one single branch from a remote repository one-off, without adding it as a new remote, you can run:

$ git fetch https://github.com/robinnewhouse/mariadb-server.git ninja-build-cracklib
From https://github.com/robinnewhouse/mariadb-server
* branch ninja-build-cracklib -> FETCH_HEAD
$ git merge FETCH_HEAD
Updating 112eb14f..c649d78a
Fast-forward
plugin/cracklib_password_check/CMakeLists.txt | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
$ git show
commit c649d78a8163413598b83f5717d3ef3ad9938960 (HEAD -> 11.5)
Author: Robin

This is a very fast and small download, which will not persist as a remote. It creates a temporary Git reference called FETCH_HEAD, which you can then use to inspect the branch history by running git show FETCH_HEAD, or you can merge it, cherry-pick, or perform other operations.

If you want to download the bare minimum, you can even operate on individual commits as raw patch files. A typical example would be to download a GitHub Pull Request as a patch file and apply it locally:

$ curl -LO https://patch-diff.githubusercontent.com/raw/MariaDB/server/pull/3026.patch
$ git am 3026.patch
Applying: Fix ninja build for cracklib_password_check
$ git show
commit a9c44bc204735574f2724020842373b53864e131 (HEAD -> 11.5)
Author: Robin

The same works for GitLab Merge Requests as well – just add .patch at the end of the MR url. This will apply both the code change inside the patch, as well as honor the author field, and use the patch description as the commit subject line and message body. However, when running git am, the committer name, email, and date will be that of the user applying the patch, and thus the SHA-sum of the commit ID will not be identical.

The latest Git has a new experimental command sparse-checkout that allows one to checkout only a subset of files, but I won’t recommend it as this post is purely about best practices and tips I myself find frequently useful to know.

Inspecting Git history and comparing revisions

The best command to view the history of a single file is:

git log --oneline --follow path/to/filename.ext

The extra --follow makes Git traverse the history longer to find if the same contents existed previously with a different file name, thus showing file contents across file renames. Using --oneline provides a nice short list of just the Git subject lines. To view the full Git commit messages as well as the actual changes, use this:

git log --patch --follow path/to/filename.ext

If there is a specific change you are looking for, search it with git log --patch -S <keyword>.

To view the project history in general, having this alias is handy:

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'"

The output shows all references, multiple branches in parallel and it is nicely colorized. If the project has a lot of messy merges, sticking to one branch may be more readable:

git log --oneline --no-merges --first-parent

However, an even better option is to use gitk --all &. This standard Git graphical user interface allows you to browse the history, search for changes with a specific string, jump to a specific commit to quickly inspect it and what preceded it, open a graphical Git blame in a new window, etc. The --all instructs gitk to show all branches and references, and the ampersand backgrounds the process so that your command-line prompt is freed to run other commands. If your workflow is based on working over SSH on a remote server, simply connect with ssh -X remote.server.example to have X11 forwarding enabled (only works on Linux). Then on the SSH command-line just run gitk --all & and a window should pop up.

laptop$ ssh -X remote-server.example.com
server$ echo $DISPLAY
:0 (X11 forwarding is enabled and xauth running)
server$ cd path/to/git/repo
server$ gitk --all &

A typical need is also to compare the files and changes across multiple commits or branches using Git diff. The nicer graphical option to it is to run git difftool --dir-diff branch1..branch2 which will open the diff program of your choice. Personally I have opted to always use Meld with git config diff.tool meld.

Committing, rebasing, cherry-picking and merging

When making a Git commit, doing it graphically with git citool helps to clearly see what changes have been made, and to select the files and even the exact lines to be committed with the click of a mouse. The tool also offers built-in spell-checking, and the text box is sized just right to visually enforce keeping line lengths within limits. Since development involves committing and amending commits all the time, I recommend having these aliases:

alias g-commit='git citool &'
alias g-amend='git citool --amend &'

Personally, I practically never commit by simply running git commit. If I commit from the command line at all, it is usually due to the need to do something special, such as change the author with:

git commit --amend --no-edit --author "Otto Kekäläinen <otto@debian.org>"

Another case where a command-line commit fits my workflow well is during final testing before a code submission when I find a flaw on the branch I am working on. In these cases, I fix the code, and quickly issue:

git commit -a --fixup a1b2c3
git rebase -i --autosquash main

This will commit the change, mark it as a fix for commit a1b2c3, and then open the interactive rebase view with the fixup commit automatically placed at the right location, resulting in a quick turnaround to make the branch flawless and ready for submission.

Occasionally a Git commit needs to be applied to multiple branches. For example, after making a bugfix with the id a1b2c3 on the main branch, you might want to backport it to release branches 11.4 and 11.3 with:

git cherry-pick -x a1b2c3

The extra -x will make Git amend the commit message with a reference to the commit id it originated from. In this case, it would state: (cherry picked from commit a1b2c3). This helps people reading the commit messages later to track down when and where the commit was first made.

When doing merges, the most effective way to handle conflicts is by using Meld to graphically compare and resolve merges:

$ git merge branch2
Auto-merging VERSION
CONFLICT (content): Merge conflict in SOMEFILE
Automatic merge failed; fix conflicts and then commit the result.
$ git mergetool
Merging:
SOMEFILE
Normal merge conflict for 'SOMEFILE':
{local}: modified file
{remote}: modified file
$ git commit -a
[branch1 e4952e06] Merge branch 'branch2' into branch1

One more thing to remember is that if a merge or rebase fails, remember to run git merge --abort or git rebase --abort to stop it and get back to the normal state. Another typical need is to discard all temporary changes and get back to a clean state ready to do new commits. For that I recommend this alias:

alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

This will reset all modified files to their pristine state from the last commit, as well as delete all files that are not in version control but may be present in the project directory.

Managing multiple remotes and branches

The most important tip for working with Git repositories is to remember at the start of every coding session to always run git remote update. This will fetch all remotes and make sure you have all the latest Git commits made since the last time you worked with the repository.

$ git remote update
Fetching origin
Fetching upstream
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 3 (delta 2), pack-reused 0
Unpacking objects: 100% (3/3), 445 bytes | 55.00 KiB/s, done.
From https://github.com/eradman/entr
e2a6ab7..6fa963e master -> upstream/master

In the example above, you can see that there isn’t just the origin, but also a second remote called upstream. Most people use Git in a centralized model, meaning that there is one central main repository on e.g. GitHub or GitLab, and each developer in the project pushes and pulls that central repository. However, Git was designed from the start to be a distributed system that can sync with multiple remotes. To understand how to control this one needs to learn the concept of tracking branches and learn the options of the git remote command.

Consider this example that has two remotes, origin and upstream, and the origin remote has 3 push urls:

$ git remote -v
origin git@salsa.debian.org:debian/entr.git (fetch)
origin git@salsa.debian.org:debian/entr.git (push)
origin git@gitlab.com:ottok/entr.git (push)
origin git@github.com:ottok/entr.git (push)
upstream https://github.com/eradman/entr (fetch)
upstream https://github.com/eradman/entr (push)
$ cat .git/config
[remote "origin"]
url = git@salsa.debian.org:debian/entr.git
fetch = +refs/heads/*:refs/remotes/origin/*
pushurl = git@salsa.debian.org:debian/entr.git
pushurl = git@gitlab.com:ottok/entr.git
pushurl = git@github.com:ottok/entr.git
[remote "upstream"]
url = https://github.com/eradman/entr
fetch = +refs/heads/*:refs/remotes/upstream/*
[branch "debian/latest"]
remote = origin
merge = refs/heads/debian/latest
[branch "master"]
remote = upstream
merge = refs/heads/master

In this repository, the branch master is configured to track the remote upstream. Thus, if I am in the branch master and run git pull it will fetch master from the upstream repository. I can then checkout the debian/latest branch, merge on upstream and do other changes. Eventually, when I am done and issue git push, the changes on branch debian/latest will go to remote origin automatically. The origin has 3 pushurls, which means that the updated debian/latest will end up on both the Debian server as well as GitHub and GitLab.

The commands to set this up were:

git clone git@salsa.debian.org:debian/entr.git
cd entr
git remote set-url --add --push origin git@salsa.debian.org:otto/entr.git
git remote set-url --add --push origin git@gitlab.com:ottok/entr.git
git remote set-url --add --push origin git@github.com:ottok/entr.git
git remote add upstream https://github.com/eradman/entr

Keeping repositories nice and tidy

As most developers use feature and bug branches to make changes and submit them for review, a lot of old and unnecessary branches will start to pollute the Git history over time. Therefore it is good to check from time to time what branches have been merged with git branch --merged and delete them.

If a branch is deleted remotely as a result of somebody else doing cleanup, you can make Git automatically delete those branches for you locally as well with git config --local fetch.prune true. You can run this one-off as well with git fetch --prune --verbose --dry-run.

When working with multiple remotes, it might at times be hard to reason what will happen on a Git pull or Git push command. To see what tags and branches are updated and how without actually updating them run:

git fetch --verbose --dry-run
git push --verbose --dry-run
git push --tags --verbose --dry-run

Using the --dry-run option is particularly important when running push or pull with --prune or --prune-tags to see which branches or tags would be deleted locally or on the remote.

Another maintenance task to occasionally spend time on is to run this command to make Git delete all unreachable objects and to pack the ones that should be kept forever:

git prune --verbose --progress; git repack -ad; git gc --aggressive; git prune-packed

To do this for every Git repository on your computer, you can run:

find ~ -name .git -type d | while read D
do
echo "===> $D: "
(cd "$D"; git prune --verbose --progress; nice -n 15 git repack -ad; nice -n 15 git gc --aggressive; git prune-packed)
done

Better Git experience with Liquip Prompt and fzf

It is not practical to constantly run git status (or git status --ignored) or to press F5 in a gitk window to be aware of the Git repository status. A much handier solution is to have the Git status integrated in the command-line prompt. My favorite is Liquid Prompt, which shows the branch name, and displays green if everything is committed and clean, red if there are uncommitted changes, and yellow if changes are not pushed.

Another additional tool I recommend is the Fuzzy Finder fzf. It has many uses in the command-line environment, and for Git this alias is handy for changing branches:

alias g-checkout="git checkout "$(git branch --sort=-committerdate --no-merged | fzf)""

This will list all local branches with the recent ones topmost, and present the list in an interactive form using fzf so you can select the branch either using arrow keys, or typing a part of the branch name.

Bash aliases

While Git has its own alias system, I prefer to have everything in plain Bash aliases defined in my .bashrc. Many of these are explained in this post, but there are a couple extra as well. I leave it up to the reader to study the Git man page to learn for example what git push --force-with-lease does.

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'"
alias g-history='gitk --all &'
alias g-checkout='git checkout $(git branch --sort=-committerdate --no-merged | fzf)'
alias g-commit='git citool &'
alias g-amend='git citool --amend &'
alias g-fixup='git commit -a --fixup'
alias g-rebase='git rebase --interactive --autosquash'
alias g-pull='git pull --verbose --rebase'
alias g-pushf='git push --verbose --force-with-lease'
alias g-status='git status --ignored'
alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

Keep on learning

As a programmer, it is not enough to know programming languages and how to write code well. You also need to understand the software lifecycle and change management. Understanding Git deeply helps you better prepare for situations where potentially hundreds of people collaborate on the same code base for years and years.

To learn more about Git concepts, I recommend reading the entire Pro Git book. The original version is over a decade old, but the online version keeps getting regular updates by people contributing to it in the open source spirit. As an example, I wrote a new section last year about automatically signing Git commits. Skimming through the Git reference documentation (online version of man pages) is also a great way to become aware of what capabilities Git offers.

What is your favorite command-line Git trick or favorite tool? Comment below.

How to make a good git commit

Sun, 26 Mar 2023 00:00:00 +0000

As a software developer, your core skill is how to improve an existing code base to make the software better iteratively, patch by patch.

To be a good software developer you need to:

understand the concept of a code patch,
know how to do code improvements in well sized and properly documented patches, and
skillfully use git version control software to manage patches.

What is a patch?

A patch defines the changes to be made to the code base. It is basically a list of code lines to be added, removed or modified in a code base. Each patch always also has an author, a timestamp when it was written, a title that describes it and a longer text body that explains why this particular patch is good and applying it on the code base is beneficial.

Example:

patch

Author: Otto Kekäläinen
Date: June 22nd, 2022 08:08:08

Make output friendlier for users

Add line break so text is readable and add a 2 second delay between
messages so it does not scroll too fast.

--- a/demo.c
+++ b/demo.c
@@ -8,7 +8,8 @@ int main()
 {
 for(;;)
 {
- printf("Hello world!");
+ printf("Hello world!\n");
+ sleep(2);
 }
 return 0;
 }

How to make a patch

You can make a patch by simply copying a file, changing something in it, and then comparing the copy to the original file using the command diff and saving the output.

$ cp demo.c demo.c.orig
$ nano demo.c
$ diff -u demo.c.orig demo.c > demo.patch
$ cat demo.patch

patch

--- demo.c.orig
+++ demo.c
@@ -8,7 +8,8 @@ int main()
 {
 for(;;)
 {
- printf("Hello world!");
+ printf("Hello world!\n");
+ sleep(2);
 }
 return 0;
 }

The patch can be sent by email or uploaded somewhere. After that, anybody can download the patch, read it, and apply it to their copy of the code base using the command patch.

shell

$ grep Hello demo.c
 printf("Hello world!");

$ curl -O https://…/demo.patch

$ patch -p0 < demo.patch
patching file demo.c

$ grep Hello demo.c
 printf("Hello world!\n");

As this is not very fast nor convenient, software developers like to use git, a version control software that automates all of this. In git, we tend to talk about git commits, which basically just means a patch that has been applied on a code base.

Examples of good git commit messages

A good git commit message typically has these characteristics (adapted from the Git Pro book):

Capitalized, short summary of what the change is
More detailed explanatory text that focuses on the 'why' to motivate
the change. Use present tense and imperative format (write "Fix bug",
not "Fixed bug"). Wrap it to about 72 characters or so. The blank line
separating the summary from the body is critical.
Further paragraphs come after blank lines.
- Bullet points are okay, too
- Typically a hyphen or asterisk is used for the bullet, followed by a
single space, with blank lines in between, but conventions vary here
- Use a hanging indent

Here are a couple of real-world examples in pure text form:

From MariaDB@ff1d8fa7:

Deb: Clean away Buster to Bookworm upgrade tests in Salsa-CI
Upgrades from Debian 10 "Buster" directly to Debian 12 "Bookworm",
skipping Debian 11 "Bullseye", fail with apt erroring on:
libcrypt.so.1: cannot open shared object file
This is an intentional OpenSSL transition as described in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755

From MariaDB@2c529441:

Deb: Run wrap-and-sort -av
Sort and organize the Debian packaging files.
Also revert 4d03269 that was done in vain.

Five requirements for a good git commit

In order of importance:

1. Commits should be atomic

The first and most important thing about a good patch or a commit is that it should be a self-standing change. If a commit fixes a bug, it should not at the same time add a new feature or fix some other completely unrelated bug, otherwise it is not atomic. If you add a new feature, the same commit should ideally also add automatic tests for the feature to ensure it won’t regress, and the same commit should update the documentation to mention the feature, as it is all related and should either go or not go into the code base along with the feature itself.

If your changes are not properly scoped and self-standing, you might end up in a situation later on where somebody decides to revert or reject the commit that introduced a new feature, but miss removing the tests or documentation about it, which would not have happened if they were added in separate commits.

There is no clear rule on what is the optimal scope for a commit; it is something you will learn by experience. Sometimes it makes sense to have several separate changes in one single commit simply because of each one of them being so small. In other cases, one single logical change might span multiple commits, because it was perhaps clearer to move or rename files in one commit and then update their commits in another. This is something you will just have to learn over time as you become a more experienced software developer.

2. The title should be descriptive, yet terse and not too long

A title starts with a capital letter and has no trailing dot. Just like the subject line in an email. The title should make sense when read in a list of commits. If the title is too long, it will be cut off. A limit of 72 characters is safe to all typical places where people will be reading it, such as in a terminal window or when browsing GitHub or GitLab, but striving for under 50 characters is even better.

3. The commit message should explain why it was made

The text should be verbose enough for anybody reviewing the commit to understand why it was made, and to be convinced that the change is good. Every commit must have a text body, even if it is very short. This forces the author to spend a few seconds thinking about the change before committing.

Note that the commit message is about the change itself, so it should answer the question ‘why’. If you want to explain how a certain line of code works, simply use an inline comment next to the code itself. That way, the documentation is in the correct context. The git commit description should have just a tiny bit of ‘what’ and ‘how’, and mostly focus on the ‘why’.

The commit body should be wrapped at about 72 characters. Proper use of empty lines and lists that are indented with a dash or star makes the body more readable.

Remember to use imperative format. Don’t write Fixed bug or Added feature. Instead write Fix bug or Add feature. The patch hasn’t added or fixed anything at the time you wrote it. Think about it like an order you give to the code base to start following. Also, try to keep your text in the current tense and passive format. Don’t write This commit makes X but simply Make X. Don’t write I changed Y but just simply Change Y.

4. Use references when available

If your code change is related to a previous commit, mention the commit ID. In most software commits, IDs will automatically become links. If the code change is related to something that was discussed or tracked elsewhere, please include the bug tracker ID or a URL to the discussion. However, the reference alone does not remove the need to write a git commit message. You cannot expect that somebody reading your commit has time or even access to open and read all references - use them only as pointers for more information.

Viewing one of the earlier git commit message examples in gitk, GitLab and GitHub illustrates how a git commit automatically becomes a link:

5. Maintain correct authorship and copyright credits

The author name and timestamp are automatic if you configure your git correctly, so this should be a non-issue. If you neglect to configure git with your real name and email, you will be muddling the waters for anybody who later wants to verify something about authorship. In the worst case scenario, all your commits might be purged from the git repository due to unclear copyright.

Also keep in mind that if you commit code on behalf of somebody else, you must tell git that the author for a particular commit was somebody else and you only committed it. Read up on git commit –author for details.

The right tools make git commits easy

Using a good tool to craft your git commits goes a long way in making the commit flawless.

My personal choice is git-citool, which is distributed together with git itself, so anybody can use it on any operating system. It does not use the native graphics of each operating system, but a cross-platform graphics library, which may look a bit ugly. It is, however, very easy and convenient to use, so I love it.

To make a new commit, simply run git citool. It starts off empty and then you can select which files you want to stage, and write the git commit message in this box, and press commit. Super easy, and it is very clear what changes you are committing.

Don’t settle for bad commits - amend them

If you are not happy with your commit and want to edit it, or to use git terminology you want to “amend”, this is possible only for the topmost git commit that does not have any child commits yet. Run git-citool --amend.

Here you can see a git commit that is really bad, so it really needs to be fixed. However, with git-citool it is easy and fast.

WIP commits: how to avoid postponing writing the perfect git commit message

Remember that you don’t have to make a perfect git commit right off the bat. Do it only once you know what you actually want to write in it. While still working on the code and saving intermediate versions of it, I recommend using WIP commits where the title is simply WIP, or if you already have some commit text draft, prefix the title with WIP:.

Use `git rebase -i` frequently

When you are done with WIP commits, you can run git rebase -i to squash them together and write the final git commit message.

For a visual explanation see the presentation A Branch in Time (a story about revision histories) by Tekin Süleyman on how to use interactive rebase in git and why rebasing and amending commits will end up making the code quality better in the long run.

A polished git commit is always worth the effort

Someone who is lazy might say that while they agree with the principles, they don’t have time to follow them. To that I respond that doing things correctly from the onset actually saves time down the road.

If your git commits are good, the job of the reviewer will be much easier. They won’t waste time on just trying to understand your change, but they will get it directly and will be able to focus their energy on actually reviewing and spotting flaws in your code. If you avoid shipping a bug, you save a lot of work not having to debug, write a fix and ship a new release.
A great git commit is also useful even if it later turns out the commit had a bug, because whoever fixes that bug will have a much easier time reading in the commit what the change was supposed to do, and understanding where it fell short, and then making the same change in the correct way. This leads to bugs being fixed much more quickly and with less effort - and most often the person doing the fix is a future you who no longer remembers what the present you was thinking while making that commit, with the future you just having to stare at the commit until it makes sense.
You don’t have to rewrite anything when it comes time to submit the commit for review. Every single code review system I have ever used will automatically use the commit title and message as the review title and message if the review is a single commit review.

Now go and build great software – patch by patch!

Now you know how to make a good git commit message. If you are proud of your work and like doing things well, you will follow these guidelines. To further learn how to polish your git commit messages, see also the post on git commit messages by example.

The Linux kernel developer’s guide has also an excellent description of how to separate changes into self-standing logical changes, and how to describe the changes.

Quick builds and rebuilds of MariaDB using Docker

Sun, 12 Mar 2023 00:00:00 +0000

The MariaDB server has over 2 million lines of code. Downloading, compiling (and re-compiling), and running the test suite can potentially consume a lot of time away from actually making the code changes and being productive. Knowing a few simple shortcuts can help avoid wasting time.

While the official build instructions on mariadb.org and mariadb.com/kb are useful to read, there are ways to make the build (and rebuild) significantly faster and more efficient.

TL;DR for Debian/Ubuntu users
Get the latest MariaDB 11.0 source code, install build dependencies, configure, build and run test suite to validate binaries work:
shell
mkdir quick-rebuilds cd quick-rebuilds git clone --branch 11.0 --shallow-since=3m \ --recurse-submodules --shallow-submodules \ https://github.com/MariaDB/server.git mariadb-server mkdir -p ccache build data docker run --interactive --tty --rm -v ${PWD}:/quick-rebuilds \ -w /quick-rebuilds debian:sid bash echo 'deb-src http://deb.debian.org/debian sid main' \ > /etc/apt/sources.list.d/deb-src-sid.list apt update apt install -y --no-install-recommends \ devscripts equivs ccache eatmydata ninja-build clang entr moreutils mk-build-deps -r -i mariadb-server/debian/control \ -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends' export CCACHE_DIR=$PWD/ccache export CXX=${CXX:-clang++} export CC=${CC:-clang} export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++} export CC_FOR_BUILD=${CC_FOR_BUILD:-clang} export CFLAGS='-Wno-unused-command-line-argument' export CXXFLAGS='-Wno-unused-command-line-argument' cmake -S mariadb-server/ -B build/ -G Ninja --fresh \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \ -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \ -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF eatmydata cmake --build build/ ./build/mysql-test/mysql-test-run.pl --force --parallel=auto
mkdir quick-rebuilds
cd quick-rebuilds
git clone --branch 11.0 --shallow-since=3m \
 --recurse-submodules --shallow-submodules \
 https://github.com/MariaDB/server.git mariadb-server
mkdir -p ccache build data
docker run --interactive --tty --rm -v ${PWD}:/quick-rebuilds \
 -w /quick-rebuilds debian:sid bash
echo 'deb-src http://deb.debian.org/debian sid main' \
 > /etc/apt/sources.list.d/deb-src-sid.list
apt update
apt install -y --no-install-recommends \
 devscripts equivs ccache eatmydata ninja-build clang entr moreutils
mk-build-deps -r -i mariadb-server/debian/control \
 -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends'
export CCACHE_DIR=$PWD/ccache
export CXX=${CXX:-clang++}
export CC=${CC:-clang}
export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++}
export CC_FOR_BUILD=${CC_FOR_BUILD:-clang}
export CFLAGS='-Wno-unused-command-line-argument'
export CXXFLAGS='-Wno-unused-command-line-argument'
cmake -S mariadb-server/ -B build/ -G Ninja --fresh \
 -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \
 -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \
 -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \
 -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF
eatmydata cmake --build build/
./build/mysql-test/mysql-test-run.pl --force --parallel=auto
To rebuild after code change simply run:
shell
eatmydata cmake --build build/
eatmydata cmake --build build/
For full details, read the whole article.

Stay organized, keep directories clean

The first step is to create the working directory and some directories inside it to:

shell

mkdir quick-rebuilds
cd quick-rebuilds
mkdir -p ccache build data

The directory ccache will be used by the tool with the same name to store build cache permanently. Build artifacts will be output in the directory build to avoid polluting the source code directory so that Git in the source tree will not accidentally commit any machine-generated files. The data directory is useful for temporary test installs.

The next step is to get the source code into this working directory.

Don’t download the whole project – use shallow Git clone

The oldest Git commit in the project is from July, 2000. Since then, MariaDB has had nearly 200 000 commits. To build the latest version and perhaps submit a Pull Request to commit your improvement to the project, you don’t necessarily need to have all those 200 000 commits available in your Git clone. You can use shallow Git clone to, for example, fetch only the history of the past 3 months:

$ git clone --branch 11.0 --shallow-since=3m \
--recurse-submodules --shallow-submodules \
https://github.com/MariaDB/server.git mariadb-server
Cloning into 'mariadb-server'...
remote: Enumerating objects: 41075, done.
remote: Counting objects: 100% (41075/41075), done.
remote: Compressing objects: 100% (29333/29333), done.
remote: Total 41075 (delta 19706), reused 20092 (delta 10708), pack-reused 0
Receiving objects: 100% (41075/41075), 75.85 MiB | 8.48 MiB/s, done.
Resolving deltas: 100% (19706/19706), done.
Checking out files: 100% (24070/24070), done.
Submodule 'extra/wolfssl/wolfssl' (https://github.com/wolfSSL/wolfssl.git) registered for path 'extra/wolfssl/wolfssl'
Submodule 'libmariadb' (https://github.com/MariaDB/mariadb-connector-c.git) registered for path 'libmariadb'
Submodule 'storage/columnstore/columnstore' (https://github.com/mariadb-corporation/mariadb-columnstore-engine.git) registered for path 'storage/columnstore/columnstore'
Submodule 'storage/maria/libmarias3' (https://github.com/mariadb-corporation/libmarias3.git) registered for path 'storage/maria/libmarias3'
Submodule 'storage/rocksdb/rocksdb' (https://github.com/facebook/rocksdb.git) registered for path 'storage/rocksdb/rocksdb'
Submodule 'wsrep-lib' (https://github.com/codership/wsrep-lib.git) registered for path 'wsrep-lib'
Cloning into '/srv/sources/mariadb/quick-rebuilds/mariadb-server/extra/wolfssl/wolfssl'...
remote: Enumerating objects: 2851, done.
remote: Counting objects: 100% (2851/2851), done.
remote: Compressing objects: 100% (2124/2124), done.
remote: Total 2851 (delta 800), reused 1576 (delta 589), pack-reused 0
Receiving objects: 100% (2851/2851), 20.91 MiB | 10.43 MiB/s, done.
Resolving deltas: 100% (800/800), done.
...
Unpacking objects: 100% (3/3), done.
From https://github.com/codership/wsrep-API
* branch 694d6ca47f5eec7873be99b7d6babccf633d1231 -> FETCH_HEAD
Submodule path 'wsrep-lib/wsrep-API/v26': checked out '694d6ca47f5eec7873be99b7d6babccf633d1231'
$ git -C mariadb-server/ show --oneline --summary
f2dc4d4c (HEAD -> 11.0, origin/HEAD, origin/11.0) MDEV-30673 InnoDB recovery hangs when buf_LRU_get_free_block
$ git -C mariadb-server submodule
4fbd4fd36a21efd9d1a7e17aba390e91c78693b1 extra/wolfssl/wolfssl (4fbd4fd)
12bd1d5511fc2ff766ff6256c71b79a95739533f libmariadb (12bd1d5)
8b032853b7a200d9af4d468ac58bb9f4b6ac7040 storage/columnstore/columnstore (8b03285)
3846890513df0653b8919bc45a7600f9b55cab31 storage/maria/libmarias3 (3846890)
bba5e7bc21093d7cfa765e1280a7c4fdcd284288 storage/rocksdb/rocksdb (bba5e7b)
275a0af8c5b92f0ee33cfe9e23f3db5f59b56e9d wsrep-lib (275a0af)
$ du -shc mariadb-server/.git/modules/{storage/*,extra/wolfssl,libmariadb,wsrep-lib} \
mariadb-server/.git mariadb-server/
30M mariadb-server/.git/modules/storage/columnstore
1M mariadb-server/.git/modules/storage/maria
20M mariadb-server/.git/modules/storage/rocksdb
40M mariadb-server/.git/modules/extra/wolfssl
2M mariadb-server/.git/modules/libmariadb
1M mariadb-server/.git/modules/wsrep-lib
80M mariadb-server/.git
548M mariadb-server/
=720M total

$ git clone --branch 11.0 --shallow-since=3m \
--recurse-submodules --shallow-submodules \
https://github.com/MariaDB/server.git mariadb-server
Cloning into 'mariadb-server'...
remote: Enumerating objects: 41075, done.
remote: Counting objects: 100% (41075/41075), done.
remote: Compressing objects: 100% (29333/29333), done.
remote: Total 41075 (delta 19706), reused 20092 (delta 10708), pack-reused 0
Receiving objects: 100% (41075/41075), 75.85 MiB | 8.48 MiB/s, done.
Resolving deltas: 100% (19706/19706), done.
Checking out files: 100% (24070/24070), done.
Submodule 'extra/wolfssl/wolfssl' (https://github.com/wolfSSL/wolfssl.git) registered for path 'extra/wolfssl/wolfssl'
Submodule 'libmariadb' (https://github.com/MariaDB/mariadb-connector-c.git) registered for path 'libmariadb'
Submodule 'storage/columnstore/columnstore' (https://github.com/mariadb-corporation/mariadb-columnstore-engine.git) registered for path 'storage/columnstore/columnstore'
Submodule 'storage/maria/libmarias3' (https://github.com/mariadb-corporation/libmarias3.git) registered for path 'storage/maria/libmarias3'
Submodule 'storage/rocksdb/rocksdb' (https://github.com/facebook/rocksdb.git) registered for path 'storage/rocksdb/rocksdb'
Submodule 'wsrep-lib' (https://github.com/codership/wsrep-lib.git) registered for path 'wsrep-lib'
Cloning into '/srv/sources/mariadb/quick-rebuilds/mariadb-server/extra/wolfssl/wolfssl'...
remote: Enumerating objects: 2851, done.
remote: Counting objects: 100% (2851/2851), done.
remote: Compressing objects: 100% (2124/2124), done.
remote: Total 2851 (delta 800), reused 1576 (delta 589), pack-reused 0
Receiving objects: 100% (2851/2851), 20.91 MiB | 10.43 MiB/s, done.
Resolving deltas: 100% (800/800), done.
...
Unpacking objects: 100% (3/3), done.
From https://github.com/codership/wsrep-API
* branch 694d6ca47f5eec7873be99b7d6babccf633d1231 -> FETCH_HEAD
Submodule path 'wsrep-lib/wsrep-API/v26': checked out '694d6ca47f5eec7873be99b7d6babccf633d1231'
$ git -C mariadb-server/ show --oneline --summary
f2dc4d4c (HEAD -> 11.0, origin/HEAD, origin/11.0) MDEV-30673 InnoDB recovery hangs when buf_LRU_get_free_block
$ git -C mariadb-server submodule
4fbd4fd36a21efd9d1a7e17aba390e91c78693b1 extra/wolfssl/wolfssl (4fbd4fd)
12bd1d5511fc2ff766ff6256c71b79a95739533f libmariadb (12bd1d5)
8b032853b7a200d9af4d468ac58bb9f4b6ac7040 storage/columnstore/columnstore (8b03285)
3846890513df0653b8919bc45a7600f9b55cab31 storage/maria/libmarias3 (3846890)
bba5e7bc21093d7cfa765e1280a7c4fdcd284288 storage/rocksdb/rocksdb (bba5e7b)
275a0af8c5b92f0ee33cfe9e23f3db5f59b56e9d wsrep-lib (275a0af)
$ du -shc mariadb-server/.git/modules/{storage/*,extra/wolfssl,libmariadb,wsrep-lib} \
mariadb-server/.git mariadb-server/
30M mariadb-server/.git/modules/storage/columnstore
1M mariadb-server/.git/modules/storage/maria
20M mariadb-server/.git/modules/storage/rocksdb
40M mariadb-server/.git/modules/extra/wolfssl
2M mariadb-server/.git/modules/libmariadb
1M mariadb-server/.git/modules/wsrep-lib
80M mariadb-server/.git
548M mariadb-server/
=720M total

With a 3-month history, the main Git data for MariaDB is about 50 MB, and the submodules as shallow clones add 30 MB more. If not using shallow cloning, the whole MariaDB repository and submodules would amount to over 1 GB of data, so using shallow clones cuts the amount of data to be downloaded by over 80%.

The checked out data is almost 550 MB, but that is unpacked from the Git data, so actual network transfer was at max 80 MB of Git data.

Build inside a throwaway container

In addition to the source code, one also needs a long list of build dependencies installed. Instead of polluting your laptop/workstation with tens of new libraries, install all the dependencies inside a container that has a working directory mounted inside it. This way your system will stay clean, but files written in the working directory will be accessible both inside and outside the container and persist after the container is gone.

Next, start the container:

shell

docker run --interactive --tty --rm \
 -v ${PWD}:/quick-rebuilds -w /quick-rebuilds debian:sid bash

This example uses Docker, but the principle is the same with any Linux container tool, such as Podman.

Inside the Debian container, use apt to automatically install all dependencies (about 160 MB download, over 660 MB when unpacked to disk) as defined in MariaDB sources file debian/control:

shell

echo 'deb-src http://deb.debian.org/debian sid main' \
 > /etc/apt/sources.list.d/deb-src-sid.list
apt update
apt install -y --no-install-recommends \
 devscripts equivs ccache eatmydata ninja-build clang entr moreutils
mk-build-deps -r -i mariadb-server/debian/control \
 -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends'

The single biggest boost to the (re-)compilation speed is gained with Ccache:

shell

export CCACHE_DIR=$PWD/ccache
ccache --show-stats --verbose

We also want to prime the environment to use Clang:

shell

export CXX=${CXX:-clang++}
export CC=${CC:-clang}
export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++}
export CC_FOR_BUILD=${CC_FOR_BUILD:-clang}
export CFLAGS='-Wno-unused-command-line-argument'
export CXXFLAGS='-Wno-unused-command-line-argument'

The first step in actual compilation is to run CMake, instructing it to look at the source in directory mariadb-server/, output build artifacts in directory build/ and use Ninja as the build system. This line also always forces a fresh configuration, discarding any previous CMakeCache.txt files, to use ccache instead of calling gcc/c++ directly, and also skip a bunch of rarely used large plugins to save a lot of compilation time.

shell

cmake -S mariadb-server/ -B build/ -G Ninja --fresh \
 -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \
 -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \
 -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \
 -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF

If you are interested in knowing all possible build flags available, simply query them from CMake with:

shell

cmake build/ -LH

Note that after the configure stage has run, there are no traditional Makefiles in ‘build/’, only a ninja.build since we are using Ninja. Thus, running make build will build. With Ninja it will be ninja -C build. However, we don’t need to call Ninja directly either but just let CMake orchestrate everything with:

$ eatmydata cmake --build build/
[173/1462] Building C object plugin/auth_ed25519/CMakeFiles/ref10.dir/ref10/ge_add.c.o

In interactive mode, Ninja will have just one line of output at the time showing progress. The numbers inside the brackets show how many files have been compiled of the total number of files to compile, and the filename after it shows which file is currently being compiled. Ninja runs by default on all available CPU cores, so there is no need to define parallelism manually. If Ninja encounters warnings or errors, it will spit them out but continue to show the one-liner status at the bottom of the terminal. To abort Ninja, feel free to press Ctrl+C at any time.

Re-starting the compilation will continue where it left off – Ninja is very smart and fast in figuring out what files need to compiled.

Running the MariaDB test suite (MTR)

While the MariaDB server does have a small amount of CTest unit tests, the main test system is the mariadb-test-run script (inherited from mysql-test-run). Each test file (suffix .test) consists mainly of SQL code which is executed by mariadb-test-run (MTR) and output compared to the corresponding file with the expected output in text format (suffix .result).

To start the MTR with CMake run:

shell

cmake --build build/ -t test-force

Alternatively, one can simply invoke the script directly after the binaries have been compiled:

shell

./build/mysql-test/mysql-test-run.pl --force

This offers more flexibility, as you can easily add parameters such as --parallel=auto (as the default is to run just one test worker on one CPU) or limit the scope to just one suite or just one individual test:

shell

./build/mysql-test/mysql-test-run.pl --force --parallel=auto --skip-rpl --suite=main

Note that all commands in this example run as root, as it is necessary to start the whole container with a root user inside it to have permissions to apt install build dependencies. However, the mariadb-test-run is actually not designed to be run as root and will end up skipping some tests when run as root. Also, when run like this, a lot of the debugging information isn’t fully shown. To make most out of the mysql-test-run/mariadb-test-run script, read more in the post Grokking the MariaDB test runner (MTR).

More build targets

As concluded above, the target test-force was for MTR, and the plainly named target test is for CUnit tests. The equivalent direct Ninja command for running target test would be ninja -C build/ test. To list all targets, run cmake --build build/ --target help or ninja -C build/ -t targets all.

MariaDB 11.0 has currently over 1300 targets. There does not seem to be a very consistent pattern in how build targets are named or how they are intended to be used. One way to find CMake targets that might be more important than others is to simply grep them from the main level CMake configuration file:

$ grep ADD_CUSTOM_TARGET mariadb-server/CMakeLists.txt
ADD_CUSTOM_TARGET(import_executables
ADD_CUSTOM_TARGET(INFO_SRC ALL
ADD_CUSTOM_TARGET(INFO_BIN ALL
ADD_CUSTOM_TARGET(minbuild)
ADD_CUSTOM_TARGET(smoketest

One of the standard targets is install, which can be run ninja -C build install or CMake:

$ cmake --install build/
-- Install configuration: "RelWithDebInfo"
-- Up-to-date: /usr/local/mysql/./README.md
-- Up-to-date: /usr/local/mysql/./CREDITS
-- Up-to-date: /usr/local/mysql/./COPYING
-- Up-to-date: /usr/local/mysql/./THIRDPARTY
-- Up-to-date: /usr/local/mysql/./INSTALL-BINARY
-- Up-to-date: /usr/local/mysql/lib/plugin/dialog.so
-- Up-to-date: /usr/local/mysql/lib/plugin/client_ed25519.so
-- Up-to-date: /usr/local/mysql/lib/plugin/caching_sha2_password.so
-- Up-to-date: /usr/local/mysql/lib/plugin/sha256_password.so
...
-- Installing: /usr/local/mysql/support-files/systemd/mysql.service
-- Installing: /usr/local/mysql/support-files/systemd/mysqld.service
-- Installing: /usr/local/mysql/support-files/systemd/mariadb@.service
-- Installing: /usr/local/mysql/support-files/systemd/mariadb@.socket
-- Installing: /usr/local/mysql/support-files/systemd/mariadb-extra@.socket
-- Up-to-date: /usr/local/mysql/support-files/systemd/mysql.service
-- Up-to-date: /usr/local/mysql/support-files/systemd/mysqld.service

To better understand the full capabilities of the build tools, it is recommended to skim through the cmake man page and the ninja man page.

Run the build binaries directly

Instead of wasting time on running the install target, one can simply invoke the build binaries directly:

$ ./build/client/mariadb --version
./build/client/mariadb from 11.0.1-MariaDB, client 15.2 for Linux (x86_64) using EditLine wrapper
$ ./build/sql/mariadbd --version
./build/sql/mariadbd Ver 11.0.1-MariaDB for Linux on x86_64 (Source distribution)

To actually run the server, it needs a data directory and a user, which can be created with:

$ ./build/scripts/mariadb-install-db --srcdir=mariadb-server
$ adduser --disabled-password mariadb
$ chown -R mariadb:mariadb ./data
$ ./build/sql/mariadbd --datadir=./data --user=mariadb &
[Note] Starting MariaDB 11.0.1-MariaDB source revision as process 5428
[Note] InnoDB: Compressed tables use zlib 1.2.13
[Note] InnoDB: Using transactional memory
[Note] InnoDB: Number of transaction pools: 1
[Note] InnoDB: Using crc32 + pclmulqdq instructions
[Warning] mariadbd: io_uring_queue_init() failed with errno 0
[Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
[Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
[Note] InnoDB: Opened 3 undo tablespaces
[Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
[Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
[Note] InnoDB: log sequence number 47391; transaction id 14
[Note] InnoDB: Loading buffer pool(s) from /quick-rebuilds/data/ib_buffer_pool
[Note] InnoDB: Buffer pool(s) load completed at 230220 20:28:45
[Note] Plugin 'FEEDBACK' is disabled.
[Note] Server socket created on IP: '0.0.0.0'.
[Note] Server socket created on IP: '::'.
[Note] ./build/sql/mariadbd: ready for connections.
Version: '11.0.1-MariaDB' socket: '/tmp/mysql.sock' port: 3306 Source distribution

It is necessary to define the custom data directory path and custom user, otherwise mariadbd will fail to start:

[Warning] Can't create test file /usr/local/mysql/data/03727bdc8fe2.lower-test
./build/sql/mariadbd: Can't change dir to '/usr/local/mysql/data/' (Errcode: 2 "No such file or directory")
[ERROR] Aborting
./build/sql/mariadbd: Please consult the Knowledge Base to find out how to run mysqld as root!
[ERROR] Aborting

To gracefully stop the server, send it the SIGTERM signal:

$ pkill -ef mariadbd
[Note] ./build/sql/mariadbd (initiated by: unknown): Normal shutdown
[Note] InnoDB: FTS optimize thread exiting.
[Note] InnoDB: Starting shutdown...
[Note] InnoDB: Dumping buffer pool(s) to /quick-rebuilds/data/ib_buffer_pool
[Note] InnoDB: Buffer pool(s) dump completed at 230220 20:29:05
[Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
[Note] InnoDB: Shutdown completed; log sequence number 47391; transaction id 15
[Note] ./build/sql/mariadbd: Shutdown complete
mariadbd killed (pid 5428)

Quick rebuilds

With this setup, you can invoke eatmydata cmake --build build/ to have the source code re-compiled as quickly as possible.

The ‘screenshot’ below showcases how Ninja/CMake will only rebuild the file with changes and its dependencies. In the case of a simple MariaDB client version string change, only 5 files needed to be re-built, and it took less than a second:

$ sed 's/*VER= "15.1"/*VER= "15.2"/' -i mariadb-server/client/mysql.cc
$ time eatmydata cmake --build build/
[5/5] Linking CXX executable client/mariadb
real 0m0.992s
user 0m0.374s
sys 0m0.353s

A similar version string change in the server leads to having to rebuild over a thousand files:

$ sed 's/MYSQL_VERSION_PATCH=1/MYSQL_VERSION_PATCH=2/' -i mariadb-server/VERSION
$ time eatmydata cmake --build build/
[0/1] Re-running CMake...
-- Running cmake version 3.25.1
-- MariaDB 11.0.2
-- Packaging as: mariadb-11.0.2-Linux-x86_64
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
== Configuring MariaDB Connector/C
-- SYSTEM_LIBS: dl;m;dl;m;/usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libz.so
-- Configuring OQGraph
-- Configuring done
-- Generating done
-- Build files have been written to: /quick-rebuilds/build
[377/1257] Generating user.t
troff: fatal error: can't find macro file m
[378/1257] Generating user.ps
troff: fatal error: can't find macro file m
[433/1257] Building CXX object storage/archive/CMakeFiles/archive.dir/ha_archive.cc.o
In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29:
/quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:91:15: warning: 'index_type' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
const char *index_type(uint inx) { return "NONE"; }
^
/quick-rebuilds/mariadb-server/sql/handler.h:3915:23: note: overridden virtual function is here
virtual const char *index_type(uint key_number) { DBUG_ASSERT(0); return "";}
^
[...]
In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29:
/quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:163:7: warning: 'external_lock' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
int external_lock(THD *thd, int lock_type);
^
/quick-rebuilds/mariadb-server/sql/handler.h:5153:15: note: overridden virtual function is here
virtual int external_lock(THD *thd __attribute__((unused)),
^
36 warnings generated.
[1257/1257] Linking CXX executable extra/mariabackup/mariadb-backup
real 2m7.786s
user 12m56.232s
sys 1m57.842s

The above example also shows how Ninja spits out warnings.

Despite the majority of the project files being re-built, it still took only two minutes, mainly thanks to ccache having a high hit-rate.

$ ccache --show-stats
Cacheable calls: 3235 / 3235 (100.0%)
Hits: 1932 / 3235 (59.72%)
Direct: 49 / 1932 ( 2.54%)
Preprocessed: 1883 / 1932 (97.46%)
Misses: 1303 / 3235 (40.28%)
Local storage:
Cache size (GB): 0.11 / 5.00 ( 2.18%)

Without ccache, the build time in the same scenario is 6–8 minutes. There are some extra flags in ccache (such as CCACHE_SLOPPINESS) which can be used to further tune the ccache speed, but when I did some experimenting, I didn’t discover any that made a visible impact.

Without eatmydata, the build takes 10-20 seconds longer, as the system calls to disk will wait for fsync and the like to complete, but which we are fine skipping since we don’t care about data durability and crash recovery as this is a throwaway environment anyway. Using regular GNU GCC instead of Clang adds another 20–40 seconds to the rebuild time.

The current two minutes for the build time on my laptop with 8-core Intel i7-8650U CPU @ 1.90GHz is not exactly instant, but it is fast enough that I can sit and wait it out without feeling the need to context switch and loose my focus.

Automatic rebuild

As showcased in the post How to code 10x faster than an average programmer, as a high-performing software developer, you don’t want to waste time on manually running a lot of commands to build and test your code, but instead you want to have a setup where you write code in your editor and have the code automatically re-compile and run when the source code file is saved.

For MariaDB, the automatic rebuild part can easily be achieved with:

shell

find mariadb-server/* | entr eatmydata cmake --build build/

To automatically rebuild and also run a binary (in this case the mariadb client), define multiple commands in quotes to the -s parameter:

shell

find mariadb-server/* | \
 entr -s 'eatmydata cmake --build build/; ./build/client/mariadb --version'

When running the server use the -r parameter to have Entr automatically restart it:

shell

find mariadb-server/* | \
 entr -r ./build/sql/mariadbd --datadir=./data --user=mariadb

If the you are developing an MTR test by editing *.test files, there is no need to recompile anything, and you can simply have Entr re-run the test every time a file is changed:

shell

find mariadb-server/* | entr -r ./build/mysql-test/mysql-test-run.pl main.connect

Conclusion

The examples above are specific to MariaDB and illustrate in detail how to be efficient and avoid wasting time compiling, but the principles of utilizing ccache/clang/ninja apply to any software project in C/C++, and entr comes in handy in a myriad of situations.

Hopefully this inspires you to raise the bar on what to expect of speed and efficiency in the future!

Grokking the MariaDB test runner (MTR)

Sun, 19 Feb 2023 00:00:00 +0000

The main test system in the MariaDB open source database project is the mariadb-test-run script (inherited from mysql-test-run). It is easy to run and does not require you to compile any source code.

While writing MTR tests is relevant only for MariaDB developers, knowing how to run MTR is useful for any database administrator running MariaDB, as it is a quick way to validate that MariaDB can run correctly on your hardware and operating system version.

TL;DR for Debian/Ubuntu users
Run the full 6000+ test suite as current user in temporary directory with multiple workers in parallel and with detailed logging on failures, typically taking over 30 minutes on modern laptop:
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb cd /usr/share/mysql/mysql-test export MTR_PRINT_CORE=detailed ./mtr --force --parallel=auto --vardir=$(mktemp -d) \ --skip-test-list=unstable-tests.amd64 --big-test
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb
cd /usr/share/mysql/mysql-test
export MTR_PRINT_CORE=detailed
./mtr --force --parallel=auto --vardir=$(mktemp -d) \
--skip-test-list=unstable-tests.amd64 --big-test
For full details, read the whole article.

Install ‘mariadb-test’ package in Debian/Ubuntu

To avoid polluting your actual system with new packages, it is convenient to run MTR in a throwaway container. Start one with some RAM-memory-backed disk allocated with --shm-size=1G so running MTR with --mem later on is possible:

shell

docker run --interactive --tty --rm --shm-size=1G debian:sid bash

This example uses Docker, but the principle is the same with any Linux container tool, such as Podman.

Next, install the MariaDB test suite package. This will also pull in the MariaDB server and all the necessary dependencies. Additionally, also install the GNU Debugger (gdb) for automatic stack traces and patch.

shell

apt update
apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb

The mariadb-test-run is not intended to be run as root and will skip some tests if run as root. Therefore, create a new user inside the container and grant it permissions to the test directory:

shell

adduser --disabled-password mariadb-test-runner
chown -R mariadb-test-runner /usr/share/mysql/mysql-test

At minimum, the test runner user needs to be able to write to the path /usr/share/mysql/mysql-test/var (unless some other path is defined with --vardir and --tmpdir), but some tests also run patch to modify the test files on-the-fly, so just grant permissions to the whole test directory. As this is a throwaway container anyway, there is no need to be prudent with the permissions.

Next, switch to the test user and test directory and start the run with default settings:

$ su - mariadb-test-runner
$ cd /usr/share/mysql/mysql-test
$ ./mariadb-test-run
Logging: ./mariadb-test-run
VS config:
vardir: /usr/share/mysql/mysql-test/var
Creating var directory '/usr/share/mysql/mysql-test/var'...
Checking supported features...
MariaDB Version 10.11.2-MariaDB-1
- SSL connections supported
- binaries built with wsrep patch
Using suites: main-,archive-,atomic-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_uuid,user_variables
Collecting tests...
...
main.subselect_innodb 'innodb' w4 [ pass ] 3359
main.subselect_sj2 'innodb' w1 [ pass ] 2737
main.subselect_sj2_jcl6 'innodb' w3 [ pass ] 2933
main.parser_bug21114_innodb 'innodb' w8 [ pass ] 15364
innodb_gis.rtree_search 'innodb' w5 [ pass ] 52379
--------------------------------------------------------------------------
The servers were restarted 1736 times
Spent 8527.863 of 1659 seconds executing testcases
Completed: All 5131 tests were successful.
995 tests were skipped, 280 by the test itself.

Defining what tests to run

If no test suite is selected, MTR will run about 6000 tests, which on my laptop takes about 30 minutes. If MTR is started with the additional --big-test parameter, it will run additional tests that are resource intensive and consume, for example, a lot of RAM memory and take a long time to run, totalling a test run that has 6100 tests and takes 43 minutes to complete (on my laptop). To only run big tests, use --big --big.

If there is a need to limit the scope, such as in build systems that want to validate that the built binary works without running all tests, typically --suite=main --skip-rpl is used. This results in about 1000 tests being run, which on my laptop takes about 3½ minutes.

Even when running without any limitations on what tests are run, many tests have code that makes them opt out automatically based on some condition being missing, and on my laptop about 1000 tests end up being skipped. Some examples:

main.connect2 [ skipped ] Requires debug build
main.mysql_client_test_comp [ skipped ] No IPv6
main.connect-abstract [ skipped ] Need Linux
main.grant_cache_ps_prot [ skipped ] Need ps-protocol
main.fix_priv_tables [ skipped ] Test need MYSQL_FIX_PRIVILEGE_TABLES
main.no-threads [ skipped ] Test requires: 'one_thread_per_connection'
main.udf_skip_grants [ skipped ] Need udf example
main.lowercase_mixed_tmpdir_innodb [ skipped ] Test requires: 'lowercase2'
main.innodb_load_xa [ skipped ] Need InnoDB plugin
mariabackup.alter_copy_excluded [ skipped ] No mariabackup
binlog.binlog_expire_warnings [ skipped ] Test needs --big-test
encryption.innodb-spatial-index [ skipped ] requires patch executable
plugins.pam_cleartext [ skipped ] Not run as user owning auth_pam_tool_dir
rpl.rpl_gtid_mdev4474 'innodb,row' [ skipped ] Neither MIXED nor STATEMENT binlog format

If you are interested in one particular test, just give it as the last argument:

$ ./mariadb-test-run main.connect
Logging: ./mariadb-test-run main.connect
Creating var directory '/usr/share/mysql/mysql-test/var'...
Checking supported features...
MariaDB Version 10.11.2-MariaDB-1
- SSL connections supported
- binaries built with wsrep patch
Collecting tests...
Installing system database...
==============================================================================
TEST RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
main.connect [ pass ] 14206
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 14.206 of 20 seconds executing testcases
Completed: All 1 tests were successful.

Optimize the MTR run for speed

There are three parameters that can greatly improve how quickly MTR runs. The primary one is --parallel=auto, which will run MTR in parallel with as many workers as there are CPUs (by default MTR runs with just one worker). On my laptop, going from one MTR worker to 8 in parallel reduced the total run time for the main suite from 17 to about 3 minutes.

Another parameter is --fast, which will make the MTR kill all server processes violently without waiting for them to gracefully shutdown. The test run restarts the MariaDB server hundreds of times, so saving half a second on every shutdown results in the main suite completing 20 seconds faster.

The parameter --mem instructs MTR to make the directory var/ a symbolic link to a subdirectory on the shared memory device (/dev/shm). This works if the container is started with --shm-size and has at least 350MB of space on the ramdisk. On my laptop, this further reduced the main test suite duration down to 2½ minutes.

Optimize the MTR run for logging

When running a single test, one might use --verbose as an additional argument to see the commands that run in the test. It is also possible to define --verbose --verbose twice, but that makes the test run so verbose that it is unusable.

When running a suite of tests, you only want to have extra output visible if the test fails. If the server fails to start and a particular test doesn’t run at all, using --verbose-restart might be beneficial. If running the test in an automated system, one might want to save results in a JUnit-compatible XML file that can, for example, be rendered by GitLab CI.

This is the command that several CI systems run MTR with:

shell

export MTR_PRINT_CORE=detailed
eatmydata perl -I. ./mariadb-test-run \
 --force --testcase-timeout=120 --suite-timeout=540 --retry=3 \
 --verbose-restart --max-save-core=1 --max-save-datadir=1 \
 --parallel=auto --skip-rpl --suite=main \
 --xml-report=mariadb-test-run-junit.xml

The command above also showcases use of eatmydata, which makes fsync and similar system calls skip memory-to-disk guarantees, but in my testing with MTR, it didn’t affect speed.

Running more tests for MariaDB

The above summarizes everything one typically needs to know for running the mariadb-test-run.

For a DBA, there are also several other tools that ship with MariaDB, which can help with testing and validating one’s own environment, such as mariadb-stress-test and mariadb-slap.

Writing MTR tests for MariaDB

If you want to write a new test or fix an existing test, there are many more additional parameters to learn, such as --record and --gcov. The official contribution docs at mariadb.org list some of the most useful MTR parameters. The mariadb.com knowledge base article lists them all. As with all commands in Linux, there is also the main page for mariadb-test-run.

The structure of the tests is actually quite easy, and requires no C/C++ skills to write. Each test file (suffix .test) consists mainly of SQL code which is executed by mariadb-test-run (MTR), with the output compared (with diff) to the corresponding file with the expected output in text format (suffix .result).

In my development workflow, writing a test and running MTR might look something like this:

If you want to contribute to the MariaDB open source project, extending the test coverage is a great place to start. To scratch your own itch, think about a MariaDB bug you encountered while using it, and consider if that can be reproduced as a test and submitted upstream so that it becomes part of the body of tests and thus easy to catch if it ever regresses again.

To learn more about writing mariadb-test-run tests, read the test case authoring guide at mariadb.org.

Ensuring software quality with GitLab CI – case MariaDB in Debian

Sun, 02 Oct 2022 00:00:00 +0000

Of all CI systems I’ve used during my software development career, both as developer and manager, GitLab CI has proven itself to be the overall best system out there.

First of all, for a CI system to fulfill its purpose, it needs to test every git commit to validate that no code change breaks the test suite no matter how small the change is. Having code hosting and CI integrated in the same system is the obvious way it should be, and GitLab CI does that integration well on all levels from git command-line option support to repository permission control to user interface in every view.

For developers to pay attention to tests that stop passing after their code changes, there need to be automatic and easy to read notifications. GitLab CI does email and dozens of integrations, such as Slack messages.

For developers to quickly find the error, root cause it and deliver a proper fix, the CI pipeline needs to be visually clean yet offer options to drill into logs and build artifacts. GitLab CI checks all the boxes here.

Finally, when it is time to review a code change, the way GitLab CI integrates with GitLab Merge Requests is seamless. For example, a human reviewer can, after reading the code change, choose the action Merge automatically if CI pipeline passed without having to attend the pipeline. Brilliant time-saver.

GitLab is also open source, so if it is missing a feature that is critical for your software development work, you can add that feature yourself (or pay somebody to do it). Being open source also brings many other benefits and ultimately sets it apart from GitHub, its traditional closed source rival.

Last but not least, GitLab has excellent documentation anybody can dive into easily. Therefore, instead of duplicating that by explaining the general GitLab features and benefits, I will instead showcase how it is used in a real-life project: MariaDB Debian package maintenance.

Salsa - Debian’s GitLab instance

Debian launched salsa.debian.org in early 2018 as a platform for Debian developers to host the source code of Debian packages. In July of 2018, Salsa-CI, which provides a standardized GitLab CI pipeline template for Debian packaging quality assurance, was launched. I adopted this in August 2018 for all the packages I maintain in Debian, of which by far the biggest is the MariaDB database server.

Over the years it has grown to a very extensive pipeline that, in addition to the inherited general Salsa-CI steps, also runs a variety of additional test jobs, including:

Building MariaDB in parallel on multiple Debian releases and processor architectures
Building consumers of the MariaDB Client C library to ensure the interface stays stable
Upgrading old versions of MariaDB to the latest one, both full server upgrades and partial small upgrades, such as the client library upgrades
Upgrading various versions of MySQL, Percona and others to ensure that cross-upgrades from MariaDB variants work
Upgrading various combinations of Debian releases and MariaDB, simulating full system upgrades
Running static analysis to detect security issues and general software quality issues

Not only does the pipeline do all this, but it is also optimized to use ccache and other techniques to run as fast as possible. For details on exactly what the pipeline does, one can simply read the file debian/salsa-ci.yml. Normal GitLab CI uses the file .gitlab-ci.yml in the project root, but since in Debian packaging one is only allowed to modify files under the debian sub-directory, the file resides at this customized path.

The fact that the structure and all the steps the CI runs are defined in the code repository itself is very powerful. Anybody inspecting a pipeline run of a specific git commit can always find the specific version of the GitLab CI definition in the source code of that exact git commit itself. This is vastly superior to the structure e.g. Buildbot or Jenkins uses, where the pipeline is defined separately from the code it tests.

Not only does this make it much easier to read the pipeline steps, but it also makes contributing to the pipeline code as straightforward as filing a Merge Request on the repository, just like with any other file. The fact that the CI code and actual software code are together in the same repository makes it much easier to enforce a rule that CI must always pass, as any commit that changes the software behavior can at the same time also update the CI pipeline to account for that intentional change in behavior. Needless to say, GitLab CI works seamlessly with the protected branches feature and Merge Requests in GitLab to ensure easy and sensible rules to enforce that the mainline always stays green.

Using standard GitLab CI features, there is also a scheduled monthly rebuild that reveals if an update in any of the dependencies causes the pipeline to fail in the absence of new code commits.

Case example: MariaDB 10.6.9 incompatibility with Percona XtraDB 5.7 temporary table format

To illustrate GitLab CI in action, let’s take a look at a recent example in which it prevented breaking upgrades for (some) Debian users. In June 2022, the Salsa-CI pipeline for MariaDB 10.6.8 was all green and upgrades from Percona XtraDB 5.7 were passing flawlessly. However, after importing a new upstream minor maintenance release on August 16th, the Salsa-CI pipeline for MariaDB 10.6.9 started failing on the Percona upgrade. From the CI job log, it was easy to see that MariaDB failed to start with the /var/lib/mysql data directory from Percona XtraDB 5.7. From the build artifacts, it was easy to inspect the precise error message from the MariaDB server (as build artifacts are configured to expire after 30 days, I can’t link to them for reference).

This quickly led to filing MDEV-29321 on August 18th. As the failure was caught by CI, it was easy to provide the steps to reproduce in the bug report, along with logs and CI job references. This helped the upstream developer to immediately pinpoint the issue and post a patch to fix it, which was validated by a Salsa-CI test run on a temporary development branch. The fix was applied on mainline in the Debian packaging repository of MariaDB on August 24nd.

Such a quick turnaround time would not have been possible without a good CI system. The process clearly benefited from the very clear user interface of GitLab CI that made it easy for all parties – even those who didn’t have prior experience of GitLab CI – to read the pipeline and inspect the logs.

Computers are good at repeating the same tasks over and over; humans are good at exploring visual things

Through the years, Salsa-CI (GitLab CI) has proven incredibly valuable – it has been able to catch the tiniest packaging mistake immediately as the commit is pushed to Salsa (GitLab) and the git mainline stays in a condition that can be shipped at any time. This makes it possible to import new upstream releases at any given time, and to upload them to Debian with a high confidence that nothing will break. Before Salsa-CI, the MySQL and MariaDB packages had hundreds of open bugs in Debian (and also Ubuntu, which inherits the packages from Debian). Now genuine new bugs are rare, staying consistently in the lower tens.

If you are developing software professionally but not using CI in all your projects, you should definitely start now. Computers excel at running repetitive tasks over and over, and CI is exactly that. No human would have the diligence to always test everything. Humans tend to test code changes only when they have doubts, and thus most bugs slip in when a human makes a small change they don’t think can break anything – and then it breaks. These cases always come as a surprise, and you don’t want to find it out in production use but rather delegate it to a CI system to validate everything.

What humans are good at is visual inspection. A large part of our brain cortex is devoted to vision, so we should leverage it. GitLab CI does a great job converting the repetitive CI test run results into pipelines with various colors and symbols, perfect for human consumption. Humans also have an eye for beauty and elegance, and I personally enjoy exploring the GitLab CI pipelines. If you haven’t already, please try it out, and enjoy the warm fuzzy feeling of seeing all green pipelines!

So easy and intuitive you can start using it without reading lengthy manuals

This blog was just a sneak peek into what GitLab CI can do – there are so many useful features, such as scheduled pipeline runs (to detect regressions introduced from updated dependencies), automatic repository mirroring, and many UI goodies, such as badges. I recommend skimming through the GitLab CI documentation and in particular the .gitlab-ci.yml reference and then just start playing around with it. GitLab.com has a free plan for basic use which (unlike GitHub.com) also includes private repositories.

In short, GitLab CI makes the life of a programmer (or a team of programmers) significantly more productive by ensuring that tests run all the time, failures are quick to detect, and the whole test system is handy to maintain and evolve together with the code. Using it feels like a breeze, both because the features work just like one would expect and because the user interface is very easy to navigate. There is rarely a need to read documentation to get started – just try GitLab CI today and discover its power yourself!

Want to read more? See also the GitLab.com blog post “Debian customizes CI tooling with GitLab”

Stop the senseless killing

Sun, 18 Sep 2022 00:00:00 +0000

In over 22 years of working with Linux systems, I have often seen people use excessive force to kill various computer programs, causing unnecessary suffering.

A typical example would be:

shell

$ killall -9 mysqld

Please stop using this command. Both killall and the parameter -9 are harmful when dealing with programs that store data, like the MySQL or MariaDB database, and better alternatives exist.

Avoid killing in vain

The 9 means signal 9, which is SIGKILL. All standard Linux signals can easily be listed with:

shell

$ kill -L
 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM

The Linux kernel sends most of these signals to the program itself, and we expect programs to have code to handle various signals. One exception is signal number 9 (SIGKILL). We use the SIGKILL signal to terminate the program immediately. The program cannot handle, block, or ignore it. Therefore, it is always fatal. If kill is issued without parameters, it will default to signal 15 (SIGTERM) and the Linux kernel will request the running process to terminate itself and clean file pointers and other resources it was using. Thus, always prefer SIGTERM and only use SIGKILL as a last resort.

Violently killing programs that deal with data storage might cause writing into a file to abruptly stop halfway, leading to inconsistent data or even unusable corrupted files. For example, even if the program is not a database or similar, senseless immediate killing leads to everything dropping mid-flight, potentially causing unnecessary harm to other programs communicating over the same network. In some cases, albeit rare, poorly terminated programs may turn into zombie processes, showing up in the Linux process listing as <defunct>. If that happens, the only way to clean them is to restart the entire operating system. So, please, do not kill without trying other means first.

Don’t be senseless

Let’s dive into the first part of the command in the opening paragraph, which is also wrong.

The command killall has been in Unix and all its descendants, such as Linux, since 1983. It does the job. However, we also have modern tools, which I prefer, like pkill from 1998.

The program pkill is superior because it has the parameter -e to display what is killed (or terminated). Those who still use killall but don’t want to be senseless also use variations of ps ax | grep <processname> to see which processes are running before and after the killing.

Compare the clarity in these identical examples:

shell

$ killall mysqld
$ killall mysqld
mysqld: no process found

The killall command remains silent if it did something and only reports errors. In contrast, pkill reports which process it terminated and is only silent if it didn’t do anything.

shell

pkill -e mysqld
mysqld_safe killed (pid 3911)
mysqld killed (pid 4011)
$ pkill -e mysqld
(nothing)

The bonus of being sensible

A bonus for system administrators using pkill is they could learn something valuable by observing the effects of their commands. For example, a database sysadmin could encounter the following:

shell

$ pkill -9 -e mariadbd
mariadbd killed (pid 5281)
$ pkill -9 -e mariadbd
mariadbd killed (pid 5383)
$ pkill -9 -e mariadbd
mariadbd killed (pid 5412)
$ pkill -e mariadbd
mariadbd killed (pid 5497)
$ pkill -e mariadbd
(nothing)

With MariaDB, if the main server abruptly dies (as with signal 9/SIGKILL), the wrapper mysqld_safe (if used) will detect that and automatically restart the server because it assumes it died during a crash and knows getting it back up and running is desirable. However, when the proper signal 15/SIGTERM is used (default in pkill when no signal is defined), the mariadbd process intentionally shuts itself down. The wrapper respects that and understands that automatically restarting it would not make sense.

The procps package

The command pkill is part of the procps suite. You can find it by default on most Linux systems, but if you don’t have it, run apt install procps or yum install procps to get it. The suite also includes the command pgrep, which replaces manual ps ax | grep <processname> invocations, as mentioned above.

An example of pgrep in action:

shell

$ pgrep -af mysqld
5577 /bin/sh /usr/bin/mysqld_safe
5676 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock
5677 logger -t mysqld -p daemon error

To see the complete list of options, run pkill --help and pgrep --help or man pkill to read the man page. I recommend all system administrators make using --help and man <command> from the command line a general habit to act more sensibly in all command-line actions.

Most importantly, don’t use signal 9 (SIGKILL) on databases other than as the last resort. Please avoid any senseless killings!

About me

Mon, 01 Jan 0001 00:00:00 +0000

I am a results-driven technology executive with 25+ years of professional experience and a rich blend of strategic leadership, product development expertise, and deep-seated passion for open-source software. I am currently working as an independent consultant. I was previously a Software Development Manager for the core engine team delivering the Amazon RDS for MySQL and MariaDB database services. Before that, I was the CEO of Seravo until 2021, and the CEO of MariaDB Foundation until 2018.

I have a strong bias for action and a proven history of transforming and growing organizations by leveraging a unique blend of technological acumen and business strategy, which I hope to share in this blog. My goal is to help readers gain a deeper understanding of open source software, software engineering in general, management in technical fields, business improvement, life hacks, and much more.

I like spending my free time hiking, running and contributing to various open source projects. I am an active Debian and Ubuntu developer. See my GitHub, GitLab and Salsa profiles for recent contributions.

You can follow me on Mastodon, Twitter, Farcaster, Bluesky or connect on LinkedIn.

Let’s Connect

Interested in discussing open source strategy, Debian contributions, or how I can help your organization?

Book a chat with me on Cal.com

You can schedule a quick 15-minute introduction call for free, or book a 1-hour consultation for in-depth problem-solving.

You can also reach me by e-mail to otto at debian.org.

Optimized by Otto

Automated security validation: How 7,000+ tests shaped MariaDB's new AppArmor profile

Regular review of denials in the system log required

AppArmor in MariaDB - not a novel thing, and not easy to implement well

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews

Now available in Debian unstable, soon Ubuntu – feedback welcome!

Systemd hardening also adopted as security features keep evolving

Stop using MySQL in 2026, it is not true open source

This is not surprising – Oracle should not be trusted as the steward for open source projects

MySQL’s technical decline in recent years

Open source is more than ideology: it has very real effects on software security and sovereignty

There are options and migrating is easy, just do it

DEP-18: A proposal for Git-based collaboration in Debian

Is collaborative software development possible without git and version control software?

Inefficient ways of working prevent Debian from flourishing

Debian should embrace git, but decision-making is slow

Debian Enhancement Proposal 18

Transparency and collaboration

Recommend, not force

Next steps

Zero-configuration TLS and password management best practices in MariaDB 11.8

How encrypting database connections with TLS differs from web server HTTP(S)

Why the ‘root’ user in MariaDB has no password and how it makes the database more secure

Create separate database users for normal use and keep ‘root’ for administrative use only

Allow MariaDB to accept remote connections and enforce TLS

Should TLS encryption be used also on internal networks?

Going full-time as an open source developer

The Evolution of Open Source

What is Special About Debian?

Grow or Die

Debian Salsa CI in Google Summer of Code 2025

Improving Salsa CI: more features, robustness, speed

Good time to also learn Debian packaging

Apply now!

Update May 8th, 2025

10 habits to help becoming a Debian maintainer

1. Read and re-read the Debian Policy, the Developer’s Reference and the git-buildpackage manual

2. Make reading man pages a habit

3. Read and write emails

4. Create and use an OpenPGP key

5. Integrate Salsa CI in all work

6. Fork on Salsa and use draft Merge Requests to solicit feedback

7. Use git rebase frequently

8. Reviews: give some, get some

9. Improve Debian by improving upstream

10. Don’t hold any habits too firmly

Advanced Git commands every senior software developer needs to know

Avoid excess downloads with selective and shallow Git clone

Inspecting Git history and comparing revisions

Committing, rebasing, cherry-picking and merging

Managing multiple remotes and branches

Keeping repositories nice and tidy

Better Git experience with Liquip Prompt and fzf

Bash aliases

Keep on learning

How to make a good git commit

What is a patch?

How to make a patch

Examples of good git commit messages

Five requirements for a good git commit

1. Commits should be atomic

2. The title should be descriptive, yet terse and not too long

3. The commit message should explain why it was made

4. Use references when available

5. Maintain correct authorship and copyright credits

The right tools make git commits easy

Don’t settle for bad commits - amend them

WIP commits: how to avoid postponing writing the perfect git commit message

Use git rebase -i frequently

A polished git commit is always worth the effort

Now go and build great software – patch by patch!

Quick builds and rebuilds of MariaDB using Docker

TL;DR for Debian/Ubuntu users

Stay organized, keep directories clean

Don’t download the whole project – use shallow Git clone

Build inside a throwaway container

Running the MariaDB test suite (MTR)

More build targets

Run the build binaries directly

Quick rebuilds

Use `git rebase -i` frequently