Percona Community Blog - learn about MySQL, MariaDB, PostgreSQL, and MongoDB

InnoDB Buffer Pool Tuning: From Rule-of-Thumb to Real Signals

Wayne Leutwyler — Thu, 02 Apr 2026 00:00:00 UTC

Introduction

Many MySQL setups begin life with a familiar incantation:

innodb_buffer_pool_size = 70% of RAM

…and then nothing changes.

That’s not tuning. That’s a starting guess.

Real tuning starts when the workload pushes back.

Visual Overview

The InnoDB buffer pool is where database performance is quietly decided. It determines whether your workload hums along in memory or drags itself across disk. If you’re not actively observing and tuning it, you’re leaving performance on the table.

This guide walks through how to monitor, understand, and tune the buffer pool using real signals instead of guesswork.

What the Buffer Pool Really Is

The buffer pool isn’t just “memory for MySQL.” It’s a living system under constant pressure:

A cache of data and indexes
A write staging area (dirty pages)
A contention zone between reads, writes, and eviction

Think of it as your database’s working memory. If your working set fits, queries glide. If it doesn’t, pages are constantly evicted and reloaded, introducing latency that rarely announces itself clearly.

A Simple Mental Model

 +---------------------------+
 | Buffer Pool |
 |---------------------------|
Reads ---> | Cached Pages |
 | |
Writes ---> | Dirty Pages (pending IO) |
 | |
Eviction -> | LRU / Free List |
 +---------------------------+
 |
 v
 Disk (slow)

Three forces are always competing:

Reads want hot data in memory
Writes generate dirty pages
Eviction makes room under pressure

Your job is to keep this system balanced.

How to Monitor the Buffer Pool

Option 1: Quick Snapshot

sql
SHOW ENGINE INNODB STATUS\G

Useful for human inspection. Look for:

Buffer pool size
Free buffers
Database pages
Modified (dirty) pages
Page read/write rates

Great for debugging. Not ideal for automation.

Option 2: Structured Metrics (Recommended)

sql
SELECT
 pool_id,
 free_buffers,
 database_pages,
 modified_database_pages
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Key fields:

free_buffers → Available pages (breathing room)
database_pages → Pages holding data
modified_database_pages → Dirty pages waiting to flush

Great for automation.

The 5 Signals That Actually Matter

1. Buffer Pool Hit Ratio (Handle With Care)

Yes, it’s widely used. No, it’s not enough.

A high hit ratio does not mean your system is healthy. It does not capture:

Page churn
Eviction pressure
Access patterns

You can have a 99% hit ratio and still be IO-bound.

Use it as a sanity check, not a decision-maker.

2. Free Buffers

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation:

Near zero during steady load → normal
Near zero + rising disk reads → pressure
Near zero while mostly idle → suspicious (possible misread or config issue)

3. Dirty Page Percentage

sql
SELECT
 (SUM(modified_database_pages) / SUM(database_pages)) * 100.0 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation (context matters):

0–5% → Very clean
5–20% → Typical
20–30%+ → Potential flushing lag

4. Disk Read Pressure (Critical Signal)

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';
-- Take two samples 60s apart and compare

Track the rate of change (reads/sec), not the absolute value.

Interpretation:

Rising reads → Working set does not fit in memory
Flat reads → Memory is absorbing the workload

5. Read Ahead / Eviction Pressure

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_ahead%';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_evicted';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';

Interpretation:

Efficient read-ahead:
- read_ahead increases
- read_ahead_evicted remains low
Inefficient read-ahead (wasted IO):
- High read_ahead_evicted / read_ahead
- Indicates access patterns defeating prefetching
Buffer pool churn:
- pages_evicted rising
- buffer_pool_reads rising
- Indicates pages are evicted and re-read from disk
Healthy vs unhealthy eviction:
- High evictions + stable reads → normal turnover
- High evictions + rising reads → memory pressure

Focus on rates of change over time, not absolute values.

Detecting Thrashing

Thrashing is when the buffer pool constantly evicts and reloads pages.

Classic Symptoms

Low or zero free buffers
Increasing disk reads
Stable (but misleading) hit ratio
Spiky query latency

Visualizing Thrash

Time --->

Memory: [FULL][FULL][FULL][FULL]
Reads: ↑ ↑↑ ↑↑↑ ↑↑↑↑
Latency: - ^ ^^ ^^^
Evictions: ↑ ↑↑ ↑↑↑ ↑↑↑↑

If you see this pattern, your working set does not fit in memory.

Tuning the Buffer Pool

Step 1: Size It Intentionally

Instead of blindly assigning 70% of RAM:

Observe working set behavior
Monitor free buffers and reads
Increase gradually

Avoid starving the OS or filesystem cache.

Step 2: Tune Flushing Behavior

innodb_max_dirty_pages_pct = 75
innodb_io_capacity = 1000
innodb_io_capacity_max = 2000

Sustained IO spikes → increase innodb_io_capacity
Dirty pages climbing → flushing lag
Sudden stalls → checkpoint pressure

What they control:

innodb_io_capacity → Expected steady-state IO throughput
innodb_io_capacity_max → Burst flushing capacity
innodb_max_dirty_pages_pct → Threshold for aggressive flushing

⚠️ These values should reflect real hardware capability.

Step 3: Buffer Pool Instances:Reduce Contention

A practical, battle-tested guideline:

Use 1 instance per ~1GB of buffer pool, up to a reasonable limit.

Buffer Pool Instances: Reducing Contention

The buffer pool can be split into multiple instances, each managing its own internal structures. This helps reduce contention under high concurrency.

Without this, all threads compete for the same buffer pool internals. With multiple instances, that load is distributed.

When It Matters

Buffer pool instances only help when contention exists. You’ll see benefits if your system has:

High concurrency (many active threads)
CPU-bound workloads
Mutex contention in InnoDB

If your workload is primarily IO-bound, this setting will have little impact.

Sizing Guidelines

General guidance:

< 1GB buffer pool → 1 instance
1GB–8GB → 2–4 instances
8GB–64GB → 4–8 instances
64GB+ → 8–16 instances

Keep Instances Large Enough

Each instance needs enough memory to function efficiently.

Avoid going below ~1GB per instance.

If instances are too small:

LRU efficiency drops
Eviction becomes more aggressive
Cache locality suffers

Example

sql
innodb_buffer_pool_size = 32G
innodb_buffer_pool_instances = 8

This gives ~4GB per instance, which is well-balanced.

Common Mistakes

Increasing instances without evidence of contention
Matching instance count to CPU cores
Using many instances with a small buffer pool
Expecting this to fix IO bottlenecks

Step 4: Understand Resizing Behavior

Buffer pool resizing is online in modern MySQL versions, but:

It happens in chunks
Controlled by innodb_buffer_pool_chunk_size

Real-World Scenarios

Scenario 1: “Everything Looks Fine… But It’s Slow”

High hit ratio
Low free buffers
Rising disk reads

Cause: Working set barely fits

Fix: Increase buffer pool size gradually

If increasing the buffer pool size does not reduce disk reads, the problem is not memory.

Scenario 2: Write-Heavy Workload

Dirty pages increasing
Periodic IO spikes

Cause: Flushing cannot keep up

Fix:

Increase innodb_io_capacity
Adjust dirty page thresholds

Scenario 3: Sudden Latency Spikes

Sharp performance drops
Disk activity surges

Cause: Checkpoint pressure

Fix:

Improve IO capacity tuning
Reduce dirty page buildup

Practical Monitoring Queries

Buffer Pool Usage (MB)

sql
SELECT
 (SUM(database_pages) * 16) / 1024 AS mb_used
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Assumes default 16KB page size (innodb_page_size).

Dirty Page Percentage

sql
SELECT
 (modified_database_pages / database_pages) * 100 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Free Buffer Check

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Common Mistakes

Treating 70% as a rule instead of a starting point
Blindly trusting hit ratio
Ignoring disk read trends
Oversizing and starving the OS
Not tuning IO capacity
Leaving defaults in write-heavy systems

Quick Checklist

If you remember nothing else:

Reads increasing? → working set too big
Free buffers always ~0? → pressure
Dirty pages high? → flushing lag
Latency spiking? → checkpoint or IO saturation

Final Thoughts

The InnoDB buffer pool doesn’t fail loudly. It degrades quietly until your disk becomes the bottleneck.

By the time you notice, you’re debugging latency instead of preventing it.

Monitor the right signals, and you’ll see problems forming before users do.

That’s the difference between reacting to performance… and controlling it.

Hardening MySQL: Practical Security Strategies for DBAs

Wayne Leutwyler — Mon, 02 Mar 2026 00:00:00 UTC

MySQL Security Best Practices: A Practical Guide for Locking Down Your Database

Introduction

MySQL runs just about everywhere. I’ve seen it behind small personal projects, internal tools, SaaS platforms, and large enterprise systems handling serious transaction volume. When your database sits at the center of everything, it becomes part of your security perimeter whether you planned it that way or not. And that makes it a target.

Securing MySQL isn’t about flipping one magical setting and calling it done. It’s about layers. Tight access control. Encrypted connections. Clear visibility into what’s happening on the server. And operational discipline that doesn’t drift over time.

In this guide, I’m going to walk through practical MySQL security best practices that you can apply right away. These are the kinds of checks and hardening steps that reduce real risk in real environments, and help build a database platform that stays resilient under pressure.

1. Principle of Least Privilege

One of the most common security mistakes is over-granting privileges. Applications and users should have only the permissions they absolutely need.

Bad Practice

sql
GRANT ALL PRIVILEGES ON *.* TO 'appuser'@'10.%';

Better Approach

sql
GRANT SELECT, INSERT, UPDATE ON appdb.* TO 'appuser'@'10.%';

Recommendations

Avoid global privileges unless absolutely required
Restrict users by host whenever possible
Separate admin accounts from application accounts
Use different credentials for read-only vs write operations

Audit Existing Privileges

sql
SELECT user, host, Select_priv, Insert_priv, Update_priv, Delete_priv
FROM mysql.user;

2. Strong Authentication & Password Policies

Weak credentials remain one of the easiest attack vectors.

Enable Password Validation

component_validate_password is MySQL’s modern password policy engine. Think of it as a gatekeeper for credential quality. Every time someone tries to set or change a password, it checks whether that password meets your defined security standards before letting it in.

It replaces the older validate_password plugin with a component-based architecture that is more flexible and better aligned with MySQL 8.x design.

sql
INSTALL COMPONENT 'file://component_validate_password';

What It Does

When enabled, it enforces rules such as:

Minimum password length
Required mix of character types
Dictionary file checks
Strength scoring

If a password fails policy, the statement is rejected before the credential is stored.

Why It Matters

Weak passwords remain one of the most common entry points in database breaches. This component reduces risk by enforcing baseline credential hygiene automatically, instead of relying on developer discipline.

Recommended Policies

Minimum length: 14+ characters
Require mixed case, numbers, and symbols
Enable dictionary checks
Enable username checks

Remove Anonymous Accounts

Find Anonymous Users

Anonymous users have an empty User field.

sql
SELECT user, host FROM mysql.user WHERE user='';

If you see rows returned, those are anonymous accounts.

Drop Anonymous Users

In modern MySQL versions:

sql
DROP USER ''@'localhost';
DROP USER ''@'%';

Adjust the Host value based on what your query returned.

Why This Matters

Anonymous users:

Allow login without credentials
May have default privileges in some distributions
Increase the attack surface unnecessarily

In hardened environments, there should be zero accounts with an empty username. Every identity should be explicit, accountable, and least-privileged.

3. Encryption Everywhere

Encryption protects data both in transit and at rest.

Enable Transparent Data Encryption (TDE)

See my January 13 post for a deep dive into Transparent Data Encryption: Configuring the Component Keyring in Percona Server and PXC 8.4

Enable TLS for Connections

sql
require_secure_transport=ON

Verify SSL Usage

sql
SHOW STATUS LIKE 'Ssl_cipher';

Encryption Areas to Consider

Client-server connections
Replication channels
Backups and snapshot storage
Disk-level encryption

4. Patch Management & Version Hygiene

Running outdated MySQL versions is equivalent to leaving known vulnerabilities exposed.

Maintenance Strategy

Track vendor security advisories
Apply minor updates regularly
Test patches in staging before production rollout
Avoid unsupported MySQL versions

Check Version

sql
SELECT VERSION();

5. Logging, Auditing, and Monitoring

Security without visibility is blind defense, enable Audit Logging.

1. audit_log Plugin (Legacy Model)

Installation

sql
INSTALL PLUGIN audit_log SONAME 'audit_log.so';

Verify

sql
SHOW PLUGINS LIKE 'audit%';

2. audit_log_filter Component (Modern Model)

Introduced in MySQL 8 to provide a more flexible and granular alternative to the older plugin model.

Installation

sql
INSTALL COMPONENT 'file://component_audit_log_filter';

Verify

sql
SELECT * FROM mysql.component;

Architecture Difference

Instead of a single global policy, you create:

Filters (define what to log)
Users assigned to filters

It’s granular and rule-driven.

Auditing Key Events

Failed logins
Privilege changes
Schema modifications
Unusual query activity

References:

Useful Metrics

sql
SHOW GLOBAL STATUS LIKE 'Aborted_connects';
SHOW GLOBAL STATUS LIKE 'Connections';

6. Secure Configuration Hardening

A secure baseline configuration reduces risk from common attack patterns.

Recommended Settings

ini
local_infile=OFF
secure_file_priv=/var/lib/mysql-files
sql_mode="STRICT_ALL_TABLES"
secure-log-path=/var/log/mysql

Why These Matter

Prevent arbitrary file imports
Reduce filesystem abuse
Restrict data export/import locations

7. Backup Security

Backups often contain everything an attacker wants.

Backup Best Practices

Encrypt backups
Restrict filesystem permissions
Store offsite copies securely
Rotate backup credentials
Verify restore procedures regularly

Example Permission Check

ls -l /backup/mysql

8. Replication & Cluster Security

Replication is not just a data distribution feature. It is a persistent, privileged communication channel between servers. If misconfigured, it can become a lateral movement pathway inside your infrastructure. Treat every replication link as a trusted but tightly controlled corridor.

Principle: Replication Is a Privileged Service Account

Replication users require elevated capabilities. They must be isolated, tightly scoped, and monitored like any other service identity.

Secure Replication Users

sql
CREATE USER 'repl'@'10.%'
 IDENTIFIED BY 'strongpassword'
 REQUIRE SSL;

GRANT REPLICATION REPLICA ON *.* TO 'repl'@'10.%';

Hardening considerations:

Restrict host patterns as narrowly as possible. Avoid % whenever feasible.
Require SSL or X.509 certificate authentication.
Enforce strong password policies or use a secrets manager.
Disable interactive login capability if applicable.

Encrypt Replication Traffic

Replication traffic may include sensitive row data, DDL statements, and metadata. Always encrypt it.

At minimum:

Enable require_secure_transport=ON
Configure TLS certificates on source and replica
Set replication channel to use SSL:

sql
CHANGE REPLICATION SOURCE TO
 SOURCE_SSL=1,
 SOURCE_SSL_CA='/path/ca.pem',
 SOURCE_SSL_CERT='/path/client-cert.pem',
 SOURCE_SSL_KEY='/path/client-key.pem';

For MySQL Group Replication or InnoDB Cluster:

Enable group communication SSL
Validate certificate identity
Use dedicated replication networks

Binary Log and Relay Log Protection

Replication relies on binary logs. Protect them.

Set binlog_encryption=ON
Set relay_log_info_repository=TABLE
Restrict filesystem access to log directories
Monitor log retention policies

Compromised binary logs can reveal historical data changes.

9. Continuous Security Reviews

Security is not a one-time checklist. Regular audits help catch configuration drift and evolving threats.

Suggested Review Cadence

Weekly: failed login review
Monthly: privilege audits
Quarterly: configuration review
Semiannually: full security assessment

Security Checklist Summary

Area	Key Action
Access Control	Least privilege grants
Authentication	Strong password policies
Encryption	TLS + encrypted storage
Updates	Regular patching
Monitoring	Audit logging enabled
Configuration	Harden defaults
Backups	Encrypt and protect
Replication	Secure replication users

Final Thoughts

Strong MySQL security doesn’t come from one feature or one tool. It comes from layers working together. Hardened configuration. Tight, intentional privilege design. Encryption everywhere it makes sense. And monitoring that actually gets reviewed instead of just written to disk.

In my experience, the strongest environments aren’t the ones trying to be unbreakable. They’re the ones built to detect, contain, and respond. Every layer should either reduce blast radius or increase visibility. If an attacker gets through one control, the next one slows them down. And while they’re slowing down, your logging and monitoring should already be telling you something isn’t right.

That’s what a mature security posture looks like in practice.

Meet Percona at KubeCon + CloudNativeCon Europe 2026

Ege Güneş — Wed, 25 Feb 2026 12:00:00 UTC

The Percona team is heading to KubeCon + CloudNativeCon Europe in Amsterdam, and we’d love to meet you in person!

You can find us at Booth 790. This is a great chance to talk with engineers working on Percona Operators.

We will be there to discuss:

Running MySQL, PostgreSQL, and MongoDB on Kubernetes
Production-ready HA setups
Backup and PITR strategies
Multi-cluster and multi-region deployments
Operators roadmap and upcoming features
Real-world troubleshooting stories

If you’re running Percona Operators in production (or just getting started), we’d love to hear your feedback and learn about your challenges.

If you’re just curious (or even suspicious) about running databases on Kubernetes, we’d love to talk and answer your questions.

Admission Tickets - 20% Off for Our Community

We have a 20% discount code available for Percona community members.
If you’re planning to attend and don’t have a ticket yet, drop a comment or message us and we’ll share the details.

Schedule a Meeting

Want dedicated time with our engineers?
Drop a comment here or reach out directly. We’re happy to schedule a meeting during the event.

See you in Amsterdam!

Pre-FOSDEM & FOSDEM 2026, Community, Databases, and Open Source

Edith Puclla — Mon, 09 Feb 2026 10:00:00 UTC

This is a recap of Percona at preFosdem and Fosdem!

Before FOSDEM officially started, the database community gathered for MySQL Belgium Days (Pre-FOSDEM), a two-day event bringing together MySQL developers, DBAs, engineers, tool builders, and open-source enthusiasts. It was an excellent space for deep technical discussions, knowledge sharing, and reconnecting with the community, hosted by the amazing Frederic Descamps. The event featured strong participation from Percona and the wider MySQL ecosystem, with talks led by Peter Zaitsev, Marco Tusa, Fernando Laudares Camargos, Arunjith Aravindan, Vinicius Grippa, Pep Pla, and Yura Sorokin.

Find the recordings of the talks here.

Also, in Belgium same week, several other events took place, including PGDay-FOSDEM, MariaDB Day, and the MySQL Summit.

At PGDay, it was a pleasure to see the PostgreSQL community together; we had several participants representing us.

MariaDB Day. We had Peter Zeitsev presenting a talk titled “What MariaDB Community can learn from PostgreSQL?”

MySQL summit

During MySQL Days in Brussels, the community gathered for an in-person MySQL Summit focused on collaboration and strengthening the MySQL ecosystem, with open discussions around its present and future driven by community involvement.

MySQL RockStars 2026

The MySQL Rockstar Award is a recognition given by the MySQL Community Team at Oracle, together with previous award winners, to members of the MySQL community who have actively contributed to promoting MySQL during the past year. MySQL Legends are long-standing community members who have made a significant and lasting impact on the adoption, development, and evolution of MySQL over many years.

This year, the MySQL RockStars selected were:

Matthias Crauwels
Marco Tusa
Umesh Shastry
Ronald Bradford
Marcelo Altmann

Congratulations to all of them! You can find previous MySQL RockStars in this list

FOSDEM 2026, Percona booth

At the Percona booth, conversations focused on open-source databases and Kubernetes, including OpenEverest’s first-ever presence at FOSDEM. Around 40 Perconians were present, a great chance to finally meet many colleagues in person. As always, we had many visitors, and it was great to see that many already knew about Percona, while others were eager to learn more and explore what we do in the world of Open Source!

FOSDEM 2026, Databases DevRoom

FOSDEM 2026 officially kicked off at the ULB Solbosch Campus, bringing together thousands of open-source contributors from around the world. The Database DevRoom (UB2.252A) was packed with high-quality talks and discussions, co-led with Matthias Crauwels and Ray Paik, and

Find more details and some of the recordings here.

Celebrating 20 Years of Percona

This year, Percona celebrates its 20th anniversary. Throughout FOSDEM and Pre-FOSDEM events, it was inspiring to meet long-time users who have relied on Percona’s open-source solutions for years and shared their positive experiences. You can explore Percona’s 20-year journey here: 👉 https://percona20.com/ If you’ve had a great experience with Percona, you’re invited to share your story via the community survey.

See you at FOSDEM 2027!

PGDay and FOSDEM Report from Kai

Kai Wagner — Wed, 04 Feb 2026 10:00:00 UTC

The following thoughts and comments are completely my personal opinion and do not reflect my employers thoughts or beliefs. If you don’t like anything in this post, reach out to me directly, so I can ignore it ;-).

I’m currently on the train on my way back home from FOSDEM this year and man, I’m exhausted but also happy. Why? Because the PG and FOSDEM community is just crazily awesome. While it’s always too much of everything, it’s at the same time inspiring to see so many enthusiastic IT nerds in one place, discussing and working on what they love - technology and engineering challenges.

PGDay FOSDEM

It all started with the usual PGDay FOSDEM the day before FOSDEM. Just in case - this has been happening for over 15 years and if you read this as a little blame that you didn’t know about it, that’s absolutely correct, as you should. It’s been a great event as usual: around 150 Postgres enthusiasts collaborating with each other. There was a great set of talks (no recording available, so yes, just join next year to not miss anything), as well as the hallway track conversations.

I was able and accepted again as a volunteer helping to make the event happen. While you might think, what’s special about it, I cannot express my gratitude for being able to help in any way. I simply love it. I’m not a great coder and I’ve never been one. I’m the one that looks at his code from a year ago and questions his technical existence and overall abilities if I should rather do something without touching a keyboard. What I am very well capable of is helping and supporting events. So it was my pleasure and I hope you do feel inspired to do the same next year or at any future event, not only in the Postgres ecosystem but in general. I strongly believe in this: doing good things will get you good things back.

After the wrap up to the PGDay and a great community dinner to collaborate and discuss further, I simply fell completely tired asleep, as the next day and FOSDEM was already waiting.

FOSDEM Day 1

The next day started with volunteering at the Postgres booth. As usual, Saturday was simply crazy. The Postgres swag like hoodies, caps, mugs, shirts, etc. was almost ripped out of our living hands. We had people waiting in line just to be able to get some swag. That fact alone shows how Postgres is viewed outside of the internal PG ecosystem community. How many times I heard the sentence “Thanks a lot for the great work you do” or “Postgres just works.” Yeah, we can all argue about the details and scenarios, but what this is about is the overall ease of use. Not everyone has terabytes of data or the most complex HA and replication scenarios on this planet. Some just need a functional and boring database and, in the best case, open source - and we all know, looking at real open source, not single-vendor owned, Postgres is the king and here to stay.

After all of this, I switched clothes and helped at the Percona booth. This wasn’t any less interesting in comparison to the PG booth. How many people stepped by, asking about what we do or thanking us for our projects and that we remained open source even after all these years and so many other companies not withstanding the quick and easy money to go with open-core or closed offerings. That’s the reason I’m proud to be part of this company. We walk the talk, since 20y and we have no incentive ever changing it. Thanks to Peter Zaitsev and Peter Farkas aka P² - for those who know, just know.

Following that I had the pleassure of being the Slonik guide again. What is a Slonik guide you might ask? Slonik, the mascot of Postgres (big blue elephant), needs some help and guidance while walking throught the crowd, as you can barely see anything while inside the costome. As usual, Slonik is a celebraty. Everyone wants a picture and taking their chance to photograph Slonik in the “wild”. As you can see, even MySQL’s Sakila couldn’t resist and had to take a picture with Slonik.

If you’re wondering, like many others, why Slonik and why an Elephant? Click here for some nice written down history lesson

After an exciting but also energy-draining day, I enjoyed a Percona crew/team dinner at BrewDog, with some great conversations and good food. Fun Fact: Did you know that BrewDog is also open source?. I couldn’t stay too long - sorry about that - but I had another date. The famous Floor Drees organized in tradition another karaoke event that I couldn’t miss. As I couldn’t make it to earlier versions of it, I definitely wanted to join. What should I say apart from thanks, Floor, for this great tradition. Yes, I had a hard time talking the next day, but damn I had fun singing Swedish, Polish, German, and English songs - and yes, I most likely misunderstood all of them as usual.

Too many songs for my voice and maybe a “soft drink or two” later, I felt in my bed like a stone, and couldn’t really accept the fact that my alarm clock went off almost five minutes later (at least that’s how it felt to me).

FOSDEM Day 2

No whining helped, just getting up and making myself ready for Day 2 of FOSDEM, which started with another round of volunteering at the Postgres and Percona booths. Both basically matched the previous feedback, apart from a definitely dropped and less crowded space - seems I wasn’t the only one singing last night ;-).

With that, thanks a lot to everyone making this great FOSDEM happen. I’ll try now if the Deutsche Bahn restaurant actually works this time, as I need coffee, a big one, maybe two… See all of you next year again or at another event this year.

Stay on top of Postgres development without the inbox overwhelm. Explore hackorum.dev today and share your feedback with us.

Tuning MySQL for Performance: The Variables That Actually Matter

Wayne Leutwyler — Sun, 01 Feb 2026 00:00:00 UTC

There is a special kind of boredom that only database people know. The kind where you stare at a server humming along and think, surely there is something here I can tune. Good news: there is.

This post walks through the most important MySQL variables to tune for performance, why they matter, and when touching them helps versus when it quietly makes things worse. This is written with InnoDB-first workloads in mind, because let’s be honest, that’s almost everyone.

1. `innodb_buffer_pool_size`

Real metrics to watch

Before touching this variable, look at these:

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read%';

Key fields:

Innodb_buffer_pool_reads – physical reads from disk
Innodb_buffer_pool_read_requests – logical reads

Rule of thumb: If reads / read_requests > 1–2%, your buffer pool is too small.

Example graph

Plot Innodb_buffer_pool_reads over time. A healthy system shows a flat or gently rising line. Spikes that look like a city skyline usually mean memory pressure or a cold cache.

If MySQL performance had a crown jewel, this would be it.

What it does

The InnoDB buffer pool caches table data and indexes in memory. Reads served from RAM are fast. Reads from disk are… character building.

How to tune it

Dedicated DB server: 60–75% of system RAM
Shared server: be conservative and leave memory for the OS and other services

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';

Pro tip

If your working set fits in the buffer pool, MySQL feels magical. If it doesn’t, no amount of query tuning will save you.

2. `innodb_buffer_pool_instances`

This one matters once memory gets big.

What it does

Splits the buffer pool into multiple instances to reduce internal mutex contention.

How to tune it

Only relevant if buffer pool is ≥ 1GB
Rule of thumb: 1 instance per 1–2GB, max 8

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_instances';

Gotcha

More is not always better. Too many instances wastes memory and can hurt performance.

3. `innodb_log_file_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_log%';

Pay attention to:

Innodb_log_waits
Innodb_log_write_requests

If Innodb_log_waits is non-zero, redo logs are too small for your write rate.

Example graph

Graph Innodb_log_waits as a rate per second. Ideally, this line hugs zero like it’s afraid of heights.

This variable controls how calmly MySQL handles write-heavy workloads.

What it does

Defines the size of redo logs. Larger logs mean fewer checkpoints and smoother writes.

How to tune it

OLTP workloads: 1–4GB total redo log is common
Large transactions benefit from larger logs

sql
SHOW VARIABLES LIKE 'innodb_log_file_size';

Warning

Changing this requires a restart. Plan accordingly or accept the wrath of your on-call future self.

4. `innodb_flush_log_at_trx_commit`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_os_log_fsyncs';

Switching from 1 to 2 often reduces fsyncs by orders of magnitude.

Example graph

Overlay two lines:

Transactions per second
Innodb_os_log_fsyncs per second

On busy systems, this graph alone can justify the change to skeptical auditors.

Performance versus durability, the eternal duel.

What it does

Controls how often redo logs are flushed to disk.

Common values

1 – Safest, slowest (flush every commit)
2 – Very popular compromise
0 – Fast, risky

sql
SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';

Reality check

For many production systems, 2 delivers massive performance gains with acceptable risk, especially with reliable storage.

5. `innodb_flush_method`

This decides how MySQL talks to your disks.

What it does

Controls whether MySQL uses OS cache or bypasses it.

Caveat

Some filesystems and older kernels behave differently. Always test.

6. `max_connections`

This is not a performance knob. It is a damage limiter.

What it does

Caps the number of concurrent client connections.

Why it matters

Each connection consumes memory. Too many and MySQL dies spectacularly.

sql
SHOW VARIABLES LIKE 'max_connections';

Advice

Set it realistically
Use connection pooling
Monitor Threads_connected

7. `thread_cache_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Threads%';

Key fields:

Threads_created
Connections

If Threads_created / Connections stays above a few percent, your cache is undersized.

Example graph

Graph Threads_created as a counter. A healthy system shows a curve that flattens over time, not a staircase.

Small change, measurable win.

What it does

Caches threads so MySQL doesn’t constantly create and destroy them.

How to tune

Watch:

sql
SHOW STATUS LIKE 'Threads_created';

If it keeps climbing, increase thread_cache_size.

8. `table_open_cache` and `table_definition_cache`

Metadata matters more than people expect.

What they do

Cache open tables and table definitions to avoid repeated filesystem access.

Symptoms of being too low

High Opened_tables
Metadata lock waits

sql
SHOW VARIABLES LIKE 'table_open_cache';
SHOW VARIABLES LIKE 'table_definition_cache';

9. `tmp_table_size` and `max_heap_table_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Created_tmp%';

Watch:

Created_tmp_tables
Created_tmp_disk_tables

If disk temp tables exceed 5–10% of total temp tables, queries are spilling to disk.

Example graph

Stacked area chart:

In-memory temp tables
Disk-based temp tables

Disk usage creeping upward usually points to reporting queries pretending to be OLTP.

Disk-based temp tables are silent performance killers.

What they do

Limit how large in-memory temp tables can grow.

How to tune

Set both to the same value:

ini
tmp_table_size=256M
max_heap_table_size=256M

Reality

This helps complex queries, but bad queries still need fixing.

10. `slow_query_log` and `long_query_time`

Not a performance variable, but a performance revelation.

Why it matters

You cannot tune what you cannot see.

ini
slow_query_log=ON
long_query_time=1

This turns guesswork into evidence.

A Note on Graphing These Metrics

You don’t need exotic tools. These work well:

performance_schema
sys schema views
Prometheus + mysqld_exporter
Percona Monitoring and Management (PMM)

Golden rule: Always graph rates, not raw counters.

Final Thoughts

Tuning MySQL is less about endless knobs and more about understanding pressure points:

Memory first
I/O second
Concurrency third

Most performance wins come from a handful of variables, not heroic config files full of folklore.

If you tune one thing today, make it the buffer pool. If you tune two, add redo logs. Everything else is refinement.

And if you’re bored again tomorrow, congratulations. You’re officially a database person.

Configuring the Component Keyring in Percona Server and PXC 8.4

Wayne Leutwyler — Tue, 13 Jan 2026 00:00:00 UTC

Configuring the Component Keyring in Percona Server and PXC 8.4

(Or: how to make MySQL encryption boring, which is the goal)

Encryption is one of those things everyone agrees is important, right up until MySQL refuses to start and you’re staring at a JSON file wondering which brace ruined your evening.

With MySQL 8.4, encryption has firmly moved into the component world, and if you’re running Percona Server 8.4 or Percona XtraDB Cluster (PXC) 8.4, the supported path forward is the component_keyring_file component.

The good news: the setup is mostly identical for Percona Server and PXC.
The bad news: PXC 8.4.4 and 8.4.5 shipped with a bug that makes this less fun than it should be.

Let’s walk through a setup that works, keeps your keys locked down, and avoids the usual landmines.

Step 1: Tell MySQL Which Component to Load

Components are registered using JSON, not traditional MySQL configuration syntax. This is important, because MySQL will not politely warn you if you get it wrong. It will simply refuse to start.

Create the file:

sudo vi /usr/sbin/mysqld.my

Add:

json
{
 "components": "file://component_keyring_file"
}

Take a second to double-check the formatting. One missing quote here will cost you more time than you want to admit.

Now lock it down:

bash
sudo chown root:root /usr/sbin/mysqld.my
sudo chmod 644 /usr/sbin/mysqld.my

This is configuration, not data. MySQL only needs to read it.

Step 2: Prepare the Keyring Directory (Handle With Care)

This directory will hold encryption keys. Treat it accordingly.

bash
cd /var/lib
sudo mkdir mysql-keyring
sudo chown mysql:mysql mysql-keyring
sudo chmod 750 mysql-keyring

A simple rule that saves headaches:

mysql owns the keys
MySQL is allowed to access them
Nobody else gets any ideas

Step 3: Configure the Keyring Component Itself

Next, move to the plugin directory:

cd /usr/lib64/mysql/plugin

Create the component configuration file:

sudo vi component_keyring_file.cnf

Add:

json
{
 "path": "/var/lib/mysql-keyring/component_keyring_file",
 "read_only": true
}

This file tells MySQL where the keyring lives and ensures it can’t be casually modified at runtime.

Set ownership and permissions:

bash
sudo chown root:root component_keyring_file.cnf
sudo chmod 640 component_keyring_file.cnf

Again: configuration belongs to root. MySQL just reads it.

Step 4: The PXC 8.4.4 / 8.4.5 Bug (Yes, There’s One)

If you’re running Percona Server, you can skip this entire section and enjoy your day.

If you’re running Percona XtraDB Cluster 8.4.4 or 8.4.5, there is a known issue with plugin paths that prevents the component keyring from loading correctly. This was fixed in PXC 8.4.6.

If upgrading isn’t an option yet, you’ll need one of the following workarounds.

Option A: Create a Symlink (Preferred)

bash
sudo ln -s \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib/plugin

Option B: Copy the Plugin Directory

bash
sudo cp -ar \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib

If you’re on PXC 8.4.6 or newer, this problem is already behind you and you can safely pretend it never existed.

Step 5: Restart MySQL

Time for the moment of truth:

sudo systemctl restart mysql

Or mysqld, depending on your system.

If MySQL starts cleanly, you’re doing well. If not, go back and check your JSON files. It’s almost always the JSON.

Step 6: Verify the Keyring Is Actually Loaded

Never assume. Always verify.

sql
SELECT *
FROM performance_schema.keyring_component_status;

You should see the component_keyring_file listed and active. If it’s there, the keyring is live.

sql
+---------------------+-----------------------------------------------+
| STATUS_KEY | STATUS_VALUE |
+---------------------+-----------------------------------------------+
| Component_name | component_keyring_file |
| Author | Oracle Corporation |
| License | GPL |
| Implementation_name | component_keyring_file |
| Version | 1.0 |
| Component_status | Active |
| Data_file | /var/lib/mysql-keyring/component_keyring_file |
| Read_only | Yes |
+---------------------+-----------------------------------------------+
8 rows in set (0.00 sec)

A Note for Percona Server Users

Percona Server may still include legacy keyring plugins such as:

keyring_file
keyring_vault

Do not mix legacy keyring plugins with component keyrings. They come from different eras of MySQL design and do not coexist peacefully.

Choose one model. For MySQL 8.4 and forward, components are the future.

Additional Steps for Percona XtraDB Cluster (PXC)

Percona XtraDB Cluster introduces one critical difference compared to standalone Percona Server: the keyring file itself is not replicated by Galera. Only metadata and transactional state are replicated. The encryption keys remain node-local filesystem artifacts and must be handled deliberately.

Node 1: Establish the Authoritative Keyring

Choose a single node to initialize the keyring. This is typically Node1, but the choice itself is not important as long as you are consistent.

On this node:

Complete all previous steps in this document
Start MySQL successfully
Verify the keyring component is loaded:

sql
SELECT *
FROM performance_schema.keyring_component_status;

Once this node is running, the file below will be created and populated:

swift
/var/lib/mysql-keyring/component_keyring_file

This file becomes the authoritative source of encryption keys for the entire cluster.

Why the Keyring File Must Be Copied

PXC ensures that encrypted data remains readable on all nodes, but it does not distribute encryption keys themselves. Each node must have access to the same key material, or encrypted tablespaces will fail to open.

If a node starts without the correct keyring file, you may see:

Tablespace open failures
Startup errors related to encryption
Inconsistent behavior during SST or IST

This is expected behavior and not a bug.

Distribute the Keyring File to Other Nodes

On each remaining PXC node:

Ensure MySQL is stopped
Create the keyring directory if it does not exist:

bash
sudo mkdir -p /var/lib/mysql-keyring
sudo chown mysql:mysql /var/lib/mysql-keyring
sudo chmod 750 /var/lib/mysql-keyring

Securely copy the keyring file from Node1:

scp /var/lib/mysql-keyring/component_keyring_file node2:/var/lib/mysql-keyring/component_keyring_file

Important: Do not modify the file. Do not recreate it. Do not allow MySQL to generate a new one on secondary nodes.

Start MySQL on Each Node and Verify

After the keyring file is in place:

sudo systemctl start mysqld

Verify the component is active:

sql
SELECT * FROM performance_schema.keyring_component_status;

Each node should report the component_keyring_file as loaded and active.

At this point:

Encrypted tablespaces will open correctly
SST and IST operations will succeed
The cluster will behave consistently during restarts

Operational Notes and Best Practices

Treat the keyring file like a secret, not configuration
Restrict access to root only
Include the keyring file in your secure backup strategy
When provisioning new nodes, copy the keyring file before first startup
Never rotate or regenerate the keyring independently on individual nodes

If the keyring is lost and encrypted data exists, recovery is not possible.

Final Thoughts

This setup works reliably for:

Percona Server 8.4
Percona XtraDB Cluster 8.4
(with the known exception of 8.4.4–8.4.5)

Most failures come down to:

Treating JSON like a .cnf file
Loose ownership on sensitive files
Forgetting the PXC-specific workaround

Once those are handled, the component keyring fades into the background where it belongs. And when it comes to encryption, boring, quiet, and uneventful is exactly the outcome you want.

What is New in Percona Toolkit 3.7.1

Sveta Smirnova — Wed, 17 Dec 2025 00:00:00 UTC

Percona Toolkit 3.7.1 has been released on Dec 17, 2025. The most important updates in this version are:

Finalized SSL/TLS support for MySQL
Added support for Debian 13 and Amazon Linux 2023
Fixed MariaDB support broken in version 3.7.0
Added options to skip certain collections in pt-k8s-debug-collector and pt-stalk
Documentation improvements
Other performance improvements

In this blog, I will outline the most significant changes. A full list of improvements and bug fixes can be found in the release notes.

SSL/TLS support for MySQL

Percona Toolkit historically did not have consistent SSL support. This was reported at https://perconadev.atlassian.net/browse/PT-191. In version 3.7.0, option s for DSN was introduced. This option instructs DBD::mysql to open a secure connection with the database. This version also adds command-line option --mysql-ssl and its short form -s to all tools. All other SSL/TLS-related options, such as ssl-ca, ssl-cert, ssl-cipher, etc, could be specified in the configuration file if necessary. This completes SSL/TLS support for MySQL. For more details and information check this blog post.

Supported Platforms Update

Percona repositories now have Percona Toolkit packages for Debian 13 and Amazon Linux 2023. To install them enable repository pt with the percona-release utility. Ubuntu Focal reached its EOL and support for this platform has been removed. More information on Percona repositories is available in the User Reference Manual.

Regression Bug Fixes

Recent major changes introducing MySQL 8.4 support missed ignore case modificator for the regular expression that checks if MySQL flavor is MariaDB. As a result, tools executed replication statements not compatible with MariaDB. Version 3.7.1 fixes the regular expression and re-adds MariaDB support back (PT-2451). Future versions of Percona Toolkit will have better MariaDB support, including MariaDB-specific versions of non-offensive replication commands.

Utility pt-sift stopped working, because dependent library alt_cmds.sh was not included (PT-2498). This was not found during previous release testing, because regression test for the tool was not run. Now this miss is fixed and the utility works properly again. Additionally, regression test is updated.

Helper utility version_cmp was written in some compiled language and source code for it was not available (PT-2469). This broke version checking on platforms not compatible with the unknown platform where the binary was originally compiled. Now this utility rewritten as a Bourne-Again shell script.

Modern MySQL Support

Percona Toolkit uses legacy MySQL syntax in many places to be compatible with older versions of MySQL. In other places, it misses modern MySQL diagnostic additions. This version makes first steps to improve this situation by adding such features as invisible index support in pt-duplicate-key-checker (PR-996) and performance_schema.threads collecton in pt-stalk (PT-1718). Currently, data from performance_schema.threads is collected along with the deprecated information_schema.processlist. In the future, support for information_schema.processlist will be deprecated, then removed.

Future versions of Percona Toolkit will have more modern MySQL diagnostic support.

Performance Improvements

pt-stalk now has new option, --skip-collection, that allows to skip one or more collections. Supported values for this option are: ps-locks-transactions, thread-variables, innodbstatus, lock-waits, mysqladmin, processlist, rocksdbstatus, transactions. To skip two or more collections, separate them with a comma. E.g., --skip-collection=processlist,innodbstatus. You will find more information at PT-2289 and in the User Reference Manual for pt-stalk.

pt-k8s-debug-collector introduces option -skip-pod-summary allowing to skip pod summary collections, such as pt-mysql-summary, pt-mongodb-summary, or pg_gather. Check PT-2453 and the User Reference Manual for pt-k8s-debug-collector.

Originally, tools output was always buffered. This is usually good for performance but you may want to disable this feature when need to see output of the tools faster. For example, if you run pt-archiver or pt-table-checksum on large table in Kubernetes, you won’t see progress (PT-2052) until the tool finishes. New option, --[no]buffer-stdout, allows to disable buffering when needed.

Incompatilbe change

Earlier, if --chunk-size was enabled for pt-online-schema-change, option --chunk-time was ignored. This caused situations when a user has to start with default automatic chunk size even if it was not effective for some tables, and wait when chunk size is adjusted in subsequent iterations. Alternatively, they had to guess fixed chunk size that implies time consuming try and error approach (PT-1423).

Starting from version 3.7.1, if both options --chunk-size and --chunk-time are specified, initial chunk size will be as specified by the option --chunk-size, but later it will be adjusted, so that the next query takes specified amount of time (in seconds) to execute.

Documentation Improvements

While working on this release we found undocumented featues such as --recursion-method=dsn support in pt-table-sync (PT-2470), broken man page for pt-secure-collect and other tools written in Go language (PT-1564), as well as minor documentation issues. Now all of them are fixed.

Community contributions

This release includes contributions from Community and Percona Engineers who do not actively work on the project. We want to thank:

Iwo Panowicz for option -skip-pod-summary in pt-k8s-debug-collector (PT-2453)
Matthew Boehm for invisible indexes support in pt-duplicate-key-checker (PR-996)
Nilnandan Joshi for collecting performance_schema.threads along with information_schema.processlist in pt-stalk (PT-1718) and fix for PT-2014 - pt-config-diff does not honor case insensitivity flag
Paweł Kudzia for the updated documentation of pt-query-digest (PR-953)
Maciej Dobrzanski for fixing PR-890 - pt-config-diff: MySQL truncates run-time variable values longer than 1024 characters
Marek Knappe for fixing PT-2418 - pt-online-schema-change 3.7.0 lost data when exe alter xxx rename column xxx and PT-2458 - remove-data-dir defaults to True
Yoann La Cancellera for his work on pt-galera-log-explainer
Nyele for restoring MariaDB support (PT-2465)
Taehyung Lim for fixing PT-2401 - pt-online-schema-change ’table does not exist’ on macos
Viktoras Agejevas for fixing PR-989 - Fix script crashing with precedence error in pt-online-schema-change
Hartley McGuire for fixing PT-2015 - pt-config-diff does not sort variable flags

MySQL Replication Best Practices: How to Keep Your Replicas Sane (and Your Nights Quiet)

Wayne Leutwyler — Wed, 03 Dec 2025 00:00:00 UTC

MySQL replication has been around forever, and yet… people still manage to set it up in ways that break at the worst possible moment. Even in 2025, you can get burned by tiny schema differences, missing primary keys, or one forgotten config flag. I’ve seen replicas drift so far out of sync they might as well live in a different universe.

This guide covers the practical best practices—the stuff real DBAs use every day to keep replication stable, predictable, and boring. (Boring is a compliment in database land.)

Always Use GTIDs. Yes, Always.

GTID-based replication is one of those features that people resist turning on, and then once they do, they never want to go back.

Why GTIDs?

Failover become sane
Reparenting replicas stops being a headache
Missing transactions are easy to detect

Your my.cnf should absolutely include:

gtid_mode=ON
enforce_gtid_consistency=ON
log_replica_updates=ON

Once GTIDs are enabled, do not mix in old-style replication. That path leads straight to confusion.

Use Row-Based Replication (RBR)

Statement-based replication is a nostalgia trip that nobody asked for. It breaks on:

NOW(), UUID(), and similar functions
Floating point differences
Collation mismatches
Triggers behaving differently

Just skip the pain and use:

binlog_format=ROW

RBR is slightly more verbose, but 100× more predictable. When something breaks, it’s never because you chose ROW.

Every Table Needs a Primary Key. No Exceptions.

If you take nothing else from this guide, take this:

Replication without primary keys is a bad time.

Row-based replication needs a way to find the row that changed. Without a PK (or at least a UNIQUE index), the server has to use every column as a lookup. That’s slow, error-prone, and sometimes impossible.

The usual symptoms:

Replication lag slowly creeping up
Replica doing full table scans on updates
Rows failing to apply
Errors like:

Error 1032: Can't find record in table

Save yourself hours of debugging and just make sure every table has a primary key.

Keep the Schema Identical Everywhere

Replication assumes that everyone’s using the same schema. MySQL will happily keep going even if your schemas don’t match—and then quietly drift out of sync.

Here are the practical ways to keep schemas aligned:

Approach A — mysqldump (most common)

Export schemas only:

mysqldump --no-data mydb > schema.sql

From both servers, then:

diff source-schema.sql replica-schema.sql

Approach B — information_schema metadata

This approach is great for automaton:

SELECT table_name, column_name, column_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'mydb'
ORDER BY table_name, ordinal_position;

Execute this query on each server and diff the results. Update mydb to match the database whose schema metadata you want to examine.

Approach C — pt-table-checksum (data only)

This doesn’t compare schemas — it catches data drift. You should consider running it on a schedule such as:

high-change OLTP DBs run weekly or even daily
huge multi-TB DBs run quarterly
some sensitive systems avoid running it during peak hours

pt-table-checksum --replicate=percona.checksums

You can fix drift with:

pt-table-sync --execute --replicate=percona.checksums

Schema checks + data checks = safe replication.

Harden Your Binary Log Settings

Your binlogs are the backbone of replication. Treat them carefully.

sync_binlog=1
binlog_row_image=FULL
binlog_expire_logs_seconds=604800 # 7 days

sync_binlog=1 is the big one—without it, a crash can corrupt binlogs or the GTID position, and that leads to a very bad day.

Protect Your Replicas with super_read_only

Never allow accidental writes to replicas, in your my.cnf set:

read_only=ON
super_read_only=ON

super_read_only closes the loophole that even SUPER users could previously use to write to replicas.

Use a Dedicated Replication User

Give the minimal permissions:

CREATE USER 'repl'@'%' IDENTIFIED BY 'strong_password';
GRANT REPLICATION REPLICA ON *.* TO 'repl'@'%';

This user should do exactly one thing: replicate. Don’t reuse app users—you’re just begging for trouble.

Replication Lag: Watch It Like a Hawk

Seconds_Behind_Source lies more often than you’d expect. It’s okay for a quick glance but don’t rely on it.

Better options:

Performance Schema: replication_applier_status_by_worker
Percona Monitoring and Management (PMM)
Custom heartbeat tables
pt-heartbeat

Lag is one of the biggest causes of outages—monitor it continuously. Lag is usually the first sign something is wrong—catch it early.

Use Parallel Replication (But Don’t Overdo It)

If your primary has multiple writers or many concurrent transactions, in your my.cnf enable parallel workers:

replica_parallel_type=LOGICAL_CLOCK
replica_parallel_workers=4

4–8 workers is a sweet spot for most systems. More workers ≠ more speed; after a point it just increases memory footprint without real benefit.

But when it helps, it really helps—like cutting lag by 80–90%.

Use SSL Anywhere Outside the LAN

Replication traffic isn’t something you want exposed.

source_ssl=1
source_ssl_ca=/path/ca.pem

Earlier versions used the master_ssl_* variables, but the idea is the same: encrypt the connection when it leaves your trusted network.

Final Thoughts

MySQL replication can be rock-solid, but only if you follow a handful of rules that experienced DBAs know by heart:

Use GTIDs
Use RBR
Always have primary keys
Keep schemas aligned
Check for data drift
Harden binlog settings
Protect replicas from accidental writes
Monitor lag properly
Use parallel workers when appropriate
Encrypt connections over untrusted networks

Follow these, and your replicas will stay healthy, consistent, and (mostly) invisible—which is exactly how you want them.

Community Recap: Percona.Connect London 2025, Building the Future of Open Source Together

Edith Puclla — Tue, 02 Dec 2025 00:00:00 UTC

Percona.Connect London 2025 brought the open-source database community together for a half-day of learning and collaboration. The event focused on providing practical, technical insights for DBAs, DevOps engineers, and developers. The main takeaway was clear: Stability, Openness, and Automation are essential for modern, large-scale data infrastructure.

Top Discussions & Key Takeaways

1. The Rise of Valkey: A Truly Open Caching Alternative

Martin Visser, Valkey Technical Lead, explained the changes to the Redis license, the community needs a trusted, open-source replacement. Valkey was highlighted as the leading solution.

Valkey was started by former Redis contributors quickly after Redis removed its open source license in 2024.
It is a true open-source project governed under the Linux Foundation.
It offers enhancements like better memory efficiency, performance, and scalability.
In a recent Percona survey of 200 DBAs, Valkey was the most preferred alternative to Redis.

2. Running PostgreSQL in a Cloud Native context

Takis Stathopoulos, Enterprise Architect, presented on running PostgreSQL in a Cloud Native context, explaining how Kubernetes Operators simplify complex deployments.

Cloud Native vs. Cloud First: Cloud Native (Kubernetes) offers Portability and No vendor lock-in, allowing you to run the database consistently across different clouds and on-premise infrastructure.
Percona Operator for PostgreSQL: This tool automates crucial operations like setting up high availability (using Patroni), backups (using pgBackrest), and scaling.
When to use Cloud Native: It’s ideal for large, microservice-based applications and teams prioritizing portability and avoiding vendor lock-in.

3. Native PostgreSQL TDE is Here: Securing Data Simply

Alastair Turner, Postgres Community Advocate, introduced the new Native Transparent Data Encryption (TDE) for PostgreSQL.

4. The Future of MySQL: Vector Search & Binlog Server

Dennis Kittrell, MySQL Product Manager, discussed two key features planned for MySQL that address major operational and feature challenges.

MySQL Binlog Server MVP: This component aims to solve the problem of quick disaster recovery by acting as a stable, reliable replication source. It enables Precise Point-in-Time Recovery (PITR) using simple time or GTID coordinates.
Native Vector Support MVP: This feature allows users to eliminate the complexity of using a separate vector database. You can store, index, and search vector embeddings directly in MySQL, allowing you to combine vector searches with standard business logic in a single, transactional query

Our Community Focus

A common theme from the use cases was that while open source adoption is high, operational teams often lack the proper support and visibility.

Percona’s goal is to support the community by providing:

Stability when under heavy load or during maintenance.
Faster Troubleshooting with better monitoring and observability.
Safer Deployments through expert configuration and security support.

Thank you to everyone who joined us in London for a dynamic event. We hope the insights gained will help you with your open source database deployments.

The conversations continue in the Percona Community! You can reach out directly to the speakers:

Martin Visser (Valkey Technical Lead) LinkedIn
Dennis Kittrell (MySQL Product Manager) LinkedIn
Alastair Turner (Postgres Community Advocate) LinkedIn
Takis Stathopoulos (Enterprise Architect) LinkedIn
Andre Pons (Enterprise Sales Manager) LinkedIn

Join the Percona Community Conversation!

The Right Tool for the Job

Wayne Leutwyler — Mon, 24 Nov 2025 00:00:00 UTC

When I first got into woodworking, my mentor shared a piece of advice that has stuck with me ever since: “Use the right tool for the job.” You wouldn’t reach for a belt sander to flatten a board when a planer can accomplish the task faster, cleaner, and with far better results.

The same principle applies in the world of database engineering. When working with MySQL or Percona Server, choosing the correct tool can be the difference between efficient diagnostics and unnecessary downtime.

In this post, I’ll highlight several of the most practical and commonly used utilities from the Percona Toolkit. While the toolkit includes many powerful commands, I’ll focus on the ones that provide the most value in day-to-day operations, troubleshooting, and gathering actionable details for support cases.

PT Summary

A Percona Toolkit utility that provides a concise, high-level overview of a system’s hardware, OS configuration and performance-related metrics. It’s designed to quickly capture the essential details needed for diagnostics or support cases—CPU, memory, disk layout, kernel parameters and more all in a single, easy-to-read report.

Example

Run pt-summary with no arguments to generate a full system summary. When possible, run it with sudo to allow the tool to collect additional details that require elevated privileges:

sudo pt-summary
# Percona Toolkit System Summary Report ######################
 Date | 2025-11-24 17:15:19 UTC (local TZ: EST -0500)
 Hostname | pi16gb
 Uptime | 41 days, 2:27, 4 users, load average: 0.00, 0.00, 0.00
 Platform | Linux
 Release | Debian GNU/Linux 12 (bookworm) (bookworm)
 Kernel | 6.12.47+rpt-rpi-2712
Architecture | CPU = 32-bit, OS = 64-bit
 Threading | NPTL 2.36
 SELinux | No SELinux detected
 Virtualized | No virtualization detected
# Processor ##################################################
 Processors | physical = 4, cores = 0, virtual = 4, hyperthreading = no
 Speeds |
 Models |
 Caches |
 Designation Configuration Size Associativity
 ========================= ============================== ======== ======================
# Memory #####################################################
 Total | 15.8G
 Free | 675.0M
 Used | physical = 5.3G, swap allocated = 512.0M, swap used = 0.0, virtual = 5.3G
 Shared | 44.7M
 Buffers | 10.6G
 Caches | 10.5G
 Dirty | 128 kB
 UsedRSS | 5.1G
 Swappiness | 60
 DirtyPolicy | 20, 10
 DirtyStatus | 0, 0
 Locator Size Speed Form Factor Type Type Detail
 ========= ======== ================= ============= ============= ===========
# Mounted Filesystems ########################################
 Filesystem Size Used Type Opts Mountpoint
 /dev/nvme0n1p1 510M 14%
 /dev/nvme0n1p2 458G 5%
 /dev/sda1 117G 16%
# Disk Schedulers And Queue Size #############################
 nvme0n1 | [none] 255
 sda | [mq-deadline] 60
# Disk Partitioning ##########################################
# Kernel Inode State #########################################
dentry-state | 107782 98346 45 0 32304 0
 file-nr | 3680 0 9223372036854775807
 inode-nr | 99614 20818
# LVM Volumes ################################################
Unable to collect information
# LVM Volume Groups ##########################################
Unable to collect information
# RAID Controller ############################################
 Controller | No RAID controller detected
# Network Config #############################################
 Controller | 00.0 Ethernet controller
 FIN Timeout | 60
 Port Range | 60999
# Interface Statistics #######################################
 interface rx_bytes rx_packets rx_errors tx_bytes tx_packets tx_errors
 ========= ========= ========== ========== ========== ========== ==========
 lo 6000000000 175000 0 6000000000 175000 0
 eth0 0 0 0 0 0 0
 wlan0 5000000000 30000000 0 15000000000 22500000 0
# Network Devices ############################################
 Device Speed Duplex
 ========= ========= =========
 eth0 Unknown! Unknown!
# Network Connections ########################################
 Connections from remote IP addresses
 192.168.1.91 1
 192.168.1.251 1
 2603 2
 Connections to local IP addresses
 192.168.1.145 2
 2603 2
 Connections to top 10 local ports
 3306 2
 6011:ef0:7260:::22 2
 States of connections
 ESTABLISHED 3
 LISTEN 6
 TIME_WAIT 1
# Top Processes ##############################################
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 95842 root 20 0 0 0 0 I 6.7 0.0 0:00.08 kworker+
 1 root 20 0 169520 13088 8672 S 0.0 0.1 0:18.56 systemd
 2 root 20 0 0 0 0 S 0.0 0.0 0:01.70 kthreadd
 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_wo+
 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
# Notable Processes ##########################################
 PID OOM COMMAND
 ? ? sshd doesn't appear to be running
# Simplified and fuzzy rounded vmstat (wait please) ##########
 procs ---swap-- -----io---- ---system---- --------cpu--------
 r b si so bi bo ir cs us sy il wa st
 2 0 0 0 1 6 100 150 0 0 100 0 0
 1 0 0 0 0 0 1750 3000 1 3 97 0 0
 1 0 0 0 0 0 250 400 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
# Memory management ##########################################
# The End ####################################################

Redirect output to a file.

pt-summary > server-summary.txt

PT MySQL Summary

A Percona Toolkit utility that collects and displays a concise overview of a MySQL or Percona Server instance, including key configuration settings, performance metrics, storage engine details, replication status, buffer pool usage, and important global variables. It provides a fast, structured snapshot of the database environment, making it ideal for troubleshooting, tuning, and preparing information for support teams.

Example

pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
 System time | 2025-11-24 17:45:54 UTC (local TZ: EST -0500)
# Instances ##################################################
 Port Data Directory Nice OOM Socket
 ===== ========================== ==== === ======
 3306 /data0/mysql/data/ 0 0 /usr/local/mysql/mysql.sock
# MySQL Executable ###########################################
 Path to executable | /usr/local/mysql/bin/mysqld
 Has symbols | Yes
# Report On Port 3306 ########################################
 User | wayne@localhost
 Time | 2025-11-24 12:45:54 (EST)
 Hostname | pi16gb
 Version | 8.4.6-6 Source distribution
 Built On | Linux aarch64
 Started | 2025-10-14 09:48 (up 41+02:57:35)
 Databases | 10
 Datadir | /data0/mysql/data/
 Processes | 2 connected, 2 running
 Replication | Is not a replica, has 1 replicas connected
 Pidfile | /usr/local/mysql/mysqld.pid (exists)
# Processlist ################################################

 Command COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 Binlog Dump GTID 1 1 3000000 3000000
 Query 1 1 0 0

 User COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 replication 1 1 3000000 3000000
 wayne 1 1 0 0

 Host COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 192.168.1.251 1 1 3000000 3000000
 localhost 1 1 0 0

 db COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 NULL 2 2 3000000 3000000

 State COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 init 1 1 0 0
 Source has sent all binlog to 1 1 3000000 3000000

# Status Counters (Wait 10 Seconds) ##########################
Variable Per day Per second 11 secs
Aborted_clients 1
Binlog_snapshot_position 350000 4
Binlog_cache_use 6000
Bytes_received 20000000 225 600
Bytes_sent 2250000000 25000 4000
[...]
Table_open_cache_misses 400
Table_open_cache_overflows 225
Threads_created 9
Uptime 90000 1 1
# Table cache ################################################
 Size | 1000
 Usage | 100%
# Key Percona Server features ################################
 Table & Index Stats | Disabled
 Multiple I/O Threads | Enabled
 Corruption Resilient | Enabled
 Durable Replication | Not Supported
 Import InnoDB Tables | Not Supported
 Fast Server Restarts | Not Supported
 Enhanced Logging | Disabled
 Replica Perf Logging | Disabled
 Response Time Hist. | Not Supported
 Smooth Flushing | Not Supported
 HandlerSocket NoSQL | Not Supported
 Fast Hash UDFs | Unknown
# Percona XtraDB Cluster #####################################
# Plugins ####################################################
 InnoDB compression | ACTIVE
# Schema #####################################################
Specify --databases or --all-databases to dump and summarize schemas
# Noteworthy Technologies ####################################
 SSL | Yes
 Explicit LOCK TABLES | No
 Delayed Insert | No
 XA Transactions | No
 NDB Cluster | No
 Prepared Statements | Yes
 Prepared statement count | 0
# InnoDB #####################################################
 Version | 8.4.6-6
 Buffer Pool Size | 8.0G
 Buffer Pool Fill | 30%
 Buffer Pool Dirty | 0%
 File Per Table | ON
 Page Size | 16k
 Log File Size | 2 * 48.0M = 96.0M
 Log Buffer Size | 64M
 Flush Method | O_DIRECT
 Flush Log At Commit | 2
 XA Support |
 Checksums |
 Doublewrite | ON
 R/W I/O Threads | 4 4
 I/O Capacity | 200
 Thread Concurrency | 4
 Concurrency Tickets | 5000
 Commit Concurrency | 0
 Txn Isolation Level | REPEATABLE-READ
 Adaptive Flushing | ON
 Adaptive Checkpoint |
 Checkpoint Age | 0
 InnoDB Queue | 0 queries inside InnoDB, 0 queries in queue
 Oldest Transaction | 0 Seconds
 History List Len | 1
 Read Views | 0
 Undo Log Entries | 0 transactions, 0 total undo, 0 max undo
 Pending I/O Reads | 0 buf pool reads, 0 normal AIO, 0 ibuf AIO, 0 preads
 Pending I/O Writes | 0 buf pool (0 LRU, 0 flush list, 0 page); 0 AIO, 0 sync, 0 log IO (0 log, 0 chkp); 1 pwrites
 Pending I/O Flushes | 0 buf pool, 0 log
 Transaction States | 3xnot started
# MyISAM #####################################################
 Key Cache | 8.0M
 Pct Used | 20%
 Unflushed | 0%
# Security ###################################################
 Users | 8 users, 0 anon, 0 w/o pw, 7 old pw
 Old Passwords |
# Encryption #################################################
No keyring plugins found
# Binary Logging #############################################
 Binlogs | 3
 Zero-Sized | 0
 Total Size | 437.9M
 binlog_format | ROW
 expire_logs_days |
 sync_binlog | 0
 server_id | 10
 binlog_do_db |
 binlog_ignore_db |
# Noteworthy Variables #######################################
 Auto-Inc Incr/Offset | 1/1
 default_storage_engine | InnoDB
 flush_time | 0
 init_connect |
 init_file |
 sql_mode | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
 join_buffer_size | 256k
 sort_buffer_size | 256k
 read_buffer_size | 128k
 read_rnd_buffer_size | 256k
 bulk_insert_buffer | 0.00
 max_heap_table_size | 16M
 tmp_table_size | 16M
 max_allowed_packet | 64M
 thread_stack | 1M
 log |
 log_error | /var/log/mysql/mysqld.log
 log_warnings |
 log_slow_queries |
log_queries_not_using_indexes | OFF
 log_replica_updates | ON
# Configuration File #########################################
 Config File | /etc/my.cnf

[mysqld]
character-set-server = utf8mb4
authentication_policy = '*'
port = 3306
socket = /usr/local/mysql/mysql.sock
pid-file = /usr/local/mysql/mysqld.pid
basedir = /usr/local/mysql/
datadir = /data0/mysql/data/
tmpdir = /data0/mysql/tmp/
general_log_file = /var/log/mysql/mysql-general.log
log-error = /var/log/mysql/mysqld.log
slow_query_log_file = /var/log/mysql/slow_query.log
[...]
innodb_data_home_dir = /data0/mysql/data/
innodb_log_group_home_dir = /data0/mysql/data/
innodb_temp_data_file_path = ../tmp/ibtmp1:12M:autoextend:max:8G
innodb_buffer_pool_size = 8G
innodb-redo-log-capacity = 2G
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity = 200
innodb_buffer_pool_instances = 8
innodb_thread_concurrency = 4

# Memory management library ##################################
jemalloc is not enabled in mysql config for process with id 788
# The End ####################################################

Redirect output to file.

pt-mysql-summary > percona-server.txt

Capture both pt-summary and pt-mysql-summary into a single file.

pt-summary > percona-server-summary.txt
pt-mysql-summary >> percona-server-summary.txt

PT Online Schema Change

A Percona Toolkit utility that performs online ALTER TABLE operations by creating a shadow copy of the table, applying the schema change to that copy, and keeping it in sync with the original using triggers until it is ready to swap. This workflow minimizes locking and reduces downtime, allowing large production tables to be altered safely with minimal impact on applications. However, it’s important to remind users that long-running queries or transactions holding metadata locks (MDL) on the table will still block the final swap, potentially delaying completion of the schema change.

Examples

Adding a New Column

pt-online-schema-change \
 --alter "ADD COLUMN status TINYINT NOT NULL DEFAULT 0" \
 D=mydb,t=orders \
 --execute

This safely introduces a new column to a busy table without blocking reads or writes. The tool handles the copy, synchronization, and final table swap automatically.

Modifying a Column Type

pt-online-schema-change \
 --alter "MODIFY COLUMN price DECIMAL(10,2)" \
 D=shop,t=products \
 --execute

Changing column definitions—especially on large datasets—can be disruptive using standard SQL. With pt-osc, the migration happens online, keeping applications responsive throughout the operation.

Dropping an Unused Column

pt-online-schema-change \
 --alter "DROP COLUMN old_flag" \
 D=analytics,t=events \
 --execute

Column drops can require a full table rebuild, making them great candidates for pt-osc. This example removes a legacy column while avoiding table locks.

Adding an Index

pt-online-schema-change \
 --alter "ADD INDEX idx_user_id (user_id)" \
 D=app,t=logins \
 --execute

Index creation is another expensive operation for large tables. Here, pt-osc allows the index to be added online, improving performance without interrupting the application.

Changing a Primary Key

pt-online-schema-change \
 --alter "DROP PRIMARY KEY, ADD PRIMARY KEY(id, created_at)" \
 D=orders,t=order_items \
 --execute

Primary key modifications usually require a full table rewrite. pt-osc makes this process safer and easier on production systems by performing the change on a temporary shadow table.

Performing a Dry Run

pt-online-schema-change \
 --alter "ADD COLUMN test INT" \
 D=mydb,t=mytable \
 --dry-run

A dry run allows you to validate the plan and review the process without making any actual changes—a critical safeguard when preparing for production schema work.

Printing SQL Changes Before Execution

pt-online-schema-change \
 --alter "ADD COLUMN updated_at TIMESTAMP NULL" \
 D=crm,t=customers \
 --print \
 --execute

Using –print provides transparency into the SQL operations the tool will perform. This is particularly useful during code reviews or change-control processes.

PT Show Grants

A Percona Toolkit utility that extracts MySQL user accounts and privileges and outputs them as clean, executable CREATE USER and GRANT statements. It normalizes and orders the privileges for readability, making it valuable for auditing security, documenting access, migrating users between servers, or preparing accurate privilege information for support and compliance purposes.

Examples

Dump All Grants for All Users

The simplest and most common use case is generating a complete privilege snapshot:

pt-show-grants

This returns normalized CREATE USER and GRANT statements for every account in the instance. It’s ideal for audits, environment comparisons, and creating human-readable privilege reports.

Show Grants for a Specific User

If you want to inspect privileges for a single account, you can filter by user/host:

pt-show-grants --accounts='user@localhost'

This makes privilege debugging and user-level audits quick and targeted.

Export All Grants to a File

To create a reusable backup of every user account:

pt-show-grants > grants.sql

The resulting file is a set of CREATE USER and GRANT statements that can be restored simply by executing:

mysql < grants.sql

This is an excellent practice before server upgrades, user cleanup, or major permission changes.

Show Grants for Multiple Accounts

You can provide a comma-separated list of accounts to extract only what you need:

pt-show-grants --accounts='app@%,reporting@localhost,backup@localhost'

This is ideal for teams that manage groups of service accounts across environments.

Ignore Specific System Accounts

For cleanup scripts or custom inventory reports, skip built-in MySQL accounts:

pt-show-grants --ignore='mysql.sys@localhost,mysql.infoschema@localhost'

This focuses output on only the accounts relevant to your application.

Summary

This post highlights the importance of using the right tool for the job—both in woodworking and in database engineering. For MySQL and Percona Server environments, the Percona Toolkit offers a set of powerful utilities that simplify diagnostics, troubleshooting, schema changes, and security audits.

It introduces four key tools:

pt-summary – Generates a high-level report of system hardware, OS settings, filesystems, networking, and performance metrics. Useful for support cases and quick environment overviews.
pt-mysql-summary – Produces a structured snapshot of a MySQL instance, including configuration, performance counters, replication status, storage engine details, and important variables. Ideal for tuning and issue analysis.
pt-online-schema-change – Enables online ALTER TABLE operations by copying and syncing the table in the background, minimizing downtime. Several examples show how to add, drop, or modify columns and indexes safely.
pt-show-grants – Extracts all MySQL users and privileges into clean, reproducible CREATE USER and GRANT statements. Helpful for audits, migrations, backups, and security reviews.

Percona Operator for MySQL Is Now GA, More MySQL Options for the Community on Kubernetes

Edith Puclla — Wed, 19 Nov 2025 11:00:00 UTC

We’re excited to share that the new Percona Operator for MySQL (based on Percona Server for MySQL) is officially in General Availability (GA)!

This release introduces native MySQL Group Replication support for Kubernetes, providing our community with another open-source option for running reliable, consistent MySQL clusters at scale.

This is about more choices for the community. Each MySQL replication technology addresses different real-world needs, and now you can choose the one that best fits your workloads.

What This Means for the Community

With this release, Percona now supports two fully open-source MySQL Operators:

1. Percona Operator for MySQL (Percona Server for MySQL), New and GA

Group Replication (synchronous)
Asynchronous replication (Technical Preview)
Native MySQL experience
Auto-failover
Kubernetes-native design

2. Percona XtraDB Cluster Operator (PXC)

Galera-based synchronous replication
Strong high availability
Auto-failover
Battle-tested for mission-critical workloads

These Operators complement each other; they are not replacements. They give users the freedom to choose the right replication model for their business and technical priorities.

This GA release is a step in that direction, and we will continue publishing technical blog posts to explain when to use each Operator, how Group Replication works, and how this all fits into real-world Kubernetes environments.

Call for Community Testing and Feedback

Asynchronous replication is now available in Technical Preview, we invite you to:

Test it in your clusters
Share your feedback
Open GitHub issues
Contribute docs or examples

Your feedback will guide the next features we bring to the Operator.

Explore Percona Operator for MySQL:

MySQL Memory Usage: A Guide to Optimization

Roman Agabekov — Tue, 11 Nov 2025 00:00:00 UTC

Struggling with MySQL memory spikes? Knowing how and where memory is allocated can make all the difference in maintaining a fast, reliable database. From global buffers to session-specific allocations, understanding the details of MySQL’s memory management can help you optimize performance and avoid slowdowns. Let’s explore the core elements of MySQL memory usage with best practices for trimming excess in demanding environments.

How MySQL Uses Memory

MySQL dynamically manages memory across several areas to process queries, handle connections, and optimize performance. The two primary areas of memory usage include:

Global Buffers

These are shared by the entire MySQL server and include components like the InnoDB buffer pool, key buffer, and query cache. The InnoDB buffer pool is particularly memory-intensive, especially in data-heavy applications, as it stores frequently accessed data and indexes to speed up queries.

Connection (per thread) Buffers

When a client connects, MySQL allocates memory specifically for that session. This includes sort buffers, join buffers, and temporary table memory. The more concurrent connections you have, the more memory is consumed. Session buffers are critical to monitor in high-traffic environments.

Why MySQL Memory Usage Might Surge

Memory spikes in MySQL often result from specific scenarios or misconfigurations. Here are a few examples:

High Traffic with Large Connection Buffers: A surge in concurrent connections can quickly exhaust memory if sort or join buffers are set too large.
Complex Queries: Queries with large joins, subqueries, or extensive temporary table usage can temporarily allocate significant memory, especially when poorly optimized.
Oversized InnoDB Buffer Pool : Setting the InnoDB buffer pool size too large for the server’s available memory can trigger swapping, severely degrading database and server performance.
Large Temporary Tables : When temporary tables exceed the in-memory limit ( tmp_table_size ), they are written to disk, consuming additional resources and slowing down operations.
Inefficient Indexing : A lack of proper indexes forces MySQL to perform full table scans, increasing memory and CPU usage for even moderately complex queries.

Best Practices for Controlling MySQL Memory Usage

When you notice MySQL using more memory than expected, consider the following strategies:

1. Set Limits on Global Buffers

Configure innodb_buffer_pool_size to 60-70% of available memory for InnoDB-heavy workloads. For smaller workloads, scale it down to avoid overcommitting memory.
Keep innodb_log_buffer_size at a practical level (e.g., 16MB) unless write-heavy workloads demand more.
Adjust key_buffer_size for MyISAM tables, ensuring it remains proportionate to table usage to avoid unnecessary memory allocation.

2. Adjust Connection Buffer Sizes

Reduce sort_buffer_size and join_buffer_size to balance memory usage with query performance, especially in environments with high concurrency.
Optimize tmp_table_size and max_heap_table_size to control in-memory temporary table allocation and avoid excessive disk usage.

3. Fine-Tune Table Caches

Adjust table_open_cache to avoid bottlenecks while considering OS file descriptor limits.
Configure table_definition_cache to manage table metadata efficiently, especially in environments with many tables or foreign key relationships.

4. Control Thread Cache and Connection Limits

Use thread_cache_size to reuse threads effectively and reduce overhead from frequent thread creation.
Adjust thread_stack and net_buffer_length to suit your workload while keeping memory usage scalable.
Limit max_connections to a level appropriate for your workload, preventing excessive session buffers from overwhelming server memory.

5. Track Temporary Table Usage

Monitor temporary table usage and reduce memory pressure by optimizing queries that rely on GROUP BY, ORDER BY, or UNION.

6. Use MySQL Memory Calculator

Incorporate tools like the MySQL Memory Calculator by Releem to estimate memory usage. Input your MySQL configuration values, and the calculator will provide real-time insights into maximum memory usage. This prevents overcommitting your server’s memory and helps allocate resources effectively.

7. Monitor Query Performance

High-memory-consuming queries, such as those with large joins or sorts, queries without indexes, can affect memory usage. Use Releem’s Query Analytics and Optimization feature to determine inefficient queries and gain insights on further tuning opportunities.

Simplifying MySQL Memory Tuning with Releem

Releem takes the guesswork out of MySQL optimization by automatically analyzing your setup and suggesting configuration changes that align with your memory limits and performance needs. Whether you’re dealing with complex workloads or simply don’t have time for manual adjustments, Releem makes it easier to keep MySQL running smoothly.

A thread through my 2025 Postgres events

Alastair Turner — Mon, 10 Nov 2025 07:00:00 UTC

I recently got back from PostgreSQL Conference Europe in Riga, marking the end of my conference activities for 2025. The speakers were great. The audience, for the Extensions Showcase on Community Day on Tuesday and my Kubernetes from the database out talk, were great. The event team was great. The singing at karaoke was terrible, but it’s supposed to be.

After attending a good few events this year, starting with CERN PGDay in mid-January, I wanted to write something about more than just the most recent event. I see a common thread across presentations and sessions at a number of events over the year, that is, scale-out Postgres and particularly, its use in non-profit scientific environments.

The (beginning and) end users

Far fewer data processing challenges require pooling the resources of many physical servers these days, with servers getting bigger and storage faster. Scientific data analysis and managing large, complex scientific facilities still do. I saw three presentations on this: Rafal Kulaga, Antonin Kveton and Martin Zemko’s on managing CERN’s SCADA data; Daniel Krefl and Krzysztof Nienartowicz at CERN on how Sendai queries variable star data; and Jaoquim Oliveira in Riga on managing the European Space Agency’s (ESA’s) survey mission data.

I admit a fondness for ESA’s GAIA catalog dataset. After I was lucky enough to do a proof of concept project on joining it with other catalog data, it has provided significant intellectual interest. Don’t let me get started on the possible ways to optimise computationally expensive inequality joins on horribly skewed data, unless you really care about the problem. My interest in a dataset discussed in two of these talks is not why the thread connecting them is worth commenting on. All three presentations had a lot of content on selecting or developing database technologies for the work they were doing. That’s worth discussing a bit further.

Getting the details right

The thread of sharded, scale out, or Massively Parallel Processing (MPP) Postgres connects end user stories at my first event of the year and my last, along with stories of building this software at events in between. At PGConf.dev in Montreal David Wein gave a very condensed explanation of how AWS’s Aurora Limitless handles distributed snapshot isolation (watch the lightning talk at on YouTube), there was also an unconference session on handling the issue in core Postgres the next day. For an in-depth explanation of of what the distributed snapshot problem is and how it may be addressed, see Ants Aasma’s talk from PGConf.EU 2024

The organisations with the data are looking for open source software solutions and bumping into issues around open core licensing, project contribution breadth, project activity levels, project governance. The Postgres developer community is working on the knottiest of the problems in this space, trying to get it absolutely right. In the mean-time, various forks and extensions are delivering useful functionality for the owners of these big, complex datasets. Useful, but could do better

If this were working out for everyone, there wouldn’t be a story to tell. Sednai are building Potgres-XZ, which builds on TBase, which built on Postgres-XL. The ESAC Science Data Centre (ESDC) is facing a decision between two single-vendor projects, where one vendor doesn’t provide support for on-premises deployments. CERN procurement sought written assurances over license terms for TimescaleDB, since the CERN facilities organisation may be viewed as a service provider to their hosted scientific projects.

This pattern of licenses built specifically to avoid “AWS stealing our innovation/lunch/…”, (and it is always AWS set up as the bogeyman in these stories), is particularly unfortunate here, because it just isn’t true for Postgres. AWS, and Azure, employ big teams of community contributors to work on open source Postgres. The progress on statistics management, asynchronous IO, and vacuum in Postgres 18 are, among others, thanks to these teams’ efforts.

No matter how positive the involvement of the hyperscalers may be for Postgres, there are organisations who will prefer to run their own databases. On-premises hosting is a clear choice for organisations with big facilities capabilities, capital-centric budgeting, extreme requirements and predictable, always on workloads. Many of these organisations are publicly funded scientific projects. It would be great if there were broad-based open source solutions to meet their data management needs.

Doing better, together

At PGConf in Riga the Percona team took a few, early steps towards building a joint effort to deliver the components of such a solution. I hope that the big, open managers of structured scientific data (or their subcontractors, depending on their engagement model) and a few vendors can come together to build event data compression, columnar storage, and all the other bits which can be implemented as extensions.

The current Postgres extensions and forks for scale out systems were built on older versions of Postgres, so they had to build features which now exist in core Postgres. Their implementation of partitioning, for instance, differs subtly from the capabilities now available in modern Postgres. As feature-specific extensions can take over capabilities which are currently intertwined with sharding (like compression in Timescale or columnar storage in Citus), users will be less locked in to vertical stacks of features, some useful to them and some not. Simple sharding can then become a proxy (like pgDog), an automation on DDL on a gateway server or even a core Postgres feature.

Which leaves those special cases where moving data between shards during execution is key to performance. This is mattering less with ever bigger servers, improving Postgres parallelism and tools like DuckDB - but when it matters it still really matters. Here the sons of the ‘plum - CloudberryDB and WarehousePG, forked from Greenplum when it closed source - work their magic (hat tip to Jimmy Angelakos for the “the ‘plum” contraction). Managing that particular capability will always be a big, complex code base. If the patches carried to make it happen shrink as Postgres and extensions fill the gap, we’ll have a more sustainable route to all good database things being openly available.

Encryption support in PMM Dump

Sveta Smirnova — Thu, 30 Oct 2025 11:00:00 UTC

The pmm-dump client utility performs a logical backup of the performance metrics collected by the PMM Server and imports them into a different PMM Server instance. PMM Dump allows you to share monitoring data collected by your PMM server with the Percona Support team securely.

Up until now dumps, created by the tool, were not encrypted. It was possible to encrypt them after they are done but this required additional actions from the user.

Starting from the upcoming PMM Dump version 0.8.0-ga released on October 29, 2025, dumps are encrypted by default.

Key points

Dump files are encrypted by default with AES-256-based encryption.
An auto-generated password is produced for each encrypted dump; it is printed at the end of the export operation or can be written to a file with --pass-filepath.
You can provide a custom password with --pass.
Disable encryption with --no-encryption only when you understand the risks.
By default, for encrypted dumps, export logging to STDOUT is suppressed; use --no-just-key to override.

Why this matters

Encrypting PMM dumps prevents accidental exposure of monitoring and query data that may contain sensitive information (query text, hostnames, metrics). It brings PMM Dump in line with secure data-handling best practices and simplifies safe sharing with Percona Support.

Quick examples

Export (encryption enabled by default):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs
...
Password: ****************
$ ls pmm-dump-.tar.gz.enc

Provide a custom password:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass='My$trongP@ss'

Save auto-generated password to file:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass-filepath=/tmp/pmm-dump.pass

Disable encryption (not recommended):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --no-encryption

Import an encrypted dump:

$ pmm-dump import --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs \
--dump-path=pmm-dump-1758017090.tar.gz.enc --pass='My$trongP@ss'

Decrypt an encrypted dump (if needed):

$ openssl enc -d -aes-256-ctr -pbkdf2 -in dump.tar.gz.enc -out dump.tar.gz

Recommendations

Prefer leaving encryption enabled.
Use --pass-filepath to store passwords securely rather than relying on terminal output.
Transfer encrypted archives over secure channels (SCP/SFTP) and share passwords via secure out-of-band channels.

Availability

Encryption support is included starting in the recent PMM Dump 0.8.0-ga release. Check your PMM Dump version (pmm-dump version) and the docs for exact version details.

Additional information

Audit Log Filters Part II

Wayne Leutwyler — Wed, 08 Oct 2025 00:00:00 UTC

In my first post on the MySQL 8.4 Audit Log Filter component, I covered how to install the component and configure a basic filter that captures all events. The Audit Log Filter framework offers a highly granular and configurable auditing mechanism, enabling administrators to log specific events based on criteria such as user, host, or event type. This selective approach enhances observability, supports compliance initiatives, and minimizes unnecessary logging overhead.

In this follow-up, we’ll take a deeper technical look at defining and optimizing audit log filters to capture only the most relevant database activities—delivering actionable audit data while significantly reducing noise and log volume.

Example 1

Audit all events:

SELECT audit_log_filter_set_filter('log_all_events', '{
 "filter": {"log": true}
}');

Once this filter is created and assigned to a user (for example, with SELECT audit_log_filter_set_user(’%’, ’log_all_events’);), every database event triggered by that user—or by all users if % is used—will be written to the audit log file.

In short:

This is the most permissive audit configuration possible. It’s typically used:

As a baseline test to verify that the audit log component is working.
In diagnostic or forensic scenarios where full visibility is required.

For production environments, however, it’s recommended to create more selective filters (e.g., by event class, command type, or user) to reduce log volume and improve performance. Which we will go into more detail in the upcoming examples.

Example 2

Log table access:

select audit_log_filter_set_filter('log_table_access', '{
 "filter": {
 "class": [
 { "name": "table_access" },
 { "name": "connection" },
 { "name": "general" }
 ]
 }
}');

Included Event Classes

table_access Logs events when MySQL reads from or writes to tables.
- Useful for tracking which users or applications are accessing specific tables.
- Helps in auditing data access patterns and detecting unauthorized data reads/writes.
connection Logs connection-related events such as user logins, logouts, and failed authentication attempts.
- Important for tracking session activity and security auditing.
general Logs general query execution events—like statements sent to the server (e.g., SELECT, INSERT, UPDATE, etc.).
- Useful for general SQL activity auditing.

What It Does Functionally

After this filter is defined and assigned to a user or host (for example, with SELECT audit_log_filter_set_user(’%’, ’log_table_access’);), MySQL will only log events that fall into one of these three classes.

All other event types—like administrative commands, stored program executions, or system-level actions—will be excluded from the audit log.

Why use this filter

This configuration strikes a balance between completeness and efficiency:

Captures key operational and access-related activity.
Avoids excessive log volume from irrelevant events.
Suitable for data access auditing, security monitoring, and compliance logging.

In short, log_table_access provides targeted visibility into table usage, connections, and general query activity—ideal for environments where tracking who accessed what data is more important than recording every internal event.

Example 3

SELECT audit_log_filter_set_filter('log_minimum', '{
 "filter": {
 "class":
 [ { "name": "connection" }, { "name": "table_access", "event": [
 { "name": "delete"}, { "name": "insert"}, { "name": "update"} ]
 } ]
 }
}');

Included Event Classes

“class”: “connection”
- Logs all connection-level events:
- connect: when a user logs in.
- disconnect: when a session ends.
- Failed logins and other connection-related actions.

Purpose: provides visibility into who connected, from where, and when.

“class”: “table_access” with “event”:
- Limits logging to specific table access events:
  - “delete” when rows are deleted.
  - “insert” when new rows are added.
  - “update” when existing rows are modified.
- Read operations (like SELECT) and metadata queries are excluded.

What It Does Functionally

Once assigned to a user or host (e.g. SELECT audit_log_filter_set_user(’%’, ’log_minimum’);), this filter will produce audit entries only when:

A user connects or disconnects from MySQL.
A user performs a DML (Data Manipulation Language) operation that changes data in a table.

All other events — such as simple SELECT queries, schema reads, or administrative commands — will be ignored.

Why Use This Filter

This is a minimalist, high-value audit configuration. It’s designed to:

Track security-relevant activity (connections and data changes).
Meet compliance requirements with low performance overhead.
Prevent excessive logging and disk usage.

In Short log_minimum is an efficient auditing strategy for production environments where you only need to know:

Who accessed the database, and
What data they changed.

It gives you essential accountability and change tracking without the overhead of logging every read or administrative event.

Example 4

SELECT audit_log_filter_set_filter('log_connections', '{
 "filter": {
 "class": [
 { "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 }
 ]
 }
}');

Included Event Classes

“class”: “connection”
- This class captures events related to user sessions and authentication.
“event”: [“connect”, “disconnect”]
- connect, Logged when a client establishes a connection to the MySQL server. Includes details like username, host, client program, IP address, and connection method.
- disconnect, Logged when that client session ends or times out. Useful for tracking session duration and identifying abnormal terminations.
Important Note on pre_authenticate Events
- The pre_authenticate events are not included in the examples above.
- These events occur before the MySQL server has received authentication information from the client—meaning no user account details are available at this stage of the connection lifecycle. Because of that, if a filter that includes pre_authenticate events is assigned to a specific user (rather than a wildcard like %) using audit_log_filter_set_user(), those events will not be filtered or logged.
- This behavior often leads to confusion, as users may expect pre_authenticate events to appear in user-specific logs. Several reports and support cases have been filed on this topic, but it is expected behavior due to the timing of authentication during connection initialization.

Why Use This Filter

This filter is particularly useful when you need to:

Monitor user logins and logouts without recording query activity.
Audit connection patterns (e.g., who connected, from where, and when).
Maintain minimal log size and low overhead.
Support security investigations or session tracking without performance impact.

In short the log_connections filter provides a focused, low-overhead auditing strategy that records only connection lifecycle events. It’s ideal for environments where you primarily need to know who connected to the database, when, and from where without capturing every SQL statement or table access.

Example 5

SELECT audit_log_filter_set_filter('log_full_table_access', '{
 "filter": {
 "class": [
 {
 "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 },
 {
 "name": "query",
 "event": [
 {
 "name": "start",
 "log": {
 "or": [
 { "field": { "name": "sql_command_id", "value": "select"} },
 { "field": { "name": "sql_command_id", "value": "insert"} },
 { "field": { "name": "sql_command_id", "value": "update"} },
 { "field": { "name": "sql_command_id", "value": "delete"} },
 { "field": { "name": "sql_command_id", "value": "truncate"} },
 { "field": { "name": "sql_command_id", "value": "create_table"} },
 { "field": { "name": "sql_command_id", "value": "alter_table"} },
 { "field": { "name": "sql_command_id", "value": "drop_table"} }
 ]
 }
 }
 ]
 }
 ]
 }
}');

This statement defines a MySQL Audit Log Filter called log_full_table_access, which is designed to capture both connection activity and all table-related SQL operations — including reads, writes, and schema changes. It provides broad visibility into how users interact with tables in the database while filtering out unrelated or low-value events.

Included Event Classes

After assigning it to users or hosts (e.g. SELECT audit_log_filter_set_user(’%’, ’log_full_table_access’);), MySQL will log:

Connection lifecycle events (connect, disconnect)
All DML statements (SELECT, INSERT, UPDATE, DELETE, TRUNCATE)
All DDL statements that create, modify, or remove tables (CREATE TABLE, ALTER TABLE, DROP TABLE)
A full list of SQL_COMMANDS can be obtained from:

SELECT NAME FROM performance_schema.setup_instruments WHERE NAME LIKE 'statement/sql/%' ORDER BY NAME;

+-----------------------------------------------+
| NAME |
+-----------------------------------------------+
| statement/sql/alter_db |
| statement/sql/alter_event |
| statement/sql/alter_function |
| statement/sql/alter_instance |
| statement/sql/alter_procedure |
| statement/sql/alter_resource_group |
| statement/sql/alter_server |
| statement/sql/alter_table |
| statement/sql/alter_tablespace |
| statement/sql/alter_user |
| statement/sql/alter_user_default_role |
| statement/sql/analyze |
| statement/sql/assign_to_keycache |
| statement/sql/begin |
| statement/sql/binlog |
| statement/sql/call_procedure |
| statement/sql/change_db |
| statement/sql/change_repl_filter |
| statement/sql/change_replication_source |
| statement/sql/check |
| statement/sql/checksum |
| statement/sql/commit |
| statement/sql/create_compression_dictionary |
[...]
+-----------------------------------------------+
168 rows in set (0.07 sec)

Everything else — administrative commands, stored procedure calls, replication control, etc. — will be excluded.

Why Use This Filter

This filter offers a comprehensive audit view of how users interact with data and schema structures — perfect for compliance, forensic analysis, or access accountability. It ensures that all table reads, writes, and structure changes are tracked without overwhelming the log with irrelevant internal events.

In Short, log_full_table_access provides a broad but targeted audit scope:

Tracks connections for user session context.
Logs all table-level operations, both data and schema-related.
Delivers complete visibility into how data is accessed and changed, making it ideal for security auditing and regulatory compliance scenarios.

Final Summary

The MySQL 8.4 Audit Log Filter component provides a powerful and flexible framework for controlling how database activity is captured and logged. By allowing administrators to define granular filters based on event class, event type, user, or host, it transforms auditing from an all-or-nothing process into a precisely tuned observability tool.

In this post, we explored a range of filter examples—from the most permissive (log_all_events) to more focused configurations like log_minimum, log_connections, and log_full_table_access. Each serves a different operational or compliance purpose:

log_all_events – Captures every event for baseline validation or forensic debugging.
log_table_access – Balances visibility and performance by logging table, connection, and general query activity.
log_minimum – Targets critical actions such as connections and data modifications, providing essential accountability with minimal overhead.
log_connections – Focuses solely on login and logout events, ideal for lightweight session auditing.
log_full_table_access – Delivers comprehensive insight into all table-level DML and DDL operations along with connection tracking, suitable for compliance and change auditing.

By tailoring filters to specific operational needs, administrators can significantly reduce log volume, improve performance, and focus on high-value security and compliance events. The result is a leaner, more informative audit log that provides actionable insight into how users and applications interact with your MySQL environment—without the burden of unnecessary data.

Reference

Special Thanks

Yura Sorokin for the collaboration that made this blog post possible.

Percona at PostgreSQL Conference Europe 2025

Edith Puclla — Fri, 03 Oct 2025 11:00:00 UTC

We’re proud to announce that Percona is a Platinum Sponsor of PostgreSQL Conference Europe (PGConf.EU) 2025, taking place October 21–24, 2025 in Riga, Latvia 🇱🇻 at the Radisson Blu Latvija Conference Center.

As a Platinum Sponsor, you can find Percona in a prime main exhibit floor location; a chance to connect directly with our PostgreSQL experts from around the world. Visitors stopping by our booth will be able to:

Enter raffles and win Percona & PostgreSQL SWAG
Watch live demos and learn from expert-led sessions
Explore open source PostgreSQL solutions built for scalability, performance, and security

As a leader in open source database management and services, Percona supports innovation, collaboration, and the sharing of knowledge that drives the open source ecosystem forward. By participating in PostgreSQL Conference Europe, Percona connects with developers, contributors, and industry leaders to discuss the latest trends, challenges, and advancements. We look forward to further showcasing Percona’s dedication to empowering organizations with robust, scalable, and secure open source database solutions.

About the event

This year’s conference is the 15th Annual PostgreSQL Conference Europe. The conference is organised by PostgreSQL Europe, with participation from most of the PostgreSQL user groups around Europe, and is intended to be an important meeting and cooperation point for users both in and out of Europe. PGConf.EU is a unique chance for European PostgreSQL users and developers to catch up, learn, build relationships, get to know each other and consolidate a real network of professionals that use and work with PostgreSQL.

Percona Speaker Agenda

Our experts are taking the stage across multiple tracks to share insights, hands-on experience, and forward-looking innovations:

Wednesday, October 22

What implementing pg_tde taught us about PostgreSQL

👤 Jan Wieremjewicz

⏰ 16:05 – 16:55 | Room: Alfa | Track: Community (45 minutes)

Thursday, October 23

Kubernetes from the Database Out

👤 Alastair Turner

⏰ 10:25 – 10:55 | Room: Omega 2 | Track: DBA (25 minutes)
TDE as an Extension: A Different Path for PostgreSQL Encryption

👤 Zsolt Parragi

⏰ 11:25 – 12:15 | Room: Beta | Track: Sponsors
Why PostgreSQL took the crown from MySQL and what lies ahead

👤 Peter Zaitsev

⏰ 17:20 – 17:35 | Room: Omega 1 | Track: Platinum Sponsor Keynotes

Friday, October 24

Lessons from two decades of hacking the proprietary value into open source databases

👤 Michal Nosek

⏰ 09:25 – 10:15 | Room: Beta | Track: Sponsors

See the full agenda here.

Community Leadership – Kai Wagner

We are also proud that Kai Wagner, Senior Engineering Manager for PostgreSQL at Percona, is serving on the PGConf.EU 2025 Selection Committee. Kai is a long-time open source contributor and speaker, actively involved in projects like Ceph, openATTIC, and PostgreSQL. As a community builder and PGConf Germany organizer, he helps shape the PostgreSQL ecosystem and ensure diverse, impactful content is represented at the conference.

Join us in Riga this October to connect, learn, and celebrate PostgreSQL with Percona and the wider open source community!

Audit Log Filter Component

Wayne Leutwyler — Thu, 18 Sep 2025 00:00:00 UTC

The audit log filter component in MySQL 8.4 provides administrators with a powerful mechanism for auditing database activity at a fine-grained level. While it offers significant flexibility—such as selectively logging events based on users, hosts, or event types—it can also be challenging to understand and configure correctly.

In this article, we will examine how the audit log filter component works, walk through its core concepts, and share practical tips for configuring and managing audit filters effectively. Our goal is to help you leverage this feature to improve observability, meet compliance requirements, and reduce unnecessary logging overhead.

Enabling Audit Log Filter

We will be using Percona Server 8.4.4 or higher in the examples below. First, we need to enable the audit log filter component. To install the audit log filter component, we need to run the following command:

mysql -u root -p < /usr/share/percona-server/mysql/share/audit_log_filter_linux_install.sql

Verify that the audit log filter component is enabled by running:

select * from mysql.component;
+--------------+--------------------+-----------------------------------+
| component_id | component_group_id | component_urn |
+--------------+--------------------+-----------------------------------+
| 2 | 1 | file://component_audit_log_filter |
+--------------+--------------------+-----------------------------------+
1 row in set (0.00 sec)

Installing the component creates two new tables in the mysql system database: audit_log_filter and audit_log_user. These tables store the audit log filter definitions and the user-to-filter mappings. Together, they are referred to as the audit log filter tables.

+------------------------------------------------------+
| Tables_in_mysql |
+------------------------------------------------------+
| audit_log_filter |
| audit_log_user |
+------------------------------------------------------+

Although the configuration is persisted in these tables, they are not usually modified directly with INSERT or UPDATE statements. Instead, MySQL provides built-in functions such as:

audit_log_filter_set_filter()
audit_log_filter_set_user()

To manage filter definitions and user assignments safely.

Configure the my.cnf file to define the desired audit log output format and specify the location of the audit.log file. In the example below, the log format is set to JSON, but other formats (e.g., NEW or OLD) can also be configured depending on your requirements. The audit log file can be written to any path accessible to the MySQL server process.

Example Changes:

[mysqld]
# auditlog
audit_log_filter.format=JSON
audit_log_filter.file=/var/lib/mysql/audit.log

Restart mysql server to apply the changes:

sudo systemctl restart mysqld

The audit log filter install is complete. Now we can start using the audit log filter component.

Creating Audit Log Filters

The audit log filter component in MySQL 8.4 provides fine-grained control over database auditing. Instead of logging all events indiscriminately, administrators can define audit log filters, which are rule sets that determine exactly which events should be captured and which should be excluded.

This allows you to:

Log only the activity relevant to security, compliance, or troubleshooting.
Reduce unnecessary noise and audit log volume.
Apply different filters to specific users, hosts, or accounts for tailored auditing.

Because filters can be customized and assigned at the user or host level, the audit log filter component offers both flexibility and efficiency, making it a powerful mechanism for monitoring and securing database activity while minimizing overhead.

Lets create a rule that will log all events:

SELECT audit_log_filter_set_filter('log_all_events', '{ "filter": {"log": true } }');

Lets assign the rule to the user:

SELECT audit_log_filter_set_user('%', 'log_all_events');
SELECT audit_log_filter_flush();

Now all users will have all their events logged.

Before proceeding, ensure that the jq utility is installed on your system. The installation commands provided in the examples below are compatible with both RHEL-based and Debian-based distributions.

RHEL builds

sudo yum install jq

Debian builds

sudo apt install jq

To validate that the audit log filter is functioning as expected, we can inspect the raw contents of the audit.log file. Since the log entries are in JSON format, using jq provides an efficient way to query and extract specific events. For example, to filter and display only connection-related events, run:

cat audit.log | jq '.[]|select(.class=="connection")'

This command streams the log file, parses the JSON structure, and returns only entries where the “class” field is equal to “connection”. This approach allows for targeted analysis, making it easier to verify filter behavior, troubleshoot issues, or monitor specific event classes without manually parsing large volumes of log data.

Example Output:

{
 "timestamp": "2025-07-24 07:18:00",
 "id": 27110,
 "class": "connection",
 "event": "connect",
 "connection_id": 415,
 "account": {
 "user": "wayne",
 "host": "localhost"
 },
 "login": {
 "user": "wayne",
 "os": "",
 "ip": "",
 "proxy": ""
 },
 "connection_data": {
 "connection_type": "socket",
 "status": 0,
 "db": ""
 },
 "connection_attributes": {
 "_pid": "717914",
 "_platform": "aarch64",
 "_os": "Linux",
 "_client_name": "libmysql",
 "os_user": "wayne",
 "_client_version": "8.4.5-5",
 "program_name": "mysql"
 }
}

I’ll cover more advanced filter configurations in a follow-up post—stay tuned for Part 2 of the Audit Log Filter Component series.

In summary, the audit log filter component in MySQL 8.4 provides administrators with a flexible and fine-grained approach to database auditing. By tailoring filters to specific users, hosts, and event types, you can ensure that only the most relevant activity is logged, making it easier to meet compliance requirements while reducing overhead. With proper configuration and careful use of filters, you can transform the audit log from a noisy data dump into a precise monitoring tool that strengthens both security and observability in your MySQL environment.

pg_stat_monitor Needs You! Join the Feedback Phase

Jan Wieremjewicz — Wed, 13 Aug 2025 00:00:00 UTC

At Percona, we believe that great open source software is built with the Community, not just for it. As we plan the next iteration of pg_stat_monitor, our advanced PostgreSQL monitoring extension, we’re taking a closer look at the current feature set and how it aligns with real-world usage.

In open source, the community isn’t just a user base, it’s the most important stakeholder. While we set the vision, your feedback is the compass that guides us. Your experiences, bug reports, and feature requests are what validate our direction and keep us focused on what matters most. Without your active involvement, it’s impossible to build a tool that truly solves the problems you face every day. Your input ensures pg_stat_monitor evolves in a way that is both innovative and genuinely useful.

Your Feedback Is the Compass

Over time, pg_stat_monitor has grown beyond its initial query performance monitoring scope.

While many features have proven to be extremely useful (especially in use with Percona Monitoring and Management (PMM)), others may see limited adoption or give no evidence of any adoption at all. To ensure we’re investing in what really matters, we want to understand what you, the users and contributors, actually rely on day-to-day.

That’s why we’re kicking off a Community feedback phase

We’re especially interested in:

What features of pg_stat_monitor are critical for your workflows?
Are you using it together with PMM, CLI or in another way?
Are there parts of the extension that feel unnecessary or unclear?
What would you love to see in the next release?

This is just the beginning of a broader review and improvement effort, one that we want to run transparently and inclusively, true to our open source values.

Whether you’re a developer, DBA, or platform engineer using pg_stat_monitor directly or via tools like PMM, know that your input matters.

👉 Leave a comment below, or reach out on our Percona Community Forums or via GitHub. Let’s shape the future of pg_stat_monitor together.

GitOps Journey: Part 4 – Observability and Monitoring with Coroot in Kubernetes

Daniil Bazhenov — Tue, 22 Jul 2025 00:01:00 UTC

Our PostgreSQL cluster is running, and the demo app is generating traffic — but we have no visibility into the health of the Kubernetes cluster, services, or applications.

What happens when disk space runs out? What if the database is under heavy load and needs scaling? What if errors are buried in application logs? How busy are the network and storage layers? What’s the actual cost of the infrastructure?

This is where Coroot comes in.

Coroot is an open-source observability platform that provides dashboards for profiling, logs, service maps, and resource usage — helping you track system health and diagnose issues quickly.

We’ll deploy it using Helm via ArgoCD, continuing with our GitOps workflow.

This is Part 4 in our series. Previously, we:

Set up ArgoCD and a GitHub repository for declarative manifests (Part 1)
Installed a PostgreSQL cluster using Percona Operator
Deployed a demo application to simulate traffic and interact with the database

All infrastructure is defined declaratively and deployed from the GitHub repository, following GitOps practices.

So far, we’ve explored cluster scaling, user management, and dynamic configuration — and now it’s time for observability.

We’ll install Coroot by following the official documentation for Kubernetes.

Steps ahead:

Install the Coroot Operator
Install the Coroot Community Edition

Let’s get started.

Project Structure

We already have a postgres/ directory for PostgreSQL manifests and an apps/ directory for ArgoCD applications.

We’ll preserve this layout and add a new coroot/ folder for clarity. You can use a different structure if preferred.

Create Manifest for Installing the Coroot Operator

The documentation recommends installing via Helm.
Since we use ArgoCD, we’ll create a manifest that installs via Helm.

Create file: coroot/operator.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-operator
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-operator
 targetRevision: 0.4.2
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

Note: I’m using version 0.4.2, which was current at the time of writing.
To check available versions, use this GitHub link or Helm CLI:

helm repo add coroot https://coroot.github.io/helm-charts
helm repo update
helm search repo coroot-operator --versions

Create Manifest for Installing Coroot Community Edition

Create file: coroot/coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-ce
 targetRevision: 0.3.1
 helm:
 parameters:
 - name: clickhouse.shards
 value: "2"
 - name: clickhouse.replicas
 value: "2"
 - name: service.type
 value: LoadBalancer
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

This chart creates a minimal Coroot Custom Resource.
I’ve added service.type: LoadBalancer to expose a public IP.

If you don’t use LoadBalancer, you’ll need to forward the Coroot port after installation:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Create ArgoCD Application Manifest

Since we manage our infrastructure via a GitHub repository, we need an ArgoCD Application that tracks changes in the coroot/ folder.

Create file: apps/argocd-coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-sync-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: coroot
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This lightweight app will monitor the folder and apply updates automatically if a change is detected (e.g. chart version bump).

Define Chart Installation Order

We have two charts: operator.yaml and coroot.yaml, and the operator must be installed first.

Create coroot/kustomization.yaml to specify resource order:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - operator.yaml
 - coroot.yaml

Publish Manifests to GitHub

Check which files were changed:

git status

Add changes:

git add .

Verify staged files:

git status

Commit:

git commit -m "Installing Coroot Operator and Coroot with ArgoCD"

Push:

git push origin main

Apply ArgoCD Application

Deploy the ArgoCD app that installs Coroot from our GitHub repository:

kubectl apply -f apps/argocd-coroot.yaml -n argocd

Validate installation and sync:

We now see coroot, coroot-operator, and coroot-sync-app deployed.

Access Coroot UI

Since we deployed Coroot using LoadBalancer, retrieve its external IP:

kubectl get svc -n coroot

Open EXTERNAL-IP on port 8080.
For example: http://35.202.140.216:8080/

If you didn’t use LoadBalancer, run port-forward:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Then visit http://localhost:8080

You’ll be prompted to set an admin password on first login.

Exploring Coroot UI

On the home page, we see a list of applications running in the cluster and resource usage.

I increased the load on the PostgreSQL cluster using the Demo App to test observability.

The PostgreSQL cluster dashboard offers several tabs:

CPU
Memory
Storage
Instances
Logs
Profiling
Tracing

Coroot displays a visual map of service interactions — showing which app connects to the PostgreSQL cluster.

The Profiling tab looks excellent and intuitive. Here’s the Demo App profiling view:

I also triggered an intentional error in the demo app.
Coroot correctly displayed it in both the home view and the app details page.

I especially liked the Logs and Costs sections in the sidebar — very well implemented.

First Incident: Storage Usage in PostgreSQL Turns Yellow

While exploring Coroot and the cluster, I increased the load on the PostgreSQL cluster using the Demo App.

After a short while, I noticed that the Postgres disk was full.

I opened the cluster details and went to the Storage tab.

By default, the cr.yaml file allocates just 1Gi of disk space — which is fine for a test setup.

Let’s increase disk size the GitOps way.

Increase Storage Size

Open the file postgres/cr.yaml and locate the section:

 dataVolumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Change storage from 1Gi to 5Gi.

Note: Backup volumes (pgBackRest) are also enabled by default and set to 1Gi.

 manual:
 repoName: repo1
 options:
 - --type=full
# initialDelaySeconds: 120
 repos:
 - name: repo1
 schedules:
 full: "0 0 * * 6"
# differential: "0 1 * * 1-6"
# incremental: "0 1 * * 1-6"
 volume:
 volumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Increase this storage to 5Gi as well.

Save changes to cr.yaml, then commit and push to the GitHub repository.

ArgoCD will automatically apply the changes. Pure GitOps magic.

Check the result in Coroot — everything looks great. The disk is increased to 5Gi and the issue is resolved.

Conclusion

We’ve installed and tested a solid monitoring tool, and it really makes a difference.

Across this 4-part series, we walked through the GitOps journey step by step:

Part 1 - Created a Kubernetes cluster, installed ArgoCD, and set up a GitHub repository.
Part 2 - Deployed a PostgreSQL cluster using Percona Operator for PostgreSQL.
Part 3 - Deployed a demo app via ArgoCD using Helm.
Installed and tested Coroot, an excellent open-source observability tool.

Managed the PG cluster through GitHub and ArgoCD — scaled replicas, created users, resized volumes, configured access, and more.

Thank you for reading — I hope this series was helpful.

The project files are available in my repository https://github.com/dbazhenov/percona-argocd-pg-coroot

I’d love to hear your questions, feedback, and suggestions for improvement.

GitOps Journey: Part 3 – Deploying a Load Generator and Connecting to PostgreSQL

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:50 UTC

We’ll deploy a demo application into the Kubernetes cluster using ArgoCD to simulate load on the PostgreSQL cluster.

This is a series of articles, in previous parts we:

Part 1 - Prepared the environment and installed ArgoCD and GitHub repository.
Part 2 - Installed Percona Operator for Postgres and created a Postgres cluster.

The application is a custom Go-based service that generates traffic for PostgreSQL, MongoDB, or MySQL.

It uses a dataset of GitHub repositories and pull requests, and mimics real-world operations like fetching, creating, updating, and deleting records.
Load intensity is configurable through a browser-based control panel.

We’ll install it using Helm, tracked and deployed via ArgoCD.

Reference repository: github-stat

Create the ArgoCD Application Manifest

Create a file named argocd-demo-app.yaml in the apps/ directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: demo-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/github-stat
 targetRevision: main
 path: k8s/helm
 destination:
 server: https://kubernetes.default.svc
 namespace: demo-app
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This will install the Helm chart from
https://github.com/dbazhenov/github-stat/tree/main/k8s/helm

By default, the service is configured as LoadBalancer, making it accessible from the internet.

To switch to NodePort (if needed), override the Helm value:

source:
 helm:
 parameters:
 - name: controlPanelService.type
 value: NodePort

We’ll keep default settings in this example.

Push the Application Manifest to GitHub

Track and commit your changes:

git status

git add .

git commit -m "Installing Demo Application in ArgoCD by HELM"

git push origin main

Expected Git output:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/argocd-demo-app.yaml

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Installing Demo Application in ArgoCD by HELM"
[main 03ce175] Installing Demo Application in ArgoCD by HELM
 1 file changed, 20 insertions(+)
 create mode 100644 apps/argocd-demo-app.yaml
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 686 bytes | 686.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 6b2dc98..03ce175 main -> main
➜ percona-argocd-pg-coroot git:(main)

Apply the ArgoCD Application

Deploy the app via:

kubectl apply -f apps/argocd-demo-app.yaml -n argocd

ArgoCD will install the app and has started tracking the app’s HELM chart

Validate the Deployment

Confirm the app status in ArgoCD UI:

Check running pods:

kubectl get pods -n demo-app

Expected pods:

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n demo-app
NAME READY STATUS RESTARTS AGE
demo-app-dataset-6d886f67-j648w 1/1 Running 0 2m52s
demo-app-load-577cff97c9-d8j99 1/1 Running 0 2m52s
demo-app-valkey-74989c9bf7-gjp4x 1/1 Running 0 2m52s
demo-app-web-5b98d4c65c-xmkq9 1/1 Running 0 2m52s

demo-app-dataset - loads dataset
demo-app-load - generates traffic
demo-app-valkey - Redis-compatible DB backend
demo-app-web - UI dashboard

Open the Application Dashboard

Retrieve the external IP:

kubectl get svc -n demo-app

Find the EXTERNAL-IP of demo-app-web-service.

Sample output:

➜ percona-argocd-pg-coroot git:(main) kubectl get svc -n demo-app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-app-valkey-service ClusterIP 34.118.235.203  6379/TCP 4m59s
demo-app-web-service LoadBalancer 34.118.232.144 34.28.221.107 80:31308/TCP 4m59s

Access the app in your browser:

Navigate to the Settings tab to configure a PostgreSQL connection.

PostgreSQL Credentials Setup

Percona Operator has already (Application and system users):

Created schema and database cluster1
Created user cluster1
Stored credentials in cluster1-pguser-cluster1 secret

Extract the password:

kubectl get secret cluster1-pguser-cluster1 -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s connect to the database from the Demo application using the given user and cluster1-pgbouncer.postgres-operator.svc host

In the Connection String field enter

user=cluster1 password='[PASSWORD]' dbname=cluster1 host=cluster1-pgbouncer.postgres-operator.svc port=5432

The connection has been successfully created, this is good.

To start generating the load, we need to import the Dataset using the Import Dataset button.

Dataset Import Error: Create Schema Denied

During import, the app tries to create a schema.
By default, pgBouncer limits user privileges, preventing this action.

Percona documentation suggests enabling proxy.pgBouncer.exposeSuperusers and creating a privileged user.

We’ll handle this via GitOps. It seems cool that we’ll be doing this with tracking in Git, as these are important settings and we shouldn’t forget about them and turn them off in the future.

Define a New PostgreSQL User

We will make changes to postgres/cr.yaml that will add a new user and also enable the proxy.pgBouncer.exposeSuperusers option.

In the postgres/cr.yaml file I found the users section, uncommented and added my user data.

In postgres/cr.yaml, add:

 users:
 - name: daniil
 databases:
 - demo
 options: "SUPERUSER"
 password:
 type: ASCII
 secretName: "daniil-credentials"

Note: In production, use scoped permissions like "LOGIN CREATE CREATEDB" rather than SUPERUSER.

I also found the proxy.pgBouncer.exposeSuperusers setting and set it to true

Update pgBouncer config:

 proxy:
 pgBouncer:
 replicas: 3
 image: docker.io/percona/percona-pgbouncer:1.24.1
 exposeSuperusers: true

Commit and push:

git status

git add .

git commit -m "Postgres cluster: Creating a new user and pgBouncer.exposeSuperusers"

git push origin main

After a couple of minutes, ArgoCD will synchronize the changes and Percona Operator will create the user and change the configuration.

Connect With the New User

Get the password:

kubectl get secret daniil-credentials -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s replace Connection String in Demo application, I got the following string

user=daniil password='iKj:e[wT3*g]OF5+f' dbname=dataset host=cluster1-pgbouncer.postgres-operator.svc port=5432

Click the “Import Dataset” button and wait a few minutes until the import is in Done status in the Dataset tab.

Enable Load Generation

Activate the load generator:

Toggle Enable Load in the connection settings
Click Update Connection

Open the Load Generator Control Panel and adjust sliders and toggles as needed:

Conclusion

In this part, we:

Deployed a demo application via Helm in ArgoCD
Connected it to our PostgreSQL cluster
Managed PostgreSQL users and access via GitHub and GitOps
Imported a dataset and activated the traffic generator through the web UI

In Part 4, we’ll deploy Coroot for observability and profiling.
It’s an impressive tool for diagnosing behavior across services in the Kubernetes cluster.

GitOps Journey: Part 2 – Deploying PostgreSQL with GitOps and ArgoCD

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:30 UTC

We’re now ready to deploy PostgreSQL 17 using GitOps — with ArgoCD, GitHub, and the Percona Operator for PostgreSQL.

If you’re a DBA, developer, DevOps engineer, or engineering manager, this part focuses on GitOps in action: deploying and managing a real database cluster using declarative infrastructure.

In Part 1, we set up the Kubernetes environment and installed ArgoCD.
Now it’s time to define and launch the PostgreSQL cluster — fully versioned and synced through Git.

We’ll follow the official Percona Operator documentation and reference the GitHub repository to build out a production-grade setup.

Preparing the Environment

There are multiple ways to install the Percona Operator and create a PostgreSQL cluster.
We’ll use the simplest and most GitOps-friendly approach:

Deploy the operator using deploy/bundle.yaml
Deploy the cluster using deploy/cr.yaml

Source files:

https://github.com/percona/percona-postgresql-operator/blob/main/deploy/bundle.yaml
https://github.com/percona/percona-postgresql-operator/blob/main/deploy/cr.yaml

Project Structure

Repository structure can vary depending on your services and infrastructure scale.
For this series, we’ll use:

postgres/ → Contains all manifests related to PostgreSQL: the operator, clusters, backups
apps/ → Contains ArgoCD application manifests that track changes in the repository

You’re free to choose a different structure. Just ensure all paths are correctly referenced in ArgoCD.

Creating the Postgres Directory and Saving Manifests

You can manually download the files from GitHub or automate it via CLI:

mkdir postgres

curl -o postgres/bundle.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/bundle.yaml

curl -o postgres/cr.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/cr.yaml

You can also separately clone the operator repository and grab the necessary files from there.

Creating the ArgoCD Application Manifest

This ArgoCD application will track the postgres/ directory and automatically sync changes from GitHub.

Create the file: apps/argocd-postgres.yaml

Content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: postgres
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: postgres
 destination:
 server: https://kubernetes.default.svc
 namespace: postgres-operator
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true 

You can also create this app manually via the ArgoCD UI or CLI, but using a manifest aligns better with GitOps principles.

Double-check your:

repoURL → matches your GitHub repository
path → corresponds to your PostgreSQL manifest directory
namespace → targets the correct namespace for operator and cluster

Managing ArgoCD Sync Order with Waves

ArgoCD applies manifests based on sync-wave annotations.

The operator (bundle.yaml) should be applied first
The cluster (cr.yaml) comes second

Add these annotations:

In bundle.yaml:

metadata:
 annotations:
 argocd.argoproj.io/sync-wave: "1"

In cr.yaml:

metadata:
 name: cluster1
 annotations:
 argocd.argoproj.io/sync-wave: "5"

This ensures a stable deployment sequence.

Later in the series (e.g. when installing Coroot), we’ll use a more advanced method: defining sync order via kustomization.yaml.

Reviewing Cluster Configuration

Before applying the manifests, review and adjust your cluster settings in cr.yaml.

Key defaults:

name: cluster1 → Cluster name
postgresVersion: 17 → PostgreSQL version

To keep the cluster lightweight and test horizontal scaling later, reduce replicas to 1:

instances:
 - name: instance1
 replicas: 1

proxy:
 pgBouncer:
 replicas: 1

You can also configure resource limits, disk sizes, backups, and users in this file.

Publishing the Configuration to GitHub

Verify your repo status:

git status

Add files:

git add .

Review staged files:

git status

Expected result:

➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/
 postgres/

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
 (use "git restore --staged ..." to unstage)
 new file: apps/argocd-postgres.yaml
 new file: postgres/bundle.yaml
 new file: postgres/cr.yaml

Commit:

git commit -m "Initial configuration of a Postgres cluster using Percona Operator for Postgres and ArgoCD"

Push to GitHub:

git push origin main

Verify files are published correctly in the repository.

Applying the ArgoCD App Manifest

To initiate the deployment, apply the previously created manifest:

kubectl apply -f apps/argocd-postgres.yaml -n argocd

After a minute or two, the ArgoCD dashboard should display the synced PostgreSQL application and the deployed cluster.

Verifying Cluster Status

Check the running pods:

kubectl get pods -n postgres-operator

Expected results:

One PostgreSQL instance pod
One pgBouncer pod

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 27m
cluster1-instance1-22vd-0 4/4 Running 0 28m
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 28m
cluster1-repo-host-0 2/2 Running 0 28m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 29m

Scaling the Cluster via GitOps

Let’s test the GitOps model by updating the cluster configuration to increase replicas to 3.

Edit postgres/cr.yaml:

 instances:
 - name: instance1
 replicas: 3

 proxy:
 pgBouncer:
 replicas: 3

Save the changes and push them:

git status

git add .

git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"

git push origin main

ArgoCD will automatically detect and apply this update.

Expected results:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
 (use "git add ..." to update what will be committed)
 (use "git restore ..." to discard changes in working directory)
 modified: postgres/cr.yaml

no changes added to commit (use "git add" and/or "git commit -a")
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"
[main 6b2dc98] Postgres cluster: Horizontal scaling from 1 replica to 3
 1 file changed, 2 insertions(+), 2 deletions(-)
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 435 bytes | 435.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 81ae9e8..6b2dc98 main -> main
➜ percona-argocd-pg-coroot git:(main)

Expected results on GitHub Repo:

Confirming the Update in ArgoCD

In the ArgoCD UI, you should now see the application synced to the latest commit with the updated replica count.

Verify pods:

kubectl get pods -n postgres-operator

Expected result — 3 PostgreSQL pods and 3 pgBouncer pods.

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 38m
cluster1-instance1-22vd-0 4/4 Running 0 39m
cluster1-instance1-q2r4-0 4/4 Running 0 3m38s
cluster1-instance1-r4s2-0 4/4 Running 0 3m39s
cluster1-pgbouncer-649b7cf845-9cppx 2/2 Running 0 3m37s
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 39m
cluster1-pgbouncer-649b7cf845-tkf9z 2/2 Running 0 3m36s
cluster1-repo-host-0 2/2 Running 0 39m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 40m

What’s Next

We’ve successfully installed the Percona Operator for PostgreSQL and deployed a cluster using GitHub and ArgoCD.

We also verified GitOps functionality by scaling the cluster through Git-controlled configuration.
All changes are tracked, versioned, and declarative — a solid foundation for modern infrastructure management.

To continue experimenting:

Connect to the Cluster
Manage Users. Note: the default user does not have SUPERUSER privileges. If your app requires creating databases, you’ll need to configure appropriate roles.
Expose the Cluster. So you can connect from external clients or apps.

We’ll do exactly that in the next part — by deploying a demo application and connecting it to the database using GitOps.

GitOps Journey: Part 1 – Getting Started with ArgoCD and GitHub

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:10 UTC

Welcome to GitOps Journey — a hands-on guide to setting up infrastructure in Kubernetes using Git and automation.

GitOps has gained traction alongside Kubernetes, CI/CD, and declarative provisioning.
You’ve probably seen it mentioned in blog posts, tech talks, or conference slides — but what does it actually look like in practice?

We’ll start from scratch: prepare a cluster, deploy a PostgreSQL database, run a demo app, and set up observability — all managed via Git and GitHub using ArgoCD.

What We’ll Build

ArgoCD — syncs manifests from a GitHub repository to your cluster
PostgreSQL — a production-ready database using Percona Operator
Demo App — a real Go-based web app connected to the database
Coroot — an open-source tool for monitoring performance, logs, and service behavior

This series is for anyone new to GitOps or Kubernetes.
Each part includes clear steps, real-world YAML, and examples you can run yourself.

This is Part 1 of the GitOps Journey.
If you already have ArgoCD and a working Kubernetes cluster, you can skip ahead:

Part 2 – Deploying PostgreSQL with Percona Operator

Part 3 – Connecting a Real App to the Cluster

Part 4 – Observability with Coroot

Copilot assisted with formatting, Markdown structure, and translation.
All ideas, architecture decisions, and hands-on implementation were created by Daniil Bazhenov.

Otherwise, let’s start by preparing the cluster and setting up ArgoCD.

Creating a Kubernetes Cluster

I’ll be using Google Kubernetes Engine (GKE), but you can use AWS, DigitalOcean, or even run Minikube locally.

You’ll also need these CLI tools installed on your machine:

kubectl - The official CLI tool for Kubernetes — used to manage clusters, view resources, apply manifests, and more.
helm - A package manager for Kubernetes — lets you install complex apps using reusable charts (like PostgreSQL, monitoring tools, etc.)

I use the following command to create a cluster in GKE

gcloud container clusters create dbazhenov-demo \
 --project percona-product \
 --zone us-central1-a \
 --cluster-version 1.30 \
 --machine-type n1-standard-8 \
 --num-nodes=3

To delete the cluster:

gcloud container clusters delete dbazhenov-demo --zone us-central1-a

Note: This command doesn’t remove your LoadBalancers, so I prefer deleting them manually in Google Cloud’s web console to ensure no resources are left running post-experiment.

Here’s the resulting setup:

➜ community git:(blog_argocd_pg) ✗ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-dbazhenov-demo-default-pool-b1b48316-8nrj Ready  6m7s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-8v14 Ready  6m6s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-zg6z Ready  6m7s v1.30.12-gke.1279000
➜ community git:(blog_argocd_pg) ✗

Installing ArgoCD

We’ll begin with ArgoCD, the GitOps engine that will deploy:

PostgreSQL database cluster

A demo app to simulate real usage

Coroot for monitoring and profiling workloads

ArgoCD supports multiple deployment methods — we’ll experiment with different ones during this series.

Install ArgoCD (based on official docs):

Create namespace:

kubectl create namespace argocd

Deploy ArgoCD:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Check the pods:

kubectl get pods -n argocd

Expected output: ArgoCD components running (server, repo, redis, controllers, etc.)

➜ community git:(blog_argocd_pg) ✗ kubectl get pods -n argocd
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 57s
argocd-applicationset-controller-6d569f7895-89kgk 1/1 Running 0 64s
argocd-dex-server-5b44d67df9-p42z5 1/1 Running 0 62s
argocd-notifications-controller-5865dfbc8-gqzwt 1/1 Running 0 61s
argocd-redis-6bb7987874-99j59 1/1 Running 0 61s
argocd-repo-server-df8b9fd78-64czj 1/1 Running 0 60s
argocd-server-6d896f6785-82tf2 1/1 Running 0 59s

Access ArgoCD UI

You have two options (or more):

Port forwarding (local only)

kubectl port-forward svc/argocd-server -n argocd 8080:443

Internet-accessible LoadBalancer

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

I will use Load Balancer by executing the command above, you need to wait a few minutes to get the IP address.

Let’s get the IP address of the ArgoCD service in the EXTERNAL-IP field.

kubectl get svc argocd-server -n argocd

➜ community git:(blog_argocd_pg) ✗ kubectl get svc argocd-server -n argocd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
argocd-server LoadBalancer 34.118.234.162 34.132.39.194 80:30549/TCP,443:32146/TCP 9m51s

Access the UI in your browser using the IP.

Getting Started with ArgoCD Login

Download Argo CD CLI

Install ArgoCD CLI (see instructions).

Get the initial password:

argocd admin initial-password -n argocd

ArgoCD recommends changing it to a new secure password, which we will do.

argocd login 34.132.39.194 --insecure

Authorize using initial-password and user admin and execute the password update command

argocd account update-password

All the steps to get and update your password are below.

➜ community git:(blog_argocd_pg) ✗ argocd admin initial-password -n argocd
0mxV6IVcF3qZDR-O

 This password must be only used for first time login. We strongly recommend you update the password using `argocd account update-password`.
➜ community git:(blog_argocd_pg) ✗ argocd login 34.132.39.194 --insecure
Username: admin
Password:
'admin:login' logged in successfully
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗ argocd account update-password
*** Enter password of currently logged in user (admin):
*** Enter new password for user admin:
*** Confirm new password for user admin:
Password updated
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗

Now log into the ArgoCD web UI using admin and your new password.

Welcome to the ArgoCD interface, we don’t have any applications right now, we will install them later.

Setting Up GitHub Repo

We’ll need a GitHub repo to store infrastructure manifests. ArgoCD will sync from this repo and apply changes.

Install git and create a GitHub account
Add your SSH key to your GitHub profile. GitHub SSH settings
Create a new GitHub repository

I recommend a public repo for this educational project — no secrets will be committed, and it simplifies ArgoCD setup. Plus, it earns you some green squares on GitHub. If you go with a private repo, make sure it’s properly linked in ArgoCD.

Clone the repo:

Clone the repository using an SSH address.

git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git

Navigate to the project directory

cd percona-argocd-pg-coroot

Expected output:

➜ gitops git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git
Cloning into 'percona-argocd-pg-coroot'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
Receiving objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
➜ gitops cd percona-argocd-pg-coroot
➜ percona-argocd-pg-coroot git:(main) ls
README.md
➜ percona-argocd-pg-coroot git:(main)

Summary

We’ve prepared everything to launch our GitOps-powered infrastructure:

Kubernetes cluster
ArgoCD deployed
GitHub repo ready

In the next posts, we’ll deploy the PostgreSQL cluster, the demo app, and add Coroot monitoring.

Stay tuned!

Using replicaSetHorizons in MongoDB

Ivan Groenewold — Tue, 22 Jul 2025 00:00:00 UTC

When running MongoDB replica sets in containerized environments like Docker or Kubernetes, making nodes reachable from inside the cluster as well as from external clients can be a challenge. To solve this problem, this post is going to explain the horizons feature of Percona Server for MongoDB.

Let’s start by looking at what happens behind the scenes when you connect to a replicaset URI.

Node auto-discovery

After connecting with a replset URI, the driver discovers the list of actual members by running the db.hello() command:

mongosh "mongodb://mongo1-internal:27017/?replicaSet=rs0"
rs0 [direct: primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877b5e18a13d54b752ff25c'),
 counter: Long('6')
 },
 hosts: [ 'mongo1-internal:27017', 'mongo2-internal:27017', 'mongo3-internal:27017' ],
...

The list of hosts returned contains the name of each member as you provided it to the rs.initialize() command.

The node identity crisis

The names are resolvable inside the same network, so all is well in this case. But what happens when connecting from outside?

Typically you would be using names like mongo1-external.mydomain.com that correctly point to the external IP addresses of the members. The problem is that after the initial connection is made, the driver will perform auto-discovery and try to connect to the names as reported by db.hello(). These are not resolvable from outside.

What if we connect by IP address directly? again, the driver will get the names from the list above, try to reach those and fail after the initial connection is made:

$ mongosh mongodb://user:pass@10.30.50.155:32768/?replicaSet=rs0
Current Mongosh Log ID: 6849eb15ba228be45a69e327
Connecting to: mongodb://@10.30.50.155:32768/?replicaSet=rs0&appName=mongosh+2.5.2
MongoNetworkError: getaddrinfo ENOTFOUND mongo1-internal

Even though mongo1-internal is not part of the connection string, the driver tries to reach it. So if the replica set members advertise their internal IPs or DNS names, clients outside can’t connect unless they can resolve that same name. We could work around that, but there’s another issue: the ports.

The port issue

In the containerized world, it is likely that you set up your containers to use default port 27017. However they might be mapped to a different external port, since you have to avoid port collisions (think about the case where containers are co-located in the same host).

We need a way for replica set members to identify themselves with different names and ports, depending on whether the client is in the same network or outside. A concept similar to split-brain DNS.

What is Horizons?

Horizons is a MongoDB feature that allows replica set members to advertise different identities depending on the client’s access context, such as internal versus external networks.

With this, you can make the same MongoDB replica set usable from:

Internal container network (using internal hostnames/IPs)
External applications (using public IPs or DNS names)

MongoDB’s horizons rely on Server Name Indication (SNI) during the TLS handshake to determine which hostname and port to advertise. At connection time, clients present the hostname they used, and MongoDB uses that to return the proper set of endpoints. For that reason TLS is required in order for horizons to work.

Let’s walk through an example.

Example Scenario: MongoDB Replica Set in Docker

You can run the following steps on your local machine to test the feature.

Get your certificates ready

Let’s start by creating the required CA and certificates using Cloudflare’s PKI and TLS toolkit.

Step 1: Create ca-csr.json

mkdir certs
cd certs
tee ca-csr.json <{
 "CN": "MyTestCA",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "US", "ST": "CA", "L": "SF", "O": "Acme", "OU": "MongoDB CA" }]
}
EOF

Generate the CA:

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

This creates:

ca.pem — CA certificate
ca-key.pem — CA private key

Step 2: Create server-csr.json for each server, specifying both internal and external names in the “hosts” section so that our certificate is valid for everything.

for i in 1 2 3; do
 name="mongo$i"

 tee "${name}-csr.json" <{
 "CN": "${name}",
 "hosts": ["${name}", "${name}.internal", "localhost", "127.0.0.1"],
 "key": { "algo": "rsa", "size": 2048 },
 "names": [
 { "O": "MongoDB", "OU": "Database", "L": "Internal", "ST": "DC", "C": "US" }
 ]
}
EOF
done

Step 3: Generate certificates using CFSSL

for i in 1 2 3; do
 name="mongo$i"

 cfssl gencert \
 -ca=ca.pem -ca-key=ca-key.pem \
 -config=<(cat <<'JSON'
{
 "signing": {
 "default": {
 "expiry": "8760h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
 }
}
JSON
) "${name}-csr.json" | cfssljson -bare "${name}"
cat "${name}.pem" "${name}-key.pem" > "${name}-combined.pem"
done

cd ..

Resulting files:

mongo{1,2,3}.pem — cert for server
mongo{1,2,3}-key.pem, key for server
mongo{1,2,3}-combined.pem, both in a single file as expected by mongo

Docker Compose Setup

Create a file with docker compose configuration:

tee test-horizons.yml <name: horizons
services:
 mongo1:
 container_name: mongo1
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27017:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo1-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo2:
 container_name: mongo2
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27018:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo2-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo3:
 container_name: mongo3
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27019:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo3-combined.pem
 --tlsCAFile /certs/ca.pem
networks:
 default:
 driver: bridge
EOF

Here we are mapping our containers to ports 27017, 27018 and 27109 externally.

Now, start the services:

$ docker-compose -f test-horizons.yml up -d

Initiate the Replica Set with Horizons

Now let’s initiate the replica set with different host names and ports for external access.

Launch a shell into one of the containers:

$ docker exec -it mongo1 /bin/bash

Authenticate and initialize the replica set with this config:

$ mongosh --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsAllowInvalidCertificates
rs.initiate({
 _id: "rs0",
 members: [
 {
 _id: 0,
 host: "mongo1:27017",
 horizons: { external: "localhost:27017" }
 },
 {
 _id: 1,
 host: "mongo2:27017",
 horizons: { external: "localhost:27018" }
 },
 {
 _id: 2,
 host: "mongo3:27017",
 horizons: { external: "localhost:27019" }
 }
 ]
})

Note: The “horizon” field here maps the external context to a different address than the internal one. Since we are going to test connecting from the local machine directly to the containers, set the horizons to localhost and the mapped ports.

Connect from Inside Docker

Spin up a new containerized client, or use one of the existing MongoDB containers:

$ docker exec -it mongo1 mongosh --host rs0/mongo1:27017,mongo2:27017,mongo3:27017 --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877deab6568339f46dfd9c4
Connecting to: mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongo1-combined.pem&tlsCAFile=%2Fcerts%2Fca.pem&appName=mongosh+2.5.0
Using MongoDB: 8.0.8-3
Using Mongosh: 2.5.0

rs0 [primary] test>

It connects using internal Docker hostnames.

Connect from Outside Docker

From your local machine:

$ mongosh "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877defabc3f9a2d054a1296
Connecting to: mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=certs%2Fmongo1-combined.pem&tlsCAFile=certs%2Fca.pem&appName=mongosh+2.3.1
Using MongoDB: 8.0.8-3
Using Mongosh: 2.3.1

rs0 [primary] test>

Check the identities returned

As we have seen, MongoDB will resolve the external horizon names and connect successfully in both cases. You can verify the advertised hostnames and ports for the external connection:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'localhost:27017', 'localhost:27018', 'localhost:27019' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'localhost:27017',
 me: 'localhost:27017',
...

Versus the internal case:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'mongo1:27017', 'mongo2:27017', 'mongo3:27017' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'mongo1:27017',
 me: 'mongo1:27017',
...

Conclusion

The horizons feature in MongoDB is a powerful tool to bridge the gap between internal and external connectivity, especially in containerized or multi-network deployments.

Horizon also has following limitations:

Using horizons is only possible with TLS connections
Duplicating domain names in horizons is not allowed by MongoDB
Using IP addresses in horizons definitions is not allowed by MongoDB
Horizons should be set for all members of a replica set, or not set at all

This feature is not listed in the official MongoDB documentation for some reason, however it is available in both Percona Server for MongoDB and MongoDB Community Edition. Also, Kubernetes users rejoice! Percona Operator for MongoDB supports horizons since version 1.16.

Percona Bug Report: June 2025

Aaditya Dubey — Mon, 30 Jun 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-9823: mysql_migrate_keyring fails with PS Components.

The failure is triggered by a missing symbol, but the underlying cause is the way keyring components are built in Percona Server. When attempting to migrate keyring data (e.g., from Vault to File), the tool fails to load the Percona Server component .so files, making the migration process unusable.

Percona Server builds a reference to the my_free symbol, which is not properly resolved in the shared libraries. In contrast, upstream MySQL builds do not include this dependency.

This issue blocks both component-to-component and component-to-plugin keyring migrations, affecting users who rely on secure key management transitions.

Reported Affected Version/s: 8.0.x, 8.4.x
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9836: There is a regression issue with audit_log_filter.so compared to audit_log.so. The audit_log_filter, whether used as a plugin (8.0) or a component (8.0 and 8.4), shows a significant performance regression. When logging everything, QPS drops by over 70%. While configuring selective logging can reduce the impact, it still results in a 30–35% drop in QPS.

For this reason, moving to audit_log_filter in 8.0 is not recommended. Additionally, this should be taken into account when planning upgrades to 8.4, as audit logging can significantly impact performance. (audit_log is not available as a component—only as a plugin.)

Reported Affected Version/s: 8.0.42-33, 8.4.5-5
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9837: A crash occurs on replica nodes during parallel replication when an INSERT is executed on a secondary index that recently had a DELETE on the same key. The issue is caused by a race condition in the secondary index reuse logic, leading to an assertion failure (row0ins.cc:268).

This issue is more likely to occur under heavy write workloads, particularly when the application frequently performs DELETE followed by INSERT on the same keys. It only affects replica servers where replica_parallel_workers > 0 and slave_preserve_commit_order=ON.

Reported Affected Version/s: 8.0.36-28, 8.0.42-33
Upstream Bug: 118334
Workaround/Fix: The user can modify their logic to use UPDATE instead of DELETE followed by INSERT, which avoids triggering the crash path.
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9861: The audit_log_filter plugin cannot be installed when component_keyring_kmip is enabled with Fortanix DSM. While testing with component_keyring_kmip, we enabled the “Allow secrets with unknown operations” option in Fortanix, which allowed the audit log installation to proceed one step further. At this point, a secret is successfully created for the audit log, but MySQL crashes upon restart.

This issue is related to bug PS-9609 and still persists when using Fortanix DSM as the KMIP server.

Reported Affected Version/s: 8.0.42-33
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: The issue has been fixed, and the fix is expected in the upcoming release of Percona Server (PS).

PS-9914: After running ALTER TABLE … ENGINE=InnoDB to rebuild a large table (~10 million rows) with ROW_FORMAT=COMPRESSED, it was observed approximately a 50% drop in write-only workload throughput (measured via sysbench), despite a reduction in .ibd file size and no changes to table structure or indexes. The table had previously undergone heavy deletions (~50%), suggesting possible fragmentation prior to the rebuild.

Reported Affected Version/s: 8.0.37-29, 8.0.42-33
Upstream Bug: 118411
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9956: PS 8.4.4-4 with group replication crashes on Oracle Linux 9 during bootstrap or failover when the audit log filter component is enabled, but does not crash on Oracle Linux 8.

Reported Affected Version/s: 8.4.4-4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation.

Percona Xtradb Cluster

PXC-4652: PXC 8.4 crashes with a SIGSEGV in unordered_map called from rpl_gtid_owned during high activity, while PXC 8.0 under the same workload and data remains stable; the crash occurs randomly during operations like COMMIT or INSERT.

Reported Affected Version/s: 8.4.3, 8.4.4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.4.5 – Pending Release

PXC-4684: An UPDATE query that joins two tables but modifies only one—e.g., UPDATE test.t2 JOIN test.t1 USING (i) SET t2.d = t2.d+1, t1.d = t1.d;—causes an MDL BF-BF conflict on other PXC nodes, even without triggers, as both tables are included in the Table_map_log_event.

Reported Affected Version/s: 8.0.41
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.0.42 – Released | 8.4.5 – Pending Release

Percona Toolkit

PT-2418: In pt-online-schema-change 3.7.0, data was lost when executing the following SQL — the value of column col_2 was unexpectedly set to NULL:

Eg:

ALTER TABLE t RENAME COLUMN col_1 TO col_2;
MySQL version: 8.0+
pt-online-schema-change --no-version-check \
 h=127.0.0.1,u=root,p=xxx,P=xxx,D=sysbench,t=sbtest1 \
 --alter="RENAME COLUMN col_1 TO col_2" \
 --execute --statistics

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2419: pt-duplicate-key-checker Ignores DESC in Index Definitions. Users running pt-duplicate-key-checker regularly observed that a newly added composite index was being incorrectly flagged as a duplicate and removed.

Before:

KEY `idx_ts` (`ts`),
KEY `idx_ts_id` (`ts` DESC, `id`)

After:

KEY `idx_ts_id` (`ts`)

The tool appears to ignore the DESC direction in index definitions, leading to incorrect de-duplication. This behaviour may affect query plans and performance in setups relying on sort order.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2425: Case-Sensitive MariaDB Detection Causes Sync Failure in pt-table-sync. In pt-table-sync 3.7.0, a case-sensitive check for the MariaDB flavor ($vp->flavor() =~ m/maria/) fails because flavor() returns “MariaDB Server”, causing the condition to evaluate incorrectly. As a result, the tool looks for source_host and source_port in $source, while the actual keys are master_host and master_port, leading to failures or uninitialized value warnings.

The Error:

Use of uninitialized value in concatenation (.) or string at /usr/bin/pt-table-sync line 7086.

Manually updating the regex to m/maria/i resolves the issue. Similar case-sensitive checks appear elsewhere in the script and may require centralizing the MariaDB detection logic.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 3.7.1 - Not Yet Released

PT-2197: In pt-online-schema-change (version 3.7.0), attempting to run an ALTER operation results in the following error:

Use of uninitialized value in string eq at /usr/bin/pt-online-schema-change line 4321

This occurs even when replica connectivity in both directions is fully functional and multiple replicas are connected. Notably, the issue does not occur in version 3.5.1, where the operation succeeds as expected (with the expected increase in connections). Schema change automation breaks unexpectedly on newer versions despite a valid replication setup.

Reported Affected Version/s: 3.5.2, 3.6.0, 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2432: While pt-replica-find includes internal logic for handling replication channels, it currently lacks a corresponding --channel command-line option. Attempting to use it results in an error:

$ pt-replica-find --channel=foo
Unknown option: channel

This prevents users from specifying a replication channel directly, limiting the tool’s usability in multi-channel replication environments.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2448: pt-k8s-debug-collector should not collect secret details of pgbouncer

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2446: When attempting to run pt-table-checksum with Group Replication enabled, and the tool returns the following error:

Error checksumming table schema.table: DBD::mysql::st execute failed: The table does not comply with the requirements by an external plugin. [for Statement "DELETE FROM percona.checksums WHERE db = ? AND tbl = ?" with ParamValues: 0=' ', 1=' '] at /bin/pt-table-checksum line 11323.

It gets suspected that this is caused by the tool attempting to set @@binlog_format := ‘STATEMENT’, which is not supported under Group Replication.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM [Percona Monitoring and Management]

PMM-13994: pmm_agent shows disconnected status despite active metrics collection, After a temporary connectivity issue, pmm_agent continues to display a Disconnected status in pmm-admin list, even though connectivity has been restored and dashboards are populating correctly.

$ pmm-admin list
...
pmm_agent Disconnected
...

Reported Affected Version/s: 2.43.2, 2.44.1, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Restarting the pmm_agent should fix the issue.
Fixed/Planned Version/s: 3.4.0 - Not Yet Released

PMM-13905: When adding both a MongoDB Cluster and a standalone MongoDB Replica Set (not part of the cluster) to the same PMM environment (e.g., “test”), the MongoDB ReplSet Summary dashboard does not allow viewing the standalone RS.

The “cluster” filter cannot be unselected, making it impossible to visualize replica sets that are not associated with a defined cluster. As a result, only RSs from the cluster are visible, while standalone RSs are excluded from the dashboard view.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Whenever possible, use separate environments when adding the cluster and standalone RS nodes in PMM (e.g., use “env1” for the cluster and “env2” for the standalone RS).
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13910: In the MongoDB Sharded Cluster Summary and Collections dashboards, several graphs fail to populate correctly. Specifically:

Top Hottest Collections by Read
Top Hottest Collections by Write

These graphs display only admin, config, and system collections, even when other collections are under heavy traffic. Additionally, the Collections dashboard shows no data across all graphs—except for the first one (Top 5 Databases By Size), which populates as expected.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13950: In both PMM 2 and PMM 3, with MySQL 5.7 and MySQL 8.0, the server_uuid is not being collected from MySQL’s global variables as expected. Despite being available via SHOW GLOBAL VARIABLES LIKE ‘server_uuid’;, the PMM agent fails to parse or capture this value.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13792: In PMM 2.44.0, the Advisor Insights incorrectly reports that journaling is not enabled for MongoDB 7.0.9-15, despite journaling being enabled by default in this version.

Attempts to explicitly enable journaling in the MongoDB config result in a startup warning:

The storage.journal.enabled option and the corresponding --journal and --nojournal command-line options have no effect in this version... Journaling is always enabled. Please remove those options from the config.

False alert may confuse users and lead to misconfiguration attempts that prevent MongoDB from starting.

Reported Affected Version/s: 2.44, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

Percona Kubernetes Operator

K8SPG-772: In the Percona PostgreSQL Operator, a runtime panic occurs when CompletedAt is nil and not properly checked before dereferencing, leading to a segmentation fault:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0]

Stack trace:

github.com/percona/percona-postgresql-operator/percona/watcher.getLatestBackup
 .../wal.go:123

github.com/percona/percona-postgresql-operator/percona/watcher.WatchCommitTimestamps
 .../wal.go:65

The CompletedAt field is not validated before being accessed in getLatestBackup(), which causes a crash during WAL watcher execution.

This panic can crash the operator’s goroutine, interrupting WAL monitoring and potentially affecting backup or failover logic.

Reported Affected Version/s: 2.5.0, 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.7.0 - Pending Release

K8SPG-792: The upstream operator includes functionality that allows cluster or operator administrators to define default PostgreSQL images for each major version using environment variables. This enables users to create clusters without explicitly specifying spec.image, as the operator will automatically apply the predefined image.

However, a recently introduced Patroni version check does not align with this behavior. It introduces a hardcoded dependency on spec.image, effectively bypassing the default image mechanism and undermining the intended feature.

Reported Affected Version/s: 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: A possible workaround exists by manually setting the Patroni version through annotations, but this is not ideal and diminishes the convenience and flexibility originally provided.
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

K8SPXC-1651: While testing the Pod Scheduling Policy feature in Everest, we encountered a situation where a PXC database pod remained in the Pending state. This occurred because Kubernetes was unable to schedule the pod on any available node due to an affinity configuration mismatch.

However, even after updating the affinity rules in the PerconaXtraDBCluster object, the new configuration was not propagated to the pod, and it remained in the Pending state.

The fact that the Pod remains stuck in Pending even after affinity is changed or removed — and only a manual kubectl delete pod resolves it — indicates that the operator fails to reconcile affinity changes properly.

Note: This issue affects other operators as well, not just PXC.

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

K8SPXC-1648: The PVC size is rounded up to the nearest whole GiB value (e.g., 1.2Gi becomes 2Gi). When a storage resize operation is triggered, the operator deletes the existing StatefulSet (STS) and recreates it with the new requested PVC size.

However, if the new requested size rounds up to the same value as the original, the operator does not recreate the STS. Instead, it attempts to update the existing STS, which leads to the following error:

Note: This issue affects other operators as well, not just PXC.

Error: failed to deploy pxc: updatePod for pxc: failed to create or update sts:
update error: StatefulSet.apps "minimal-cluster-pxc" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals',
'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

PBM [Percona Backup for MongoDB]

PBM-1499: Restore to Missing Backup Fails with Unclear Error in Restore Custom Resource Status.
When attempting to restore from a backup that does not exist in the main storage, the operator logs correctly report the failure:

define base backup: get backup metadata from storage: get from store: no such file

However, the status.error field in the PerconaServerMongoDBRestore custom resource only shows a generic message:

error: 'define base backup: %v'

This results in a misleading or unclear error message being surfaced to the user through the custom resource, even though the logs contain the full and accurate description of the issue.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1502: In PBM 2.9.0, running pbm profile sync fails with the error:

Error:  or --all must be provided

This occurs even when a valid profile name is given, such as:

pbm profile sync azure-blob

The issue affects all defined profiles (azure-blob, gcp-cs, minio) and prevents syncing individual profiles. This appears to be a bug where the CLI fails to recognize the passed argument.

Reported Affected Version/s: 2.9.0
Upstream Bug: Not Applicable
Workaround/Fix: Use pbm profile sync --all instead
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1538: Backup is marked as successful, despite the oplog not being uploaded.

Reported Affected Version/s: 2.4.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1551: In a single-node PSMDB replica set with one PBM agent, PBM occasionally re-executes the last issued command, causing errors like:

active lock is present

This typically occurs when the database is under load.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1553: When restoring a 33-shard physical backup from a mixed (MongoDB Enterprise + Percona) production cluster into a Percona-only test cluster, PBM intermittently fails during the “clean-up and reset replicaset config” stage. Some shards restore successfully, while others restore only partially or fail entirely.

Both clusters run MongoDB 6, with FCV set to 5.
Restore uses --replset-remapping due to different replica set names
Issue affects restores regardless of matching node count per shard.

The problem appears tied to the restore logic handling replica set configuration cleanup.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1564: A user environment experiences repeated failures during incremental backups with the error:

[ERROR: cannot use the configured storage: source backup is stored on a different storage]

Full, base incremental, and logical backups succeed without issues. There is no indication of recent storage or configuration changes, and pbm status shows backup attempts occur close together.

The issue temporarily resolves after running pbm config --force-resync, suggesting a possible bug in storage metadata syncing or internal state handling.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

What's new in PMM 3.2.0: Five major improvements you need to know

Catalina Adam — Tue, 03 Jun 2025 00:00:00 UTC

PMM 3.2.0 brings some long-awaited fixes and new capabilities. You can now install PMM Client on Amazon Linux 2023 with proper RPM packages, get complete MySQL 8.4 replication monitoring, and track MongoDB backups directly in PMM.

Here’s what’s most important in this release:

1. Native Amazon Linux 2023 support - no more workarounds

What’s new: If you’ve been running PMM Client on AL2023 and dealing with complex manual installations, those days are over. You can now install PMM Client through native RPM packages just like any other supported platform.

What this means for you: Streamlined setup means you can get your Amazon Linux 2023 environments monitored faster.

2. Complete MySQL 8.4 replication monitoring

What’s new: PMM now fully supports replication monitoring for MySQL 8.4, including key metrics like IO Thread status, SQL Thread status, and Replication Lag. MySQL 8.4 changed how these metrics are exposed, and earlier PMM versions couldn’t track them accurately.

What this means for you: With the upgraded MySQL Exporter (v0.17.2), you now get complete replication monitoring across all supported MySQL versions (5.7, 8.0, and 8.4) without any visibility gaps.

3. MongoDB backup monitoring dashboard

What’s new: The new PBM Details dashboard lets you monitor MongoDB backups directly in PMM using the PBM collector. Instead of switching between PMM and separate backup tools, you now get a real-time, unified view of backup activity across replica sets and sharded clusters.

What this means for you: Easily track backup status, configuration, size, duration, PITR status, and recent successful backups—all in one place. No more tool-hopping to stay on top of your backup operations.

4. Grafana 11.6 upgrade with enhanced capabilities

What’s new: PMM now ships with Grafana 11.6, delivering enhanced visualization capabilities and improved alerting workflows.

Key features include:

Alert state history for reviewing historical changes in alert statuses
Improved panel features and visualization actions
Simplified alert creation with better UI workflows
Recording rules for creating pre-computed metrics
Navigation bookmarks for quick dashboard access

What this means for you: These enhancements make your monitoring dashboards more interactive, your alerting more sophisticated, and your overall monitoring workflow more efficient.

5. Dramatically improved Query Analytics performance

What’s new: We’ve optimized QAN filter loading performance to reduce the number of processed rows by up to 95% in large environments.

What this means for you: Filters on the PMM Query Analytics page now load much faster, making the interface more responsive to improve your troubleshooting efficiency.

Additional improvements worth noting

Beyond these five major enhancements, PMM 3.2.0 also introduces:

Secure ClickHouse connections with authenticated credential support
MongoDB Feature Compatibility Version (FCV) panels for better cluster version visibility
Nomad integration laying groundwork for future extensibility
Numerous bug fixes improving stability across ProxySQL, PostgreSQL, and MySQL monitoring

Getting started with PMM 3.2.0

Ready to experience these improvements? Set up your PMM 3.2.0 instance using our quickstart guide or upgrade your existing installation following our migration documentation.

For existing users with external PostgreSQL databases, make sure to review the external PostgreSQL configuration migration guide before upgrading.

Questions or feedback? We’d love to hear from you! Connect with the Percona community through our forums or join the conversation on our community channels.

PostgreSQL 18 - Top Enterprise Features (fast read)

Jan Wieremjewicz — Mon, 26 May 2025 00:00:00 UTC

So the Beta1 is available for PostgreSQL 18 and while not all the features have to make it to GA, we can surely hope they do!

Taking a close look at what’s coming, here below is the selection of what excites me in particular:

1. OAuth 2.0 authentication support

→ Finally aligns with modern enterprise SSO and identity standards (e.g., Okta, Azure AD). A major win for security teams and regulatory compliance.

2. Logical replication from standbys now with conflict logging

→ Now you can replicate from replicas not only primary nodes and thanks to conflict logging troubleshooting issues moves closer to what the users have been asking for. It’s a big step toward robust, native, HA-friendly logical replication. Not yet there, but on the right path!

3. Asynchronous I/O (AIO)

→ Modern async reads improve performance, especially under heavy parallel workloads. Foundation for future IO improvements, and also a feature that scratches an itch for a lot of cloud deployments.

4. Faster & safer major upgrades

→ pg_upgrade enhancements like parallel upgrade checks (--jobs), safer upgrades (--swap), and planner stats carried forward = faster version adoption and smoother upgrades for large clusters.

5. Observability++

→ Enhanced EXPLAIN statement and pg_stat_io improvements enhance understanding and optimize I/O behavior across tables, indexes, and WAL. This reduces the need for external monitoring tools.

Other Notable Features

While these are the top mentions, the goodies are not limited to only these. Some smaller improvements are just as exciting. Looking at these, the top interesting ones for me are:

New extension_control_path Server Variable

→ Enables operators to manage PostgreSQL extensions via Kubernetes image volumes (Kubernetes 1.33 image volumes) without modifying the base image.

→ Huge win for immutable image strategies and GitOps-friendly operator design.
Easier online constraints management

→ Add new NOT NULL constraints without locking large tables using NOT VALID

→ Use NOT ENFORCED foreign keys and CHECK to model relationships without runtime overhead
Smarter index maintenance (bottom-up deletion)

→ Reduces bloat, lowers vacuum overhead.
SQL/JSON path support + JSON performance gains

→ Enables document-style querying at scale.
Better insights into queries and vacuum

→ EXPLAIN now includes buffer usage in subplans, triggers, and functions which helps spot slow parts

→ pg_stat_all_tables now tracks how much time vacuum and autovacuum take per table
Indexing improvements

→ Parallel GIN builds help speed up full-text and vector searches key for hybrid full-text and vector search

→ B-tree skip scans make range and selective queries faster

What enterprises would want in PostgreSQL 19+

There is already a lot to like in this release, but based on what we hear from users, customers, and our own teams, here is what is still high on the list. First up and this one’s close to home is that we’d love to see the Transparent Data Encryption (TDE) patches from Percona Server for PostgreSQL make their way upstream. That would allow users to benefit from pg_tde directly in Community PostgreSQL Server. The rest of the list is a mix of long standing asks and forward looking ideas. It is a wishlist for sure, but one we hope to help make real over time:

Built-in Logical Conflict Resolution Algorithms

→ Support for conflict-handling strategies (e.g., last-write-wins, column-level rules) would simplify bidirectional replication and eliminate the need for custom frameworks, opening the door for fully open-source multi-master setups.
Logical failover orchestration

→ Seamless promotion and failover in logical topologies, with less reliance on external tooling. This would be great from the perspective of both Kubernetes deployments as well as the ease of use for HA solutions out there.
Better integration with external auth systems

→ Automatic PostgreSQL user creation based on OAUTH/LDAP roles at login, reducing operational burden for large-scale identity management and central access control.
Pluggable or columnar storage support

→ Native support or better extension hooks for OLAP and hybrid workloads, closing the gap with cloud-native alternatives like Citus or Redshift.
Sharding to provide horizontal scaling

→ Transparent sharding is a highly desired capability that becomes critical as workloads scale. While not always needed on day one, having built-in sharding means teams can grow without reinventing the wheel. Lack of it makes horizontal scaling complex, requiring more expertise and introducing higher operational overhead for DBA teams.

OS Platform End of Life (EOL) Announcement for Ubuntu 20.04 LTS

Julia Vural — Thu, 22 May 2025 00:00:00 UTC

Ubuntu 20.04 LTS (Focal Fossa) is scheduled to reach its official end of life on May 31, 2025. In alignment with the upstream vendor’s lifecycle, we are also ending platform support for Ubuntu 20.04 for all our MySQL related product offerings. This date and others are published in advance on our Percona Release Lifecycle Overview page.

As part of our support policy, Percona will continue to provide advisory support for databases running on EOL platforms, but effective April 1, 2025, we have discontinued:

Delivery of new packages or binary builds
Distribution of hotfixes or bug fixes for Percona software on Ubuntu 20.04
OS-level support for issues not related to the database itself

However, all existing packages will remain available for download.

We are committed to ensuring a smooth and seamless experience for our users. Migrating to a supported operating system will ensure that you continue to receive security updates, bug fixes, and new features for our products.

We encourage you to take action promptly and plan your migration from Ubuntu Focal to a supported operating system. Each operating system vendor has different supported migration or upgrade paths to their next major release. Please contact us if you need assistance migrating your database to a different supported OS platform – we will be happy to assist you!

Note: If requested by a customer, we may provide updated Percona packages for up to six months beyond the Percona EOL date, as a courtesy grace period.

Percona Bug Report: April 2025

Aaditya Dubey — Mon, 19 May 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-8846: The ALTER INSTANCE RELOAD TLS thread gets stuck. This happens in instances with a high new connections rate (>60/s) but not in all instances. Percona and Upstream did not fix it properly because the current implementation of ALTER INSTANCE RELOAD TLS requires all existing SSL connections to be closed. “A thread that executes ALTER INSTANCE RELOAD TLS tries to acquire an RCU Lock(Read-Copy-Update), waiting for the number of readers to become 0. In other words, when the server has a constant flow of new incoming SSL connections, the chances of acquiring this lock are pretty low.” Therefore, Percona and Oracle only partially fixed this; this fix should improve this behaviour.

Reported Affected Version/s: 8.0.32-24, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9609: The audit_log_filter can’t be installed when the server is using component_keyring_kmip

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9628: The binlog_encryption does not work with component_keyring_kmip

Reported Affected Version/s: 8.0.40-31

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9664: With a very simple workload, MyRocks allocates a lot of memory and does not free it when the workload finishes. All instrumentation available either does not provide information about memory allocated or provides only part of it. As a result, users cannot predict how much RAM to install on the server that runs the MyRocks storage engine. With InnoDB same workload requires about 1.7G and frees about 0.5G once the job is finished.

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9719: When changing binlog_transaction_dependency_tracking in high load workload, MySQL got a segmentation fault.

Reported Affected Version/s: 8.0.40, 8.0.41-32, 8.0.33

Upstream Bug: 117922

Workaround/Fix: “set global binlog_transaction_dependency_tracking = commit_order;”

Fixed/Planned Version/s: 8.0.42-33, 8.4.5-5

PS-9768: An unexpected duplicate error occurs when running a select query with a group by JSON data.

Reported Affected Version/s: 8.0.41-32, 8.4.4-4

Upstream Bug: 117927

Workaround/Fix: Rebuilding the table with “alter table db_name.table_name = rocksdb;” can fix the issue.

Fixed/Planned Version/s: To Be Determined

Percona Xtradb Cluster

PXC-4512: When DDLs run against tables with foreign key references when there is a write load simultaneously. The issue is typically triggered during pt-online-schema-change execution, and after a dozen or so iterations, random PXC nodes will terminate with MDL BF-BF conflict. Sometimes, the writer fails, and sometimes, the other nodes, but it can be reproducible with just the RENAME query.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27, 8.0.36-28, 8.0.37-29, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Only a full write stop would help.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4648: After upgrading from 8.0.41 to 8.4.3, the node can’t join the group with the following error.

[ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.4.3/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():343: Group requested gcs_proto_ver: 5, max supported by this node: 4.Upgrade the node before joining this group.Need to abort.

The problem is that 8.0.41 and 8.4.4 use Galera 26.4.21. It introduced the GCS protocol version 5, 8.0.40 and 8.4.3 using Galera 26.4.20. So, the node that doesn’t understand protocol 5 tries to connect to a cluster that uses protocol 5.

It is fixed as a documented bug, which can be seen here.

Reported Affected Version/s: 8.4.3

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.4.4

PXC-4638: The binlog_utils_udf plugin fails to access binlog files correctly after an SST (State Snapshot Transfer) due to inconsistencies in the mysql-bin.index file. After an SST, the first entry in the mysql-bin.index file can be incorrectly formatted with a relative path, while subsequent entries use absolute paths. This inconsistency can prevent the binlog_utils_udf plugin from locating the correct binlog files.

The resulting mysql-bin.index content after SST might appear as:

mysql-bin.000010
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000011
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000012 

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Rewriting binlog.index and adding the “./” prefix to the first entry, and running flush binary logs resolved this issue.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-3576: Deploying a new installation using the setting lower_case_table_names=1 on the startup generates the following entry on the log:

[Warning] [MY-010324] [Server] 'db' entry 'percona_schema mysql.pxc.sst.role@localhost' had database in mixed case that has been forced to lowercase because lower_case_table_names is set. It will not be possible to remove this privilege using REVOKE.

Looking at the mysql.db, we can see the deployment was able to create the mysql.pxc.sst.role mapping an uppercase database name:

mysql> select * from mysql.db where user='mysql.pxc.sst.role';
+-----------+--------------------+--------------------+------+
| Host | Db | User | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Create_tmp_table_priv | Lock_tables_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Execute_priv | Event_priv | Trigger_priv |
+-----------+--------------------+--------------------+-------------+-------------+-------------+-------------+-------------+-----------+------------+-------------+
| localhost | PERCONA_SCHEMA | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |
| localhost | percona_schema | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |

This produces a duplicate entry for the percona_schema database and a warning message on the mysql.log.

Reported Affected Version/s: 8.0.21-12.1, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4657: When executing DML and DDL on the same table, the DML will get a deadlock error. If the DML does not change the data but matches, it won’t be replicated, for example.

mysql > UPDATE test.t SET d = d LIMIT 1;
Query OK, 0 rows affected (0.01 sec)
Rows matched: 1 Changed: 0 Warnings: 0

If the table contains a trigger, the DML does not get a deadlock error and will be replicated to other nodes.

CREATE TRIGGER `t_on_update`
AFTER UPDATE ON `t`
FOR EACH ROW
BEGIN
 INSERT INTO t_history (`d`)
 VALUES (NEW.`d`);
END

When other nodes apply the DML and DDL, the applier threads will get an MDL BF-BF conflict.

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Using a single applier thread will skip the bug, but it may introduce a performance issue, such as flow control.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4664: Converting thd->rli_slave to target type Slave_worker* causes a segmentation fault.

Reported Affected Version/s: 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Toolkit

PT-2392: pt-online-schema-change resume functionality doesn’t work with ADD INDEX

Reported Affected Version/s: 3.6.0, 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2322: The issue reports that pt-mysql-summary does not correctly detect and display the jemalloc memory management library, even when it is enabled. Despite jemalloc being loaded and visible in the process memory map (/proc//maps), the output from pt-mysql-summary is missing this information in some cases, unlike version 3.2.1, which correctly identifies and reports it.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2422: When using the –history option with pt-online-schema-change (pt-osc), the query responsible for updating the history entry with the new_table_name is not appropriately constrained by a primary or unique key. As a result, this UPDATE operation can inadvertently modify all entries in the history table, rather than just the intended row.

This behavior can lead to significant issues when running multiple schema change operations in parallel, as the history entries for different migrations may interfere with each other, causing data consistency problems.

Additionally, if a migration is paused and later resumed, this lack of key constraint can result in only a subset of the data being correctly copied to the new table, potentially leading to partial data loss or corruption when the final table swap occurs.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2442: Multiple security vulnerabilities have been identified in the latest version of Percona Toolkit, including:

CVE-2024-56171: Use-After-Free Vulnerability in libxml2

CVE-2024-12797: OpenSSL Raw Public Key Authentication Vulnerability

CVE-2022-37967: Windows Kerberos Elevation of Privilege Vulnerability

CVE-2025-24928: Stack-Based Buffer Overflow in libxml2

It is recommended that the associated advisories be reviewed and the necessary patches or upgrades be applied to mitigate the risk.

Reported Affected Version/s: 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.0-1

PMM (Percona Monitoring and Management)

PMM-13694: When using a non-default pg_stat_statements.max value, the calculated QPS displayed in QAN may be wrong.

Reported Affected Version/s: 2.38.0, 2.44.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13807: pmm-agent crashed at query.Fingerprint due to query including a column named “value”

Reported Affected Version/s: 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13847: PMM 3.0 doesn’t support running on a different uid/gid in Kubernetes

Reported Affected Version/s: 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-13984: Percona Monitoring and Management (PMM) version 3.1 with OVA image currently cannot be imported into VMware environments.

Reported Affected Version/s: 3.1.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-12784: This error invalid GetActionRequest.ActionId: value length must be at least 1 runes. occurs in the QAN (Query Analytics) dashboard, indicating that the ActionId field in the GetActionRequest is empty or improperly formatted. This typically results from a missing or incorrectly populated Action ID parameter, which is required for retrieving query data.

Reported Affected Version/s: 2.33.0, 2.41.0, 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

Percona XtraBackup

PXB-3421: XtraBackup fails when the –databases parameter contains a very long list of databases or has a large amount of whitespace before the actual database names. When the –databases parameter is provided with 1859 whitespace characters before a table name (e.g., db01.t1), XtraBackup crashes with a signal error. If the number of whitespace characters is reduced to 1858 or fewer, the backup proceeds successfully without error.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PXB-3426: Using KMIP component causes double free of memory on error paths.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.35-33, 8.4.0-3

PXB-3392: xtrabackup doesn’t pick up –innodb-log-group-home-dir config parameter

Reported Affected Version/s: 8.0.35-30, 8.0.35-31, 8.0.35-32

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Kubernetes Operator

K8SPG-703: When using ttlSecondsAfterFinished, there is a potential race condition where the backup jobs may be deleted before the Percona operator has had sufficient time to reconcile the perconapgbackups objects. This issue can occur even with relatively long timeouts like 1m, 5m, or 30m, not just extremely short intervals.

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.7.0

K8SPSMDB-1263: While creating a 1.3TB logical backup, the replica’s state changes to “errored” after approximately 16 hours with the message:

“failed to find CERTIFICATE”

despite the backup continuing to run and eventually completing. This raises concerns about the validity of the backup and the ability to restore from it without manually altering its status to “success.”

Reported Affected Version/s: 1.13.0, 1.14.0, 1.15.0, 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1292: When spec.tls.mode is set to requireTLS, physical backup restores fail with a “server selection timeout” error. This occurs because the operator cannot establish a secure connection to the MongoDB server, resulting in closed socket errors and the inability to disable Point-in-Time Recovery (PiTR).

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.21.0

K8SPSMDB-1294: When using MCS on GKE 1.30, an API version mismatch occurs, resulting in the error:

“no matches for kind ‘ServiceImport’ in version ’net.gke.io/v1alpha1’”

This indicates that the expected ServiceImport kind is not available in the specified API version, preventing proper service discovery.

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1336: Restoring a backup into a new Kubernetes cluster can lead to “Time monotonicity violation” errors on config servers and mongos, causing the pods to restart. This occurs when the restored chunk version timestamps are earlier than the expected timestamps in the new cluster, resulting in tripwire assertions and persistent crashes.

Reported Affected Version/s: 1.19.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

K8SPXC-1548: Failed to delete old backups on Google Cloud Storage

Reported Affected Version/s: 1.14.0, 1.15.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.18.0

PBM (Percona Backup for MongoDB)

PBM-1482: Selective Restore with replset-remapping hangs on oplog replay and doesn’t finish

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1487: Error Location6493100 on mongos after successful logical restore or PITR

Reported Affected Version/s: 2.8.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1531: PBM Restore getting randomly stuck

Reported Affected Version/s: 2.6.0, 2.7.0, 2.8.0, 2.9.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Setting Up and Monitoring MongoDB 8 Replica Sets with PMM 3 Using Docker: A Beginner-Friendly Guide

Daniil Bazhenov — Tue, 18 Mar 2025 00:00:00 UTC

This guide explains how to set up a MongoDB 8 Replica Set and monitor it using PMM 3, all within Docker. We’ll guide you through the steps to create a local environment, configure the necessary components, and connect them for effective monitoring and management.

The guide is written in detail for beginners. In the conclusion section there are ready configurations for the experienced.

The recent release of Percona Monitoring and Management 3 introduces several new features:

Upgraded Grafana version for an improved user experience.
Rootless containers for enhanced security.
ARM support for the pmm-client.
Monitoring capabilities for MongoDB 8, along with new dashboards.

This article is intended for developers and DBAs who want to experiment with these tools locally using Docker. We will cover the following steps to set everything up and test the functionality:

Launch the PMM 3 pmm-server for monitoring and open it in a browser.
Install MongoDB, starting with a standalone server, and then convert it into a Replica Set with three nodes. Percona Server for MongoDB images are used in this article.
Configure the pmm-client for MongoDB to send metrics to the pmm-server.
Explore the PMM 3 dashboards.

We will use Docker Compose to define and run multiple containers efficiently through a single docker-compose.yaml file.

If you’re ready to dive into the world of Dockerized database monitoring and management, let’s get started!

Step Zero: Preparation

To get started, you need a terminal to run Docker commands and a text editor to modify the docker-compose.yaml file.

You also need Docker installed on your system. If Docker is not installed, follow these instructions to set it up:

Docker Desktop: This application includes both Docker and Docker Compose. It is available for multiple operating systems. This guide uses Docker Desktop on macOS with an Apple Silicon ARM processor. Download Docker Desktop
Docker and Docker Compose (separately): If preferred, install Docker and Docker Compose individually. Use the following links for guidance:
- Download Docker for your OS
- Download Docker Compose

Command to Verify Installation:

Verify Docker Compose Installation: If you’re using Docker Desktop, Docker Compose is included. If you installed Docker Compose separately, you can verify the installation with:

docker-compose --version

This should return the version of Docker Compose installed.

sh
➜ community git:(main) ✗ docker-compose --version
Docker Compose version v2.21.0-desktop.1

Once Docker and Docker Compose are installed and verified, you are ready to move on to the next steps of deploying PMM 3 and MongoDB in Docker.

Project Directory

Create a directory where we’ll store the configuration and necessary files. For example, I created a directory named pmm-mongodb-setup and navigated into it.

bash
mkdir pmm-mongodb-setup
cd pmm-mongodb-setup

Step One: Starting PMM 3 Using Docker Compose

To start with, we will launch PMM 3, specifically the pmm-server, which we will later access via a browser.

Create the Docker Compose Configuration File

First, create a file named docker-compose.yaml in your project directory. Then, copy and paste the following configuration into the file to set up PMM 3:

yaml
version: '3'

services:
 pmm-server:
 image: percona/pmm-server:3
 platform: "linux/amd64" # Specifies that Docker should use the image for the amd64 architecture, which is necessary if the container doesn't support ARM and your host system is ARM (e.g., Mac with Apple Silicon).
 container_name: pmm-server
 ports:
 - 8080:80
 - 443:8443
 healthcheck: # Defines a command to check the container's health and sets the timing for executions and retries.
 test: ["CMD-SHELL", "curl -k -f -L https://pmm-server:8443 > /dev/null 2>&1 || exit 1"]
 interval: 30s
 timeout: 10s
 retries: 5

Explanation:

We define the first pmm-server service to use the percona/pmm-server:3 image

platform: This parameter ensures compatibility with ARM-based processors, such as my Mac with Apple Silicon, by instructing Docker to use the image for the amd64 architecture.

healthcheck: This parameter performs a check to confirm that the container has started successfully.

Start the PMM 3 Container

Save the docker-compose.yaml file, and then use the following command to start the PMM 3 container:

docker-compose up -d

This command will download the PMM 3 image if it is not already available locally and start the container in detached mode.

Expected result in the terminal:

shell
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 2/2
 ✔ Network pmm-mongodb-setup_default Created 0.1s
 ✔ Container pmm-server Started 0.9s
➜ pmm-mongodb-setup

Expected result in Docker Desktop:

Open PMM in a browser

Now you can open PMM in your browser at http://localhost. Use admin/admin as the username and password to log in. When prompted to change the password, skip this step by clicking the Skip button. Since this is a test setup, we need a simple password for other containers.

Now, PMM 3 is successfully running using Docker Compose.

Step Two: Starting MongoDB

We start by launching a standalone MongoDB service. This simple configuration allows us to quickly understand how it operates before moving on to a more advanced setup with a Replica Set.

To keep the database data persistent, even after a restart, a volume is used for MongoDB data storage. Add the following configuration to the docker-compose.yaml file under the pmm-server service:

yaml
 mongodb:
 image: "percona/percona-server-mongodb:8.0-multi"
 volumes:
 - mongodb-data:/data/db
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 healthcheck:
 test: ["CMD-SHELL", "mongosh --eval 'db.adminCommand(\"ping\")' --quiet"]
 interval: 30s
 timeout: 10s
 retries: 5

volumes:
 mongodb-data: # MongoDB data storage volume

Explanation:

environment: Configures the root user’s credentials (username and password).

command: Sets additional parameters for MongoDB:

bind_ip_all: Allows external access to the database, for instance, through MongoDB Compass.

profile: Enables profiling settings to support Query Analytics (QAN) in PMM.

healthcheck: Ensures the container starts successfully by executing a health check command.

volumes:: Creates the specified volume for MongoDB data storage. Defines a Docker volume to store MongoDB data persistently.

Launching the Configuration

Save the updated docker-compose.yaml file and launch the MongoDB service by running the following command:

docker-compose up -d

Docker Compose checks the configuration and starts the MongoDB container.

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Volume "pmm-mongodb-setup_mongodb-data" Created 0.0s
 ✔ Container pmm-mongodb-setup-mongodb-1 Started 0.1s
 ✔ Container pmm-server Running 0.0s
➜ pmm-mongodb-setup

Verifying MongoDB in Docker Desktop:

Step Three: PMM Client

At this point, we have both the PMM Server, which is accessible in the browser, and MongoDB running. To transfer metrics from MongoDB to the PMM Server, a container with pmm-client needs to be started.

Add another service pmm-client to the docker-compose.yaml file, right after mongodb. Note that `volumes:`` must remain at the bottom of the file.

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --username=databaseAdmin --password=password --host=mongodb --port=27017 --query-source=profiler

Explanation:

depends_on: Ensures that pmm-client starts only after pmm-server and mongodb have started successfully and passed the healthcheck.

PMM_AGENT_PRERUN_SCRIPT: The pmm-admin add mongodb command adds the MongoDB service to PMM for monitoring.

PMM_AGENT_SERVER_ADDRESS: Specifies the PMM Server address and uses port 8443.

PMM_AGENT_SERVER_USERNAME and PMM_AGENT_SERVER_PASSWORD: Update these values if you have changed the PMM login credentials.

Applying the Updated Configuration

Run the following command to apply the updated docker-compose.yaml configuration and start the pmm-client service:

docker-compose up -d

After running the command, you should see the pmm-client container start successfully:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Container pmm-mongodb-setup-mongodb-1 Healthy 0.0s
 ✔ Container pmm-server Healthy 0.0s
 ✔ Container pmm-client Started 0.1s
➜ pmm-mongodb-setup

Open PMM in your browser. On the homepage, you should now see MongoDB listed as a monitored service:

Step Four: Convert a Standalone MongoDB to a Replica Set

A single MongoDB instance works well for development and testing. At this point, you can already connect to the database from your application or tools such as MongoDB Compass and run various NoSQL queries.

However, the goal is to deploy a Replica Set consisting of three replicas, which is recommended for production and operational setups. For now, we set up the Replica Set on a single machine with a single docker-compose, which is intended for testing and development purposes only.

Both the mongodb and pmm-client services need to be updated.

Stopping All Services

First, stop all the currently running services:

docker-compose down

Result:

sh
➜ pmm-mongodb-setup docker-compose down
[+] Running 4/4
 ✔ Container pmm-client Removed 0.3s
 ✔ Container pmm-mongodb-setup-mongodb-1 Removed 0.4s
 ✔ Container pmm-server Removed 4.5s
 ✔ Network pmm-mongodb-setup_default Removed

Generating a Key File

To run three MongoDB replicas that can securely communicate with each other, a key file is required.

Create a secrets folder next to the docker-compose.yaml file and generate the mongodb-keyfile:

sh
mkdir secrets
openssl rand -base64 128 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile

In this case, the mongodb-keyfile will be generated inside the secrets folder. If your operating system does not support these commands, manually create the secrets folder and add a mongodb-keyfile with the following content:

rVLhIK2PhZKGxysjwMR4t1OmNppqdAzEs408hrbzg95D146mn9YENixId6pvIGCA
Cy9hc1k6OKKabbv7Rm347NwSFxbdPPx0/jnaO80U/a6/mv0XqSmEl8wdR91b4jIm
d98LobplwRs4b7g9cnLMUAIULr0WG+J36NtKIA6q4eE=

Add three mongodb services for Replica Set

Remove the existing mongodb service from the docker-compose.yaml file and replace it with three Replica Set services:

yaml
 mongodb-rs101:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs101
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-101:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs102:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs102
 ports:
 - "28017:28017"
 command: ["mongod", "--port", "28017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-102:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 28017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs103:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs103
 ports:
 - "29017:29017"
 command: ["mongod", "--port", "29017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-103:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 29017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

Key Points:

./secrets:/etc/secrets:ro: Mounts the key file from your disk into the container.

ports:: Each replica runs on a separate port since they are hosted on the same machine.

command::

--keyFile: Enables replication and uses the key file for secure communication between replicas.

--replSet: Defines a Replica set parameter named rs.

healthcheck:: Ensures that pmm-client starts only after the Replica Set is initialized.

Adding Volumes

Define three separate volumes for data storage:

yaml
volumes:
 mongodb-data-101:
 mongodb-data-102:
 mongodb-data-103:

Initializing the Replica Set

Add another service to initialize the Replica Set (after mongodb-rs103):

yaml
 mongodb-rs-init:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: rs-init
 depends_on:
 - mongodb-rs101
 - mongodb-rs102
 - mongodb-rs103
 entrypoint: [
 "sh", "-c",
 "until mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'print(\"waited for connection\")'; do sleep 5; done && \
 mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'config={\"_id\":\"rs\",\"members\":[{\"_id\":0,\"host\":\"mongodb-rs101:27017\"},{\"_id\":1,\"host\":\"mongodb-rs102:28017\"},{\"_id\":2,\"host\":\"mongodb-rs103:29017\"}],\"settings\":{\"keyFile\":\"/etc/secrets/mongodb-keyfile\"}};rs.initiate(config);'"
 ]
 volumes:
 - ./secrets:/etc/secrets:ro

This service connects to one of the replicas and initializes the Replica Set configuration.

Updating pmm-client

Finally, modify the pmm-client service to register all three Replica Set nodes:

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb-rs101:
 condition: service_healthy
 mongodb-rs102:
 condition: service_healthy
 mongodb-rs103:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --service-name=mongodb-rs101 --username=databaseAdmin --password=password --host=mongodb-rs101 --port=27017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs102 --username=databaseAdmin --password=password --host=mongodb-rs102 --port=28017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs103 --username=databaseAdmin --password=password --host=mongodb-rs103 --port=29017 --query-source=profiler

Explanation:

depends_on: Ensures pmm-client waits until all replicas and pmm-server are initialized.

PMM_AGENT_PRERUN_SCRIPT: Adds all three replicas to PMM monitoring with the pmm-admin add mongodb command.

Launching the Configuration

Start all services with:

docker-compose up -d

Expected Output:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 10/10
 ✔ Network pmm-mongodb-setup_default Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-101" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-102" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-103" Created 0.0s
 ✔ Container mongodb-rs103 Healthy 0.1s
 ✔ Container pmm-server Healthy 1.1s
 ✔ Container mongodb-rs101 Healthy 0.1s
 ✔ Container mongodb-rs102 Healthy 0.1s
 ✔ Container rs-init Started 0.1s
 ✔ Container pmm-client Started 0.0s
➜ pmm-mongodb-setup

Expected Output in Docker Desktop:

Verifying in PMM

Open PMM and explore dashboards such as the MongoDB Replica Set Summary, which displays information about your replicas and various metrics:

For example, I experimented by restarting one of the MongoDB services in Docker Desktop. The Replica Set switched the Primary replica, and when I simulated a failure by stopping a service, the monitoring dashboard reflected this event:

Connecting to MongoDB and Useful Commands

Connecting to MongoDB

There are several ways to connect to MongoDB depending on your setup and tools:

Using Docker Desktop

If you are using Docker Desktop, you can select a container and open the Exec tab. This opens a terminal within the container.

From there, you can connect to MongoDB using the mongosh shell with the following command:
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Using Docker CLI

If you’re running Docker without Docker Desktop, you can connect to the container using the following command:
bash
```
docker exec -it  bash
```
when you connect to the container, connect to MongoDB
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Connecting from Applications or Tools

If you are connecting through an application or a tool like MongoDB Compass, you can use connection strings tailored to your setup:

Here are the MongoDB connection strings tailored for your use cases:
1. Standalone MongoDB: Connects directly to a single MongoDB instance.
  sh
  mongodb://databaseAdmin:password@localhost:27017
2. Primary Node: Forces a direct connection to the primary node in the Replica Set using the directConnection=true option.
  sh
  mongodb://databaseAdmin:password@localhost:27017/?directConnection=true
3. Full Replica Set: Lists all Replica Set members and enables automatic failover using replicaSet=rs
  sh
  mongodb://databaseAdmin:password@localhost:27017,localhost:28017,localhost:29017/?replicaSet=rs

Useful Commands

Here are some helpful commands for managing and troubleshooting your MongoDB setup:

Check the status of the Replica Set. After connecting to MongoDB, use the following command to retrieve the status of the Replica Set:
sh
```
rs.status()
```
Check MongoDB runtime configuration. Use this command to view the configuration options for the running MongoDB instance:
sh
```
db.adminCommand({ getCmdLineOpts: 1 })
```

Conclusion

In this article, we explored deploying MongoDB using Docker and Docker Compose. We covered both a Single Instance MongoDB for simple setups and a MongoDB Replica Set for high availability, while integrating them with Percona Monitoring and Management (PMM) for monitoring.

Here are the final configurations you can download:

Single Instance MongoDB + PMM 3: Link
Replica Set MongoDB + PMM 3: Link

Thank you for reading! I look forward to your comments and questions.

Join Us Online: Stream About Percona Toolkit for MySQL!

Sveta Smirnova — Fri, 14 Mar 2025 00:00:00 UTC

Are you passionate about databases and want to stay on top of the latest advancements in MySQL and Percona Toolkit? You’re in luck! We are excited to invite you to our upcoming online stream, where we’ll delve into some exciting changes and updates.

Date: March 27, 2025
Time: 13:30 GMT / 9:30 ET
Streaming Live on: LinkedIn and YouTube

About the Event

Our featured speaker, Sveta Smirnova, Principal Support Engineering Coordinator at Percona, will share her insights on the recent updates in MySQL, focusing on the removal of offensive replication statements like START/STOP SLAVE. As the maintainer of the Percona Toolkit, Sveta had to rewrite numerous tools and libraries to accommodate these changes, resulting in significant updates to 511 files.

This event is perfect for developers, DBAs, and anyone interested in databases. Sveta, an expert in MySQL and Percona Toolkit, is also an accomplished author and a speaker at international conferences on development and databases. During the stream, she will discuss the challenges she faced while renewing legacy code, including supporting SSL and handling the deprecation of certain tools.

We invite you to join the discussion and share your experiences, questions, and insights. Engage directly with Sveta and other participants to gain a deeper understanding of the Percona Toolkit and its latest enhancements.

Discussion Topics

Introduction to Percona Toolkit:
- History and evolution of Percona Toolkit.
- Overview of the most popular tools and their uses.
Changes in MySQL:
- Overview of new and legacy syntax.
- Implications of dropped offensive replication statements.
Adapting Percona Toolkit:
- Challenges in renewing legacy code.
- Solutions implemented, including SSL support and deprecation of certain tools.
Practical Insights:
- Detailed explanation of the changes made to 511 files.
- Fine-tuning and migration strategies for Percona Toolkit users.

Why Attend?

Insightful Content: Gain valuable knowledge on the latest changes in MySQL and how they impact Percona Toolkit.
Expert Guidance: Learn directly from Sveta Smirnova, an industry expert with extensive experience in database management.
Interactive Session: Have the opportunity to ask questions live and engage directly with the speaker.

How to Join

Tune in on March 27, 2025, at 13:30 GMT / 9:30 ET, and watch the live stream on LinkedIn and YouTube. Don’t miss this chance to enhance your understanding of MySQL and Percona Toolkit while gaining practical insights from one of the best in the field.

Mark your calendars, spread the word, and get ready for an informative session! We look forward to seeing you there!

If you have any questions or need further information, feel free to reach out to us.

📅 Add to Calendar
🔗 Join the LinkedIn Stream
🔗 Join the YouTube Stream

We can’t wait to see you at the event!

Percona Monitoring and Management 3 and rootless containers

Sergey Pronin — Wed, 19 Feb 2025 00:00:00 UTC

In today’s landscape, where security breaches are a constant concern, reducing potential attack vectors is a top priority for any organization. Percona Monitoring and Management (PMM) has established itself as a reliable solution for database performance monitoring. With the release of PMM version 3, Percona has significantly strengthened its security posture, notably by introducing support for rootless container deployments. This advancement directly addresses a crucial security challenge and enhances the overall robustness and reliability of PMM.

The inherent risks associated with root privileges are well-documented. While many applications, including those containerized, have historically relied on root access, this practice presents a substantial security vulnerability. In the event of a successful exploit, an attacker gains comprehensive control over the host system. This risk is further exacerbated in environments with outdated software or complex configurations. Essentially, while the root user offers extensive capabilities, it also represents a significant potential liability that should be carefully mitigated.

In this blog post we will look at the exact differences between PMM versions 2 and 3 and how they behave.

Enforcing Pod Security Standards

The Pod Security Standards define three different policies to broadly cover the security spectrum. These policies are cumulative and range from highly-permissive to highly-restrictive.

To enforce restrictions on a specific namespace we will create the following YAML manifest:

apiVersion: v1
kind: Namespace
metadata:
 name: secure-namespace
 labels:
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/warn: restricted
 pod-security.kubernetes.io/audit: restricted

This YAML creates a namespace named secure-namespace. The pod-security.kubernetes.io/enforce: restricted label instructs Kubernetes to deny any pods in this namespace that violate the “restricted” PSS profile. The warn and audit labels are also very useful for monitoring and testing before fully enforcing the restricted policy.

Deploy PMM

We will execute a series of deployments to demonstrate the difference between PMM2 and PMM3 behavior in insecure and secure environments. All files can be found in this github repository: spron-in/blog-data/pmm3-rootless

Regular namespace

PMM2

% kubectl apply -f 01.pmm2.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 45s

I can connect to my PMM2 server with a Service that I created.

PMM3

% kubectl apply -f 02.pmm3.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 20s

I can connect to my PMM3 server with a Service that I created.

Secure namespace

Let’s try to deploy both versions of Percona Monitoring and Management servers in a secure namespace. For both we will see the following:

% kubectl apply -f 01.pmm2.yaml -n secure-namespace
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
No resources found in secure-namespace namespace.

There are no Pods created and if you describe the StatefulSet, you are going to see a similar error:

Warning FailedCreate 5s (x15 over 87s) statefulset-controller create Pod pmm-server-0 in StatefulSet pmm-server failed error: pods "pmm-server-0" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The error will be the same for PMM2 and PMM3 manifests.

Secure namespace and security contexts

We are now going to apply Pod and Container Security Contexts to both manifests.

Under spec.containers we add everything that Kubernetes suggested us:

 - name: pmm-server
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop:
 - ALL
 runAsNonRoot: true
 seccompProfile:
 type: 'RuntimeDefault'

PMM2

% kubectl -n secure-namespace apply -f 03.pmm2-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 0/1 Error 2 (15s ago) 28s

Even though the PMM2 server Pod can be created now, it is failing to start. If you check the logs, you are going to see the following:

% kubectl -n secure-namespace logs pmm-server-0
...
Error: Can't drop privilege as nonroot user
For help, use /usr/local/bin/supervisord -h

PMM3

% kubectl -n secure-namespace apply -f 04.pmm3-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 32s

PMM3 starts just fine.

Helm

The recommended approach to deploy PMM3 in Kubernetes is via Helm. You can find our helm charts in percona/helm-charts github repository and more in our documentation.

To deploy PMM3 in a namespace or environment with strict security (like OpenShift), you need to pass similar security context parameters. You can find how in 05.pmm3-helm.yaml values manifest.

Then the deployment will look like this:

helm repo update
helm install pmm3 percona/pmm -f 05.pmm3-helm.yaml --namespace secure-namespace

Conclusion

PMM 3’s rootless design excels where PMM 2 falters in secure Kubernetes environments. Enforcing Pod Security Standards, we saw PMM 3 deploy successfully, while PMM 2 failed, even with security contexts. This highlights PMM 3’s enhanced security, crucial for modern, hardened deployments. Using Helm simplifies secure PMM 3 deployments, ensuring robust database monitoring without compromising security.

Try out Percona Monitoring and Management version 3, 100% open source database observability solution, and learn more about its enhancements.

Tell us what you think in our forum or let us know if you are looking for commercial support.

How to Use IAM Roles for Service Accounts (IRSA) with Percona Operator for MongoDB on AWS

Natalia Marukovich — Mon, 17 Feb 2025 00:00:00 UTC

Introduction

Percona Operator for MongoDB is an open-source solution designed to streamline and automate database operations within Kubernetes. It allows users to effortlessly deploy and manage highly available, enterprise-grade MongoDB clusters. The operator simplifies both initial deployment and setup, as well as ongoing management tasks like backups, restores, scaling, and upgrades, ensuring seamless database lifecycle management.

When running database workloads on Amazon EKS (Elastic Kubernetes Service), backup and restore processes often require access to AWS services like S3 for storage. A key challenge is ensuring these operations have secure, least-privileged access to AWS resources without relying on static credentials. Properly managing these permissions is crucial to maintaining data integrity, security, and compliance in automated backup and restore workflows.

IAM Roles for Service Accounts (IRSA) is the recommended approach to solve this problem. IRSA allows Kubernetes pods to securely assume IAM roles, eliminating the need for hardcoded credentials, long-lived AWS keys, or excessive permissions. Instead, it leverages OpenID Connect (OIDC) authentication, ensuring that only the right workloads get access to AWS services.
By implementing IRSA, you enhance the security posture of your Kubernetes workloads while simplifying IAM management. In this article, we’ll walk through how IRSA works, why it’s beneficial, and how to configure it properly for the Percona Operator for MongoDB in EKS clusters.

IRSA Installation and Configuration for Percona Operator for MongoDB

IRSA requires an OpenID Connect (OIDC) provider associated with your EKS cluster.
So, you should create an OIDC provider for your EKS cluster.

Creating an OIDC provider for your EKS cluster involves several steps. This setup allows your EKS cluster to use IAM roles for service accounts, which makes it possible to grant fine-grained IAM permissions to pods.

shell
# Check if OIDC is already set up:

aws eks describe-cluster --name  --query "cluster.identity.oidc.issuer" --output text

https://oidc.eks.eu-west-3.amazonaws.com/id/7AA1C67941083331A80382E464EB2F1F

# If it is not already set up, create an OIDC provider:

eksctl utils associate-iam-oidc-provider --region  --cluster  --approve

Here oidc-id is 7AA1C67941083331A80382E464EB2F1F. We will use it under role creation.

Create an IAM Policy to access s3 buckets. Substitute with your correct bucket name:

shell
# Define the required permissions in an IAM policy JSON file: 


cat s3-bucket-policy.json

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:*"
 ],
 "Resource": [
 "arn:aws:s3:::",
 "arn:aws:s3:::/*"
 ]
 }
 ]
}

# Create the IAM policy:

aws iam create-policy --policy-name  --policy-document file://s3-bucket-policy.json

Create an IAM Role and Attach the Policy:

shell
# Role example. Replace  with account id and  with cluster’s OIDC ID


cat role-trust-policy.json



{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/"
 },
 "Action": "sts:AssumeRoleWithWebIdentity",
 "Condition": {
 "StringEquals": {
 "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com"
 }
 }
 }
 ]
}

# Create role:

aws iam create-role --role-name  --assume-role-policy-document file://role-trust-policy.json --description "Allow access to s3 bucket"

Attach the policy to the role.

shell
# Please update ,  and  with the corresponding values.

aws iam attach-role-policy --role-name  --policy-arn arn:aws:iam:::policy/

Install the operator and deploy Percona Server for MongoDB in your EKS cluster (skip this step if you already have the operator and the database cluster installed).
To ensure proper functionality, we need to annotate both the operator service account (default: percona-server-mongodb-operator) and the cluster service account (default: default).

🔴 Warning: The cluster and operator won’t restart automatically; therefore, a manual restart is necessary to apply the changes.

shell
# Get service accounts:

$ kubectl get sa -n 
NAME SECRETS AGE
default 0 25m
percona-server-mongodb-operator 0 25m


# Get role_arn:

aws iam get-role --role-name  --query "Role.Arn" --output text

# Annotate service account. Please update role_arn with appropriate value.

kubectl annotate serviceaccount default \
 eks.amazonaws.com/role-arn=""

kubectl annotate serviceaccount percona-server-mongodb-operator \
 eks.amazonaws.com/role-arn=""

To verify that the settings have been applied, inspect service accounts and the environment variables in both the operator and replica set (RS/Config) pods. The variable AWS_ROLE_ARN should be properly set.

shell
# Check annotation in service account

$ kubectl get sa -n  percona-server-mongodb-operator -o yaml

$ kubectl get sa -n  default -o yaml

# Check the variable inside container

$ kubectl exec -ti  -n  bash

bash-5.1$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

$ kubectl exec -ti  -n  bash

[mongodb@some-name-rs0-0 db]$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

Configure the backup/restore settings as usual, but do not provide s3.credentialsSecret for the storage in deploy/cr.yaml. For detailed instructions please refer to Configure storage for backups.

shell
# backup section in cr.yaml example 
 storages:
 aws-s3:
 type: s3
 s3:
 region: 
 bucket: 

Conclusion

Using IAM Roles for Service Accounts (IRSA) in an Amazon EKS cluster is a best practice when running database operators in Kubernetes. By integrating IRSA, database operators—such as the Percona Server for MongoDB Operator—can securely access AWS services like S3 for backups without relying on static credentials.

IRSA enhances security by enforcing the principle of least privilege, ensuring that database operators in EKS have access only to the specific AWS resources they require. This approach reduces the risk of unauthorized access while also improving manageability by eliminating the need to store and rotate AWS credentials within Kubernetes secrets. By adopting IRSA in Percona Server for MongoDB Operator , organizations can create a more secure, scalable, and automated environment for managing MongoDB databases.

Join Percona for Google Summer of Code 2025 – Explore, Innovate, and Contribute!

Radoslaw Szulgo — Wed, 05 Feb 2025 00:00:00 UTC

Are you passionate about open-source databases, AI/ML, and security? Do you want to work on real-world projects that impact thousands of developers and enterprises worldwide? Percona is excited to invite students to participate in Google Summer of Code 2025 (GSoC) and help advance our cutting-edge open-source database solutions!

Why Contribute to Percona?

At Percona, we believe that open world is a better world! GSoC is an excellent opportunity to work with seasoned developers, gain hands-on experience, and contribute to powerful database tools used by businesses globally.

For 2025, we’re especially interested in projects that focus on AI/ML and security—two critical areas shaping the future of databases. Whether you’re passionate about automating database performance insights with AI or hardening security for mission-critical data, we have exciting challenges for you!

Percona mentors are going to help you realize your ideas or one of the ideas below. We invite you to contribute to products and projects such as:

Percona XtraDB Cluster
Percona Server for MySQL
Percona Distribution for PostgreSQL
Percona Operator for PostgreSQL
Percona Server for MongoDB
Percona Backup for MongoDB
Percona Operator for MongoDB
Percona Everest
Percona Monitoring and Management (PMM)
pg_tde: Transparent Database Encryption for PostgreSQL
As well as CI/CD related projects with the Percona Build Engineering.

Project Ideas for GSoC 2025

Below are some suggested project ideas categorized by Percona software:

Percona Distribution for PostgreSQL

Snapshot-based PostgreSQL backups

Database users are often very familiar with their storage provider’s storage snapshot capabilities. These snapshots are very handy and performant to use, hence their popularity among users. Backups for other databases (e.g., MongoDB) are often configured via this capability as it provides many performance benefits for large-scale data, especially on Cloud deployments. Having such technology supported across the backup solutions for multiple databases makes it possible to leverage this effectively for Percona Everest via the Percona Operators.

Comments from Crunchy on what needs to be glued together to get snapshots and pgBackRest working together better. Additionally, Timescale on how they use snapshots and pgBackRest together in their hosted managed service.

Deliverables: Have an API available to Percona Distribution for PostgreSQL to effectively use storage snapshots to create backups and restore from storage snapshot-based backups. Preferably, have it added to/complementary to the currently recommended solution of pgBackRest.

Have Percona Operator for PostgreSQL expose the storage snapshot-based backup/restore so that Percona Everest can leverage it.

Required/preferred skills: C++, PostgreSQL, Kubernetes Duration: 350 hours **Difficulty level: **Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

pgBackRest to Barman close gap improvements

PostgreSQL has two main backup tools: Barman and pgBackRest. Both are powerful backup and restore tools, each with its own strengths. pgBackRest is generally considered more advanced in terms of parallelism, performance, and flexibility. Barman does offer some advantages in certain areas. While pgBackRest is maintained by Community, Barman is a tool mainly maintained by one company and is less popular. Barman does have UX improvements over pgBackRest, especially for non-expert users:

direct WAL archiving with PostgreSQL’s built-in archive_command,
simpler backup and recovery process, especially for standby creation,
clearer logging and monitoring for backup integrity,
simpler configuration in small to medium deployments,
better native tools for cloud backups

It would be beneficial for Percona, which uses pgBackRest in the Percona Distribution for PostgreSQL, to have the backup tool close any functionality gaps in Barman. Percona customers sometimes use Barman and expect Percona to support it. Having a way to migrate off Barman to pgBackRest, reducing any potential friction for the users, would be beneficial.

Deliverables: Provide a close-gap set of improvements based on the list available in the description

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Tool to investigate PostgreSQL locks for dummies

Currently, there is no tool that allows users with low experience to detect and understand all types of locks on their PG database, which may lead to many issues in deployments not managed by expert PostgreSQL users. As described in the blog posts below, understanding how locks work is difficult:

Deliverables:

Detect ddl with mixed strong locks and others. i.e. allow to review locked PIDs as pg_locks will not work
Present all locks in a GUI
(Streched) have the GUI integrated in PMM

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Medium **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Session continuity for PgBouncer for the zero downtime upgrades

Percona is looking to introduce zero downtime upgrades capability to the Percona Operator and later on to Percona Everest. The assumption is to base on pgBouncer and our HA solution utilizing the replica with the new database and a switch from the previous version to the new version.

Such a solution provides a zero downtime upgrade capability and a Rollback capability. To provide true zero downtime major upgrades for the current Percona Distribution for PostgreSQL, there needs to be an improvement that takes over the switching of sessions between the databases: the previous version and the new version

In the future, this tool should also make it possible to zero downtime and migrate to Everest.

Deliverables: Extend pgBouncer to ensure that the sessions can be switched between the databases without downtime for the users but only a potential performance drop

Required/preferred skills: C++, Kubernetes Duration: 175 hours Difficulty level: Hard **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Percona Software for MongoDB

Interactive Shell Installer for Percona Software for MongoDB

This project aims to develop an interactive shell-based installer for Percona Server for MongoDB and Percona Backup for MongoDB. The installer will simplify the installation, configuration, and initial setup process, making it easy for users to deploy these open-source enterprise solutions efficiently. The primary goal is to enhance the user experience by reducing manual setup steps and ensuring proper configuration through guided prompts and automation.

Deliverables:

A command-line-based interactive installer script.
Automated dependency checks and installation.
Interactive prompts for configuration choices (e.g., authentication, replication, sharding).
Seamless installation of both Percona Server for MongoDB and Percona Backup for MongoDB.
Integration with package managers for major Linux distributions (Debian, Ubuntu, RHEL, CentOS).
Logging and validation mechanisms to ensure correct setup.
Documentation and user guide for the installer.

Required/preferred skills: C++ or Go

Duration: 350 hours

Difficulty level: Medium

Mentors: @radoslaw.szulgo

Relevant repository and resources: https://github.com/percona/percona-backup-mongodb

Percona Backup for MongoDB backup speed throttling

On large scale deployments, backups may significantly impact network performance - speficially network bandwidth may be heavily utilized, if the backup storage is fast, causing performance degradation of the database itself. Database reliability engineers would like to reduce the network load by slowing down physical backups with Percona Backup for MongoDB (PBM) configuration. The scope of the project is to implement a network bandwidth rate limiter and perform load testing showing the impact or rate limiting on backup time.

Deliverables: The expected outcome of this project is insurance that backup will not degrade network bandwidth impacting the database. As a result a participant needs to provide proposed code changes in a form of fork of PBM and a create a report with load test results.

Required/preferred skills: Go
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

Percona Backup for MongoDB Golang SDK

The project’s purpose is to extend capabilities and reduce maintenance in monitoring, managing, and automating backup and restores of MongoDB from Percona Monitoring and Management (PMM) tool. In the scope of the project there’s a migration from Percona Backup for MongoDB CLI to a dedicated PBM golang client library in PMM. The client library (aka SDK) has to be implemented and map all current CLI operations to Go API functions. As a stretch goal, the project can be extended into implementing a backup progress reporting using the created SDK.

Deliverables: The expected outcome of the project is the reduced maintenance of backup integration in PMM project and enabling extensibility of backup management. As a result of the project a new open-source SDK should be create

Required/preferred skills: Go
Duration: 350 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/pmm

CEPH Storage support in Percona Backup for MongoDB

Ceph is an open source Software Defined Storage(SDS) software that is massively scalable and reliable. It’s one of the most popular storage technologies in Kubernetes and Openshift. The project aims to enable users to store their Percona Backups for MongoDB data on a Ceph storage which would be very convenient as they wouldn’t need to manage other additional storages. The scope of the project includes building a workspace setup on Kubernetes and Percona Operator for MongoDB, research on current challanges using Ceph storage, and implement necessary changes to make it work in a performant way. At the end document the solution.

Deliverables: The project’s deliverables are technical documentation, Percona Operator for MongoDB changes, and instructions on setting up an environment with Ceph storage.

Required/preferred skills: Go, Kubernetes
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

BoostFS storage support in Percona Backup for MongoDB

Dell Data Domain Boost File System (BoostFS) provides a general file system interface to the DD Boost library, allowing standard backup applications to take advantage of DD Boost features. In this project we’d like to extend our open-source Percona Backup for MongoDB to leverage that storage technology to reduce backup and restore time - and the same time help users to reduce their Recovery Time Objective (RTO) and Recovery Point Objective (RPO). In the scope of the project there’s a preparation of the workspace setup with Percona Server for MongoDB, Percona Backup for MongoDB and mounted BoostFS disk volumes on Google Cloud Platform and documenting the architecture of the environment. Additionally, the project includes a research on how PBM works with that storage and implementing necessary changes to PBM to make it work. Finally, a simple benchmark should be performed that proves the performance boost.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support BoostFS in PBM, and report incl. performance benchmark results and comparison to other storage systems.

OpenStack Swift storage support in Percona Backup for MongoDB

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound. Swift is very convenient to use as a backup storage for MongoDB workloads running on OpenStack platform. The scope of the project includes building a workspace environment on Google Cloud Platform with OpenStack clusters and running there Percona Server for MongoDB. Then implementing required changes in Percona Backup for MongoDB to support Swift storage. Finally, performing benchmark tests and comparison to GCP native storages.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support OpenStack Swift in PBM, and report incl. performance benchmark results and comparison to other storage systems.

Required/preferred skills: Go, GCP, OpenStack
Duration: 175 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repositories and resources:

Percona Server for MySQL and Percona XtraDB Cluster

Automating Code Merges with AI

The regular and manual merge from Oracle’s GitHub repository process is time-consuming, complex, and prone to errors, particularly due to merge conflicts. Careful attention is required to avoid introducing regressions into Percona’s open-source products. While this project is specific to Percona’s needs, it addresses a common challenge in open-source software development, as many projects rely on upstream repositories for their code. Therefore, the solution can be generalized and could benefit other open-source projects with similar code integration needs.

This GSoC project aims to develop an intelligent system using Artificial Intelligence to automate the MySQL fork merge process. Percona has been performing these merges for 18 years, accumulating a wealth of historical data (code changes, merge resolutions, conflict histories, test results) that can be leveraged to train an AI model.

The core objective is to create a tool that can:

Analyze upstream changes: Process and understand the changes introduced by Oracle in their MySQL repository.
Identify merge conflicts: Identify conflicts between upstream changes and Percona’s modifications.
Suggest merge resolutions: Propose solutions for resolving identified conflicts, drawing on patterns from historical merge data.
Automate merges: Automatically apply upstream changes with the suggested merge resolutions.
Learn and adapt: Continuously improve its performance and accuracy by learning from new merge data and feedback.

Deliverables:

Reduced merge time and effort: Automating the merge process will free up developer time for other critical tasks.
Improved merge accuracy: AI can potentially identify subtle conflicts that might be missed by manual review.
Faster release cycles: Streamlining the merge process will enable quicker releases of updated Percona products.
Open-source contribution: The resulting tool will be open-sourced, benefiting other projects that maintain forks of MySQL or similar databases. This problem is not unique to Percona; other open-source projects facing similar merging challenges can utilize this solution.

As a result of this project, you’re expected to deliver:

A working prototype of the AI-powered merge tool.
Well-documented code and training data.
Comprehensive test suite and evaluation results.
A report detailing the project’s methodology, findings, and future directions.

Required/preferred skills: Python, Machine Learning libraries and frameworks (e.g., TensorFlow, PyTorch, scikit-learn), C++, Git, database systems Duration: 350 hours Difficulty level: Hard Mentors: Julia Vural, Oleksiy Lukin Relevant repositories and resources:

Percona Everest

Easier Troubleshooting on database clusters in Percona Everest

The main goal is to provide tools to Percona Everest users to troubleshoot database clusters. This project will require the implementation of log collection, rotation, UI, and possibly an AI helper to analyze those logs. If users have a centralized log collection implemented, this tool needs to be able to integrate with it.

Deliverables:

Full user flow to support database cluster troubleshooting process (UI, backend, API, integrations).

Log Collection & Rotation System:
- Implement a mechanism to collect logs from Percona Everest-managed database clusters.
- Ensure efficient log rotation to manage storage and performance impact.
- Enable compatibility with external log aggregation tools (e.g., Elasticsearch, Grafana Loki, or OpenTelemetry)
User Interface for Log Access:
- Develop a UI within Percona Everest to allow users to view and analyze logs.
- Include search, filtering, and visualization options for better troubleshooting.
AI-Powered Log Analysis (Stretched scope)
- Explore AI-driven log analysis to provide users with insights, anomaly detection, and recommendations.
- Implement basic AI-assisted troubleshooting if feasible within the project timeline.
Documentation & Testing:
- Deliver user and developer documentation covering installation, usage, and troubleshooting.
- Include test cases and automation scripts to ensure system reliability.

Required/preferred skills: Kubernetes, Go, CI/CD Duration: 350 hours **Difficulty level: **Medium Mentors: @Diogo_Recharte, @Mayank_Shah Relevant repository and resources: https://github.com/percona/everest

Percona Everest RBAC policies management UI

Create a user interface to create and manage role-based access control policies

Deliverables:

Role-Based Access Control (RBAC) UI:
- Develop a user-friendly interface in Percona Everest to create, update, and manage RBAC policies.
- Implement role assignment and permission configuration for database clusters.
Documentation & Testing:
- Deliver comprehensive user and developer documentation.
- Include test cases and automation scripts to ensure reliability.

Required/preferred skills: Front-end, CI/CD tools Duration: 90 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Context sensitive help

The Percona Everest documentation contains valuable information, hints, and tips, but we lack a way to present relevant information to our users. This project aims to work with the UX and Docs teams to solve this problem.

Deliverables:

Implement a mechanism to display relevant documentation, hints, and tips based on the user’s current action or screen within Percona Everest.
Ensure seamless integration with the existing UI for a non-intrusive experience.
Enable contextual tooltips, pop-ups, or side panels that present relevant documentation without requiring users to leave the interface.
Support links to full documentation pages when needed.
Optionally, explore AI-driven suggestions based on user behavior and past queries.
Allow users to control the level of help they receive (e.g., enable/disable tips, adjust verbosity).
Provide user and developer documentation on how the system works and how to extend it.
Ensure thorough testing to validate the accuracy and relevance of displayed help content.

Required/preferred skills: Front-end, CI/CD tools Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Backups and restore timeline visualization

Databases are usually long-living services, and investigating issues with them is easier when you can see events like backups and restores of this service on a timeline.

Deliverables:

Develop a visual timeline within Percona Everest to display backup and restore events for database clusters.
Ensure the timeline is intuitive, zoomable, and supports different time ranges (e.g., last 24 hours, 7 days, custom range).
Retrieve and display backup and restore events from Percona Everest’s database and logs.
Include metadata such as timestamps, duration, status (success, failure), and associated users or processes.
Allow users to filter events by type (full backup, incremental backup, restore, etc.).
Enable color-coding or icons to differentiate event types at a glance.
Deliver comprehensive user and developer documentation.
Ensure automated tests for data accuracy, UI performance, and usability.

Refactor test automation using page object model

Our project currently has a functional end-to-end (E2E) UI test suite that ensures the stability and correctness of our application. However, the test suite does not follow the Page Object Model (POM) design pattern, making it harder to maintain, scale, and debug.

Deliverables:

Restructure existing test automation to follow the Page Object Model (POM) design pattern.
Ensure better separation of test logic and UI elements for improved maintainability.
Implement modular and reusable page object classes for different UI components and workflows.
Standardize naming conventions and best practices for test scripts.
Improve error handling and logging to make test failures easier to diagnose.
Ensure the refactored test suite runs efficiently in CI/CD pipelines.
Validate test performance improvements and maintain test coverage.

Required/preferred skills: Playwright, Typescript, Kubernetes Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Tomislav_Plavcic, Edith Puclla Relevant repository and resources: https://github.com/percona/everest

Percona Monitoring and Management (PMM)

Queryable backup and restore of Percona Server for MongoDB

The project aims to equip MongoDB Database Administrators with Percona Monitoring and Management (PMM) extension that allows to query data directly from a backup. It solves their pain when they want to inspect just a single document in a collection from a few Terabytes size backup. The time associated with downloading the snapshot, decompressing it, getting it running in a local MongoDB node, and finally running the query would be significant. Not only that, but there are obvious nontrivial costs — both monetary and operational — associated with having to quickly spin up new environments. In scope of the project there is a backend Go application server implementation to run on-demand ephemeral mongodb instance, load data from backup, and enable user to run a DB query from a UI interface.

Deliverables: It’s expected to deliver source code changes to PMM in form of PMM fork that extends PMM functionality for queryable MongoDB backup. Specifically, it is expected to prepare a solution design, implementation, unit and/or integration tests based on a Docker environment, and documentation.

Required/preferred skills: Go, MongoDB
Duration: 350 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repository:
- https://github.com/percona/pmm
- https://github.com/percona/percona-backup-mongodb

Design Engineering for PMM

Percona Monitoring and Management (PMM) is a long-standing open-source software, but its age also comes with some UX and UI debt. Facing new goals to help innovate PMM, the team is excited to look forward to starting “renovating the house” and swapping the GUI with a more modern one built in-house. We are looking for experts in design engineering to help with:

Making/refining/cataloging UI components that we will need for QA and production;
Creating functional prototypes ad hoc from written ideas or designs;
Convert old PMM pages like for like into new PMM pages (with new UI).

Deliverables:

Contributing with ideas to help make the code library more easy to contribute to;
Contributing to the code library with at least one new component;
Create at least one code prototype for one of the team’s ongoing ideas;
Convert at least one existing PMM functionality page UI into the new UI.

Required/preferred skills: CI/CD, Git, Storybook, React, MUI, Figma

Duration: 350 hours

Difficulty level: Medium

Mentor: @pedro.fernandes

PMM UI for PostgreSQL backups - create, restore, check, monitor

Backup management without UI is not an easy task for the users. Having a tool that could be a tool of choice for backup and restore management could provide a unification layer for multiple backup/restore tools as well as provide a very important value that’s often overlooked: backup monitoring.

As it turns out, many DBAs are often worried about the state of their backups and make it a daily routine task to check how the backups they have configured are. As the environments scale, this task becomes increasingly tiresome. Also, having to check whether the backups they create are effective makes it another routine task that needs either extra automation scripting to run a restore or a manual task.

Deliverables:

Extend PMM UI to add existing backups so that they can be monitored
Extend PMM UI to create backups
Create a tool to automate the backup testing (check whether the backups created are usable)
Extend PMM to monitor and alert on backup irregularities
Integrate the backup management with external schedulers like cron.

Required/preferred skills: C++, Go, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

LLM-Powered Test Scenario Generation for Open Source Contributions

Open-source projects thrive on community contributions, but ensuring that each pull request (PR) has adequate test coverage is a major challenge. Many PRs introduce changes without proper regression tests, leading to bugs and unstable releases.

This project aims to build an LLM-powered tool that analyzes code changes in GitHub PRs and automatically generates relevant test scenarios. Using an Open Source LLM (e.g., DeepSeek, Mistral, LLaMA), the system will identify impact areas, suggest missing test cases, and recommend regression tests based on past commits. The goal is to integrate this into GitHub workflows, enabling maintainers to quickly assess PR test coverage and guide contributors in writing better tests.

Deliverables:

The student will work on developing a system with the following features:

PR Analysis & Impact Assessment
Extract and analyze code diffs in pull requests.
Identify affected functions, dependencies, and modules.
Predict impact areas using a dependency graph.

Test Scenario Generation using LLMs

Use an Open Source LLM (DeepSeek, Mistral, etc.) to generate test cases.
Recommend unit tests, integration tests, and regression scenarios.
Compare new tests with existing ones to detect gaps in coverage.

GitHub Bot for Automated Suggestions

Implement a bot that comments on PRs with test recommendations.
Provide interactive feedback to contributors and maintainers.
Integrate with GitHub Actions for CI/CD automation.

Regression Test Identification

Identify existing test cases that need to be re-run.
Suggest additional tests based on historical PRs and past bug reports.

Evaluation Metrics & Benchmarking

Measure effectiveness by tracking missed bugs before/after integration.
Collect feedback from maintainers and contributors.

Future Scope:

Extend beyond GitHub to GitLab, Bitbucket, and other version control systems.
Support additional test types, such as security and performance tests.
Implement self-learning mechanisms to improve accuracy over time.

Required/preferred skills: Strong programming skills in Python or JavaScript, Experience with GitHub APIs & Pull Request Workflows, Understanding of Machine Learning / LLMs (DeepSeek, Mistral, LLaMA, etc.), Familiarity with Software Testing & QA Automation, Experience with CI/CD Pipelines & GitHub Actions (Bonus). Duration: 350 hours Difficulty level: Hard Mentor: Peter Sirotnak, @vasyl.yurkovych

Percona Build Engineering

SBOMs for Percona database software - MySQL, PostgreSQL, and MongoDB

A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a nested inventory, a list of ingredients that comprise software components. The project aims to adapt Percona’s build pipelines to generate SBOMs for Percona Software for MySQL, PostgreSQL, and MongoDB. This will enable organizations using Percona software to be more secure and avoid software supply chain vulnerabilities that were very harmful in late 2020 with the discovery of the Solar Winds cyberattack or later with the Log4j security flaw.

Deliverables: At the end of the project, a running staging pipeline in Jenkins and Trivy should produce complete SBOMs for Percona Server for MySQL, PostgreSQL, and MongoDB, Percona Backup for MongoDB, Percona Xtra Backup for MySQL. SBOMs are uploaded automatically to the Percona repository and are downloadable publicly. Additionally, technical documentation on how the process works is expected to be created.

Required/preferred skills: Jenkins, Trivy
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:

Evolving CI/CD: Automating Build, Test, and Release for Robust Software Delivery

Continuous Integration and Continuous Deployment (CI/CD) pipelines are the backbone of modern software development, ensuring rapid, reliable, and repeatable delivery. However, many pipelines still operate in fragmented stages, where builds and tests are automated, but releases remain a manual or semi-automated process.

This project aims to transform our CI/CD pipelines into a true end-to-end automated system, seamlessly integrating build, test, and release stages. By implementing best practices in CI/CD automation, we will ensure that only thoroughly tested software progresses to release, minimizing human intervention and reducing the risk of deployment failures.

Deliverables: The successful completion of this project will result in a fully automated and robust CI/CD pipeline that seamlessly integrates build, test, and release processes. The key outcomes will include:

Fully Automated CI/CD Pipeline

A redesigned pipeline where builds, testing, and releases are interconnected and automated. Code changes will automatically trigger builds, run tests, and, if successful, deploy releases without manual intervention. Comprehensive Test Integration

The pipeline will incorporate unit tests, integration tests, security scans, and other quality assurance mechanisms. Ensuring that faulty builds do not reach production by enforcing test-driven deployment. Automated Release Process

A mechanism that automatically releases software only if all tests pass. Versioning, tagging, and artifact management will be streamlined. The release process will be documented and configurable for different environments (e.g., staging, production). Infrastructure as Code (IaC) & Deployment Automation

Documentation & Guides

Clear technical documentation detailing the new pipeline’s workflow and configuration. A step-by-step guide for developers and DevOps engineers on how to use and extend the pipeline.

Required/preferred skills: CI/CDl like Jenkins, GitHub Actions, GitLab CI, or similar; Docker; Testing frameworks and automated deployment strategies; infrastructure as code (IaC) and cloud environments is a plus.
Duration: 350 hours
Difficulty level: Medium
Mentors: @Evgeniy_Patlan , @Vadim_Yalovets
Relevant repository: https://github.com/Percona-Lab/jenkins-pipelines

Build Automation for Open-Source Databases

Building and maintaining multiple database forks—such as MySQL, MongoDB, and PostgreSQL—often involves redundant build scripts, leading to inefficiencies, inconsistencies, and maintenance overhead. Currently, each database has its own set of build scripts despite sharing many common steps.

This project aims to develop a modular, extensible build system that allows for streamlined compilation and packaging of different database forks. The system will provide a flexible framework where users can select required modules, specify target OS distributions, and automate the build process with minimal configuration.

By implementing a plugin-based architecture, this modular builder will simplify cross-database maintenance, reduce duplication, and improve consistency across different builds.

Deliverables:

Modular Build Framework – A reusable, pluggable system that dynamically selects required modules for MySQL, MongoDB, and PostgreSQL builds.
Multi-OS Support – Automated builds for multiple Linux distributions (Debian, Ubuntu, CentOS, RHEL) with configurable OS selection.
Automated Package Creation – DEB and RPM package generation with standardized versioning and tagging.
Configurable & Scalable Builds – Easy customization of build parameters, allowing extension to new database forks or patches.
CI/CD Integration – Optional support for Jenkins, GitHub Actions, or GitLab CI to enable fully automated builds.
Comprehensive Documentation – User and developer guides with example configurations for quick adoption and extension.

Required/preferred skills: Bash/Python, CMake, Makefiles, Autotools, Linux and packaging (DEB/RPM), dependency management, CD/CD tools are a plus

Duration: 175 hours

Difficulty level: Medium

Mentors: @Evgeniy_Patlan , @Vadim_Yalovets

Relevant repositories:

More ideas are coming soon!

Suggest your ideas in the comments of the post or on the forum.

GSoC isn’t just about working on predefined ideas—it’s about innovation! If you have a project idea that aligns with Percona software, AI/ML, security, or database performance, submit your proposal, and our mentors will be happy to discuss it with you.

Do you have questions? Visit our Community Forum or join our chat channels to connect with potential mentors.

Ready to get started? See our Google Summer of Code 2025: Contribution guide.

See you in GSoC 2025!

Percona Operator for MongoDB 1.19: Remote Backups, Auto-Generated Passwords, and More!

Sergey Pronin — Fri, 31 Jan 2025 00:00:00 UTC

The latest release of the Percona Operator for MongoDB, version 1.19, is here. It brings a suite of enhancements designed to streamline your MongoDB deployments on Kubernetes. This release introduces a technical preview of remote file server backups, simplifies user management with auto-generated passwords, supports Percona Server for MongoDB 8.0, and includes numerous other improvements and bug fixes. Let’s dive into the details of what 1.19 has to offer.

Remote Backups with Network File System (Technical Preview)

Backing up your MongoDB data is crucial, and Percona Operator for MongoDB 1.19 introduces a powerful new option for backup storage: the filesystem type. This feature, currently in technical preview, allows you to leverage a remote file server, mounted locally as a sidecar volume, for your backups. This is particularly useful in environments with network restrictions that prevent the use of S3-compatible storage or for organizations using non-standard storage solutions that support the Network File System (NFS) protocol.

Setting Up Remote Backups

To use this new capability, you’ll need to add your remote storage as a sidecar volume within the replsets section of your Custom Resource (and configsvrReplSet for sharded clusters). Here’s how:

replsets:
 ...
 sidecarVolumes:
 - name: backup-nfs-vol
 nfs:
 server: "nfs-service.storage.svc.cluster.local"
 path: "/psmdb-my-cluster-name-rs0"
 ...

Then, configure the mount point and sidecar volume name in the backup.volumeMounts section:

YAML:

backup:
 ...
 volumeMounts:
 - mountPath: /mnt/nfs/
 name: backup-nfs-vol
 ...

Finally, set up a filesystem type storage in the backup.storages section, pointing it to the mount point:

YAML:

backup:
 enabled: true
 ...
 storages:
 backup-nfs:
 type: filesystem
 filesystem:
 path: /mnt/nfs/

See more in our documentation about this storage type.

Simplified User Management with Auto-Generated Passwords

Managing user credentials just got easier. Percona Operator for MongoDB 1.19 enhances declarative management of custom MongoDB users by adding the ability to generate passwords automatically. Now, when defining a new user in your deploy/cr.yaml file, you can omit the reference to an existing Secret containing the password, and the Operator will handle the generation for you:

YAML:

...
users:
 - name: my-user
 db: admin
 roles:
 - name: clusterAdmin
 db: admin
 - name: userAdminAnyDatabase
 db: admin

The Operator will create a Secret to store the generated password securely. It is important to note that the Secret will be created after the cluster is in the Ready state.

Get the user credentials: Find the Secret resource named -custom-user-secret Get the user password with this one-liner:

kubectl get secret my-cluster-name-custom-user-secret -o jsonpath='{.data.my-user}' | base64 -d

You can find more details on this automatically created Secret in our documentation.

Percona Server for MongoDB 8.0 Support

Staying up-to-date with the latest MongoDB versions is essential for performance and security. Percona Operator for MongoDB 1.19 now officially supports Percona Server for MongoDB 8.0, in addition to 6.0 and 7.0. This means you can leverage the latest features and improvements from MongoDB 8.0, combined with the enterprise-grade enhancements and open-source commitment of Percona Server for MongoDB.

Check out this blog post to learn more about the features in MongoDB 8.0.

Streamlined AWS S3 Access with IAM Roles for Service Accounts (IRSA)

Percona Operator for MongoDB 1.19 adds support for IAM Roles for Service Accounts (IRSA), simplifying secure access to AWS S3 for backups on Amazon EKS. IRSA lets you grant granular S3 permissions to specific Pods via their associated Kubernetes service accounts. This approach ensures that only the Pods that require S3 access receive it, adhering to the principle of least privilege. Furthermore, each Pod can only access credentials linked to its service account, providing strong credential isolation. For enhanced security, all S3 access is tracked through AWS CloudTrail, enabling comprehensive auditability. All of this happens without the need to manually manage and distribute AWS credentials.

Configuration Steps

Create an IAM Role: Define an IAM role with S3 access permissions. See AWS documentation.
Identify Service Accounts: The Operator uses percona-server-mongodb-operator and your cluster uses default (customizable in deploy/cr.yaml).

Annotate Service Accounts: Link the IAM role to both service accounts:

$ kubectl -n  annotate serviceaccount default eks.amazonaws.com/role-arn:  --overwrite
$ kubectl -n  annotate serviceaccount percona-server-mongodb-operator eks.amazonaws.com/role-arn:  --overwrite

Configure S3 Storage: Set up S3 storage in deploy/cr.yaml without s3.credentialsSecret. The Operator will use IRSA.

Important: IRSA credentials take precedence over IAM instance profiles, and S3 credentials in a Secret override both.

IRSA streamlines S3 access, enhancing security and manageability for your MongoDB backups on EKS. Learn more in our documentation.

Conclusion

Percona Operator for MongoDB 1.19 delivers a significant step forward in simplifying and automating the management of your MongoDB clusters on Kubernetes. With features like remote backups, auto-generated passwords, and support for Percona Server for MongoDB 8.0, this release empowers you to deploy, manage, and scale your databases with greater ease and efficiency.

We encourage you to explore the full release notes and try out the new features. As always, your feedback is invaluable to us. Please share your thoughts and contribute to the project on our GitHub repository or our Community Forum.

Percona Monitoring and Management 3.0.0-GA

Ondrej Patocka — Wed, 29 Jan 2025 00:00:00 UTC

We’re excited to announce the release of Percona Monitoring and Management (PMM) 3.0.0 GA.

The Percona Monitoring and Management (PMM) 3.0.0 release delivers major security and stability enhancements. Notable security improvements include rootless deployments and encryption of sensitive data, along with improved API authentication using Grafana service accounts. Deployment options have expanded with official ARM support and the ability to use Podman for rootless deployments, providing flexibility and better security. Additionally, the introduction of containerized architecture has increased stability, and a streamlined upgrade process ensures reliability and ease of maintenance.

User experience has been significantly improved with more flexible monitoring configurations and UI-based upgrades for Podman installations. This release also includes new features such as monitoring for MongoDB 8.0 and integration with Watchtower for automated container updates. These enhancements aim to provide users with a more secure, stable, and user-friendly monitoring and management experience.

Release notes

To see the full list of changes, check out the 3.0.0 GA Release Notes

Percona Monitoring and Management (PMM) 3.0.0 Release Notes:

Security Enhancements:
- Implementation of rootless deployments to enhance security.
- Encryption of sensitive data to ensure information confidentiality.
- Improved API authentication with Grafana service accounts, increasing access security.
Deployment Options:
- Official PMM Client ARM support, allowing the use of PMM on ARM architecture devices.
- Rootless deployments using Podman, providing flexibility and security.
- Support for deployments using Helm, Docker, Virtual Appliance, and Amazon AWS for various use cases.
Stability Improvements:
- Increased stability through containerized architecture, providing isolation and manageability.
- Streamlined upgrade process, reducing the risk of failures during updates and enhancing reliability.
User Experience:
- Flexible monitoring configurations, allowing users to tailor the system to their needs.
- UI-based upgrades for Podman installations, making the update process more convenient and intuitive.
New Features:
- Monitoring for MongoDB 8.0, ensuring support for the latest database versions.
- Integration with Watchtower for automated container updates, simplifying management and keeping the system up-to-date.

We invite you to install and try the new PMM 3.0

Quickstart guide

Get started with PMM

Multiple installation options

About PMM installation

MySQL 8.4 Support in Percona Toolkit 3.7.0

Sveta Smirnova — Mon, 06 Jan 2025 00:00:00 UTC

Percona Toolkit 3.7.0 has been released on Dec 23, 2024. The main feature of this release is MySQL 8.4 support.

In this blog, I will explain what has been changed. A full list of improvements and bug fixes can be found in the release notes.

TLDR;

Replication statements in 8.4 are fully supported by the Percona Toolkit
pt-slave-delay has been deprecated.
pt-slave-find has been renamed to pt-replica-find. The old name has been deprecated but exists in the repository as an alias of the pt-replica-find.
pt-slave-restart has been renamed to pt-replica-restart. Old name has been deprecated but exists in the repository as an alias of the pt-replica-restart.
Basic SSL support has been added to the tools where it was not working before (see https://perconadev.atlassian.net/browse/PT-191 ), and Percona Toolkit now supports caching_sha2_password and sha256_passwordauthentication plugins. Full implementation of https://perconadev.atlassian.net/browse/PT-191 is planned for the next version.

Replication Statements

MySQL 8.4 removed earlier deprecated offensive language, such as SLAVE or MASTER. This made tools written for earlier versions not compatible with the new version. Percona Toolkit was also affected, and I had to rewrite it.

However, Percona Toolkit should be able to run not only with MySQL 8.4 but also with older versions. So, the change was not a simple grep and replace of offensive words. It is not even possible for version MySQL 8.0 because new syntax was first introduced in 8.0.23 for the CHANGE REPLICATION SOURCE and START/STOP REPLICA commands. Earlier versions weren’t aware of this change.

Another challenge was the fact that I could replace all occurrences of the word SLAVE with REPLICA. Still, I could not do the same for the MASTER and SOURCE pairs because replication source-related commands are mapped differently:

Legacy syntax	Syntax without offensive words
`CHANGE MASTER`	`CHANGE REPLICATION SOURCE` (since 8.0.23)
`SHOW MASTER STATUS`	`SHOW BINARY LOG STATUS` (since 8.4.0)
`RESET MASTER`	`RESET BINARY LOGS[ AND GTIDS]` ( since 8.4.0)
`MASTER` in other commands	`SOURCE` (partially since 8.0.23, fully since 8.4)

So, I added selectors that use the correct command depending on the MySQL server version.

I intentionally implemented new syntax for version 8.4 only, so I do not have to check every single minor version of 8.0. I also did not implement new syntax for MariaDB. This may happen in the future.

However, all messages displayed to the user use the new syntax. If you rely on old syntax somewhere in your scripts, adjust them.

Internally, most of the functions were renamed to use the new syntax, but the important module lib/MasterSlave.pm kept its name.

Deprecated and Outdated Tools

As a result of this change, pt-slave-delay has been deprecated. The tool stays in the repository and works as before when connected to MySQL 8.0 or an earlier version. However, it refuses to work with MySQL 8.4. The tool will be removed in one of the future versions.

Tools pt-slave-find and pt-slave-restart were renamed to pt-replica-find and pt-replica-restart. Aliases with old names still exist, so you have time to change your scripts. However, expect that these aliases will be removed in one of the future versions as well.

Tool pt-variable-advisor has been updated to reflect current default values.

Basic SSL Support

Percona Toolkit did not have consistent SSL support: some of the tools were able to connect using SSL, and others did not. This was reported at https://perconadev.atlassian.net/browse/PT-191. In this version, I added option “s” for DSN that instructs DBD::mysql to open a secure connection with the database. As a result, Percona Toolkit now supports caching_sha2_password and sha256_password authentication plugins. But other SSL options are still missed. Full SSL support will be added in the next version.

Conclusion

Percona Toolkit fully supports MySQL 8.4. If you use pt-slave-find and pt-slave-restart, consider calling them by their new names pt-replica-find and pt-replica-restart. Tool pt-slave-delay has been deprecated and will be removed in future versions. Use built-in feature delayed replication instead.

Percona Monitoring and Management 3.0.0-Beta - Tech Preview

Ondrej Patocka — Mon, 02 Dec 2024 00:00:00 UTC

Percona Monitoring and Management 3.0.0 Beta - Tech Preview

We’re excited to announce the Tech Preview (Beta) release of Percona Monitoring and Management (PMM) 3.0.0-Beta.

This release is intended for testing environments only, as it’s not yet production-ready. The GA (General Availability) release will be available through standard channels in the upcoming months.

Release notes

To see the full list of changes, check out the 3.0.0-Beta - Tech Preview Release Notes

Installation options

PMM Server

Docker

Server: docker pull perconalab/pmm-server:3.0.0-beta
Docker installation guide

VM

PMM Client

Docker images

AMD 64 + ARM 64: docker pull perconalab/pmm-client:3.0.0-beta

Binary packages

Package Manager installation

Enable testing repository via Percona-release: percona-release enable pmm3-client testing
Install relevant pmm-client package using your system’s package manager

Percona Bug Report: October 2024

Aaditya Dubey — Mon, 25 Nov 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-8057: When max_slowlog_size is set to above 4096, then it gets reset to 1073741824. This overwrites the slow log file path with a different file name, which becomes like node_name.log.000001. Due to this issue, your path defined at slow_query_log_file won`t be useful. This issue has started happening since MySQL Version 8.0.32.

E.g.:

MySQL 8.0.36 is running with the following set of configurations:

slow_query_log = ON
slow_query_log_file = /home/user/sandboxes/msb_ps8_0_36/data/slow
long_query_time = 10
max_slowlog_files = 2
max_slowlog_size = 510000000

Check the log_file path:

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+------------------------------------------------------------+
| Variable_name | Value |
+---------------------+------------------------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/localhost.log.000001 |
+---------------------+------------------------------------------------------------+
1 row in set (0.00 sec)

Interestingly, you will see that this file,/home/user/sandboxes/msb_ps8_0_36/data/localhost.log, 000001, was not even created.

user@localhost:~/sandboxes/msb_ps8_0_36/data$ ll | grep slow
-rw-r----- 1 adi adi 355518891 Aug 13 16:45 localhost-slow.log
-rw-r----- 1 adi adi 1079653421 Jul 30 18:13 localhost-slow.log.old
-rw-r----- 1 adi adi 255 Aug 13 16:48 slow
-rw-r----- 1 adi adi 255 Aug 13 16:45 slow.000001

After removing max_slowlog_size = 510000000

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+--------------------------------------------+
| Variable_name | Value |
+---------------------+--------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/slow |
+---------------------+--------------------------------------------+
1 row in set (0.01 sec)

Reported Affected Version/s: 5.7.36-39, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Workaround/Fix: Use “set global slow_query_log_file =’’;”

PS-9214: INPLACE ALTER TABLE might fail with a duplicate key error if concurrent insertions occur; there have been many bugs reported here and in MySQL bugs regarding duplicate key errors while doing an online alter table operation on tables with primary and unique keys indexes. The bug is not as easy to reproduce but involves ONLY the primary key and includes an atomic sequence that cannot create a duplicate. It seems to be related to page splits/merges.

Reported Affected Version/s: 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Upstream Bug: 115511

Workaround/Fix: Use ALTER TABLE … ALGORITHM=COPY instead.

PS-9275: When querying based on a function, MySQL does not use the available functional index when using the LIKE operator, which results inconsistent query plans when functional Indexes are used.

E.g.:

CREATE TABLE `test` (
 `id` int NOT NULL AUTO_INCREMENT,
 `a` varchar(200) DEFAULT NULL,
 `test02` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 `!hidden!test01!0!0` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 PRIMARY KEY (`id`),
 KEY `test01` ((monthname(from_unixtime(`a`)))),
 KEY `test02` (`test02`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE MONTHNAME(FROM_UNIXTIME(a)) Like 'April%';
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | ALL | NULL | NULL | NULL | NULL | 13 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE `!hidden!test01!0!0` Like 'April%';
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | range | test01 | test01 | 39 | NULL | 2 | 100.00 | Using where |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

Reported Affected Version/s: 8.0.36-28, 8.4.X

Upstream Bug: 104713

Workaround/Fix: Use the indexes created on virtual fields explicitly.

PS-9286: KMIP Component leaves keys in a pre-active state.

Reported Affected Version/s: 8.0.X, 8.3.0-1, 8.4.0-1

Fixed Version: PS 8.0.39-30, 8.4.2-2

PS-9314: The database crashed due to the SELECT statement. Since the JSON is invalid, the command should return ERROR 3146, an Invalid data type for JSON, but unfortunately, it crashed the instance with Signal 11 using JSON_TABLE.

Reported Affected Version/s: 8.0.36-28, 8.0.37-29, 8.0.39-30

Fixed Version: PS 8.0.39-30, 8.4.2-2

E.g.:

mysql> show global variables like 'version%';
+-------------------------+-----------------------------------------------------+
| Variable_name | Value |
+-------------------------+-----------------------------------------------------+
| version | 8.0.36-28 |
| version_comment | Percona Server (GPL), Release 28, Revision 47601f19 |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
| version_compile_zlib | 1.2.13 |
| version_suffix | |
+-------------------------+-----------------------------------------------------+
6 rows in set (0.01 sec)

mysql> SELECT ele AS domain FROM JSON_TABLE('["TEST'+(select load_file('test'))+'"]', "$[*]" COLUMNS (ele VARCHAR(70) PATH "$" )) AS json_elements ;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server

PS-9369: The audit plugin causes memory exhaustion after a few days; disconnecting threads and disabling the audit plugin is undesirable. This workaround can not be used since it requires scheduling an application outage. Even when small, it’s a recurrent event.

Reported Affected Version/s: 8.0.37-29

Fixed Version: PS 8.0.40-31 [Yet to Release]

Percona Xtradb Cluster

PXC-4453: In 3 Node PXC cluster, node01 has active flow control(FC). Active FC blocks user sessions to insert a message into the channel queue (session waits on send monitor (conn->sm)); send monitor is blocked because FC is active. The idea behind the logic is that applier threads, when consuming messages from the queue conn->recv_q, should check if FC is active, and if the queue level is below conn->lower_limit, FC should be disabled, and the user connection thread waiting on the sending monitor should be woken up. In other words, disabling the FC signal is driven by the consumption of events from recv_q by applier threads.

In this case, it seems that recv_q is empty, but FC is active, so nothing can be added to recv_q. We have a vicious circle of some kind of deadlock, and due to this race condition, we are seeing cluster hangs.

Reported Affected Version/s: 5.7.25, 5.7.44, 8.0.36-28

Fixed Version: PXC 8.0.37-29, 8.4.0

PXC-4404: wsrep_preordered=ON causes protocol violations, which cause a node to crash when the group view changes on a cluster with a node acting as an async replica.

Reported Affected Version/s: 5.7.44-31

Workaround/Fix: Set wsrep_preordered=OFF; however, you may experience a delay in async replication.

Note: Option wsrep_preordered is deprecated in MySQL-wsrep: 8.0.19-26.3, MariaDB: 10.1.1

PXC-4362: The PXC node evicted when creating a function by the user doesn`t have the super privilege, and binary logging is enabled.

Reported Affected Version/s: 8.0.34-26

Fixed Version: PXC 8.0.36-28, 8.4.0

Workaround/Fix: Setting log_bin_trust_function_creators is the workaround. Note that log_bin_trust_function_creators is deprecated by MySQL 8.0.34 and will be removed in the future.

PXC-4365: PXC nodes leave clusters when the row size is too large and have more than 3 nvarchar columns.

Reported Affected Version/s: 8.0.35-27

Fixed Version: PXC 8.0.36-28, 8.3.0

Percona Toolkit

PT-2325: pt-table-sync does not produce the correct SQL statements to sync tables containing JSON columns properly.

E.g.:

pt-table-sync emits the following SQL:

DELETE FROM `test`.`test_to` WHERE `id`='2' AND `data`='{"baz": "quux"}' LIMIT 1;

INSERT INTO `test`.`test_to`(`id`, `data`) VALUES ('1', '{"foo": "bar"}');

The INSERT statement works fine, but the DELETE fails to delete the row with id=‘2’, because the AND data=’{“baz”: “quux”}’ portion of the WHERE clause will result in the query matching zero rows.

Verify the incorrect contents of the test_to table with the following:

# Examine the state of our test tables.
$ docker exec -it mysql_5_7_12_test mysql -utest -ptest -e "
 use test;
 select * from test_to;"

That should return the following output:

+------+-----------------+
| id | data |
+------+-----------------+
| 2 | {"baz": "quux"} |
| 1 | {"foo": "bar"} |
+------+-----------------+

Witness that the row with id=2 still exists in the table and was not deleted as it should have been. With JSON columns, the DELETE statement would need to look like this:

DELETE FROM `test`.`test_to`
WHERE `id`='2'
AND `data`=CAST('{"baz": "quux"}' AS JSON)
LIMIT 1;

Reported Affected Version/s: 3.5.7

PT-2329: During the run, pt-archiver will ignore columns that are camelCase during the insert, but it will get all the columns during select.

It could be confirmed by using a dry-run:

pt-archiver --source [...] --dest [...] --where "1=1" --statistics --progress=10000 --limit=1000 --no-delete --no-safe-auto-increment --no-check-columns --columns=addressLine1,addressLine2,city,state,postalCode,country,customerNumber --why-quit --skip-foreign-key-checks --dry-run

Here are the results:

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) ORDER BY `customernumber` LIMIT 1000

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) AND ((`customernumber` > ?)) ORDER BY `customernumber` LIMIT 1000

INSERT INTO `classicmodels`.`addresses`(`city`,`state`,`country`) VALUES (?,?,?)

Reported Affected Version/s: 3.5.7

Workaround/Fix: The solution is to include all columns in lowercase in the param –columns until the bug is fixed.

PT-2344: pt-config-diff compares mysqld options, but it fails if the [mysqld] section is in uppercase, even though that is a valid way of setting mysqld variables. Since [MYSQLD] is acceptable for MySQL, pt-config-diff should compare the options under that section.

Reported Affected Version/s: 3.5.7

Workaround/Fix: Use [mysqld] as lowercase until the bug is fixed.

PT-2355: Table data is lost if we accidentally resume a previously failed job that has null boundaries. pt-online-schema-change should not resume a job with empty boundaries.

Reported Affected Version/s: 3.6.0

Fixed Version: PT 3.7.1

Workaround/Fix: Do not run pt-online-schema-change with job id having null boundaries.

PT-2356: If you run pt-online-schema-change, which results in an error, then subsequent runs will create new tables that won`t be cleaned up.

Reported Affected Version/s: 3.6.0

PT-2349: pt-table-sync is failing to sync data from PXC to the async environment, and trigger errors include “WSREP detected deadlock/conflict and aborted the transaction.”

Reported Affected Version/s: 3.3.1, 3.5.2, 3.6.0

PT-1726: pt-query-digest is not distinguishing queries when an alias is used

Reported Affected Version/s: 3.6.0

E.g.:

Queries from slow query log:

Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.b=5;

# Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.c=5;

The fingerprints for the above queries are the same, which is incorrect behaviour:

select a,b,c from table? as t? where t?=? and t?=?

PT-2374: If we say –ignore=bob, every combination of the bob user will be ignored. This includes bob@localhost, bob@::1, bob@foobar, etc. But this is not the case. Only bob@% is ignored; pt-show-grants –ignore does not ignore all accounts.

Reported Affected Version/s: 3.6.0

PT-2375: When pt-table-sync is used on a table with a GENERATED AS column, it fails because we cannot REPLACE/INSERT values into a GENERATED column.

E.g.:

`requestStatus` tinyint(1) GENERATED ALWAYS AS (if((`provRequired` = 0),(`httpSyncStatus` not between 200 and 299),(`httpAsyncStatus` not between 200 and 299))) VIRTUAL,

ERROR 3105 (HY000): The value specified for generated column 'requestStatus' in table 'qqq' is not allowed.

The —-ignore-columns parameter specifically states that if a REPLACE/INSERT is needed, all columns will be used. Due to this, the pt-table-sync does not work with the generated columns.

Reported Affected Version/s: 3.6.0

PMM [Percona Monitoring and Management]

PMM-12013: If we add many RDS instances to the PMM server, say 200+, and Change the prom scrape.maxScrapeSize to the value that allows the VM to parse the reply from the exporter, then the metrics are gathered unreliably, there are gaps, and the exporter`s RSS feed goes to like 5GB for instance. This concludes that rds_exporter is unreliable for large deployments.

Reported Affected Version/s: 2.35.0

Fixed Version: PMM 3.0.0-Beta available as Tech Preview]

PMM-12161: In the Mongodb cluster summary page, Under QPS of Config Services dashboard, it is being clubbed configRS, mongoS and mongod servers. This results in too many configuration services under the QPS of the Config Services dashboard.

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 3.1 [Yet to Release]

PMM-12993: In PMM, CPU metrics have a label “mode” to identify between CPU info: sys, iowait, nice, user, idle, etc. With 1 rds instance, the metric is perfectly fine. However, after adding more instances, the CPU metric is still collected, but the “mode” label is empty, which breaks the graphs in the Advanced Data Exploration dashboard.

Reported Affected Version/s: 2.41.1, 2.41.2

PMM-13148: If we run the queries without using the schema name, then we don`t see such queries in the QAN.

E.g.:

mysql [localhost:8036] {msandbox} ((none)) > update test.joinit set g=100,t="06:44:50" where i=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

Here, we can see we did not explicitly select the database name using the USE command and executed the query directly. This results in QAN not being able to capture such queries for analytics.

Reported Affected Version/s: 2.41.2

Workaround/Fix: Run queries with USE ; ;

PMM-13252: A 500 error message is returned while creating the role with the existing name.

E.g.:

Enable Access roles in PMM Settings
Open the Access role page and create a role with the name “Test. “
Try to create a new role with the name “Test. “

It returns with Internal server error 500:

in logs: msg="RPC /accesscontrol.v1beta1.AccessControlService/CreateRole done in 1.409839ms with unexpected error: pq: duplicate key value violates unique constraint "roles_title_key""

Reported Affected Version/s: 2.42.0

Workaround/Fix: It is expected to be fixed in PMM 3

PMM-13277: When we try to launch PMM using AWS AMI as mentioned in our docs. However, the AWS webpage works fine, and it logins, but every graph and details are blank with “Server error 502” The same can be seen in the log for Victoria metrics:

The following error will be seen:

2024-07-27T06:40:22.848Z panic /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/mergeset/part_header.go:88 FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory
panic: FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 2.43.0

Percona XtraBackup

PXB-3302: If the number of GTID sets is absolutely large on a MySQL instance, the output “GTID of the last change” in the Xtrabackup log is truncated compared to the full output in xtrabackup_binlog_info and xtrabackup_info. This can be an issue for external tools obtaining the GTID coordinates from the log as it would be impractical to get the coordinates from xtrabackup_binlog_info or xtrabackup_info on a large, compressed xbstream file.

Here is a snippet of a backup log:

2024-06-03T10:18:59.678581+08:00 0 [Note] [MY-011825] [Xtrabackup] MySQL binlog position: filename 'mysql-bin.000002', position '10197', GTID of the last change ** REDACTED **,9f18624b-214f-11ef-871f-b445068273a0:1,9f1bf5ae-214f-11ef-871f-b445068273a0:1,9f1f6fac-214f-11ef-871f-b445068273a0:1,9f231076-214f-11ef-871f-b445068273a0:1,9f26d153-214f-11ef-871f-b445068273a0:1,9f2a5fdd-214f-11ef-871f-b445068273a0:1,9f2df6e8-214f-11ef-871f-b445068273a0:1,9f318143-214f-11ef-871f-b445068273a0:1,9f353351-214f-11ef-871f-b445068273a0:1,9f38f96c-214f-11ef-871f-b445068273a0:1,9f3cdc53-214f-11ef-871f-b445068273a0:1,9f40fcc0-214f-11ef-871f-b445068273a0:1,9f44955a-214f-11ef-871f-b445068273a0:1,9f481188-214f-11ef-871f-b445068

Snippet of xtrabackup_binlog_info:

mysql-bin.000002 10197 ** REDACTED **,9fdd9048-214f-11ef-871f-b445068273a0:1,9fe138a3-214f-11ef-871f-b445068273a0:1,9fe4c3d8-214f-11ef-871f-b445068273a0:1,9fe82e39-214f-11ef-871f-b445068273a0:1

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-32

PXB-3283: When xtrabackup takes a backup and exports a tablespace, xtrabackup gets the wrong table definition from the ibd for tables that have changed the charset-collation in MySQL before backup.

Eg:

CREATE TABLE test.a (
 a datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_as_ci;

the collation_id is 8 (latin1_swedish_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When MySQL converts the charset on a table, it converts the date and time data types columns in the ibd file but not the data dictionary cache. The collation in the ibd does not match that of the data dictionary.

ALTER TABLE test.a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci;

The collation_id becomes 224 (utf8mb4_unicode_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 224

The collation_id of the copied table is 8 (latin1_swedish_ci)

create table xb.a like test.a;

shell> ibd2sdi /var/lib/mysql/xb/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When xtrabackup exports the tablespace, the collation_id is 224 in ibd. Xtrabackup will write it to cfg metadata file.

When MySQL imports a tablespace, MySQL gets an error Column %s precise type mismatch because the collation_id of MySQL does not match that of xtrabackup.

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-31

PXB-2797: When importing a single table (IMPORT TABLESPACE) from a backup made using xtrabackup and the table contains a full-text index, the import process will error out with:

ERROR 1808 (HY000) at line 132: Schema mismatch (Index xxxxxx field xxxxxx is ascending which does not match metadata file which is descending)

Reported Affected Version/s: 8.0.28-20

Fixed Version: PXB 8.0.35-31

PXB-3210: PXB fails to build on macOS since 8.0.33-28 due to FIND_PROCPS()

CMake Error at cmake/procps.cmake:29 (MESSAGE):
 Cannot find proc/sysinfo.h or libproc2/meminfo.h in . You can pass it to
 CMake with -DPROCPS_INCLUDE_PATH= or install
 procps-devel/procps-ng-devel/libproc2-dev package
Call Stack (most recent call first):
 storage/innobase/xtrabackup/src/CMakeLists.txt:24 (FIND_PROCPS)

Reported Affected Version/s: 8.0.33-28, 8.0.34-29, 8.0.35-30

Fixed Version: PXB 8.0.35-31

PXB-3130: Performing upgrade from PS 8.0.30 -> PS 8.0.33 using PXB

Use PXB 8.0.30 on PS 8.0.30
Copy to new host
Prepare using PXB 8.0.33
Start PS 8.0.33

Which results in the Assertion:

I0825 22:33:01.738917 05155 ???:1] xtrabackup80-apply-log(stderr) - InnoDB: Assertion failure: log0recv.cc:4353:log.m_files.find(recovered_lsn) != log.m_files.end()

Reported Affected Version/s: 8.0.30

Fixed Version: PXB 8.0.35-31

Percona Kubernetes Operator

K8SPXC-1398: Scheduled PXC backup job pod fails to complete the process successfully in a random/sporadic fashion.

Error Returns As:

+ EXID_CODE=4
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessfull'
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 4 (Interrupted system call)2024-05-03 09:39:08 [ERROR] Backup was finished unsuccessfull
+ exit 4

Reported Affected Version/s: 1.13.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Note: Since we don`t have steps to reproduce the issue, it is hard to confirm whether the fix is working as expected. Please feel free to provide feedback or create a Jira if required.

K8SPXC-1397: The operator`s default configuration makes the cluster unusable if TDE (Transparent data encryption) is used; the entry point of the PXC container configures the parameter binlog_rotate_encryption_master_key_at_startup. As a workaround, binlog_rotate_encryption_master_key_at_startup should be disabled. However, it has security implications.

Reported Affected Version/s: 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

K8SPXC-1222: Upgrading Cluster Fails When Dataset Has Large Number Of Tables. When the operator replaces the first pod with one with the new version, it fails to start up and gets stuck in a loop that restarts every 120 seconds.

The problem looks like from pxc-entrypoint.sh:

for i in {120..0}; do
 if echo 'SELECT 1' | "${mysql[@]}" &>/dev/null; then
 break
 fi
 echo 'MySQL init process in progress...'
 sleep 1
 done

Reported Affected Version/s: 1.11.0, 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Orchestrator

DISTMYSQL-406: Orchestrator 3.2.6-11 shows the MySQLOrchestratorPassword variable value in the error log and when accessing the web interface.

E.g.:

Create a MySQL and Orchestrator node

mysql -e "CREATE USER 'orchestrator_srv'@'%' IDENTIFIED BY 'orc_server_password'; GRANT ALL ON orchestrator.* TO 'orchestrator_srv'@'%';"

Configure Orchestrator to use node0 as MySQL backend database

vi /usr/local/orchestrator/orchestrator.conf.json

Add the following lines and remove sqlite options:

 "MySQLOrchestratorHost": "node_0_IP",
 "MySQLOrchestratorPort": 3306,
 "MySQLOrchestratorDatabase": "orchestrator",
 "MySQLOrchestratorUser": "orchestrator_srv",
 "MySQLOrchestratorPassword": "orc_server_password",

On node1, there are several messages showing the backend password:

Feb 28 23:26:03 XX-XX-node1 orchestrator[4262]: 2024-02-28 23:26:03 ERROR 2024-02-28 23:26:03 ERROR QueryRowsMap(orchestrator_srv:orc_server_password@tcp(10.124.33.138:3306)/orchestrator?timeout=1s&readTimeout=30s&rejectReadOnly=false&interpolateParams=true) select hostname, token, first_seen_active, last_seen_Active from active_node where anchor = 1: dial tcp 10.124.33.138:3306: connect: connection refused

Reported Affected Version/s: 8.0.36(PS)

Fixed Version: 8.4.0(PS)

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

About Percona:

As the only provider of distributions for all three of the most popular open source databases—PostgreSQL, MySQL, and MongoDB—Percona provides expertise, software, support, and services no matter the technology.

Whether its enabling developers or DBAs to realize value faster with tools, advice, and guidance, or making sure applications can scale and handle peak loads, Percona is here to help.

Percona is committed to being open source and preventing vendor lock-in. Percona contributes all changes to the upstream community for possible inclusion in future product releases.

Percona Monitoring and Management 2.43.0 Preview Release

Ondrej Patocka — Thu, 12 Sep 2024 00:00:00 UTC

Percona Monitoring and Management 2.43.0 Tech Preview Release

Hello everyone! Percona Monitoring and Management (PMM) 2.43.0 is now available as a Tech Preview Release.

We encourage you to try this PMM preview release in testing environments only, as these packages and images are not fully production-ready. The final version is expected to be released through the standard channels in the coming week.

To see the full list of changes, check out the PMM 2.43.0 Tech Preview Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.43.0-rc

PMM client package installation

Download AMD64 or Download ARM64 pmm2-client tarball for 2.43.0.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.43.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-0db618c7da6e202f4

Timeline for Database on Kubernetes

Edith Puclla — Tue, 16 Jul 2024 00:00:00 UTC

The Evolution

Since its inception in June 2014, Kubernetes has dramatically transformed container orchestration, revolutionizing the management and scaling of applications. To mark its tenth anniversary, the Data on Kubernetes Community (DoKC) released an infographic showcasing key milestones and community contributions to the evolution of operators for managing stateful applications. This project was made possible by the collaboration of DoKC members Edith Puclla, Sergey Pronin, Robert Hodges, Gabriele Bartolini, Chris Malarky, Mark Kember, Paul Au, and Luciano Stabel.

Explore the infographic to see how Kubernetes has shaped the future of database management on Kubernetes.

Adoption and Impact

The CNCF says that 84% of organizations are using or considering Kubernetes, with 70% running stateful applications on it in production. The number of users and containers has grown, showing that more people are contributing, adopting cloud-native technologies, and finding new ways to use Kubernetes to manage stateful applications.

Looking Ahead

As we celebrate ten years of Kubernetes, the way databases are integrated keeps improving, thanks to community efforts and new technology. Percona Everest is an excellent example of this progress. It’s more than just a tool for databases; it represents the future of running databases on Kubernetes. It’s open-source and makes running any database on cloud-based Kubernetes clusters easy. If you want to try it, visit our Percona Everest GitHub Repository and give us a star if you like it. For feedback or comments, join the Percona Forum for Percona Everest discussion.

Percona Joins Community Over Code 2024 in Bratislava, Slovakia

Edith Puclla — Fri, 21 Jun 2024 00:00:00 UTC

Last week, I participated as a speaker for the first time at Community Over Code 2024.

Community Over Code

Community Over Code is a key principle at Apache, highlighting the importance of having a solid and collaborative community rather than just focusing on the code. While good code is essential, the community’s strength and resilience keep a project going and growing. I love how this is expressed in the “Apache Way.”

Renaming ApacheCon to “Community Over Code” reflects this idea, emphasizing the central role of community in Apache’s approach. This year, Community Over Code was in Bratislava, Slovakia. My talk was in the Community track, where I spoke about Outreachy Internship, the Apache Software Foundation, and Apache Airflow. My goal was to celebrate the success of the Outreachy program, which has surpassed 1,000 internships, and I am proud to be one of them.

I talked about the community, shared success stories, and provided clear examples, such as Bowrna Prabhakaran and Ephraim Anierobi, and shared my personal experience. I also explained how Outreachy, along with my mentors from Apache Airflow, Jarek Potiuk, and Elad Kalif, boosted my professional career in Open Source 03 years ago.

Working at Percona

Now I work for Percona, which has been recognized as one of Inc. Magazine’s 2024 Best Workplaces! I love my job and the amazing things we are building at Percona. Percona is an open-source software company that fully believes that open source should remain open throughout. This belief inspires me to work at Percona.

What We’re Focused on Now

If you’re interested in improving database performance on Kubernetes, you will love Percona Everest. It is our cloud-native solution for efficiently managing and running databases on Kubernetes. With a very user-friendly interface, you can run any database on any cloud provider or on-premises.

Here are some use cases where you might use Percona Everest.

Seeking ways to make the development of internal platforms easier and faster.
Looking for affordable alternatives to public Database-as-a-Service (DBaaS) offerings.
Building multi-cloud or hybrid cloud setups to meet data compliance needs for multi-regional businesses.
Wishing to leverage Kubernetes for its scalability and stability to run databases efficiently.
Transitioning from monolithic to microservices architecture to modernize legacy database infrastructure.

Any feedback is welcome on our Percona Everest forum, and if you like it, give us a start on GitHub: github.com/percona/everest.

Let's take a look at Percona Everest 1.0.0 RC

Daniil Bazhenov — Fri, 14 Jun 2024 00:00:00 UTC

Hi, the Percona Everest 1.0.0-rc1 release was published on GitHub.

Percona Everest is the first open source cloud-native platform for provisioning and managing PostgreSQL, MongoDB and MySQL database clusters.

I want to tell you how to install it so you can try it out.

RC builds aren’t meant for the general public; we don’t support upgrading from RC to stable versions. This means that this is only for testing and familiarizing yourself with the features. RC builds are not stable and are often buggy. There will be no upgrade. :)

To get started, you will need a Kubernetes cluster. Right now, Percona Everest is in Beta. Don’t use production clusters; use test clusters in the cloud like GKE or local in Minikube, k3d, or Kind.

I created a test cluster in GKE with the command:

gcloud container clusters create test-everest-rc --project percona-product --zone us-central1-a --cluster-version 1.27 --machine-type n1-standard-4 --num-nodes=3

Delete it after the test with the command:

gcloud container clusters delete test-everest-rc --zone us-central1-a

Now, we need Everest CLI for the RC version; download it from GitHub for your operating system.

I downloaded it, renamed it to everestctl, and copied it to a folder for experimentation.

Now, we need to make it executable

chmod +x ./everestctl

Let’s check that everestctl works and that we have the correct version.

./everestctl version

We can now install Percona Everest.

./everestctl install --version-metadata-url https://check-dev.percona.com

Note that I use the --version-metadata-url parameter https://check-dev.percona.com; this is required for RC builds.

During the installation process, you must set one or more namespaces and databases.

Once the installation is complete, the new user authentication feature is the first significant change. You will be offered two commands.

Command to retrieve the admin user password that was generated automatically during installation:

./everestctl accounts initial-admin-password

Command to set a new password:

./everestctl accounts set-password --username admin

Now that we know the admin user password, we can open Percona Everest in a browser. Run the following command to use kubectl port forwarding to connect to Percona Everest without exposing the service:

kubectl port-forward svc/everest 8080:8080 -n everest-system

Take a Clone it will last longer

Wayne Leutwyler — Sun, 02 Jun 2024 00:00:00 UTC

So cloning is a great subject. I mean we clone sheep, we can clone human organs in time we might be able to clone humans, but thats a topic for scientist and philosophers.

What is MySQL Replicaion:

The MySQL clone plugin can be used to replicate data from a MySQL server to another MySQL server, and it supports replication. The cloning process creates a physical snapshot of the data, including tables, schemas, tablespaces, and data dictionary metadata. It tracks replication coordinates from the source server and transfers them to the replica, which allows replication to begin at a consistent position in the replication stream. This data includes the binary log position (filename, offset) and the gtid_executed GTID set. The replication metadata repositories are also copied during the cloning operation.

The clone plugin that was released with MySQL Version 8.0.17. Its quick and very easy to setup. You can use it for so many different solutions. I’ve listed some common ones below, but I know that there are many more use cases.

Create a new replica.
Recover a replica which is out of sync with the primary.
Quickly deploy MySQL servers with data set already in place.

In this article I will cover the basics of setting up and running a clone from and existing MySQL server. I will be using MySQL version 8.0.36.

Prepare the Source Server

Install the clone plugin.
Create a clone user.

Install Clone plugin

source > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

source > show plugins;

Create Clone User

Now we need to create a user to run the clone process. I highly suggest you create a user for the cloning. Please don’t use an ID with full admin rights. Create a user with the least amount of privileges needed.

source > CREATE USER 'clone_user'@'%' IDENTIFIED BY 'S3k3rtPassWd';
source > GRANT BACKUP_ADMIN ON *.* TO 'clone_user'@'%';

Verify your new clone user.

source > show grants for clone_user;

Preparation on the source server is complete. Lets move on to the clone.

Prepare the Clone Server

We need to start by installing the clone plugin.
Then we need to define the source that clone will be based on.

Install plugin

clone > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

clone > show plugins;

Define Source Server

You can use the source host name or host IP address.

clone > SET GLOBAL clone_valid_donor_list='SOURCE_HOSTNAME:3306';

Start and monitor the cloning process

We are now ready to kick off the cloning.

clone > CLONE INSTANCE FROM clone_user@SOURCE_HOSTNAME:3306 IDENTIFIED BY 'S3k3rtPassWd';

Cloning should start now. If you have any issues, check your log files and very the steps above. Now that the cloning is running we can monitor the cloning process using this command.

clone > SELECT * FROM performance_schema.clone_progress\G

As you can see in the output below, that the cloning has completed DROP DATA and is now working on FILE COPY.

Observations

Just recently I worked with a customer who moved a 5.6TB database from a Galera Cluster to standard MySQL replication. The data was moved from the source to replicas in two different locations. Timings are detailed below.

Source to Replica in same location: 5.6TB in approxmently 50 mins.
- 112GB per minute.
Source to Replica in different geographic locations: 5.6TB in approxmently 1 hour and 45 mins.
- 53.33GB per minute.

MySQL Clone Plugin’s robust performance and efficiency in managing large data transfers, making it an excellent tool for environments requiring quick and reliable data replication.

Summary

The MySQL Clone Plugin offers several benefits, including:

Fast Data Copying:
- Enables rapid cloning of MySQL instances, facilitating quick data replication and environment setup.
Consistent Data State:
- Ensures data consistency during cloning, avoiding issues that can arise from manual copying or inconsistent states.
Reduced Downtime:
- Minimizes downtime during cloning operations, crucial for maintaining service availability.
Ease of Use:
- Simplifies the cloning process through straightforward commands, reducing the complexity for database administrators.
Automated Cloning Process:
- Automates many steps involved in the cloning process, reducing the potential for human error and increasing efficiency.

Reference

The Clone Plugin

Image by kjpargeter on Freepik

Release Roundup May 15, 2024

David Quilty — Wed, 15 May 2024 00:00:00 UTC

Percona software releases and updates April 30 - May 15, 2024.

Percona is a leading provider of unbiased, performance-first, open source database solutions that allow organizations to easily, securely, and affordably maintain business agility, minimize risks, and stay competitive, free from vendor lock-in. Percona software is designed for peak performance, uncompromised security, limitless scalability, and disaster-proofed availability.

Our Release Roundups showcase the latest Percona software updates, tools, and features to help you manage and deploy our software. It offers highlights, critical information, links to the full release notes, and direct links to the software or service itself to download.

Today’s post includes releases and updates that have been released since April 29, 2024. Take a look.

Percona Distribution for MongoDB 6.0.15

Percona Distribution for MongoDB 6.0.15 was released on April 30, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Distribution for MongoDB 6.0.15

Percona Server for MongoDB 6.0.15-12

On April 30, 2024, we released Percona Server for MongoDB 6.0.15-12. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 6.0.15. It is based on MongoDB 6.0.15 Community Edition and supports the upstream protocols and drivers. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Server for MongoDB 6.0.15-12

Percona XtraDB Cluster 5.7.44-31.65.2

Percona XtraDB Cluster 5.7.44-31.65.2 was released on May 2, 2024. This release is part of MySQL 5.7 Post-EOL Support from Percona, and the fixes are available to MySQL 5.7 Post-EOL Support from Percona customers.

That’s it for this roundup, and be sure to follow us on Twitter to stay up-to-date on the most recent releases! Percona is a leader in providing best-of-breed enterprise-class support, consulting, managed services, training, and software for MySQL, MongoDB, PostgreSQL, MariaDB, and other open source databases in on-premises and cloud environments and is trusted by global brands to unify, monitor, manage, secure, and optimize their database environments.

How to Provision a MongoDB Cluster in Kubernetes with Percona Everest Summary

Edith Puclla — Thu, 02 May 2024 00:00:00 UTC

Kubernetes continues evolving, and the complexity of deploying and managing databases within the ecosystem is a topic of considerable discussion and importance these days. This article summarizes a detailed discussion between Piotr Szczepaniak and Diogo Recharte, who offer insights and live demonstrations to simplify database operations on Kubernetes with a new technology for cloud-native applications: Percona Everest. If you want to watch the full video, check out How to Provision a MongoDB Cluster in Kubernetes Webinar.

Peter mentions that Initially, people were doubtful about using virtual machines for databases, just like they were skeptical about Kubernetes. However, the topic brings together many people who run databases on containers to share their use cases and new discussions at events like Data on Kubernetes Day at Kubecon.

The introduction of StatefulSets and Persistent Volumes has altered the perception of Kubernetes from being purely ephemeral to being capable of handling persistent data. This change is important for database applications that require data retention over time.

The Kubernetes ecosystem is rapidly expanding. This growth is thanks to its open-source nature and the continuous addition of new functionalities, such as support for specialized hardware like GPUs, which are crucial for AI and machine learning applications.

Peter also mentioned that its complexity is the main barrier to Kubernetes adoption for databases. Organizations often need help with the layer added by Kubernetes on top of database management. Also, failure in initial attempts to integrate Kubernetes can discourage organizations from further attempts, primarily due to a lack of internal expertise.

Benefits of Database as a Service (DBaaS)

DBaaS significantly reduces the time required for database provisioning, which is particularly useful in organizations needing rapid deployment. Public and private DBaaS solutions offer scalability, which is crucial for handling varying workloads and organizational growth without compromising performance.

Private vs. Public DBaaS

Private DBaaS offers more extensive customization options and control over databases, which is essential for companies with specific needs that public solutions cannot meet.

Data security and compliance with regulations are more manageable in a private DBaaS because it operate within the company’s internal infrastructure.

Demo to Deploying MongoDB on Kubernetes

Diogo presented a demo of deploying a MongoDB database using Percona’s Everest platform on Kubernetes, where he showed how to handle daily operations and disaster recovery scenarios efficiently. Watch the Percona Everest Demo on YouTube

The session explained how Kubernetes operators and custom resources help manage databases more easily. They do this by simplifying complex processes and automating regular tasks.

Some questions that users asked in this presentation are:

How do we handle the PV when the Pods go down?

The PV will remain in place; this is a standard functionality of a stateful set. After the Pod goes down, the replacement Pod will attach to the PVC, which is standard behavior for a stateful set.

What happens if the node in Kubernetes goes down?

It depends on the storage layer that you have configured in your cluster. If the storage class you are using is tied to that node, then placing it on a new node will provision a new one, and some reconciliation will occur within the database itself.

What is the current state of Percona Everest?

Percona Everest is currently in a Beta stage, and Percona aims to release a GA version. The project is fully open source, and anyone can join our project on GitHub. We appreciate feedback from the community.

Do you want to send us feedback or contribute in this cool project? We are completely open-source, you can visit Percona Everest on GitHub.

Using ProxySQL Query Mirroring to test query performance on a new cluster

Isobel Smith — Wed, 01 May 2024 00:00:00 UTC

ProxySQL is an SQL aware proxy, which gives DBA’s fine grained control over clients’ access to the MySQL cluster. A key part of our DBA team’s process in testing and preparing for major MySQL version upgrades is comparing query plans using ProxySQL query mirroring. This feature allows us to mirror queries to another cluster / host, by configuring query rules. What makes mirroring particularly useful is the ability to selectively mirror queries based on the query digest, or client user. Results from the queries that are mirrored are not returned to the client, and are sent to /dev/null.

Before configuring ProxySQL for Query Mirroring, ensure that the clients that you want to mirror the queries for, are able to connect to both the current, and the new cluster. You should also ensure that the ProxySQL monitor can connect to the new cluster, otherwise ProxySQL will mark the new hosts as offline, and the queries will not be mirrored there.

To set up query mirroring in ProxySQL:

In order to set up query mirroring, you need to add the new hosts into the mysql_server table in ProxySQL. This is how the current mysql_servers table looks, before we add the new host that we want to mirror the queries to:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
+--------------+--------------+
4 rows in set (0.01 sec)

It is important to choose a hostgroup_id that is not yet in use. You can double check the currently configured host groups in the mysql hostgroups table, as you do not want to inadvertently add the mirror hosts into the production traffic!

MySQL> select * from mysql_replication_hostgroups;
+------------------+------------------+------------+----------------------+
| writer_hostgroup | reader_hostgroup | check_type | comment |
+------------------+------------------+------------+----------------------+
| 10 | 20 | read_only | Async Cluster |
+------------------+------------------+------------+----------------------+
1 row in set (0.00 sec)

Please note, in our example, we are using async replication, so we check the mysql_replication_hostgroups table, but the hostgroups table you need to check, depends on the cluster architecture you are using:

async replica clusters check the mysql_replications_hostgroups table.
galera clusters check the mysql_galera_hostgroups table
group replication check the mysql_group_replication_hostgroups table.

We are using hostgroup 10 for the writer hostgroup, and hostgroup 20 for the reader. For this example, we will choose 100 for the mirror hostgroup_id. Once you have decided on an unused hostgroup ID, add the new clusters’ nodes to the mysql_servers table in ProxySQL.

MySQL> INSERT INTO mysql_servers(host, hostgroup, comment) VALUES ("10.12.0.987", 100, "mirror_cluster");
LOAD MYSQL SERVERS TO RUN;
SAVE MYSQL SERVERS TO DISK;

The mysql_servers table will now include the new host:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
| 100 | 10.12.0.987 |
+--------------+--------------+
4 rows in set (0.01 sec)

In order to enable query mirroring, you need to update the mirror_hostgroup column in the mysql_query_rules table. When mirroring is not enabled, the value of the mirror_hostgroup column is NULL. Our query rules before enabling query mirroring are defined as:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+------------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+------------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | NULL |
+---------+------------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

To enable mirroring, we just need to update the mirror_hostgroup. For this example, we will mirror all the SELECT queries made by myApplicationUser:

UPDATE mysql_query_rules SET mirror_hostgroup = 100 where rule_id=2;
LOAD mysql query rules TO RUN;
SAVE mysql query rules TO DISK;

The rules should now be updated:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+-----------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+-----------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | 100 |
+---------+-----------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

The incoming queries that match the query rule, (in our example above, this is all queries as matching the regular expression ‘^SELECT’, for myApplicationUser, excluding queries matching ‘^SELECT.*FOR UPDATE’), will now be mirrored to the new cluster. You can verify this by checking the MySQL processlist on the new cluster.

The stats_mysql_query_digest table on ProxySQL holds statistics for the queries that are being processed by ProxySQL. To use the stats_mysql_query_digest table, the global variables mysql-commands_stats and mysql-query_digests must be set to true, which is the default.

Comparing query performance between two clusters

Query the stats_mysql_query_digest table to compare the performance per query between the current and the new cluster:

MySQL> select
 (b.count_star+a.count_star)/2 as count,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int) as percent,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int)*(b.count_star+a.count_star)/2 as load ,
 substr(a.digest_text,1,150)
from
 stats_mysql_query_digest a
inner join
 stats_mysql_query_digest b on
 a.digest = b.digest
where
 a.hostgroup = 10
 and b.hostgroup = 100
order by
 percent ASC;

In this example, the current production cluster has hostgroup 10, and the new mirror cluster was assigned hostgroup 100. The queries with a percentage above 100 are the queries that perform slower on the new cluster, and may be worth investigating, while the queries with a percentage below 100 are more performant on the new cluster. To investigate queries, you can compare the EXPLAIN plan of the query on the current and the new cluster. We use PMM Query Analytics to compare query analytics and the explain plan of the queries on the two separate clusters.

It is worth noting, that you should allow enough time for the MySQL buffer pool to get filled, before checking the stats_mysql_query_digest table. Otherwise, the query times on the new cluster can be skewed, as the active dataset may not yet be in memory (whereas on the current cluster it might be). Also, keep in mind that if you are mirroring only a subset of queries, the load on the new cluster will be different to the current cluster, and could affect the query performance on the new cluster, so that they appear significantly faster. Checking the execution plan of the query to see whether it has changed, is therefore more important than looking at overall load.

To conclude, using query mirroring to test queries on a new system, before making the migration, allows you to compare latency and query plans per normalised query, and proactively detect any necessary alterations before switching live traffic to the new cluster.

Release Roundup April 30, 2024

David Quilty — Tue, 30 Apr 2024 00:00:00 UTC

Percona software releases and updates April 17 - April 30, 2024.

Today’s post includes releases and updates that have been released since April 15, 2024. Take a look.

Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Distribution for MySQL 8.3.0-1 (PS-based variant) was released on April 16, 2024. This release is based on Percona Server for MySQL 8.3.0-1 and merges the MySQL 8.3 code base. It introduces the following changes:

Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).
PS-9044: Adds the following variables to MyRocks:
Changes the default values for the following variables:
- rocksdb_compaction_sequential_deletes from 0 to 14999
- rocksdb_compaction_sequential_deletes_count_sd from OFF to ON
- rocksdb_compaction_sequential_deletes_window from 0 to 15000
- rocksdb_force_flush_memtable_now from ON to OFF
- rocksdb_large_prefix from OFF to ON

Download Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Server for MySQL 8.3

On April 16, 2024, we released Percona Server for MySQL 8.3. It includes all the features and bug fixes available in the MySQL 8.3 Community Edition in addition to enterprise-grade features developed by Percona. This release merges the MySQL 8.3 code base. Within this merge, Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).

Download Percona Server for MySQL 8.3

Percona Distribution for MongoDB 7.0.8

Percona Distribution for MongoDB 7.0.8 was released on April 24, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Bug fixes and improvements provided by MongoDB are included in Percona Distribution for MongoDB. Note: a number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Distribution for MongoDB 7.0.8

Percona Server for MongoDB 7.0.8-5

Percona Server for MongoDB 7.0.8-5 was released on April 24, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 7.0.8. A number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Server for MongoDB 7.0.8-5

Percona Bug Report: April 2024

Aaditya Dubey — Mon, 29 Apr 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-9092: A query over an InnoDB table that uses a backward scan over the index occasionally might return incorrect/incomplete results when changes to the table (for example, DELETEs in another or even the same connection followed by asynchronous purge) cause concurrent B-tree page merges.

Reported Affected Version/s: 5.7.44, 8.0.35, 8.0.36

Upstream Bug: 114248

Workaround/Fix: Use descending indexes for the primary key. E.g.:

CREATE TABLE bugTest.testTable (key int unsigned, version bigint unsigned, rowmarker char(3) not null default 'aaa', value MEDIUMBLOB, PRIMARY KEY (key DESC, version DESC)) Engine=InnoDB;

PS-9107: A delete/insert into the secondary index is being change-buffered. This causes an insert into the ‘ibuf’ tree. There is a limit on the maximum size of the ibuf. So, on every ibuf insert, there is a compaction of ibuf tree (ibuf_contract). As part of ibuf_contract, we randomly open an ibuf page and apply the ibuf entries to the actual secondary index pages. After applying these ibuf entries from ibuf index, the ibuf tree goes on merging pages (optimistic vs pessimistic btree operations). To do this ibuf pessimistic delete on the tree, we save the cursor position, commit mtr and do a restore. This restore does a search and position again on the ibuf entry we were processing earlier, causing [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.cc:3833:ib::fatal triggered thread.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.37-29 [Yet to Release]

Upstream Bug: 114135

Workaround/Fix: Disable innodb_change_buffering

PS-9115: MySQL crashes due to getting a native index from get_mutex_cond in group replication, and before the crash, the following set of warnings/Errors is generated:

[Warning] [MY-011630] [Repl] Plugin group_replication reported: 'Due to a plugin error, some transactions were unable to be certified and will now rollback.'

[ERROR] [MY-011631] [Repl] Plugin group_replication reported: 'Error when trying to unblock non certified or consistent transactions. Check for consistency errors when restarting the service'

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Workaround/Fix: We can not guarantee that the bug will be avoided 100%, but shutting down/stopping group replication off-hours when the workload recedes should prevent the situation.

PS-9121: InnoDB updates the primary index but not the spatial index, which eventually corrupts the spatial index. The MySQL server crashes with “[ERROR] [MY-013183] [InnoDB] Assertion failure: row0ins.cc:268:!cursor->index->is_committed().” The update query changes the data from point(0.0000000000000099,0) to point(0.00000000000001, 0). When Innodb updates the record containing a spatial index, It updates the clustered index. This issue can be repeated using the following set of SQL statements.

CREATE TABLE a

 (

 id INT PRIMARY KEY,

 a GEOMETRY NOT NULL,

 SPATIAL KEY (a)

 )

ENGINE=InnoDB;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

UPDATE a SET a = Point(0.00000000000001, 0);

DELETE FROM a WHERE id = 1;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

Reported Affected Version/s: 8.0.36-28, 8.X [Innovative Release]

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Upstream Bug: 114252

Workaround/Fix: Please consider resetting the shape to a different value from the new values.

PS-9109: The Percona server’s slow query rate is not accurately logged, which is controlled via the Log_slow_rate_type and Log_slow_rate_limit variables. Due to this issue, the slow query log records every query regardless of these variables’ values.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

Percona Xtradb Cluster

PXC-4380: In a large cluster, for example, 15 nodes, in case one node has network issues and disconnects/re-connects/disconnects, the cluster might be sent to a non-primary state if the evs.install_timeout is reached.

Reported Affected Version/s: 5.7.42-31-65, 8.0.33-25, 8.0.35-27

Fixed Version: In such a big cluster, reaching consensus between nodes takes more time, so evs.install_timeout has to be adjusted. We can also configure the cluster to evict unresponsive nodes by setting evs.auto_evict=1. So, after investigation, we found that no fix is required, as everything works as expected/designed.

Workaround/Fix: Increasing the evs.install_timeout might fix the issue. The maximum value is 15S, which can still be reached depending on cluster size and how bad the flapping is.

The default value for evs.install_timeout is evs.inactive_timeout/2.

The minimum value is evs.join_retrans_period

The maximum time is evs.inactive_timeout + 1

So, If we keep defaults:

evs.join_retrans_period = 1s

evs.inactive_timeout = 15s

evs.install_timeout (default) = 7.5s

evs.install_timeout(min) = 1s

evs.install_timeout(max) = 16s

Please note that 15s is not a hard limit and is determined by evs.inactive_timeout

We have set wsrep_provider_options=“evs.install_timeout=PT15S” for all nodes in the test environment and are now unable to reproduce the issue. So, we are on some timeout boundaries, and the above configuration parameters were introduced to fine-tune in such environments.

PXC-4367: Innodb semaphore wait timeout failure seen after upgrade from 8.0.34 to 8.0.35. This issue is a possible side effect of this patch, Where PXC node acts as the async replica to some master and in parallel, the row is updated on PXC node and via replication, the PXC node hangs. To avoid the issue, upgrading to PXC 8.0.36 is recommended.

Reported Affected Version/s: 8.0.35-27

Fixed Version: 8.0.36-28

PXC-4363: Concurrent CREATE and DROP USER queries on different nodes lead to permanent lock.

| 16 | system user | db1 | Query | 411 | Waiting for table metadata lock | drop user IF EXISTS `msandbox_rw11`@`localhost` | 411380 | 0 | 

These queries cannot be cancelled or killed, and nodes refuse to be gracefully restarted. The shutdown is stuck with this forever. Only the forcible service kill helps restore the cluster.

2024-01-18T17:49:24.470428Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4399: FLUSH TABLES during writes to the table with unique keys stall the cluster node; due to the stall, it’s not possible to abort/kill any of the above connections. The node sends a permanent flow control pause. The only way to get out of this stall is to kill the node.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4418: In MySQL, the optimizer creates a temp table definition with indexes, where 2 of them have the same name (, , ). When the query is executed, a temp table is created, then MySql tries to access . In InnoDB, we search for index by name (dict_table_get_index_on_name()), which returns the wrong . Then row_sel_convert_mysql_key_to_innobase() crashes as structures are not aligned. Please note this bug affects only very complicated queries with many JOINs and subqueries. To repeat this bug, the internal temporary table needs to be created at least for two parts of the query.

Reported Affected Version/s: 8.0.32-24, 8.0.34-26

Fixed Version: 8.0.36-28

Percona Toolkit

PT-2190:The pt-show-grants use SHOW CREATE USER command to obtain grants from the MySQL server. By default, this query returns values as they are stored in the mysql.user table. When using caching_sha256_password, such hash of the password could contain a special character. Therefore, it would not be possible to use output printed by pt-show-grants to re-create users in the database. Since version 3.6.0, pt-show-grants checks if it runs against MySQL version 8.0.17 or higher and sets session option print_identified_as_hex to true before running SHOW CREATE USER command. This allows to print commands that could be used to re-create users.

Reported Affected Version/s: 3.5.1

Fixed Version: 3.6.0 [It is expected to be released soon]

PT-2215: pt-table-sync does not recognize the privileges in roles for MariaDB

Reported Affected Version/s: 3.5.2

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2316: pt-config-diff with –pid option is broken with “Can’t locate object method “make_PID_file” via package “Daemon” at /usr/bin/pt-config-diff line 5522” on Ubuntu 20.04

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2314: pt-online-schema-change fails due to duplicate constraint names when it attempts to make a table copy for alteration.

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

Workaround/Fix: Do not duplicate constraints name.

PT-2322: pt-mysql-summary does not detect jemalloc when installed as systemd.

Reported Affected Version/s: 3.5.6, 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PMM [Percona Monitoring and Management]

PMM-11583: On MySQL 8.0, innodb_redo_log_capacity supersedes innodb_log_files_in_group and innodb_log_file_size, which eventually breaks InnoDB Logging graphs. Due to this issue, the user can not determine whether the combined InnoDB redo log file size has to be increased or not.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-13017: For certain db.collection.find(query, projection, options) queries, the Explain tab for QAN returns an error message saying, “error decoding key command: invalid JSON input; expected value for 64-bit integer.” Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.35.0, 2.37.1, 2.41.1

Fixed Version: The fix is expected to be included in an upcoming release of PMM

PMM-12522: When adding data relatively large chunks of data to MongoDB sharded cluster, pmm-agent log starts flooded with “level=error msg="cannot create metric for changelog… & level=error msg="Failed to get database names:…” which eventually shows MongoS as disconnected in PMM UI (PMM-Inventory/Services).

Reported Affected Version/s: 2.39.0, 2.40.0, 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12880: pmm-admin –tls-skip-verify does not work when x509 authentication is used.

Reported Affected Version/s: 2.41.0

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12989: PMM agent logs flooded with wrong log entries when monitoring auth-enabled arbiters. Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

Percona XtraBackup

PXB-3251: When PXB fails to load the encryption key, the xtrabackup_logfile is still created in the target dir. This causes a second attempt at running PXB to fail with a new error. The xtrabackup_logfile file contains data needed to run the –prepare process. The bigger this file is, the longer the –prepare process will take to finish. So, PXB should not create any files on disk until the encryption key is loaded.

Reported Affected Version/s: 8.0.35-30

Fixed Version: The fix is expected to be included in future releases of PXB

Percona Kubernetes Operator

K8SPG-496: When a PostgreSQL Database is set to a paused state via spec, the operator waits until all backups for the Database finish. After the backups finish, the PostgreSQL Database shall be paused, which is not happening.

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-494: High vulnerabilities found for pgbackrest, Postgres & pgbouncer package.

For pgbackrest and postgres: CVE-2023-38408

For pgbouncer: CVE-2023-32067

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-521: The upgrade path described in the documentation leads to disabled built-in extensions(pg_stat_monitor, pg_audit).

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

K8SPG-522: Cluster is broken if PG_VERSION file is missing during the upgrade from 2.2.0 to 2.3.1.

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: In order to fix the issue, please do the following:

Create PG_VERSION with contents 14 in DB instance pod.

echo 14 > /pgdata/pg14/PG_VERSION
Apply crd and rbac

kubectl apply --force-conflicts --server-side -f crd.yaml

kubectl -n operatornew apply --force-conflicts --server-side -f rbac.yaml
Restart the operator deployment.

kubectl -n operatornew rollout restart deployment percona-postgresql-operator

At this moment, patronictl show-config starts to enlist extensions.

$ kubectl -n operatornew exec cluster1-instance1-gsvs-0 -it -- bash

Defaulted container "database" out of: database, replication-cert-copy, postgres-startup (init), nss-wrapper-init (init)

bash-4.4$ patronictl show-config

loop_wait: 10

postgresql:

 parameters:

 archive_command: pgbackrest --stanza=db archive-push "%p"

 archive_mode: 'on'

 archive_timeout: 60s

 huge_pages: 'off'

 jit: 'off'

 password_encryption: scram-sha-256

 pg_stat_monitor.pgsm_query_max_len: '2048'

 restore_command: pgbackrest --stanza=db archive-get %f "%p"

shared_preload_libraries: pg_stat_monitor,pgaudit

Now extensions appear in the \dx output without pod restarts, but all Postgresql servers will be restarted:

K8SPG-547: The pgbackrest container can’t use pgbackrest 2.50. This is because pgbackrest 2.50 requires libssh2.so.1, which requires epel. Without that fix, microdnf installs pgbackrest 2.48, which creates inconsistency with the Postgresql container.

Reported Affected Version/s: 2.2.0

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: This patch can be used until it is fixed.

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Download Percona Distribution for MySQL 8.0.36 (PXC-based variant)

Deploying Percona Everest on GCP with Kubectl for Windows 11 Users

Edith Puclla — Fri, 19 Apr 2024 00:00:00 UTC

Welcome to this blog post! Today, our primary goal is to guide you through deploying Percona Everest on GCP using Kubectl, specifically for users on Windows 11. It’s been some time since I last used Windows, so this will be an excellent opportunity to do it from scratch.

Let me tell you a little bit about Percona Everest. You may have already heard it recently. It is the new open source tool launched by Percona and is already well-received by Kubernetes database users.

Percona Everest is an open source cloud-native database platform that helps developers deploy code faster, scale deployments rapidly, and reduce database administration overhead while regaining control over their data, database configuration, and DBaaS costs. It is designed for those who want to break free from vendor lock-in, ensure optimal database performance, enable cost-effective and right-sized database deployments, and reduce database administration overhead.

If you use Windows and want to try the deployment and use of Percona Everest, you are in the right place.

This image shows what Percona Everest does and what we want to achieve:

Let’s start it!

Install WSL

We will use Kubectl to run commands on our Kubernetes clusters. There are many ways to use Kubectl on Windows.

I will use WSL (Windows Subsystem for Linux) to use the Linux environment directly on Windows. It is beneficial because “kubectl” and other Kubernetes tools often have better support.

In Windows 11, open PowerShell as Administrator and run:

wsl --install

This command will install WSL using the default options, including Ubuntu distribution and enabling the WSL 2 version.

Then restart your computer and open the newly installed Linux distribution from the Start menu. Complete the initial setup by creating a user account and password. Then update and upgrade your Linux distribution:

sudo apt update && sudo apt upgrade

Woolaa! We have Ubuntu running on Windows!

Installing WSL allows your Windows machine to run kubectl and other Linux-only applications smoothly. This setup is beneficial for developers and system administrators who work with both Windows and Linux systems.

Install Kubectl

In our Ubuntu terminal on Windows, we will use official documentation to install it using the native package management system.

bash
# Download the latest release with the command:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Test to ensure the version you installed is up-to-date:
kubectl version --client

Creating a Kubernetes Cluster in Google Cloud

To create a Kubernetes Cluster with GKE, you need to have access to Google Cloud. Ensure it functions correctly and that you can access your Google project and create Kubernetes clusters. Also, ensure you have the gke-gcloud-auth-plugin installed. You can check if this is installed by running the “gcloud components list” command. If it is not installed, follow the official documentation.

I have already set it up. Now, I will proceed to create my Kubernetes cluster.

bash
gcloud container clusters create percona-everest --zone europe-west2-c --machine-type n1-standard-4 --num-nodes=3

Install Percona Everest

A prerequisite for installing Percona Everest is having a Kubernetes cluster. I have one that I created with GKE. To verify the Kubernetes cluster, run the following:

bash
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-percona-everest-default-pool-1f7a9664-b3hd Ready  1h11m v1.27.8-gke.1067004
gke-percona-everest-default-pool-1f7a9664-b5c3 Ready  1h11m v1.27.8-gke.1067004
gke-percona-everest-default-pool-1f7a9664-nck4 Ready  1h11m v1.27.8-gke.1067004

Before running the commands in the Installation section, note that Everest will search for the kubeconfig file at the ~/.kube/config path

bash
export KUBECONFIG=~/.kube/config

The time to install Percona Everest. To install it, run the following command:

bash
curl -sfL "https://raw.githubusercontent.com/percona/everest/v0.9.1/install.sh" | bash

After installing it, you will see an output similar to the one on the left. In your browser, you can directly open 127.0.0.0:8080. Voilà! We now have Percona Everest up and running!

As the output indicates, the Percona Everest app will be available at http://127.0.0.1:8080. We use the authorization token to access the Everest UI and API.

We don’t have a database, so let’s create a new one!

This is the amazing thing about Percona Everest… you can create MySQL, MongoDB, and PostgreSQL databases on Kubernetes! Woohoo!!!

You can configure the resources for a new database, set up backups, monitoring, point-in-time recovery, and more:

And this is how it looks: your database is in Kubernetes!

Conclusion

Deploying Percona Everest on GCP using kubectl from a Windows 11 platform demonstrates the versatility and robust capabilities of managing databases on Kubernetes. The process should help you set up a powerful cloud-native database platform efficiently. We’ve walked through setting up your environment, installing necessary tools, creating a Kubernetes cluster, and finally deploying Percona Everest. Now, you can take full advantage of everything Percona Everest offers, from operational flexibility to cost efficiency.

If Percona Everest seems cool, feel free to contribute—it’s open source! Find Percona Everest on GitHub. If you encounter any issues during installation or have more questions, write to us in our community forum. If you prefer learning visually through videos, we have a friendly playlist of Percona Everest.

Release Roundup April 17, 2024

David Quilty — Wed, 17 Apr 2024 00:00:00 UTC

Percona software releases and updates April 2 - April 17, 2024.

Today’s post includes those releases and updates that have come out since April 1, 2024. Take a look.

Percona Distribution for MySQL 8.0.36 (PXC-based variant)

Percona Distribution for MySQL 8.0.36 (PXC-based variant) was released on April 3, 2024. It is the most stable, scalable, and secure open source MySQL distribution, with two download options: one based on Percona Server for MySQL and one based on Percona XtraDB Cluster. This release is focused on the Percona XtraDB Cluster-based deployment variation and is based on Percona XtraDB Cluster 8.0.36. Release highlights include improvements and bug fixes provided by Oracle for MySQL 8.0.36:

The hashing algorithm employed yielded poor performance when using a HASH field to check for uniqueness. (Bug #109548, Bug #34959356)
All statement instrument elements that begin with statement/sp/%, except statement/sp/stmt, are disabled by default.

Percona XtraDB Cluster 8.0.36

Percona XtraDB Cluster 8.0.36 was released on April 3, 2024. It supports critical business applications in your public, private, or hybrid cloud environment. Our free, open source, enterprise-grade solution includes the high availability and security features your business requires to meet your customer expectations and business goals.

Download Percona XtraDB Cluster 8.0.36

Percona Distribution for MongoDB 7.0.7

Percona Distribution for MongoDB 7.0.7 was released on April 4, 2024. It is a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together.

It includes the following components:

Percona Server for MongoDB is a fully compatible source-available, drop-in replacement for MongoDB.
Percona Backup for MongoDB is a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets.

Download Percona Distribution for MongoDB 7.0.7

Percona Distribution for MongoDB 5.0.26

April 9, 2024, saw the release of Percona Distribution for MongoDB 5.0.26. This release is based on Percona Server for MongoDB 5.0.26-22 and Percona Backup for MongoDB 2.4.1.

Download Percona Distribution for MongoDB 5.0.26

Percona Distribution for MongoDB 4.4.29

On April 2, 2024, we released Percona Distribution for MongoDB 4.4.29. This is the last minor release in Percona Distribution for MongoDB 4.4.

Download Percona Distribution for MongoDB 4.4.29

Percona Server for MongoDB 7.0.7-4

Percona Server for MongoDB 7.0.7-4 was released on April 4, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 7.0.5 and includes the improvements and bug fixes of MongoDB 7.0.6 Community Edition and MongoDB 7.0.7 Community Edition.

Download Percona Server for MongoDB 7.0.7-4

Percona Server for MongoDB 5.0.26-22

Percona Server for MongoDB 5.0.26-22 was released on April 9, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 5.0.x Community Edition. Percona Server for MongoDB 5.0.26-22 includes both improvements and bug fixes of MongoDB 5.0.25 Community Edition and MongoDB 5.0.26 Community Edition.

Download Percona Server for MongoDB 5.0.26-22

Percona Server for MongoDB 4.4.29-28

Percona Server for MongoDB 4.4.29-28 was released on April 2, 2024. This is the last minor release in Percona Server for MongoDB 4.4.

Download Percona Server for MongoDB 4.4.29-28

Release Roundup April 2, 2024

David Quilty — Tue, 02 Apr 2024 00:00:00 UTC

Percona software releases and updates March 18 - April 2, 2024.

Today’s post includes those releases and updates that have come out since March 18, 2024. Take a look.

Percona Monitoring and Management 2.41.2

Percona Monitoring and Management 2.41.2 was released on March 22, 2024. It is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL, and MongoDB. Starting with PMM 2.41.2, we now offer pmm-client packages for the latest version of Debian. You can install these packages by following the instructions in our documentation. We have also added several experimental dashboards, which are subject to change and recommended for testing purposes only.

Download Percona Monitoring and Management 2.41.2

Percona Operator for MySQL based on Percona Server for MySQL 0.7.0

Percona Operator for MySQL based on Percona Server for MySQL 0.7.0 was released on March 25, 2024. Percona Operator for MySQL allows users to deploy MySQL clusters with both asynchronous and group replication topology. This release includes various stability improvements and bug fixes, getting the Operator closer to the General Availability stage. Version 0.7.0 of the Percona Operator for MySQL is still a tech preview release and it is not recommended for production environments.

Download Percona Operator for MySQL based on Percona Server for MySQL 0.7.0

Percona XtraBackup 8.3.0-1

Percona XtraBackup 8.3.0-1 was released on March 26, 2024. Percona XtraBackup 8.3.0-1 is based on MySQL 8.3 and fully supports the Percona Server for MySQL 8.3 Innovation series and the MySQL 8.3 Innovation series. This release allows taking backups of Percona Server 8.3.0-1 and MySQL 8.3. This Innovation release is only supported for a short time and is designed to be used in an environment with fast upgrade cycles. Developers and DBAs are exposed to the latest features and improvements. Patches and security fixes are included in the next Innovation release instead of a patch release or fix release within an Innovation release. To keep your environment current with the latest patches or security fixes, upgrade to the latest release.

Download Percona XtraBackup 8.3.0-1

Percona Distribution for MongoDB 6.0.14

Percona Distribution for MongoDB 6.0.14 was released on March 26, 2024. It is a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together. Release highlights include:

Fixed the issue with missing peer certificate validation if neither CAFile nor clusterCAFile is provided.
Fixed the issue with multi-document transactions missing documents when the movePrimary operation runs concurrently by detecting placement conflicts in multi-document transactions.
Allow a clustered index scan in a clustered collection if a notablescan option is enabled.
Fixed tracking memory usage in SharedBufferFragment to prevent out of memory issues in the WiredTiger storage engine.
Added an index on the process field for the config.locks collection to ensure update operations on it are completed even in heavy loaded deployments.
Fixed the incorrect hardware checksum calculation on zSeries for buffers on stack.

Download Percona Distribution for MongoDB 6.0.14

Percona Server for MongoDB 6.0.14-11

Percona Server for MongoDB 6.0.14-11 was released on March 26, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 6.0.14. It is based on MongoDB 6.0.14 Community Edition and includes improvements and bug fixes provided by MongoDB.

Download Percona Server for MongoDB 6.0.14-11

Percona Backup for MongoDB 2.4.1

On March 25, 2024, Percona Backup for MongoDB 2.4.1 was released. It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set), and for restoring those backups to a specific point in time.

This release fixes the issue of failing incremental backups. It was caused by the backup metadata document reaching the maximum size limit of 16MB. The issue is fixed by introducing the new approach to handling the metadatada document: it no longer contains the list of backup files which is now stored separately on the storage and is read by PBM on demand. The new metadata handling approach applies to physical, incremental, and snapshot-based backups.

Download Percona Backup for MongoDB 2.4.1

Creating a Standby Cluster With the Percona Operator for PostgreSQL

Zsolt Parragi — Wed, 27 Mar 2024 00:00:00 UTC

In this video, Nickolay Ihalainen, a Senior Scaling Specialist at Percona Global Services, explains how to set up replication with standby clusters for Kubernetes databases using Percona’s open-source tools, including the Percona Operator for PostgreSQL

A Standby Cluster is a backup version of your main database. It’s there to keep your data safe and make sure your database can keep running even if something goes wrong with the main one.

For this demo, we use Percona Operators for PostgreSQL to create the clusters, which facilitates high availability setups and database management by automating deployment, scaling, and management tasks within Kubernetes environments.

Nickolay created a primary node and the configuration of replication to standby clusters for PostgreSQL, ensuring data redundancy and availability. This primary node has two standby databases in the same primary node.

Then, we have the object storage (S3) for backups, highlighting the importance of having offsite backups in different geographical locations to safeguard against data loss. This is where the primary node will access to make a data replication.

This demo also includes using Patroni to manage this process, enabling replication and failover between primary and standby servers, and PgBouncer, a tool that manages how applications are aligned to communicate with a PostgreSQL database.

Watch our complete hands on in this YouTube video:

You can find more instructions on how to deploy a standby cluster for Disaster Recovery, and also you can Create a Standby Cluster With the Percona Operator for PostgreSQL.

Is you have questions o feedback, write to us in our Percona Comunity Forum

Release Roundup March 18, 2024

David Quilty — Mon, 18 Mar 2024 00:00:00 UTC

Percona software releases and updates March 5 - March 18, 2024.

Today’s post includes those releases and updates that have come out since March 4, 2024. Take a look.

Percona Server for MySQL 5.7.44-49 (Post-EOL support version)

This release is Percona Server for MySQL 5.7.44-49 (Post-EOL support version), and the fixes are available to MySQL 5.7 Post-EOL Support from Percona customers. Community members can build this release from the source. Percona Server for MySQL 5.7.44-49 contains the fix for CVE-2024-20963 and a portability fix.

Download Percona Server for MySQL 5.7.44-49 (Post-EOL support version)

Percona Distribution for MySQL 8.0.36 (PS-based variant)

On March 4, 2024, Percona Distribution for MySQL 8.0.36 (PS-based variant) was released. It is the most stable, scalable, and secure open source MySQL distribution, with two download options: one based on Percona Server for MySQL and one based on Percona XtraDB Cluster. This release is focused on the Percona Server for MySQL-based deployment variation. This release fixes the Orchestrator issues.

Download Percona Distribution for MySQL 8.0.36 (PS-based variant)

Percona Server for MySQL 8.0.36

Percona Server for MySQL 8.0.36 was released on March 4, 2024. It includes all the features and bug fixes available in the MySQL 8.0.36 Community Edition, and enterprise-grade features developed by Percona. Improvements and bug fixes provided by Oracle for MySQL 8.0.36 and included in Percona Server for MySQL are the following:

The hashing algorithm employed yielded poor performance when using a HASH field to check for uniqueness. (Bug #109548, Bug #34959356)
All statement instrument elements that begin with statement/sp/%, except statement/sp/stmt, are disabled by default.

Download Percona Server for MySQL 8.0.36

Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0

Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0 was released on March 4, 2024. It contains everything you need to quickly and consistently deploy and scale Percona XtraDB Cluster instances in a Kubernetes-based environment on-premises or in the cloud. Among other new features, a custom prefix for Percona Monitoring and Management (PMM) allows using one PMM Server to monitor multiple databases, even if they have identical cluster names.

Download Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0

Percona Distribution for PostgreSQL 13.14

Percona Distribution for PostgreSQL 13.14 was released on March 6, 2024. It is a solution of a collection of tools from the PostgreSQL community that are tested to work together and assist you in deploying and managing PostgreSQL. A release highlight is that the Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 13.14

Percona Distribution for PostgreSQL 12.18

On March 11, 2024, we released Percona Distribution for PostgreSQL 12.18. This release of Percona Distribution for PostgreSQL is based on PostgreSQL 12.18.

Download Percona Distribution for PostgreSQL 12.18

Percona Backup for MongoDB 2.4.0

On March 5, 2024, Percona Backup for MongoDB 2.4.0 was released. It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set), and for restoring those backups to a specific point in time. A release highlight is that you can now delete backup snapshots of a specific type. For example, delete only logical backups that you might have created and no longer need. You can also check what exactly will be deleted with the new --dry-run flag. This improvement helps you better meet the organization’s backup policy and improves your experience with cleaning up outdated data.

Download Percona Backup for MongoDB 2.4.0

Release Roundup March 4, 2024

David Quilty — Mon, 04 Mar 2024 00:00:00 UTC

Percona software releases and updates February 21 - March 4, 2024.

Today’s post includes those releases and updates that have come out since February 20, 2024. Take a look.

Percona Distribution for PostgreSQL 16.2

Percona Distribution for PostgreSQL 16.2 was released on February 27, 2024. It provides the best and most critical enterprise components from the open source community in a single distribution, designed and tested to work together. This release is based on PostgreSQL 16.2. A release highlight is that a Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 16.2

Percona Distribution for PostgreSQL 15.6

On February 28, 2024, we released Percona Distribution for PostgreSQL 15.6, which is based on PostgreSQL 15.6. A Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Percona Distribution for PostgreSQL also includes the following packages:

llvm 12.0.1 packages for Red Hat Enterprise Linux 8 and compatible derivatives. This fixes compatibility issues with LLVM from upstream.
supplemental ETCD packages which can be used for setting up Patroni clusters. These packages are available for the following operating systems:

Download Percona Distribution for PostgreSQL 15.6

Percona Distribution for PostgreSQL 14.11

Percona Distribution for PostgreSQL 14.11 was released on March 1, 2024, and is based on PostgreSQL 14.11. A release highlight is a Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 14.11

Setting Up Your Environment for Kubernetes Operators Using Docker, kubectl, and k3d

Edith Puclla — Mon, 04 Mar 2024 00:00:00 UTC

If you are just starting out in the world of Kubernetes operators, like me, preparing the environment for their installation should be something we do with not much difficulty. This blog will quickly guide you in setting the minimal environment.

Kubernetes operators are invaluable for automating complex database operations, tasks that Kubernetes does not handle directly. Operators make it easy for us – they take care of essential tasks like backups and restores, which are crucial in database management.

If you want an introduction to Kubernetes Operators, I cover it in this 5-minute blog post, Exploring the Kubernetes Application Lifecycle With Percona.

Now that we know why Kubernetes Operators are essential let’s prepare our environment to install some of them. We are going to base this installation on Linux for now. Prerequisites: For this, we will need a basic understanding of Kubernetes concepts and some Linux command line skills. We also need Docker Engine to be able to use K3d at all for containerization. To test, make sure this command runs appropriately:

docker run hello-world

Installing kubectl

To manage and deploy applications on Kubernetes, we will need kubectl tool, which is included in most Kubernetes distributions. If it’s not installed, let’s do it following the official installation instructions:

To install the kubectl binary with curl on Linux, we need to download the latest release of kubectl using the command:

bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

The previous binary installs kubectl in /usr/local/bin/kubectl. We need root ownership and specific permissions for secure execution.

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

To test the installation, we use the following:

kubectl version --client

kubectl version --client --output=yaml

If you receive a response like this, it indicates that you are ready to use kubectl.

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Installing K3d

k3d is a lightweight tool that simplifies running k3s (Rancher Lab’s minimal Kubernetes distribution in Docker), enabling easy creation of single and multi-node k3s clusters for local development.

Install the current latest release of k3d with curl:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

To test the installation, you can use the following:

k3d --help

If you see a message similar to this, you are ready to create your k3d Kubernetes clusters.

bash
https://k3d.io/
k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker.
Nodes of a k3d cluster are docker containers running a k3s image.
All Nodes of a k3d cluster are part of the same docker network.
Usage:
k3d [flags]
k3d [command]
Available Commands:
cluster Manage cluster(s)
completion Generate completion scripts for [bash, zsh, fish, powershell | psh]
config Work with config file(s)

Starting the Kubernetes cluster

Let’s use K3d and create a Kubernetes cluster with three nodes. Using the flag -a, you can specify the number of nodes you want to add to the k3d cluster.

k3d cluster create database-cluster -a 3

Now, list details for our k3d cluster.

bash
k3d cluster list database-cluster

NAME SERVERS AGENTS LOADBALANCER
database-cluster 1/1 3/3 true

Now, our environment is ready to begin installing our Percona Kubernetes Operators.

Conclusion

In this tutorial, we chose k3d over Minikube due to its efficiency and speed in setting up Kubernetes clusters with multiple nodes, which are essential for effectively testing Kubernetes operators in a local environment. Although it’s possible to perform tests on a single node with both systems, k3d makes it easier to simulate a more realistic distributed environment, allowing us to utilize our resources more efficiently.

Take a look at our GitHub repository for our Percona Kubernetes Operators:

They are fully Open Source. And if you are looking for a version with a graphical interface, we have Percona Everest, our cloud-native database platform: docs.percona.com/everest

What’s Next? Let’s install our Kubernetes Operators!

Release Roundup February 21, 2024

David Quilty — Wed, 21 Feb 2024 00:00:00 UTC

Percona software releases and updates February 5 - February 21, 2024.

Today’s post includes those releases and updates that have come out since February 5, 2024. Take a look.

Percona Distribution for MySQL (PS-based variation) 8.2.0

Percona Distribution for MySQL (PS-based variation) 8.2.0 was released on February 5, 2024. It is a bundling of open source MySQL software enhanced with carefully curated and designed enterprise-grade features. Percona Distribution for MySQL offers two download options; this one is based on Percona Server for MySQL.

This release merges the MySQL 8.2 code base, introducing several significant changes:

Removes remains of Percona-specific encryption features (support for custom Percona 5.7 encrypted binlog format).
Removes the deprecated rocksdb_strict_collation_check and rocksdb_strict_collation_exceptions RocksDB system variables.

Download Percona Distribution for MySQL (PS-based variation) 8.2.0

Percona Distribution for MongoDB 6.0.13

Percona Distribution for MongoDB 6.0.13 was released on February 20, 2024. It includes the following components:

Percona Server for MongoDB is a fully compatible source-available, drop-in replacement for MongoDB.
Percona Backup for MongoDB is a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets.

This release of Percona Distribution for MongoDB is based on the production release of Percona Server for MongoDB 6.0.13-10 and Percona Backup for MongoDB 2.3.1.

Download Percona Distribution for MongoDB 6.0.13

Percona Distribution for MongoDB 4.4.28

Percona Distribution for MongoDB 4.4.28 was released on February 7, 2024. It’s a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together. In addition to bug fixes and improvements provided by MongoDB and included in Percona Server for MongoDB, Percona Backup for MongoDB 2.3.1 enhancements include the following:

Support for Percona Server for MongoDB 7.0.x
The ability to define custom endpoints when using Microsoft Azure Blob Storage for backups
Improved PBM Docker image to allow making physical backups with the shared mongodb data volume
Updated Golang libraries that include fixes for the security vulnerability CVE-2023-39325.

In addition, Percona Server for MongoDB 4.4.28-27 is no longer available on Ubuntu 18.04 (Bionic Beaver).

Download Percona Distribution for MongoDB 4.4.28

Percona Distribution for MongoDB 4.2.25

On February 8, 2024, Percona Distribution for MongoDB 4.2.25 was released. Release highlights include:

Optimized the construction of the balancer’s collection distribution status histogram
Fixed the query planner logic to distinguish parameterized queries in the presence of a partial index that contains logical expressions ($and, $or).
Improved performance of updating the routing table and prevented blocking client requests during refresh for clusters with 1 million of chunks.
Avoided traversing routing table in balancer split chunk policy
Fixed the issue that caused the modification of the original ChunkMap vector during the chunk migration and that could lead to data loss. The issue affects MongoDB versions 4.4.25, 5.0.21, 6.0.10 through 6.0.11 and 7.0.1 through 7.0.2. Requires stopping all chunk merge activities and restarting all the binaries in the cluster (both mongod and mongos).

Download Percona Distribution for MongoDB 4.2.25

Percona Server for MongoDB 6.0.13-10

Percona Server for MongoDB 6.0.13-10 was released on February 20, 2024. It is based on MongoDB 6.0.13 Community Edition and supports the upstream protocols and drivers.

Release highlights include:

Percona Server for MongoDB packages are available for ARM64 architectures, enabling users to install it on-premises. The ARM64 packages are available for the following operating systems:

Ubuntu 20.04 (Focal Fossa)
Ubuntu 22.04 (Jammy Jellyfish)
Red Hat Enterprise Linux 8 and compatible derivatives
Red Hat Enterprise Linux 9 and compatible derivatives

Download Percona Server for MongoDB 6.0.13-10

Percona Server for MongoDB 4.4.28-27

Percona Server for MongoDB 4.4.28-27 was released on February 7, 2024. It is a source available, highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 4.4.28 Community Edition enhanced with enterprise-grade features. Release highlights include these bug fixes, provided by MongoDB and included in Percona Server for MongoDB:

Fixed the issue with the data and the ShardVersion mismatch for sharded multi-document transactions by adding the check that no chunk has moved for the collection being referenced since transaction started
Improved cluster balancer performance by optimizing the construction of the balancer’s collection distribution status histogram
Fixed the issue with blocking acquiring read/write tickets by TransactionCoordinator by validating that it can be recovered on step-up and can commit the transaction when there are no storage tickets available
Investigated a solution to avoid a Full Time Diagnostic Data Capture (FTDC) mechanism to stall during checkpoint

Percona Server for MongoDB 4.4.28-27 is no longer available on Ubuntu 18.04 (Bionic Beaver).

Download Percona Server for MongoDB 4.4.28-27

Percona Server for MongoDB 4.2.25-25

Percona Server for MongoDB 4.2.25-25 was released on February 7, 2024. A release highlight is that Percona Server for MongoDB includes telemetry that fills in the gaps in our understanding of how you use Percona Server for MongoDB to improve our products. Participation in the anonymous program is optional. You can opt-out if you prefer not to share this information. Read more about Telemetry.

Percona Server for MongoDB 4.2.25-25

Percona Bug Report: January 2024

Aaditya Dubey — Mon, 19 Feb 2024 00:00:00 UTC

At Percona, we believe that transparency is key to improving our products. We are dedicated to creating top-of-the-line open-source database solutions and providing support for any issues that may arise. We encourage feedback and bug reports to help us continually improve.

We stay updated on bug reports through our own platform as well as other sources to ensure we have the most up-to-date information. To make it easier for you, we have compiled a central list of the most critical bugs for your reference in this edition of our bug report.

In this episode of our bug report, we provide the following list of bugs.

Percona Server/MySQL Bugs

PS-8983: System variable group_replication_view_change_uuid introduced in MySQL 8.0.26 which corrected the issue Bug#103641 in where data is inconsistent between nodes after killing primary node in group replication, However there is still an issue where these events are also generated on the standby/secondary cluster in a ClusterSet thus creating errant transactions, and if binlogs containing these events are purged, then it will not be possible to perform a failover between clusters.

Reported Affected Version/s: 8.0.26

Fixed Version: 8.0.31-23

PS-9048: When innodb_optimize_fulltext_only is enabled and running OPTIMIZE TABLE which has fulltext index actually causing assertion in Percona server debug build, Please note issue is specifically happening when PARSER is ngram.

Reported Affected Version/s: 5.7.42, 8.0.34 Fixed Version: N/A [Fix in Progress]

PS-9018: When replica has a non-replicated database then during intensive workload from the source where multi-threaded slave applier (MTS) is enabled and log_slave_updates=0 then DDL executed against this non-replicated database completely stalls the replica instance.

Upstream Bug: 113727

Reported Affected Version/s: 8.0.19+ Fixed Version: N/A [Fix in Review] Workaround : Use log_slave_updates=1, Please note enabling this may produce huge binlog volume on the replica which may or may not be feasible with respect to storage.

PS-9083: Percona server crashes when server is running with slow_query_log in conjunction with long_query_time, log_slow_verbosity = profiling,query_info variables.

Reported Affected Version/s: 8.0.35 Fixed Version: 8.0.36 [Pending Release] Workaround : Remove “query_info” from log_slow_verbosity

PS-9081: Materializing happens when a query is being executed against performance_schema.data_locks can lead to excessive memory usage and OOM.

Reported Affected Version/s: 8.0.34+ Fixed Version: It is expected to be fixed by PS 8.0.37 Workaround : Putting a LIMIT clause to read queries.

Percona Xtradb Cluster

PXC-4341: Execution of prepared statement after FLUSH TABLES makes the node abort from the cluster.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36 Workaround : There is no straight forward workaround but one can run the prepared statement and FLUSH TABLES statement separately.

PXC-4316: Network loss may lead to node’s logs flooded with “changed identity” events which eventually let primary node go non-primary, and reconnect another node. It will keep non primary nodes so we ended with all nodes as non primary.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36

PXC-4348: Cluster state interrupted with MDL BF-BF conflict when forcing deadlock. To hit the crash we are required to run queries on multiple sessions where one session should run “optimize table ;” multiple times so mysqlslap is the right candidate to repeat this behavior and other sessions will run delete/insert on the same table.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36

Percona Toolkit

PT-2217: When running pt-mongodb-summary against psmdb6.0/psmdb7.0 it gives error “BSON field ‘getCmdLineOpts.recordStats’ is an unknown field” Please note that PT tool does not work with MongoDB 6.0+.

Reported Affected Version/s: 3.5.X Fixed Version: It is expected to be fixed by PT 3.6.0

PT-2309: When the primary key is a UUID binary 16 column pt-table-sync hits with error “Cannot nibble table db_name.table_name because MySQL chose no index instead of the PRIMARY”

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.5.8

PT-2305: pt-online-schema-change should error out if server is a slave/replica in row based replication. This can lead to source/replica becoming inconsistent if there are writes on source when the tool runs on replica.

Please find the example below where data loss is seen:

Set-up classic source-replica
Make sure binlog_format=row
Create a table on master and add sufficient data so that pt-osc takes a little bit of time to run.
Then start pt-osc on slave, and execute updates/deletes on master.
Once pt-osc is done, check table checksum or table count to verify the data differences. Please check the below output with row differences:

master [localhost:22536] {msandbox} (test) > select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 999999 |
+----------+
1 row in set (0.58 sec)

slave1 [localhost:22537] {msandbox} (test) > select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
1 row in set (0.40 sec)

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.6.0

PT-2284: When running pt-kill with the –daemonize option, if the query has character like ‘柏木’, pt-kill process exists with message “Wide character in printf at /usr/bin/pt-kill line 7508.”

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.6.0 Workaround : Running pt-kill without –daemonize option manually.

PT-2089: When SHOW ENGINE INNODB STATUS reports garbled UTF characters then pt-deadlock-logger crashes with “server ts thread txn_id txn_time user hostname ip db tbl idx lock_type lock_mode wait_hold victim query”

Reported Affected Version/s: 3.3.1, 3.5.7

Percona Monitoring and Management (PMM)

PMM-12806: We can’t tune VictoriaMetrics running inside PMM since PMM does not honor the environment variables for VictoriaMetrics. So PMM pre-defines certain flags that allow users to set all other VictoriaMetrics parameters as environment variables.

Example:

To set downsampling, use the downsampling.period parameter as follows:

-e VM_downsampling_period=20d:10m,120d:2h

This instructs VictoriaMetrics to deduplicate samples older than 20 days with 10 minute intervals and samples older than 120 days with two hour intervals.

Reported Affected Version/s: 2,40.1 Fixed Version: It is expected to be fixed by PMM 2.41.2

PMM-12805: When monitoring MongoDB servers, logs might get filled with the a CommandNotSupportOnView message, As a result, disk space fills up.

Reported Affected Version/s: 2,40.0, 2.41.0 Fixed Version: It is expected to be fixed by PMM 2.41.2

PMM-12809: Common Vulnerabilities and Exposures (CVE) found in PMM gRPC(Remote Procedure Call (RPC)) which impacts PMM v2.40.1+

Reported Affected Version/s: 2.41.0+ Fixed Version: It is expected to be fixed by PMM 2.41.2

Percona XtraBackup

PXB-3024: Backups are not reliable when running on a secondary node of Group Replication(GR) since –lock-ddl does not have any effect on secondary node of GR.

Reported Affected Version/s: 8.0.28-20, 8.0.31-24 Fixed Version: 8.0.32-26

PXB-2928: Xtrabackup crashes with signal 11 when taking a backup using –page-tracking option. So if you are using this option while taking backup then upgrading to PXB 8.0.31 is recommended since there is no workaround available to this issue at the moment.

Reported Affected Version/s: 8.0.29-22 Fixed Version: 8.0.31-24

PXB-3037: In order to assure a consistent replication state, –safe-slave-backup option stops the replication SQL thread and waits to start backing up until slave_open_temp_tables in SHOW STATUS is zero. If there are no open temporary tables, the backup will take place, otherwise the SQL thread will be started and stopped until there are no open temporary tables. The backup will fail if slave_open_temp_tables does not become zero after –safe-slave-backup-timeout seconds (defaults to 300 seconds). The replication SQL thread will be restarted when the backup finishes, But due to this bug if backup fails in between then SQL thread is not getting restarted. So restarting the SQL thread manually is required.

Reported Affected Version/s: 8.0.31-24, 8.0.35-30 Fixed Version: No ETA

PXB-2733: backup-lock-timeout and backup-lock-retry-count do not work.

Reported Affected Version/s: 2.4.24, 8.0.27-19, 8.0.35-30 Fixed Version: No ETA

Percona Kubernetes Operator

K8SPG-492: Restore job created by PerconaPGRestore doesn’t inherit .spec.instances[].tolerations since restore Job pod get stuck in pending and causing down time.

Reported Affected Version/s: 2.2.0 Fixed Version: It is expected to be fixed by PG operator 2.4.0 Workaround: Remove taint, wait until the restore container is scheduled and re-add it again. K8SPSMDB-958: PMM fails to monitor mongos due to lack of permission.

Reported Affected Version/s: 1.14.0 Fixed Version: 1.15.0

K8SPG-291: Modifying existing backup schedule does not work with PG operator v1.3.0

Reported Affected Version/s: 1.3.0 Fixed Version: 1.4.0

K8SPG-286: When requiring TLS for all connections, PMM client fails to connect with “no pg_hba.conf entry”.

Reported Affected Version/s: 1.2.0, 1.3.0, 2.0.0 Fixed Version: 1.4.0

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Unexpected Stalled Upgrade to MySQL 8.0

Alexandre Vaniachine — Fri, 26 Jan 2024 00:00:00 UTC

A multi-tenant database is a database that serves multiple clients, or tenants, who share the same database schema but have separate data sets. One way to achieve data isolation for each client is to create a separate MySQL database for each tenant.

Some advantages of this approach are:

It allows for easy backup and restore of individual tenant data.
It simplifies the database administration and maintenance tasks, as each database can be managed independently.
Scaling is easily achieved by adding more database servers and distributing tenant databases across them.

This approach requires a large number of tables on each server. Combined with the default value of innodb_file_per_table=ON, this results in a large number of files that affects crash recovery time or Percona XtraBackup execution.

This blog post describes how to take care of a large number of files when upgrading to MySQL 8.0 in-place.

Version Selection

A steady stream of MySQL 8.0 minor releases provides improvements and refactoring of new MySQL 8.0 features. However, some of these releases introduce incompatibilities that require corresponding changes on the application side. Limiting scope of the application-side changes, we chose MySQL 8.0.25 version. This was our first step towards the major version 8.0.

Upgrade In-Place

A new MySQL 8.0 option is the upgrade in-place procedure. According to the upgrading guide:

An in-place upgrade is performed by using existing data on the server and involves the following actions:

Stopping the MySQL 5.7 server

Replacing the old binaries with MySQL 8.0 binaries

Starting the MySQL 8.0 server with the same data files.

While an in-place upgrade may not be suitable for all environments, especially those environments with many variables to consider, the upgrade should work in most cases.

As an exception, in the case of an environment with a large number of tables, the upgrade in-place may get stalled for weeks.

Below we describe how to debug and resolve such issue.

Encountering the Issue

In our test environment, we encountered a similar issue. Despite steady CPU usage, the in-place upgrade looks stalled. We monitored the upgrade progress by counting files modified in the last 24 hours. Monitoring revealed low modification rates, like

find /var/lib/mysql -name "*.ibd" -mtime -1 | wc -l
14887

that were decreasing. Although the InnoDB files continued to be modified, the decreasing modification rate was too low to be practical.

Investigating the Issue

To debug this problem we used the Linux perf tool. While the mysqld process was running during upgrade:

we collected perf data

perf record -F 10 -o mysqld.perf -p $(pidof mysqld) -- sleep 20;
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.256 MB mysqld.perf (1016 samples) ]

and produced the perf report

perf report --input mysqld.perf --stdio

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 1K of event 'cpu-clock'
# Event count (approx.): 101600000000
#
# Overhead Command Shared Object Symbol
# ........ ....... .................. ................................
#
 34.55% mysqld libc-2.17.so [.] __sched_yield
 14.86% mysqld [kernel.kallsyms] [k] __raw_spin_unlock_irq
 11.32% mysqld [kernel.kallsyms] [k] system_call_after_swapgs
 11.32% mysqld mysqld [.] Fil_shard::reserve_open_slot
 ...

To find out why mysqld process stuck in the Fil_shard::reserve_open_slot call, we checked the Percona Server source code that shows:

the function code

/** Wait for an empty slot to reserve for opening a file.
@return true on success. */
bool Fil_shard::reserve_open_slot(size_t shard_id) {
 size_t expected = EMPTY_OPEN_SLOT;

 return s_open_slot.compare_exchange_weak(expected, shard_id);
}

and the corresponding comments

The data structure (Fil_shard) that keeps track of the tablespace ID to
fil_space_t* mapping are hashed on the tablespace ID. The tablespace name to
fil_space_t* mapping is stored in the same shard. A shard tracks the flushing
and open state of a file. When we run out open file handles, we use a ticketing
system to serialize the file open, see Fil_shard::reserve_open_slot() and
Fil_shard::release_open_slot().

Apparently, the stalled upgrade process hit the open files limit, given the large number of files in our environment.

Resolving the Issue

To prevent the mysqld upgrade process from running out of open file handles, we followed Percona guidance for setting open files limit

Counted files as

find /var/lib/mysql/ -name "*.ibd" | wc -l
324780

and added another 1000 to this number for other miscellaneous open file needs.

Increased the innodb_open_files limit in two places:

added a corresponding line to configuration file /etc/my.cnf like

innodb_open_files = 325780

added a corresponding line to the systemd configuration file such as /etc/systemd/system/mysqld.service.d/override.conf like

[Service]
LimitNOFILE = 325780

With these adjustments our upgrade completed processing of a terabyte of data in just a few hours. To provide more visibility into the upgrade process we also increased the default level of error log verbosity by adding another line to the /etc/my.cnf file:

log_error_verbosity = 3

Increased verbosity enabled progress monitoring in the mysql error log during upgrade, like:

2023-10-28T00:27:09.331924Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.25-15) starting as process 16034
...
2023-10-28T00:27:09.353871Z 1 [System] [MY-011012] [Server] Starting upgrade of data directory.
2023-10-28T00:27:09.353986Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
...
2023-10-28T00:27:19.412572Z 1 [Note] [MY-012206] [InnoDB] Found 324780 '.ibd' and 0 undo files
2023-10-28T00:27:19.412757Z 1 [Note] [MY-012207] [InnoDB] Using 17 threads to scan 324780 tablespace files
2023-10-28T00:27:28.764032Z 0 [Note] [MY-012200] [InnoDB] Thread# 0 - Checked 15615/20298 files
...
2023-10-28T00:27:31.718051Z 0 [Note] [MY-012201] [InnoDB] Checked 20298 files
2023-10-28T00:27:31.718440Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 324780 files.
...
2023-10-28T00:27:48.821432Z 1 [Note] [MY-012922] [InnoDB] Waiting for purge to start
2023-10-28T00:27:48.878058Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-10-28T00:27:48.885203Z 1 [Note] [MY-011088] [Server] Data dictionary initializing version '80023'.
2023-10-28T00:27:49.187508Z 1 [Note] [MY-010337] [Server] Created Data Dictionary for upgrade
...
2023-10-28T01:57:55.312683Z 2 [System] [MY-011003] [Server] Finished populating Data Dictionary tables with data.
...
2023-10-28T01:59:15.187709Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80025' started.
...
2023-10-28T03:01:09.880932Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80025' completed.
...
2023-10-28T03:01:13.905459Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25-15' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 15, Revision a558ec2.

Discussion

While we were contemplating if this is a feature or a bug, MySQL release 8.0.28 refactored the related innodb_open_files code. Further details are provided in the corresponding open source commit WL#14591 InnoDB: Make system variable innodb_open_files dynamic:

- The `innodb_open_files` system variable can now be set with a dynamic SQL procedure `innodb_set_open_files_limit(N)`. If the new value is too low, an error is returned to client with the minimum value presented. If the value is out of bounds or of incorrect type, it will be reported as error also.
- `Fil_system::set_open_files_limit` was added to allow changes to the global opened files limit. The `Fil_system::m_max_n_open` is atomic now and extracted to a separate class `fil::detail::Open_files_limit`, instantiated as `Fil_system::m_open_files_limit`.
...
- `Fil_shard::reserve_open_slot`, Fil_shard::release_open_slot and static Fil_shard::s_open_slot were removed. Now we have CAS-based system of assuring the opened files will not exceed the limit set.

Thus, the new MySQL 8.0.28 feature – dynamic innodb_open_files variable – eliminated the need for open files limit adjustments in preparation for MySQL 8.0 upgrade.

Conclusions

Lessons learned:

Prepare for MySQL 8.0 upgrade in-place by taking a backup of the data directory.
Take advantage of the Percona Server open source code.
Follow guidance and advice posted in Percona blogs.

The content in this blog is provided in good faith by members of the open source community. Percona has not edited or tested the technical content. Views expressed are the authors’ own. When using the advice from this or any other online resource test ideas before applying them to your production systems, and always secure a working back up.

Our Top Picks from the Kubernetes 1.29 Release

Sergey Pronin — Fri, 12 Jan 2024 00:00:00 UTC

The latest Kubernetes version, 1.29, was released on December 13th 2023. Inspired by the Mandala and symbolizing universal perfection, it concludes the 2023 release calendar. This version comes with various exciting improvements, many of which will be helpful for users who run databases on Kubernetes.

Figure 1 - Mandala created in Excalidraw, not perfectly symmetrical.

Here, we highlight this latest release’s four key features and improvements. Let’s take a look at them together.

In-Place Update of Pod Resources

This alpha feature allows users to change requests and limits for containers without restarting. It simplifies scaling by a lot and opens new opportunities for auto scaling tools like HPA, VP, and Kubernetes Event-driven Autoscaling (KEDA). It removes the barriers of scaling the applications that were not easy to restart.

When resource resizing is not possible in-place, there are clear strategies for users and controllers (like StatefulSets, JobController, etc.) to handle the situation effectively.

It was first introduced in 1.27 but moved back to alpha as it requires additional architectural changes. It also has performance improvements and comes with Windows containers support. Read more about this in its Kubernetes Enhancement Proposals (KEP) and the issue #1287 created to add this feature.

Kubernetes VolumeAttributesClass ModifyVolume

The Kubernetes v1.29 release introduces an alpha feature enabling modification of volume attributes, like IOPS and throughput, by altering the volumeAttributesClassName in a PersistentVolumeClaim (PVC). This simplifies volume management by allowing direct updates within Kubernetes, avoiding the need for external provider API management. Previously, users had to create a new StorageClass resource and migrate to a new PVC; now, changes can be made directly in the existing PVC.

Discover further details in the KEP and issue #1287, which was established for the inclusion of this feature.

ReadWriteOncePod PersistentVolume Access Mode

Kubernetes offers access modes for Persistent Volumes (PVs) and Persistent Volume Claims (PVCs), including ReadWriteOnce, ReadOnlyMany, and ReadWriteMany. In particular, ReadWriteOnce restricts volume access to a single node, enabling multiple pods on that node to read from and write to the same volume concurrently. This setup ensures exclusive volume access on a per-node basis while allowing shared volume usage within the node. However, this introduces a potential issue, especially for applications that require exclusive access by a single pod. In this release, the ReadWriteOncePod access mode for PersistentVolumeClaims has become stable. Now that it is stable, a PVC can be configured to be mounted by a single Pod exclusively.

Here are the Kubernetes Enhancement Proposal (KEP) and issue #2485 that led to the development of this feature.

Make Kubernetes aware of the LoadBalancer behavior

kube-proxy’s handling of LoadBalancer Service External IPs is set to change. Traditional methods, such as IPVS and iptables, bind these IPs to nodes, optimizing traffic but causing issues with certain cloud providers and bypassing key Load Balancer features.

There are numerous problems with existing behavior:

Some cloud providers (Scaleway, Tencent Cloud, …) are using the LB’s external IP (or a private IP) as source IP when sending packets to the cluster. This is a problem in the ipvs mode of kube-proxy since the IP is bounded to an interface, and healthchecks from the LB is never coming back.
Some cloud providers (DigitalOcean, Scaleway, …) have features at the LB level (TLS termination, PROXY protocol, …). Bypassing the LB means missing these features when the packet arrives at the service (leading to protocol errors).

The solution would be to add a new field in the loadBalancer field of a Service’s status, like ipMode. This new field will be used by kube-proxy in order to not bind the Load Balancer’s External IP to the node (in both IPVS and iptables mode). The value VIP would be the default one (if not set, for instance), keeping the current behavior. The value Proxy would be used to disable the shortcut. This change allows more flexible handling of External IPs, maintaining current behavior as the default and offering an alternative to avoid these issues.

Read more about this in its Kubernetes Enhancement Proposals (KEP)

If you are interested in learning about databases on Kubernetes, you can start by running MySQL in Kubernetes. Explore the solutions, and weigh the pros and cons. Also, discover our predictions for Cloud Native technologies for this year.

Data on Kubernetes Community initiatives: Automated storage scaling

Sergey Pronin — Wed, 10 Jan 2024 00:00:00 UTC

In the world of Kubernetes, where everything evolves quickly. Automated storage scaling stands out as a critical challenge. Members of the Data on Kubernetes Community have proposed a solution to address this issue for Kubernetes operators.

If, like me, this is your first time hearing about Automated storage scaling, this will help you understand it better:

Storage scaling in Kubernetes Operators refers to the ability of an application running on Kubernetes to adjust its storage capacity automatically based on demand. In other words, it is about ensuring that an application has the right amount of storage available at any given time, optimizing for performance, cost, and operational efficiency, and doing this as automatically as possible.

As databases grow increasingly integral, the absence of unified solutions for storage scaling is becoming more evident. Let’s explore some existing solutions and their limitations:

pvc-autoresizer

This project detects and scales PersistentVolumeClaims (PVCs) when the free amount of storage is below the threshold. pvc-autoresizer It is and active open source project on GitHub.

There are certain downsides:

Works with PVCs only. It does not work with StatefulSet and does not have integration with Kubernetes Operator.
It requires Prometheus stack to be deployed.

Percona wrote a blog post about pvc-autoresizer to automate storage scaling for MongoDB clusters on Kubernetes.

EBS params controller

This controller provides a way to control IOPS and throughput parameters for EBS volumes provisioned by EBS CSI Driver with annotations on corresponding PersistentVolumeClaim objects in Kubernetes. It also sets some annotations on PVCs backed by EBS CSI Driver representing current parameters and last modification status and timestamps.

Find more about EBS params controller on GitHub.

Kubernetes Volume Autoscaler

This automatically increases the size of a Persistent Volume Claim (PVC) in Kubernetes when it is nearing full (either on space OR inode usage). It is a similar solution to pvc-autoresizer. Check out more about Kubernetes Volume Autoscaler

Kubernetes Event-driven Autoscaling(KEDA)

KEDA performs horizontal scaling for various resources in k8s, including custom resources. The metric tracking component is already figured out, but unfortunately, it does not work with vertical scaling or storage scaling yet. We opened an issue in GitHub to start the discussion.

As you can see, there are some limitations to performing Automated storage scaling, and to address this gap, the Data on Kubernetes community wants to develop a solution that solves practical problems and contributes to the open source community.

We’re tackling the significant challenge of unexpected disk usage alerts and potential system shutdowns due to insufficient volume space, a common issue in Kubernetes-based databases.

Possible Solutions

The following possible solutions were proposed:

Operators must be capable of changing the storage size when Custom Resource is changed.
Operators must create resources following certain standards, like applying annotations with indications of which fields should be changed
3rd party component (Scaler) will take care of monitoring the storage consumption and changing the field in the CR of the DB

Our goal as a community is to develop a fully automated solution to prevent these inconveniences and failures.

Final Thoughts

Once a new solution is validated and proven functional, it will benefit many communities, enabling them to integrate it with their operators. Additionally, it will present an excellent opportunity for Percona to incorporate it into our Operators, enhancing efficiency and facilitating automated storage scaling.

We invite those interested, especially in this particular project, to join us. This is an opportunity to be at the forefront of shaping the automated scaling solutions in Kubernetes. You can join the Data on Kubernetes community on Slack, specifically on the #SIG-Operator.

Are you interested in understanding Storage Autoscaling in databases? Explore our detailed example of Storage Autoscaling using the Percona Operator for MongoDB. For questions or discussions, feel free to join our experts on the Percona Community Forum

Volunteering as a Program Committee Member for Data on Kubernetes Day Europe 2024

Edith Puclla — Wed, 10 Jan 2024 00:00:00 UTC

The Data on Kubernetes Day Europe 2024 Program Committee is a group of professionals and experts responsible for organizing the Data on Kubernetes Day Europe 2024 content for the upcoming co-located events at Kubecon in Paris on 19 March.

As Data on Kubernetes community members, Sergey Pronin (Group Product Manager at @Percona) and I (Tech Evangelist) volunteered to evaluate proposal topics submitted for the event through the Sessionize platform. Not only us but also many other members of the Data on Kubernetes community participated as volunteers.

Community members who participate in this Program Committee evaluate proposals for talks, workshops, and other sessions submitted by potential speakers. This involves identifying each submission’s relevance, quality, and originality and being completely transparent and honest when reviewing a set of talks for the Data On Kubernetes Community co-located event.

Being a program committee means adhering to guidelines and following the Linux Foundation’s code of conduct.

Be professional and courteous.
Even express feedback constructively, not destructively.
Be considerate when choosing communication channels

After all program committee members completed their evaluations, the Data on Kubernetes Day Co-located Events Europe 2024 schedule was announced.

Look at this promising agenda:

We feel great to be a part of these efforts and to contribute to something significant by making it a reality at an in-person event. Thanks to the Linux Foundation for recognizing our efforts as Program Committee Members and for considering Percona, an active member of the DoK community.

In recognition of this support, we earned a badge from The Linux Foundation.

If you want to know more initiatives Percona have in the DoK community, read Data on Kubernetes Community initiatives: Automated storage scaling.

If you have any questions, remember to visit our Percona Community Forum.

Percona Bug Report: November 2023

Aaditya Dubey — Tue, 19 Dec 2023 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-8086 : Increased memory usage in LRU manager with ROW_FORMAT=COMPRESSED, so it seems that after evicting uncompressed frames for a compressed table, Percona Server LRU manager uses more memory to track them than upstream MySQL.

Reported Affected Version/s: 5.7.x, 8.0.26, 8.0.32

PS-8737 / [Bug #110706] : Data will be lost when you perform table rebuild immediately after INSERT or DELETE commands. This means that all ALTER TABLE operations that require table rebuild, including a “null” alteration; that is, an ALTER TABLE statement that “changes” the table to use the storage engine that it already has Eg: “ALTER TABLE t1 ENGINE = InnoDB;”, So after INSERT and DELETE, please do not execute table rebuild statement immediately.You can find the full list of ALTER operations that require table rebuild at https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html Check for column “Rebuilds Table”. All operations for which this column contains “Yes” or “No*” are affected.

Reported Affected Version/s: 8.0.[28/29/30/31/32]

PS-8428 : ALTER TABLE t ADD FULLTEXT crashes the server when –innodb_encrypt_online_alter_logs=ON. The problem has nothing to do with either innodb_encrypt_online_alter_logs, or Parallel Threads for Online DDL Operations. The issues turned out that in binaries built with OpenSSL 3.0.x my_aes_crypt() function has a flaw and can no longer decrypt data encrypted with the same function previously.

Reported Affected Version/s: 8.0.30-22

Fixed version: 8.0.30-22

Please don’t get confused with the affected & fix version is the same for this bug since this bug was reported internally during testing of the release build & that’s the reason it gets the same affected & fix version.

PS-8987 / [Bug #112935] : This bug results in inconsistency seen between MYISAM and MEMORY for simple CREATE and SELECT operation.

Please check the below scenario where result inconsistency is seen:

Added sample data to MyISAM/InnoDB Tables.
Executed SELECT statement which is a bit complex so can’t be added here but it can be seen in the bug report.
Empty set return when SELECT executed against MyISAM/InnoDB Tables & 4 rows return when same query executed against Memory Engine.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27

PS-8990 / [Bug #112979] : MySQL server does not respect system variable binlog_transaction_compression_level_zstd so it sets the compression level for binary log transaction compression on this server, which is enabled by the binlog_transaction_compression system variable. The value is an integer that determines the compression effort, from 1 (the lowest effort) to 22 (the highest effort). If you do not specify this system variable, the compression level is set to 3. As the compression level increases, the data compression ratio increases, which reduces the storage space and network bandwidth required for the transaction payload.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27

PS-9015 / [Bug #113256] : “DATA_FREE” shows a different value when comparing information_schema.TABLES vs information_schema.PARTITIONS. It is hard to say which result set is correct since we don’t have a source of truth.

Reported Affected Version/s: 5.7.43-47, 8.0.34-26

PS-9011 / [Bug #112946] : Prior to 8.0.29 INSTANT column exists on a non-system table with NULL columns in the MySQL SCHEMA which eventually leads to a corruption post 8.0.30+ upgrades. Although it’s probably not a best practice to create tables in mysql SCHEMA, it should not lead to corruption, especially when INSTANT is the default algorithm.

Percona Xtradb Cluster

PXC-4343 : Occasionally, during SST, InnoDB tablespace gets silently corrupted, resulting in the later Xtrabackup failure with the following error [MY-012224] [InnoDB] Header page contains inconsistent data in datafile. The triggering condition appears to be PXC 5.7 => 8.0 upgrade, where the corruption manifests in a 2nd node that joins later from the upgraded node. The corruption gets discovered once the 3rd node tries to join from the 2nd node as the donor or when a regular backup is taken from the 2nd node. To workaround this issue, always specify the first upgraded node as a donor.

Reported Affected Version/s: 8.0.34-26

PXC-4237 : When adding a new node, with PXC tarball installation error is being reported saying [WSREP] Failed to read 'ready ' from: wsrep_sst_xtrabackup-v2. However, this issue is expected to be fixed by the upcoming release of PXC 8.0.35, and fortunately, below workaround can fix the issue quickly:

Create /var/run/mysqld/ folder owned by mysql OS user.

Backup wsrep_sst_common file before editing:

shell> cp -nvp /usr/local/mysql/bin/wsrep_sst_common /usr/local/mysql/bin/wsrep_sst_common.orig.for-bug-PXC-4237

Implement the change:

shell> sed -i '297s/.*/set +e; & ; set -e/' /usr/local/mysql/bin/wsrep_sst_common

Verify the changes:

shell> sed -n '297p' /usr/local/mysql/bin/wsrep_sst_common.orig.for-bug-PXC-4237
 MYSQLD_PATH=$(readlink -f /proc/${WSREP_SST_OPT_PARENT}/exe)
shell> sed -n '297p' /usr/local/mysql/bin/wsrep_sst_common
set +e; MYSQLD_PATH=$(readlink -f /proc/${WSREP_SST_OPT_PARENT}/exe) ; set -e

Reported Affected Version/s: 8.0.32-24, 8.0.34-26

PXC-4318 : PXC cluster stalls and eventually crashes due to a long semaphore wait, which is happening because ha_commit_low does not commit a transaction that does not perform any changes such as an empty transaction and it can’t be controlled since it is an internal process. Fortunately, the upcoming release of PXC 8.0.35 will fix the issue.

Reported Affected Version/s: 8.0.33-25

PXC-4336 : PXC node eviction when a new CHECK CONSTRAINT is created which violates the condition, Eg. table is created with one entry say id 100 and after creation we added CHECK CONSTRAINT using ALTER TABLE t ADD CONSTRAINT CHK_id CHECK (id <=75); Here 100 is not less than 75 which violate the conditions and eventually node become inconsistent and get disconnected/evicted from the cluster. To avoid this scenario make sure to avoid violation of CHECK CONSTRAINT conditions.

Reported Affected Version/s: 8.0.34-26

PXC-4034 : When PXC cluster uses as a source and an async replication as a replica where “set @@session.sql_log_bin = off;” is used, this introduces a GTID gap in the “gtid_executed” set on the PXC source. As a workaround to the issue avoid using statement with “set @@session.sql_log_bin = off;” in the source/PXC

Reported Affected Version/s: 5.7.38-31.59, 8.0.28-19, 8.0.34-26

Percona Toolkit

PT-1724 : Percona toolkit unable to work if user using ‘caching_sha2_password’ Authentication plugin

Reported Affected Version/s: 3.0.13, 3.5.5

PT-2030 : pt-heartbeat is not compatible with PostgreSQL throwing Cannot get MySQL var character_set_server: DBD::Pg::db selectrow_array failed: ERROR: syntax error at or near “LIKE” LINE 1: SHOW VARIABLES LIKE ‘character_set_server’

Reported Affected Version/s: 3.3.1, 3.5.5

PT-2083 : when running pt-archiver with –charset option in MySQL 8.0 does not work.

Reported Affected Version/s: 3.3.1, 3.5.5

Fixed version: 3.5.6 [Pending Release]

PT-2106 : In pt-online-schema-change adding column to table (parent table) with having foreign key reference which triggers rebuilding constraints and can cause inconsistencies.

Reported Affected Version/s: 3.3.1, 3.5.5

PT-2207 : pt-archiver doesn’t work when ANSI_QUOTES is set in sql_mode

Reported Affected Version/s: 3.5.2, 3.5.5

Fixed version: 3.5.6 [Pending Release]

Percona Monitoring and Management (PMM)

PMM-4712 : PMM frequently crashes due to out of memory kills with postgres_exporter consuming 20-30GB of RAM and to debug it pprof endpoints to postgres_exporter was missing

Fixed version: 2.41.0 [Pending Release]

PMM-12013 : rds_exporter unreliable for large deployments which generate gaps in the gathered metrics and some improvement fix done here at PMM-11727

Reported Affected Version/s: 2.35.0

PMM-12349 : ReplicaSet Summary shows wrong data when a node is gone

Reported Affected Version/s: 2.40.1

PMM-12631 : Route of /logs.zip crashes with reflect: call of reflect.Value.NumField on string

Reported Affected Version/s: 2.40.1 Fixed version: 2.41.0 [Pending Release]

PMM-12738 : File certificate.conf is required, but not mentioned anywhere in helm charts but in fact, it should not be required.

Reported Affected Version/s: 2.40.1

Percona XtraBackup

PXB-2860 : Xtrabackup keeps locking table even using –tables-exclude and –lock-ddl-per-table.

Reported Affected Version/s: 8.0.33-28 Fixed version: 8.0.34-29

PXB-3168 : Under high write load, backup fails with “log block numbers mismatch” error

Reported Affected Version/s: 8.0.33-28, 8.0.34-29 Fixed version: 8.0.35-30, 8.2

PXB-3079 : Prepare skips rollback on encrypted tables and completes successfully if the keyring plugin is not loaded.

Reported Affected Version/s: 8.0.33-27 Fixed version: 8.0.34-29

PXB-3147 : Xtrabackup failed to execute query ‘DO innodb_redo_log_consumer_register(“PXB”); if sql_mode=’ANSI_QUOTES’ is used.’ This results in the Xtrabackup execution failure.

Reported Affected Version/s: 8.0.33-28 Fixed version: 8.0.35-30, 8.2

PXB-2954 : Xtrabackup failing with “[ERROR] [MY-011825] [Xtrabackup] innodb_init(): Error occurred” to prepare in case of orphan ibd

Reported Affected Version/s: 8.0.28-21 Fixed version: 8.0.32-26

Percona Kubernetes Operator

K8SPG-404 : Upgrade from percona PostgreSQL Operator 1.3 to 1.4 is ending up with a cluster without any replicas.

Reported Affected Version/s: 1.4.0 Fixed version: 1.5.0 [Pending Release]

K8SPG-420 : Ending up in multiple shared repo after cluster pause and unpause.

Reported Affected Version/s: 1.4.0 Fixed version: 1.5.0 [Pending Release]

K8SPG-435 : Pod is recreated when /tmp is filled

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

K8SPG-443 : Only english locale is installed, missing other languages support in Postgres

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

K8SPG-453 : pg_stat_monitor hangs primary instance and it’s impossible to disable it

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Percona Monitoring and Management 2.41 preview release

Ondrej Patocka — Wed, 06 Dec 2023 00:00:00 UTC

Percona Monitoring and Management 2.41 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.41 is now available as a preview release.

To see the full list of changes, check out the PMM 2.41 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.41.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.41.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.41.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-0a04085f4c721e913

The Importance of Anti-Affinity in Kubernetes

Edith Puclla — Thu, 30 Nov 2023 00:00:00 UTC

Last week, I embarked on the task of deploying our Percona Operator for MongoDB in Kubernetes. After completing the deployment process, I noticed that the status of the Custom Resource Definition for Percona Server for MongoDB was still displaying as ‘initializing’ and two of our Pods remained in a Pending state.

bash
edithpuclla@Ediths-MBP % kubectl get perconaservermongodbs.psmdb.percona.com -n mongodb
NAME ENDPOINT STATUS AGE
my-db-psmdb-db my-db-psmdb-db-mongos.mongodb.svc.cluster.local initializing 4m58s

bash
edithpuclla@Ediths-MBP % kubectl get pods -n mongodb
NAME READY STATUS RESTARTS AGE
my-db-psmdb-db-cfg-0 2/2 Running 0 109m
my-db-psmdb-db-cfg-1 2/2 Running 0 108m
my-db-psmdb-db-cfg-2 0/2 Pending 0 107m
my-db-psmdb-db-mongos-0 1/1 Running 0 106m
my-db-psmdb-db-mongos-1 1/1 Running 0 106m
my-db-psmdb-db-rs0-0 2/2 Running 0 109m
my-db-psmdb-db-rs0-1 2/2 Running 0 108m
my-db-psmdb-db-rs0-2 0/2 Pending 0 107m
my-op-psmdb-operator-77b75bbc7c-qd9ls 1/1 Running 0 118m

Upon further inspection of the pod in pending status, I discovered a clear indicator of the error:

bash
kubectl describe pod my-db-psmdb-db-cfg-2 -n mongodb
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal NotTriggerScaleUp 3m53s (x62 over 13m) cluster-autoscaler pod didn't trigger scale-up:
 Warning FailedScheduling 3m27s (x4 over 13m) default-scheduler 0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

I took a closer look at the YAML configuration of our CRD in the Replsets section, particularly drawn to the Affinity subsection.

kubectl describe perconaservermongodbs.psmdb.percona.com my-db-psmdb-db -n mongodb

Here’s what I discovered:

Affinity and Anti-Affinity are key parts of the scheduling process in Kubernetes, and both focus on ensuring that Pods are correctly assigned to Nodes in the cluster. You can configure a Pod to run on a specific node or group of nodes. There are several ways to achieve this, nodeSelector is the simplest way to constrain Pods to nodes with specific labels. Affinity and anti-affinity expand the types of constraints you can define and give you more flexibility.

Let’s explore what it means to specify anti-affinity rules for the ReplicaSets.

The key kubernetes.io/hostname is a well-known label in Kubernetes that is automatically assigned to each node in the cluster. It usually holds the value of the node’s hostname. When used as a topology key in anti-affinity rules, it implies that the rule should consider the hostname of the nodes. In simpler terms, it’s telling Kubernetes to not to schedule the pods of this ReplicaSet on the same physical or virtual host (node).

If we review our cluster, it has two nodes for installing the Percona Operator for MongoDB.

bash
edithpuclla@Ediths-MBP ~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-mongo-operator-test-default-pool-7c118de9-b9vc Ready  68m v1.27.4-gke.900
gke-mongo-operator-test-default-pool-7c118de9-ts16 Ready  68m v1.27.4-gke.900

In the context of databases like MongoDB, High availability is often achieved through replication, ensuring that the database can continue to operate even if one or more nodes fail. Within a MongoDB Replica Set, there are multiple copies of the data, and these copies are hosted on different Replica Set members. The default HA MongoDB topology is a 3-member Replica Set. Percona Operator for MongoDB deploys MongoDB in the same topology by default. With anti-affinity set to kubernetes.io/hostname, that means at least 3 Kubernetes worker nodes are needed to deploy MongoDB.

We created the minimum three nodes that the Percona Operator for MongoDB needs. We see that we don’t have the error with Antiaffinity in the Pods because each Pod was located appropriately in different nodes. Now our operator and our database were deployed correctly.

bash
edithpuclla@Ediths-MBP ~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-mongo-operator-test-default-pool-e4e024a8-1dj3 Ready  76s v1.27.4-gke.900
gke-mongo-operator-test-default-pool-e4e024a8-d6j2 Ready  74s v1.27.4-gke.900
gke-mongo-operator-test-default-pool-e4e024a8-jkkr Ready  76s v1.27.4-gke.900

If we list all the resources in our namespace, we can see that all pods are running properly and all the resources have been created.

bash
edithpuclla@Ediths-MBP ~ % kubectl get all -n mongodb
NAME READY STATUS RESTARTS AGE
pod/my-db-psmdb-db-cfg-0 2/2 Running 0 4m40s
pod/my-db-psmdb-db-cfg-1 2/2 Running 0 4m2s
pod/my-db-psmdb-db-cfg-2 2/2 Running 0 3m20s
pod/my-db-psmdb-db-mongos-0 1/1 Running 0 2m56s
pod/my-db-psmdb-db-mongos-1 1/1 Running 0 2m39s
pod/my-db-psmdb-db-rs0-0 2/2 Running 0 4m39s
pod/my-db-psmdb-db-rs0-1 2/2 Running 0 3m59s
pod/my-db-psmdb-db-rs0-2 2/2 Running 0 3m28s
pod/my-op-psmdb-operator-77b75bbc7c-q2rqh 1/1 Running 0 6m47s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-db-psmdb-db-cfg ClusterIP None  27017/TCP 4m40s
service/my-db-psmdb-db-mongos ClusterIP 10.72.17.115  27017/TCP 2m56s
service/my-db-psmdb-db-rs0 ClusterIP None  27017/TCP 4m39s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-op-psmdb-operator 1/1 1 1 6m47s

NAME DESIRED CURRENT READY AGE
replicaset.apps/my-op-psmdb-operator-77b75bbc7c 1 1 1 6m47s

NAME READY AGE
statefulset.apps/my-db-psmdb-db-cfg 3/3 4m41s
statefulset.apps/my-db-psmdb-db-mongos 2/2 2m58s
statefulset.apps/my-db-psmdb-db-rs0 3/3 4m40s

In conclusion, affinity and anti-affinity in Kubernetes are tools for strategically placing pods in a cluster to optimize factors such as performance, availability, and compliance, which are critical for the smooth and efficient operation of containerized applications, also setting up anti-affinity rules like failure-domain.beta.kubernetes.io/zone in Kubernetes is a key strategy for keeping clusters running, especially in production environments. This approach spreads pods across different availability zones, which means if one zone has an issue, the others can keep the system running. It’s a smart way to ensure your cluster can handle unexpected outages, making it a popular choice for those who need their Kubernetes setups to be reliable and available at all times.

Learn more about our Percona Operator for MongoDB, and if you have questions or comments, you can write to us on our Percona Community Forum.

Day 02: The Kubernetes Application Lifecycle

Edith Puclla — Mon, 20 Nov 2023 00:00:00 UTC

If you are in the world of application development, you know that every application has a lifecycle. An application lifecycle refers to the stages that our application goes through from initial planning, building, deployment, monitoring, and maintenance in different environments where our application can be executed.

On the other hand, the Kubernetes Application Lifecycle refers exclusively to applications deployed and managed in Kubernetes clusters. This differs from the normal application lifecycle because Kubernetes introduces new principles, practices, and tools for managing applications on containers.

In this blog post, we will talk about these phases Day 0, Day 1 and Day 2 in the lifecycle of an application in Kubernetes and we will focus specifically on the phase of Day 2.

Image 1: Day 0, Day 1 and Day 2 in the Kubernetes Application Lifecycle

Day 0 refers to the preparation stage before deploying applications in Kubernetes. It’s the stage of identifying goals, planning the infrastructure. Ensuring that the development team has knowledge about Kubernetes and best practices. It’s a stage for investment in training. And the evaluation of the application components to determine which are suitable for use within containers and Kubernetes.

Day 1 is the stage that involves deploying the application in Kubernetes clusters and the creation of Kubernetes resources: deployments, pods, services. Additionally, it includes configuration management and the implementation of basic monitoring following the decisions made on Day 0.

Finally Day 2, our application is already running in Kubernetes clusters by reaching this stage. Day 2 refers to the management, monitoring, and optimization of our Kubernetes clusters over the long term.

Day 2 involves:

Gathering information from our Kubernetes clusters through monitoring and logging.
Scaling our application, either horizontally or vertically.
Application of security best practices and compliance with policies.
Establishing backups and recovery processes to protect our data and application from future disasters.

Day 2, activities focus on sustainability, efficiency, and long-term continuous improvement to ensure the stability of our application and meet customer expectations.

Let’s see how Percona takes charge of Day 2

For Percona, a company specializing in the management of open source databases like MySQL, PostgreSQL, MongoDB, Day 2 refers to the ongoing efforts to ensure that database systems are running efficiently securely, and in alignment with business objectives.

Here are some examples of how Percona handles this phase:

To achieve Performance Monitoring, if you use our Percona Kubernetes Operators, you can integrate it with Percona Monitoring and Management (PMM) to check the performance of your databases in real time. Monitor query execution times, resource utilization, and server health. PMM helps to identify bottlenecks and inefficiencies, allowing for timely optimization and tuning.

Image 2: this is what the PMM Dashboard interface looks like when monitoring your database resources.

If we discuss data protection and disaster recovery, using Percona XtraBackup, an open-source backup utility for MySQL-based servers. In that case, you can ensure that your database remains fully accessible during scheduled maintenance periods.

As for scaling strategy and high availability, adopting solutions such as Percona XtraDB Cluster or Percona Server for MySQL enables us to secure the database and efficiently manage increased workloads, all while keeping downtime to a minimum.

These were just some examples of what Percona does for Day 2 to maintain tasks crucial for the business and that relies on databases to keep critical applications and services running.

Are you interested in learning more about Kubernetes or need assistance with your cloud-native strategy? With Percona Kubernetes Operators, you can manage database workloads on any supported Kubernetes cluster running in private, public, hybrid, or multi-cloud environments. They are 100% open source, free from vendor lock-in, usage restrictions, and expensive contracts, and include enterprise-ready features by default. Learn more about Percona Kubernetes Operators

Data On Kubernetes

Edith Puclla — Fri, 10 Nov 2023 00:00:00 UTC

If you’ve attended one of the Kubecon talks or related events, you’ve likely encountered the phrase Data on Kubernetes. To understand what this means, let’s explore some fundamental concepts related to Kubernetes, workload, stateless, and stateful applications.

Kubernetes, workload, stateless and stateful applications

Kubernetes is a container orchestration tool that has already become an industry standard. When we talk about “container orchestration”, we are referring to the automated management and coordination of containers using Kubernetes.

Now, let’s explore what a workload is in the context of Kubernetes. A workload represents an application running on Kubernetes. An application may consist of a single component or multiple components working together. These components are packaged into containers operating within a group of Pods.

There are two types of workloads, depending on the nature of the application: Stateless and Stateful.

In a stateless application, the client session data is not stored on the server. This is because the application doesn’t need to retain past interactions to function. However, in a stateful application, storing client session data is essential as it is necessary for subsequent interactions within the application.

Now, we are already familiar with Kubernetes, workloads, stateless and stateful applications, and we also understand that Pods are responsible for managing these types of workloads.

Built-in Workload Resources in Kubernetes

In a Kubernetes cluster, we can have thousands of Pods, and we don’t need to directly manage them individually. Instead, we utilize workload resources to manage a group of Pods and choose what workload resource depends on the type of workload we are dealing with, Stateless or Stateful.

For example, if we have stateless applications, we can use the Deployment and ReplicaSet resources, which are well-suited for this type of workflow. On the other hand, the StatefulSet resource allows us to run Pods that need to maintain state.

Data on Kubernetes refers to the management and storage of data within the Kubernetes ecosystem. Kubernetes provides a robust framework for handling data, making it a versatile platform for both stateless and stateful applications while ensuring data durability, availability, and security.

The challenge

Kubernetes was initially designed to run stateless applications. However, the number of stateful applications running on Kubernetes has increased significantly. There are many challenges when it comes to running applications with state in Kubernetes, such as data management strategies, volume persistence, and others. According to the 2021 Data on Kubernetes report of more than 500 executives and technology leaders, 90% believe it is ready for stateful workloads, and a large majority (70%) are running them in production, with databases topping the list. This gives rise to initiatives aimed at standardizing the requirements for managing stateful applications on Kubernetes.

This is how Data on the Kubernetes Community emerges. The Data on Kubernetes (DoKc) community was established in spring 2020. It is an openly governed group of curious and experienced practitioners, drawing inspiration from the Cloud Native Computing Foundation (CNCF) and the Apache Software Foundation. They aim to help create and improve techniques for using Kubernetes with data.

There are several organizations that are part of the Data on Kubernetes community, and Percona is part of it as well. Percona is adding efforts in DoKC Operator SIG(Special Interest Groups), where we discuss gaps in information around K8s operators for the industry-at-large & co-creates projects to fill the gap. Watch the Kubernetes Database Operator Landscape panel discussion to learn more about the community efforts in Data on Kubernetes.

Conclusion

Data on Kubernetes is a crucial concept in the Kubernetes ecosystem. Kubernetes was initially designed for the stateless application, now faces the challenge of managing stateful workloads, and is more notable in databases. The Data on Kubernetes (DoKc) community has emerged to address these challenges and standardize the management of stateful applications, drawing inspiration from industry standards like CNCF and Apache Software Foundation.

If you want to be part of them, you are welcome to join DoKC. Also, check this outstanding Kubernetes Database Operators Landscape, where members of DoKC talk about operations for data workloads on Kubernetes.

Exploring Kubernetes Operators

Edith Puclla — Fri, 03 Nov 2023 00:00:00 UTC

The concept of Kubernetes Operators was introduced around 2016 by the CoreOS Linuxdevelopment team. They were in search of a solution to improve automated container management within Kubernetes, primarily with the goal of incorporating operational expertise directly into the software.

According to the Cloud Native Computing Foundation, “Operators are software extensions that use custom resources to manage applications and their components”.

Kubernetes is designed for automation, offering essential automation features. It can automatically deploy and manage workloads. The definition provided by CNCF regarding operators, as mentioned above, highlights the flexibility we have to customize the automation capabilities made possible by Kubernetes Operators using custom resources.

In Kubernetes, certain applications require manual attention as Kubernetes can not autonomously manage them. This is especially the case with databases. This is where Operators come into play. Databases are complex entities that also have complex database operations and features that Kubernetes itself may not inherently understand. While deploying a database in Kubernetes manually isn’t a problem, the true strength of operators shines during Day 2 operations, which include tasks such as backups, failover, and scaling. Operators automate these manual tasks for applications within Kubernetes.

The main challenge that arises when implementing containerized databases is the problem of data persistence. This is the challenge for containers in general, and it is more critical in the context of databases despite ongoing advances in container maturity. Kubernetes operators are designed to address this gap. While it is possible to use Kubernetes resources like Persistent Volume Claims (PVCs) without operators, operators simplify the process by providing a higher level of abstraction and automation.

It is possible to create new operators using the Kubernetes operator pattern concept. This allows you to extend cluster behavior without modifying the Kubernetes code by linking controllers to one or more custom resources. These Operators use and extend the Kubernetes API, a key component within the Kubernetes architecture, with the essential concepts for users to interact with the Kubernetes cluster. They create custom resources to add new functionality according to the needs of an application to be flexible and scalable. This is how we automate workloads using Kubernetes Operators.

One of the primary benefits of operators is the automation of repetitive tasks that are often managed by human operators, eliminating errors in application lifecycle management.

Conclusion

In this article, we explored an overview of what Kubernetes Operators are; we saw why they are necessary and the benefits of using them. I hope you have gained a general understanding of why Kubernetes Operators are valuable.

If you want to know more about Kubernetes operators designed specifically for databases, you can visit the Percona website, where you will find Kubernetes operators created by Percona for MongoDB, PostgreSQL, and MySQL.

Are you considering creating your own operator? Start by using the Operator-SDK. Additionally, you can watch Sergey Pronin’s (Group Product Manager At Percona) talk at the DoK Community about Migrating MongoDB to Kubernetes, where he discusses the reasons why Percona created an Operator for MongoDB.

Building and Running Percona Everest From Source Code

Daniil Bazhenov — Mon, 30 Oct 2023 00:00:00 UTC

Digging deeper into the architecture of an open source product

Recently, Percona team announced the public alpha version of a new open source product – Percona Everest. It allows you to create database clusters on Kubernetes cluster.

I have installed Percona Everest several times and tried its features. Standard installation is very simple and takes a few minutes.

But, to understand the product deeper, I came up with the idea to explore repositories, build and run Percona Everest from source.

In this post, I will explain what I did step by step and what components and frameworks are used in the development of Percona Everest.

Architecture, components, and tools

At the top level, we have two components or tools on the user side: Percona Everest App and everestctl CLI tool.

The Percona Everest App is a basic application that provides a web interface for database creation and management functions. Percona Everest App can be installed on your computer or remote server. The App consists of two major components:

Frontend is a browser-based application providing a web interface for managing clusters and interacting with backend APIs. It is developed with React and TypeScript.
Backend API that process requests from frontend, interact with Kubernetes clusters and databases. It is developed on Golang and PostgreSQL as a database.

everestctl is a CLI tool for provisioning of Percona Everest on Kubernetes clusters. It is used to install Percona Everest components such as database operators on the Kubernetes cluster. It is developed in Golang and is provided as a ready-made executable, but, in this post, we will also build it from source code.

Remember that normally, when you install Percona Everest following the instructions in the documentation, the frontend and backend are built and integrated into a single container and run as a single unit.

Let’s get started with our experiments.

Frontend installation and launch

Percona Everest Frontend is developed using the Bit framework.

Bit is an open source toolchain for the development of composable software using React library and TypeScript.

Bit is used by about 100K developers, 250+ community plugins and has 16K+ stars on GitHub.

Clone the Frontend repository

You need to clone a repository with the Percona Everest frontend: https://github.com/percona/percona-everest-frontend

git clone git@github.com:percona/percona-everest-frontend.git

Install Bit

Bit requires the npm package manager to install. npm is a popular registry of JavaScript packages and libraries. The npm registry contains over 800,000 code packages. Open source developers use npm to share software. Installing npm on your operating system is straightforward. I’m sure you can handle it. It installs along with Node.js.

Open the Percona Everest Frontend source directory and run the following commands.

npm i -g @teambit/bvm

bvm install 1.0.0

bit install --recurring-install

Launching the frontend application

Moving forward, we have two options:

Run the frontend application using Bit.
Build a ready application and copy it to the backend.

The Percona Everest frontend repository contains versioning branches release-[version] and the current version in development in the main branch. We will run the latest dev version from main.

Let’s run the command:

bit run everest --skip-watch

As a result, we can open localhost:3000 in the browser.

The Frontend is now built, and we can move on to the Backend.

Additional information

There is the other way to build a frontend to work with backend. You will need two commands:

bit snap --build

bit artifacts percona.apps/everest --out-dir build

In this case, the build will be done to the folder:

build/percona.apps_everest/artifacts/apps/react-common-js/everest/public/

You need to copy all the files to the public/dist folder of the backend repository. We will talk about backend in the next section.

The installation process may change over time, so I recommend to keep track of the up-to-date commands in the files:

README.md
Makefile
CI/CD of GitHub configuration, file .github/workflows/ci.yml in the repository.

Backend

So we’ve launched Frontend, and now it shows an error because it sends requests to the Backend API, and we don’t have it yet.

We will need to clone the repository with Percona Everest Backend

https://github.com/percona/percona-everest-backend

Percona Everest Backend is developed in Golang using the Echo framework. The Echo repository has over 26k stars on GitHub.

Generally, it is an API that interacts with the frontend, processing requests and sending them to the Kubernetes cluster.

Let’s get it up and running.

Run PostgreSQL locally

You need Docker to run it. I hope you have Docker installed.

Let’s run one of the two commands in the repository directory:

make local-env-up

docker-compose up --detach --remove-orphans

Using Docker for this process will be replaced by Kubernetes. You can see the YAML manifest in the file:

/deploy/quickstart-k8s.yaml

Launch the Go app

We have two options, I use:

go run cmd/main.go

But you can also use:

make run-debug

Starting with version 0.4.0, you will need to add the SECRETS_ROOT_KEY environment variable before starting the application. The secret key must be used on restarts if you do not start from the scratch. export SECRETS_ROOT_KEY=$(openssl rand -base64 32)

Now we can open the localhost:3000 in the browser again and check that the backend is running. But we see that we have no Kubernetes clusters connected and configured.

Everestctl and Kubernetes cluster

Another important component of Percona Everest is everestctl. everestctl is a CLI tool responsible for provisioning Percona Everest on Kubernetes clusters.

We will need:

Kubernetes cluster
Build and run everestctl

Preparation of the Kubernetes Cluster

You can use Kubernetes Cluster on AWS, Google Cloud, or minikube.

The Percona Everest documentation says:

You must have a publicly accessible Kubernetes cluster to use Percona Everest. EKS or GKE is recommended, as it may be difficult to make it work with local installations of Kubernetes such as minikube, kind, k3d, or similar products.

The documentation provides instructions on how to run the test cluster

In this post, we use minikube which is preliminarily installed.

The Makefile of the Percona Everest Backend repository contains the make k8s command to start the cluster in minikube.

Let’s open the directory of the backend repository and launch minikube:

make k8s-macos

make k8s

As a result, I see in the console this message:

🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

I also see that I have a minikube cluster with three nodes. It is time to install Percona Everest components using everestctl.

➜ percona-everest-backend git:(main) ✗ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 3m44s v1.26.3
minikube-m02 Ready  3m22s v1.26.3
minikube-m03 Ready  3m3s v1.26.3

Build everestctl

Let’s clone the repository: https://github.com/percona/percona-everest-cli/

Open the repository folder and run build:

make build

everestctl is built in the binary file /bin/everest

Grant execution privileges

chmod +x ./bin/everest

Let’s run the Percona Everest installation:

./bin/everest install operators

That command will start the installation wizard.

I will leave all default values, just pressing Enter. Otherwise, you can experiment with the settings, it is your choice.

As a result, the following processes will run on the cluster:

Creating namespace percona-everest.

Installing Operator Lifecycle Manager (OLM).
Installing Percona OLM Catalog.
Installing Percona Operators for the databases selected in the wizard.
Installing everest-operator operator.
Creating services and roles.

That’s it! We’ve installed Percona Everest completely. You can open it in a browser and create a database.

What’s next?

Try creating databases with different configurations.

Repeat the installation with a different cluster or settings.

If you face any problems or have ideas on how to improve components, create Issues on GitHub in the appropriate repositories.

Stop and remove Percona Everest

Once you finished your experiments, you can:

Stop Frontend - stop Bit running by pressing CTRL+C in the console.
Stop Backend API - stop Golang script by pressing CTRL+C in the console.
Stop and remove PostgreSQL in Docker - run make local-env-down in the backend repository, or use Docker Desktop to stop.
Remove Kubernetes cluster.

Updates

Every day, developers make changes to the code and publish to repositories on GitHub.

You can stop the component, pull the changes with git pull, and start a new version. It’s just for experimentation and development. Some versions of components will not be compatible; uninstall all components and start over using the appropriate versions. Detailed instructions on how to upgrade will appear later.

You can see changes to the build process or parameters in the repositories.

Couple of useful commands

List of databases

kubectl -n percona-everest get db

List of pods

kubectl -n percona-everest get pods

Conclusion

I hope you were able to make it this far, and it was interesting for you.

I’d love if you leave your feedback in the comments.

Kubernetes Community Days UK: Keynote Cilium and eBPF

Edith Puclla — Tue, 24 Oct 2023 00:00:00 UTC

This week, at Kubernetes Community Days UK in London. Liz Rice, Chief Open Source Officer at Isovalent, delivered a keynote on Cilium, eBPF, and the new feature of Cilium: Mutual Authentication.

Figure 1. Liz Rice Keynote KCD UK, London. Tuesday 17, 2023

Cilium is an eBPF-powered open source, cloud native solution for delivering, securing, and observing network connectivity between workloads.

eBPF is a technology that allows us to create modules to modify the behavior of the Linux kernel, but why would we want to change the Linux kernel?

Some use cases for observability, security, and networking require tracking and monitoring our application, but we don’t want to constantly modify our application with these changes. It’s better to add a program that can observe the behavior of our application from the kernel.

But changing the Linux kernel can be, well, hard.

Figure 2. Adding features to the kernel (cartoon by Vadim Shchekoldin, Isovalent)

However, eBPF enables you to modify the kernel’s behavior without directly altering the kernel itself. It might sound unconventional, but eBPF makes this possible through the creation of programs for the Linux kernel. The Linux kernel accepts eBPF programs that can be loaded and unloaded as needed.

Figure 3. Adding kernel features with eBPF (cartoon by Vadim Shchekoldin, Isovalent)

To ensure that these eBPF programs written by us are secure, there is a mechanism in place that allows these programs to be verified safe for execution. This can be seen in eBPF verification and security. You don’t have to restart the kernel to deploy or remove eBPF applications, which makes eBPF one of the technology tools of the moment.

Liz also announced that Cilium recently graduated from CNCF. This means Cilium is considered stable and has been successfully used in production environments.

Figure 4. CNCF Project Maturity Levels

After understanding what eBPF is, let’s move on to the actual topic of Liz’s keynote. She spoke about Mutual Authentication with Cilium.

Mutual Authentication with Cilium was the last significant feature missing from Cilium Service Mesh. It’s a somewhat more complex topic related to mTLS (Mutual transport layer security).

mTLS is a mechanism that ensures the authenticity, integrity, and confidentiality of data exchanged between two entities in the network.

In Cilium 1.14, one of the most significant releases, Cilium introduces support for a feature that many developers have requested: mutual authentication. This feature simplifies the process of achieving mutual authentication between two workloads. It now only requires adding two lines of code to the YAML in the Cilium Network Policy to authenticate communication between two workloads.

Slightly more complex, isn’t it? Let’s explore Mutual Authentication with Cilium in a second blog post very soon. We’ll also examine how this is related to Kubernetes and why it matters when running databases on Kubernetes.

Check what the other Graduated and Incubating Projects are at CNCF, and don´t forget to subscribe to our Percona Community Blog to read more about Open Source, CNCF Projects, and Database.

Percona Monitoring and Management 2.40 preview release

Taras Kozub — Tue, 03 Oct 2023 00:00:00 UTC

Percona Monitoring and Management 2.40 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.40 is now available as a preview release.

To see the full list of changes, check out the PMM 2.40 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.40.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.40.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.40.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-09895e9b605f14cbc

Open Source And Recession – An Economic Outlook

Ninad Shah — Thu, 14 Sep 2023 00:00:00 UTC

Initially, I dropped an idea of writing this blog by reckoning the amount of time and energy required to perform research. However, my mentor Ramona Gerum motivated me and provided inputs to write this blog. This would not have been possible without her help. Thank you very much Ramona.

Introduction

In October 2022, the term recession gained popularity on Google trends. This happened as many economists already predicted a slowdown, and it started flashing on news channels world-wide. In the following year, many giant corporations, such as Google, Microsoft, Amazon and others, announced massive layoffs; this was followed by other chain of events from stock market crashes to high inflation rates across the globe. This development reinforced people’s belief about recession.

Majority of organisations who ran several rounds of layoffs claimed to have overstaffing issues as the hiring was done to anticipate the upcoming tide of work that never came, and they did not fire any technical/software experts. In such cases, open source products often rise as an alternative to proprietary ones. This blog will focus on the effects of a recession on open source products.

As this blog focuses on recession, it is important to understand some economic terms to understand this blog properly.

GDP(Gross Domestic Product):

This is the gross monetary value of goods and services produced within a country. Every country produces various products, such as food, heavy machinery, oil and others. Those commodities would give some value upon selling them in the market. The gross total of the values of all such goods is GDP.

Generally, it is calculated on an annual basis.

GDP Growth:

The change in GDP compared to the previous GDP figure is GDP growth. In general, economists calculate them on a quarterly and yearly basis as they represent the economic position of a country and predict its growth in the near and distant future.

Developed countries:

This one is a little complex as there are more than one key factors that are decision-makers. For a country to become a developed one, all the factors from GDP to standard of living and HDI(Human Development Index) should be taken into consideration. In terms of economics, countries with a GDP of 12,000 USD(some experts put it at 24,000 USD) per capita(person) are considered a part of the emerged world.

Boom:

A period of good economic growth: rising number jobs, increasing wages, increased profitability. This is characterized as a short stint of wealth creation and accumulation.

Recession:

A period when economic growth is very low or nil. In this period, higher unemployment, lower demand in the job market, higher inflation and layoffs are commonly observed in such a period.

In economics, negative GDP growth for 2 consecutive quarters is perceived as a recession, and if the condition of recession persists for 3 years, it is called an economic depression. However, economists analyze other indicators as well to decide if the recession prevails or not.

Inflation rate:

An average increase in percentage in the gross price of various commodities from the previously obtained number. For example, if the milk was being sold at 1.5 USD in 2018, and in 2019, it is being sold at 1.55 USD, the price rise is nearly 3.33% that is the inflation rate of milk for a year. Usually, the gross percentage is considered to determine the condition of a nation. It is one of the important indicators.

Ideally, 3-4% inflation rate is considered to be a healthy one.
Higher than 5% affects the livelihood of people and many people would be pushed below the poverty line.
While it is lower than 2%, it should be considered that the economy is in a reverse gear, which is not good for the growth of any country.

Third-world countries:

This term is often used in a pejorative manner; it was originally used for those countries who did not prefer to align themselves with either the USSR(socialist) or the USA(capitalist) in the era of the cold war that surfaced after the second world war. However, as they did not choose any side, they failed to grow like developed countries, hence the term “third-world countries” became synonymous with underdeveloped or developing countries.

Causes of a recession

A recession is caused by any disruption in an existing established economic flow. As a country grows, various channels are established through which money flows from sources to sources, and that is how money keeps flowing through the system. Whenever the rotation of money is affected, it gradually affects the economy of a country.

For instance, consider the below existing channel.

Farmers grow crop
During crop harvesting seasons, farmers employ additional people
Farmers sell goods to wholesalers
Wholesalers sell it further to retail stores and business houses
Retail stores sell it consumers, and business houses produce goods for consumer

Here, if a country is affected by a drought and there is no other method of farming available, people in the above hierarchy from farmers to employees of retail stores and business houses will become jobless for that year and the flow of money gets disrupted. This is to note that food is an essential commodity, and due to crop failure, the demand for food items exponentially rises and the prices of food items also skyrocket.

Here, just to understand, effects of recession will not be visible immediately after the drought; it takes time to surface. Although the above one is just taken as an example, this is not a completely hypothetical scenario.

As we saw a classical example of drought, there are various phenomena that can drive the economy to recession. It completely depends on the economical condition of a country.

In the third-world countries:

This group is already facing a number of internal problems and depends on prosperous countries. They may or may not have adequate resources, however if they have, they do not know how to utilise them efficiently. Also, due to internal clashes, poverty and economic instability, they witness recessions very frequently. Some evident factors for recession in such countries are

Drought
Heavy rainfall
Wars
Epidemic
International sanctions
Political instability
Uncontrollable inflation
Corruption and scams
Strikes

In the developed countries:

The developed world is often perceived as the utopia as processes are well-established there. The scope of recession appears limited, however this is just one side of the coin. There is a dialogue in the movie Wall Street: “Greed is good! Greed is progress!….”. Why I quoted this here is because greed is hardwired in the human gene, which propels humans to earn more money and exploit the system’s loopholes. Over the period, it causes an economic slowdown. Some of the driving factors for recession in such countries are

Policy loopholes
Stock market crash
Bad loans
Wars
Epidemic

Also, any economic slowdowns in influential countries resonate world-wide. Considering the classical case of the great recession of 2008, it started due to the housing bubble that resulted in the subprime mortgage crisis. In other words, due to sudden hikes in the prices of houses, banks started approving loan amounts with cheaper mortgage assets/commodities. For example, the loan amount is 100K USD, while the mortgaged item is worth 10K USD only. Also, people with poor credit rating got loans. This resulted into a burst of a bubble, which was an impetus to economic turmoil, millions of job losses and closure of businesses.

Repetitions And Durations Of Recessions

As described in the last paragraph, reasons for recessions may vary from country-to-country, also recessions in various countries may not necessarily coincide. Hence, the stint and repetition pattern for every recession are different.

Earlier, it was believed that recessions and booms are followed by each other. However, over the period, the political structure and law system in developed countries have evolved in such a way that their economies may stay resilient against recession.

Taking the case of the US, before the 1980s, the period of economic slowdowns were too frequent, nearly 3-5 years. However, since the 80s, the gap between the 2 recessions has significantly widened.

The snippet below taken from Investopedia shows the period of recession(“NBER Recessions” and “Length of Recession(Months)”) in the USA.

The condition of recession may not prevail across multiple countries at the same time, having said that, the great recession of 2008 was the first case of global recession, which cannot be considered as an exceptional one as it was the beginning of the era of globalisation. New waves of recessions will have global impact.

Performance of open source during and after recessions

During every recession, it has been observed that organisations lean towards less costly products, so that they can cut down the cost and maintain their profitability. Open source software emerges as a boon for them because they can save licensing cost for the product cost.

The amount required to acquire the license is very high. During a recession period, it is not at all an affordable option for companies, having said that, companies may run without new development work for a certain period and may eliminate some staff members, however platforms and databases are very crucial; hence, they cannot be eliminated as it is tantamount to shutting down a company.

Deploying open source products in an environment is the most viable option in such cases as they may continue running their businesses.

A survey was conducted by Linux foundations on OSS(Open Source Software) in which nearly 431 people, who were from “Fortune 500 group of companies”, working from middle-level to top-level management participated. They were asked various questions about open source and its benefits. When they were asked about adopting open source products, their answers are as mentioned in the chart below.

Source – Page 10

By analysing the chart, we can see surges in demand for open source in both the dot-com bubble burst(2001) and the great recession (2008). Besides, a sharp rise from 2016 can be noticed as well. However, it can be seen that the demand for open source has always been moderate.

In this section, it can be seen that open source software is resilient against recessions and keeps on flourishing during and after odd times as well.

Can Open Source Keeps Recessions At Bay?

This is a really interesting question that requires further research. Since the last few years, companies have become more optimistic about open source software products, such as MySQL, PostgreSQL, Kubernetes and so on. This is because these products provide all those features that meet industry standards and requirements. Over the period, open source databases, OS and web/app servers have evolved by working with various industries, and now, they meet all the requirements that companies give. Due to which, open source successfully carved a niche in the market.

The noticeable thing is that organisations do not need to pay a penny to use these products. Also, if one can afford to hire a small team of developers, the company comes up with its own version of the product. In case this is not the viable option, there are communities and a number of small-scale to medium-scale companies that provide support of open source.

If we can focus on the cost of products, it would give a more clear view on open source’s ability to stem recession.

The case of 2008’s great recession:

We all have heard tales of 2008 recessions: heavy pay cuts, job losses, suicides, loss of property and so on and on and on. As I mentioned in this blog earlier with an example, a disruption in the flow of money causes recession, and in the case of 2008, banks sanctioned loans that cannot be repaid. Due to which, banks were defaulted and it had badly hit the economy of the USA, which affected the whole world.

The most peculiar thing the Washington Post highlighted about its impact and losses.

Loss in the global economy – nearly 2 trillion USD
The US stock market loss – around 8 trillion USD
Gross individual losses – around 9.8 trillion USD

A hypothetical scenario with an open source in 2008:

Had it changed anything if organisations had placed open source products instead of proprietary software products back in 2008? As we can understand, the stock market loss and individual losses cannot be recovered due to this, because it has no relation with the product in the backend or frontend.

To understand this part, we have to check out some revenues. The revenue of Microsoft was nearly 52 billion USD in 2008, and if we start looking into revenues of other proprietary software producing organisations, their collective revenue may hardly reach half of the global economy loss(2 trillion USD). If we consider all the losses incurred due to proprietary products only, even in that case, open source could not have staved off the recession as the number itself is very huge.

As per Statitsa, the gross revenue of open source services was 32 billion USD in 2022. Considering this figure, it could have definitely helped mitigate the situation up to certain extent in 2008, and many of jobs could have been saved and many organisations might have avoided closures.

Open source’s role after 2008:

Due to loopholes in policy, the situation of economic turbulence arose, hence the US government took corrective actions and introduced a number of overhauls in economic policies, which made the economy more stable and immune to unwanted slowdowns.

Along with these things, open source gained popularity and percolated into different industries. Many companies have moved away from proprietary products and adopted open source products as they found them cost-effective with all the required features available. Of course, a penny saved is a penny earned. However, it seems no proper analysis was made to notice the economical impact of open source’s entry in the market. Also upon looking at the potential of open source a number of giant corporations, such as Google, Microsoft and others, have started contributing to the open source community.

But, if those people who contribute to open source are actually getting benefited? The answer lies in the below survey result.

Source – Page 13

Many economists predicted recession in the US between 2015-2018, which did not actually happen. Also, while I am writing this blog, experts have already announced a recession in 2023-24, which is yet to come true. Though I firmly believe that open source is pushing the recession back, it is difficult to prove it in absence of surveys and data. I hope some experts may pick this topic and make a detailed analysis on this part; I am sure it will give a great push to the open source community.

Conclusion

Booms and recessions were repeating on regular intervals earlier, however over the period, due to economic advancements, recessions became infrequent and started repeating after several years. It is not necessary that all countries may face economic slowdown at the same time, also reasons for slowdowns may differ as well. During such a period, organizations have to bear loss and fall in revenues, however open source becomes more adoptive during recessions. Also, it is possible that open source is keeping recessions away, but there is no data to back this claim.

Dolphie, your real-time MySQL monitoring assistant

Charles Thompson — Tue, 22 Aug 2023 00:00:00 UTC

For as long as I can remember, Innotop has been the go-to terminal tool for real-time MySQL monitoring. It is an invaluable addition to any DBA’s toolkit, but unfortunately, it’s not really actively maintained these days, except for addressing critical issues, and it hasn’t kept pace with the evolving capabilities of modern terminals. With no viable alternatives except for InnotopGo, which is also no longer actively maintained and limited to MySQL 8 (while many still use 5.7), I decided to build my own in Python.

I call it, Dolphie

Initially, I relied on Python’s Rich package for the user interface. However, I recently stumbled upon Textual a few months ago, and it piqued my interest. It’s a framework that extends the capabilities of Rich, opening up a world of possibilities in the terminal. After experimenting with it for a few days, it inspired me to redevelop Dolphie with it, and I’ve been thoroughly pleased with the results. It has allowed me to showcase many of the features that will be displayed in this blog post!

Getting started

When you first start Dolphie, you’ll be greeted with a dashboard displaying various important MySQL metrics, along with a sparkline below it to measure the QPS (Queries per second) + process list. There are multiple ways to manipulate the process list, such as changing how it sorts, filtering by user/host/query text/database/time, killing threads, and much more.

There are currently four panels that can be toggled interchangeably for display:

Dashboard
Process list
Replication/Replicas
Graph Metrics

A big perk of transitioning to Textual is the integration of graphs. It’s as if I’ve incorporated a mini-PMM (Percona Monitoring and Management) right into Dolphie! The switches you see can be toggled on and off to display or hide their corresponding metrics on the graph.

Buffer Pool Requests Graph + Replication Panel

Checkpoint Graph

Redo Logs Graph

How are your redo logs performing? Dolphie shows you how much data is being written per second, the active count of redo logs (MySQL 8 only), and how much data is being written to it per hour (inspired by this blog post)

DML Graph

Thread data

Dolphie lets you display a thread’s information with an explanation of its query along + transaction history

Kill threads

Dolphie lets you terminate threads using a selected option. Notice how it autocompletes the input for you. This is a feature across the board. It will autocomplete any input that it can

Quick switch host

After using Dolphie extensively myself, I realized the need to simplify host switching. I found myself restarting it frequently just to change the host. This feature saves all the hosts you’ve connected to, allowing for autocomplete when you want to switch

Error log

In MySQL 8, I was delighted to see that the error log was in performance_schema. Of course, I had to support it! It has switches to toggle on/off event types and search functionality

Errant transactions

The Replicas panel will let you know if your replicas have any errant transactions and what they are

These are just some of the features that Dolphie has. There are many more that I haven’t covered, which you can discover for yourself and try out!

If you’d like to try Dolphie, it’s just a pip away:

pip install dolphie

I’m open to feedback and suggestions so don’t be a stranger :) If you’d like to contribute to the project, I’d be delighted to have you!

You can find Dolphie on its GitHub

PMM Client on Raspberry Pi 4

Wayne Leutwyler — Tue, 15 Aug 2023 00:00:00 UTC

This will be the third in my series of Percona Products on a Raspberry Pi. My previous posts:

Before I get started I would like to thank guriandoro, for the work he did in compiling the PMM Client tools in his 2021 blog.

My regular readers know how much I love the Raspberry Pi and MySQL. I have several hobby projects that collect data, and MySQL on Pi is great
solution!

I recently decided that I needed to be able to monitor my two MySQL Server.

The steps below should work on the Raspberry Pi 3+ and 4.

What better tool to use than PMM

In this blog we will be using PMM Client 2.27 and PMM Server version 2.39.

Prerequisites

Important Note: Your Raspberry Pi must be on Raspbian Bullseye.

I will assume if you have made it this far into the BLOG post, then you already have a running PMM Server. If not, you can read about installing a PMM Server here.

Add the pmm user to your MySQL Server

CREATE USER 'pmm'@'127.0.0.1' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, PROCESS, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';

The following steps should be run on your Raspberry Pi

Download the pre-complied pmm-client tools

These were complied based on 2.27 source code. You can see the steps I followed to compile the tools here.

cd
wget https://github.com/cetanhota/pi4-arm-pmm-client/archive/refs/heads/main.zip
unzip main.zip

Once we have the client unzipped its a good idea to verify that the tools will work on your Raspberry Pi.

cd ~/pi4-arm-pmm-client-main/raspbian-bullseye
./pmm-agent --version

You should get output like below.

ProjectName: pmm-agent
ProjectName: pmm-agent
Version: 2.27.0-84-g78b8519-dirty
PMMVersion: 2.27.0-84-g78b8519-dirty
Timestamp: 2023-08-16 00:57:34 (UTC)
FullCommit: 78b85198ad1e2c319d4012ef5ae338f389e2000a
Branch: main

If the pmm-agent displayed info like above, then you good to keep moving down the document. If you ran into issues please post in the comments and we can see what can be done to get you working. The only problem I ran in to was that the root user could not execute the client tools.

Setup install directories

We need to create the install directory and move files to the correct locations.

sudo mkdir -p /usr/local/percona
sudo mkdir -p /usr/local/percona/pmm2
sudo mkdir -p /usr/local/percona/pmm2/tools
sudo mkdir -p /usr/local/percona/pmm2/config
sudo mkdir -p /usr/local/percona/pmm2/exporters
sudo mkdir -p /usr/local/percona/exporters
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/high-resolution
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/medium-resolution
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/low-resolution

Copy files into directories

cd ~/pi4-arm-pmm-client-main/raspbian-bullseye
sudo cp pmm-admin pmm-agent /usr/local/bin
sudo cp node_exporter vmagent /usr/local/percona/pmm2/exporters

Configure the PMM Client

Its time to configure and test the PMM Client for the MySQL server where we are install PMM Client.

Please make sure you have the following three items:

PMM Server Hostname or IP Address
PMM Server admin ID
PMM Server admin Password

sudo pmm-agent setup --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml\
 --server-address= --server-insecure-tls\
 --server-username= --server-password=

Now that the PMM Client has been configured, lets move on to starting the PMM Client.

Start the PMM Client

 sudo pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml

Verify by reviewing the client is running correctly. If all is well you should see output like:

INFO[2023-08-15T17:00:34.294-04:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml. component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=main
INFO[2023-08-15T17:00:34.296-04:00] Using /usr/local/percona/pmm2/exporters/vmagent component=main
INFO[2023-08-15T17:00:34.296-04:00] Starting... component=client
INFO[2023-08-15T17:00:34.296-04:00] Starting local API server on http://127.0.0.1:7777/ ... component=local-server/JSON
INFO[2023-08-15T17:00:34.296-04:00] Connecting to https://admin:***@192.168.1.127:443/ ... component=client
INFO[2023-08-15T17:00:34.298-04:00] Started. component=local-server/JSON
INFO[2023-08-15T17:00:34.320-04:00] Connected to 192.168.1.127:443. component=client
INFO[2023-08-15T17:00:34.320-04:00] Establishing two-way communication channel ... component=client
INFO[2023-08-15T17:00:34.341-04:00] Two-way communication channel established in 21.349589ms. Estimated clock drift: -84.399159ms. component=client

At this point press ctl+c to stop the pmm-agent.

If you want to start the pmm-agent again and running the background use this command:

sudo nohup pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml &

You may choose a different startup process for the PMM Client based on personal preference.

Add pt-summary to the PMM Client

cd /usr/local/percona/pmm2/tools
sudo wget percona.com/get/pt-summary
sudo chmod ugo+x pt-summary

Let’s see summary in action:

Add your Raspberry Pi MySQL Server to PMM Server

I am sure most of you know how to add a server to your existing PMM Server but if you have not done this before here is what you need to do.

Connect to your PMM Server.
Use pmm-admin command to add server for monitoring.

/usr/local/percona/pmm2/bin/pmm-admin add mysql --service-name= --server-insecure-tls --server-url=https://:@localhost --username= --password= --host=

PMM Server Screen Shoot

In Closing

I really loved working on this little project. I hope you find some use for the information.

DoKC Operator SIG Update

Edith Puclla — Mon, 14 Aug 2023 00:00:00 UTC

Before our meeting, we started with a question to begin the morning: What board game or tabletop game have you played that you would recommend to others?

Itamar Marom suggests that Catan as a good board game, which takes a lot of time, super annoying when you lose, but generally a lot of fun. So highly suggested!

Other members like Hugh Lashbrooke and Robert Hodges prefer Monopoly, where you must reach a higher level of monopoly awareness to enjoy it fully. The idea is that you can make pretty much any deal with anyone else within the rules of monopoly. That’s when it gets fun.

Terraforming Mars and Meadow are other favorite games for members of DoK.

Let’s get into our agenda for this meeting.

For our first agenda, Itamar Marom (From AppsFlyer and DoKC Ambassador) proposes joining more data operators’ maintainers and developers in the community.

Itamar was analyzing the data technology map and saw that some workloads are very common and act differently from what is found in SIG Operator (Special Interest Group) in the case of Spark and Kafka. It would be nice to see the maintainers and hear their opinions, especially in meetings like our one with Google, where they have very different and exciting use cases.

We’ve had folks from these communities present at our virtual meetups. Bringing them to the SIG to learn, share, and find ways to collaborate would benefit the broader DoK ecosystem.

Jim Halfpenny from Stackable mentions that they have several operators for open source projects, including Apache Kafka, Apache NiFi, Apache Superset, Trino, and more. And they will love input from the community on the direction the operators should take. His aim is that our operators should play nicely with others!

Melissa Logan was part of several discussions with the Argo community, and maybe there’s a chance they could join us as well.

Itamar will prepare a proposal for this project in which the group can collaborate and will share it soon.

The second item on our agenda was Carrier Hardening and Security Project Update by Robert Hodges (DoKC Ambassador).

This project is a guide to establishing a baseline for secure data management on Kubernetes by fortifying the database operators. The guide aims to identify the typical attack surfaces that exist for databases running on Kubernetes. It will establish a collection of best practices for enhancing their security using operators.

Robert is working on an August 12 talk at DataConLA on the topic: Tips for Sleeping Well with State-of-the-Art Data Management. It will contain the framing of the operator hardening guide.

Robert is playing around with a couple of approaches to divide up the problem space. One of them is to deal with security concerns as follows:

The database itself - E.g., setting passwords safely.
Kubernetes - Security it from outside attackers - E.g., encrypted client connections.
Data outside Kubernetes - Object storage used for backups, logs forwarded to log management systems.

Robert Will share a draft of the talk in the #sig-operator channel for discussion.

The last topic on our agenda is ArgoCD and general CI/CD compatibility for operators by Robert Hodges.

Robert created a public repository Argocd-examples-clickhouse, example of ArgoCD application definitions for ClickHouse analytic applications.

In this project, you’ll find ArgoCD applications and instructions to stand up a full analytic stack based on ClickHouse in a Kubernetes cluster.

Following suggestions from community members, a new Slack channel #topic-devops was created 🎉 . This is a channel to talk about CI/CD integration, specific solutions like ArgoCD & Flux, etc.

To learn more about our meetings, join the Data on Kubernetes. An Open Community for Data on Kubernetes. We host weekly live meetups where technologists share their stories, wisdom, and practical advice for running data on Kubernetes.

Percona Monitoring and Management 2.38 preview release

Taras Kozub — Fri, 30 Jun 2023 00:00:00 UTC

Percona Monitoring and Management 2.38 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.38 is now available as a preview release.

To see the full list of changes, check out the PMM 2.38 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.38.0-rc

Important: To use the DBaaS functionality during the PMM preview release, add the following environment variable when starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.38.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.38.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.38.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-09895e9b605f14cbc

Percona University in Peru 2023

Edith Puclla — Tue, 20 Jun 2023 00:00:00 UTC

Peru is a country in South America home to a section of the Amazon rainforest and Machu Picchu ⛰️, an ancient Incan city high in the Andes mountains. Percona decided to hold the first event of Percona University 2023 in Lima, the capital of Peru, on June 10.

Percona University is a series of free technical events organized by Percona in various cities around the world since 2013. Percona uses these events to share its unbiased expertise on open source databases with the community, users, and organizations. The last Percona University was in 2022 in Istanbul, Turkey.

Something charming was designed for this first session of Percona University; there were all kinds of stickers and prizes at the end of the event.

The talks were related to open source databases and kubernetes. Let’s summarize them:

Let’s start with Peter Zaitsev, who talked about the Cloud of Serfdom vs. the Cloud of Freedom; why open source will win in the Cloud Age, spoke about the relationship between Cloud and Open Source, showing the historical changes in the impact of the Cloud on Open Source and examining the current state of affairs, and advocating for a specific approach to using Cloud and Open Source together.
Peter’s next talk was about 17 reasons to migrate to MySQL 8 MySQL 8 was different from previous major MySQL versions, as it underwent significant changes from its initial release in 2018 to the version in 2023. Many features were introduced through subsequent minor version releases. In the presentation, Peter focused on the most important Modern MySQL 8 features that had emerged since the initial release of MySQL 8.

The next on the agenda was Michael Villegas. He talked about Useful tools from the Percona Toolkit for DBAs. This talk was aimed at DBAs and developers of any level of experience responsible for MySQL database administration. They learned how to use some very useful tools within the Percona Toolkit to solve common problems within their databases. Michael showed these tools allowed them to save time and effort in resolving these issues.

After the following talk, we had a coffee break provided by Percona, it was an opportunity to chat with Peter and network with other attendees.

The next talk was by Peter Zaitsev about Advanced MySQL optimization and troubleshooting using PMM. Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBAs. The databases power their applications need to handle changing traffic workloads while remaining responsive and stable, ensuring an excellent user experience. Additionally, DBAs are expected to find cost-efficient solutions to these issues. In the presentation, Peter Zaitsev showed advanced options of PMM version 2, which allowed DBAs to address these challenges.

The next talk was for Edith Puclla (me). I did an Introduction to Kubernetes Operators. I provided a simplified overview of Kubernetes Operators, focusing on making the concept understandable for those new to the subject. I started explaining Kubernetes and why they are relevant in managing applications, then we go for Kubernetes Operators example, the reason behind that, and the benefits of using Operators.

Our last talk was about Deep Dive Into Query Performance by Peter Zaitsev. In this presentation, Peter explored this seemingly simple aspect of working with databases in detail. Peter answered questions like when you should focus on tuning specific queries or when it is better to focus on tuning the database (or just getting a bigger box). Peter also showed other ways to minimize user facing response time, such as parallel queries, asynchronous queries, queueing complex work, and as often misunderstood response time killers such as overloaded networks, stolen CPU, and even limits imposed by this pesky speed of light.

The event was well received. Many graduates, professionals, students, and open source enthusiasts attended the event.

And also nice to share moments with Peter and his fans.

We also thank ESAN University for providing us with the venue for the event.

Don’t miss our next Percona University event, in Istanbul!

See you at Percona University Peru in 2024!

Data on Kubernetes Meetup May 23

Edith Puclla — Thu, 01 Jun 2023 00:00:00 UTC

Percona has started to participate in Data on Kubernetes (DoK) meetings about Kubernetes Operators. These meetings are an initiative of DoK meetups that spotlight DoK case studies. In this blog post series, I will summarize the topics covered in each meeting.

On May 23, very interesting topics were discussed on the agenda. Let’s begin to summarize it.

We start with a new project proposal, which is called: Distributed Systems Operator Interface (DSOI). It is proposed by Adheip Singh from DataInfra, Nitish Tiwari from Parseable, and Itamar Marom from AppsFlyer.

This project is a set of best practices for building Kubernetes operators for distributed systems. The spec defines standard practices that can help define custom resources (CR). It consists of Kubernetes-native CRDs and specs and is not bound to any specific application. There are already two operators built using this set of practices:

If you want to contribute, send proposals, join datainfra-workspace or raise bugs in the GitHub repository of DSOI.

As the second item on the agenda, we have an update about the DoK Operator SIG Project Proposal - Security & Hardening Guide. This project is proposed by Robert Hodges, Altinity Inc.

This project is a guide to establishing a baseline for secure data management on Kubernetes by fortifying the database operators. The guide aims to identify the typical attack surfaces that exist for databases running on Kubernetes. It will establish a collection of best practices for enhancing their security through the utilization of operators.

Robert mentioned that he connected to TAG Security, which in turn led to a link to BadRobot, which is a scanner that checks operators for excessive privileges. Also, Robert presented to DoK Bay Area last week to introduce the problem of operator security.

For the Operator security & hardening guide, we can raise issues on sig-operator. They are currently seeking volunteers and contributors for their project; Find operator-security-hardening project on GitHub, or feel free to write Robert Hodges rhodges@altinity.com.

Finally, we have an update of the Operator Feature Matrix (OFM)

The Operator Feature Matrix (OFM) is a project from the Data on Kubernetes Community to create a standardized and vendor-neutral feature matrix for various Kubernetes operators that manage stateful workloads. This project is proposed by Alvaro Hernandez, and it is definitely a good project to contribute if you are looking to improve the end-user experience with the use of workloads in Kubernetes.

CloudNativePG project sent feedback to improve OFM. CloudNativePG is the Kubernetes operator that covers the full lifecycle of a highly available PostgreSQL database cluster. Planning to create a website for end-user adoption

There are other (non-Postgres) technologies, like Apache Druid, jumping on OFM. This is a work in progress.

The end of June is being considered for a 1.0 freeze, before which it is required to get as much feedback as possible. If you are interested, feedback can be as simple as opening an issue to discuss something; or sending a PR requesting improvements (or both). Feel free to do it on OFM GitHub Repo.

What experts said at Kubecon about Data on Kubernetes

Edith Puclla — Wed, 31 May 2023 00:00:00 UTC

Melissa Logan, managing director of Data on Kubernetes (DoK), led one of the best panels I’ve been to at a conference at Kubecon EU in Amsterdam about challenges with and the state of the art of running databases on Kubernetes.

This panel united the Data on Kubernetes Community Operator SIG and Kubernetes Storage SIG to discuss key features of Kubernetes database operators. Xing Yang from VMware, Sergey Pronin from Percona, and Álvaro Hernández from OnGres came together to discuss what works, what doesn’t, and where the industry is going. They also presented a feature matrix to help end users compare many database Operators.

If you are new to the topic of Kubernetes Operators, I wrote a blog post about Kubernetes Operators in a nutshell, you can read the first part of this article.

Let’s start by summarizing the challenges the panelists mentioned when running stateful applications on Kubernetes.

Some Operators have certain limitations, and there are security concerns. Database users always think about data encryption: is the data safe? What happens if the node goes down? What happens if we lose the storage?
The Operator model is very extensible and flexible, which is great, but on the other hand, there are so many Operators, and it becomes a challenge to choose the right one for our use cases.
People who are developing Operators also find challenges because every database has its native way of doing backups, but if you want to support more than one type of database, then it’s more challenging to find a generic way.

Does the framework capture what is needed for data workloads well?

There is a capability model for Operators that classify them into five levels. There is room for improvement in this model. It can be improved to build test compatibility and more objective measures of these capability levels if you look at level five, the top one for data workloads.

Security is the number one criterion that people use to evaluate Operators. How are they addressing security in Kubernetes Operators?
Users want to ensure that the Operator does not get a lot of privileges or does not interfere with other tenants in the Kubernetes cluster.
Now users are looking for more sophisticated ways with their existing security key-value storage. They can be sure they are safe.
The framework should provide ransomware protection, so when you back up your databases, you also want to have one immutable copy to provide your protection and recover from that.

Now Let’s talk about solutions

DoK started an Operator Special Interest Group (SIG), and community members have been meeting to discuss how as a group and as an industry, to collaborate to come up with solutions for some of the challenges end users face. The Operator SIG works with the Storage Technical Advisory Group (TAG), Storage SIG, and Security SIG.

According to data on Kubernetes’s (DoK) first report, 70% of responses are running workloads in production. More data workloads are running on Kubernetes, so it is essential to know what works well. Sharing knowledge and leveraging that expertise is vital at this stage.

There are things that SIG Operators detail in a document, like common patterns and features used when running databases on Kubernetes, best practices, the criteria for running a good operator, why observability is so important in the cloud-native environment and security.

The Operator Feature Matrix is a big initiative to help end users find operators based on different criteria to choose what they need. It is a project to compare different Operators. They are starting with database Operators, and one project is already defined: Postgres Operator Feature Matrix. Feel free to contribute to OFM.

New to Kubernetes Operators and databases? Check out Percona’s Operators for MySQL, PostgreSQL, and MongoDB.

Easy Way to Start Contributing to Open Source With PMM Documentation

Thu, 18 May 2023 00:00:00 UTC

If you are a user of Percona Monitoring and Management and noticed any typo or inaccurate information in its documentation, you can easily correct it yourself in the repository following detailed instructions in README.md. But if you are not experienced in open source contributions, you may still feel uneasy about following those steps. This post is for you! We will walk through the main steps with pictures and explanations.

Create a Fork

First, you need to create a fork from the main repository to your account. In the top-right corner of the page, click Fork - Create a new fork.

Build Documentation With Docker

The easiest way is to build documentation with Docker. If you don’t have it installed, download it from the Docker official website and follow the instructions. The process of installation is quick, and it is no more difficult than the installation of any other app.

Open your fork on GitHub and clone that repository to your local environment.

git clone git@github.com:{user-name}/pmm-doc.git

Change directory to pmm-doc.

cd pmm-doc

To check how our edits will look like, we need to build documentation for live previewing. Run:

docker run --rm -v $(pwd):/docs -p 8000:8000 perconalab/pmm-doc-md mkdocs serve --dev-addr=0.0.0.0:8000

Wait until you see INFO - Start detecting changes. When the documentation is ready to work with, it will be available at http://0.0.0.0:8000 in your browser, and it will reflect all changes that you make locally.

Make Changes

In a new Terminal tab, create a new branch and make your changes. Save them, create a commit, and push it to your fork.

Create a pull request to the main repository. You will also need to sign the CLA, so we could merge your changes.

You did it! Congratulations! Now wait for the feedback from the Percona team. If there is no problem with your PR, it will be merged into the main repository.

Next Steps

To make further changes, you need to keep your repository up-to-date with the upstream one. There are several ways to do it. You can find the information here. The simplest way is to do it using the GitHub interface. Just click on Sync fork and then Update branch.

After that, you will be able to update your local repository with git pull command.

If you face any problems with contributions to Percona repositories, don’t hesitate to contact us at community-team@percona.com or ask your question on the Percona Forum.

My Experience at Kubecon Europe in Amsterdam

Edith Puclla — Thu, 11 May 2023 00:00:00 UTC

Kubecon is the most significant event focused on the Kubernetes ecosystem. It takes place once a year in North America, Europe, and Asia. It is a perfect opportunity to learn from experts, meet friends, grow your network, and attend talks at a varied technical level and meetings focused on CNCF communities. This time I attended Kubecon in Amsterdam. The theme for this version of Kubecon was: community-in-bloom because we are still healing from COVID, and people are getting back to feeling comfortable participating in events.

It is not my first time attending Kubecon; this is my fourth time! What was different this time was that Percona, the company I work for, sponsored Kubecon. This means we also had a booth at the event to share what we do at Percona.

Yessss!! Percona had a booth in Kubecon, Amsterdam.

Percona is a 100% remote company, and Kubecon was the opportunity to meet part of the team I work with daily.

We started by meeting with friends in one of the most popular places in Amsterdam. I met people from different tech communities and CNCF ambassadors.

If you are in Amsterdam and don’t ride a bicycle, it is an incomplete experience. We rode a bike to the convention center and collected the event badge.

After that, there were three intense days, many activities, meetings, and sessions.

Community

I met with several members of the Docker and CNCF communities. This is the meeting of the Docker captains and various members of the Docker team. Such an amazing experience to know them in person.

And this is the group of photos that all the CNCF ambassadors had at Kubecon. There are 155 CNCF ambassadors around the world and contributors and advocates of the CNCF ecosystem.

Lightning Talks

I attended a large part of the lightning sessions, which were very inspiring. Each speaker has less than 5 minutes to explain a specific topic.

Kevin Patrick talked about the Armada as a Sandbox project in the CNCF, in which the principal goal is enabling batch processing across multiple Kubernetes clusters. Kevin made an introduction to Armada and showed the integration with Apache Airflow.

The next talk was about The CNCF Board Game Rules, where Peter O’Neill made an abstract about the world of the CNCF and imagined it as a role-playing game (RPG) board game. It was pretty nice for Peter to show us the adventure of CNCF with games.

This was one of my favorite lightning talks: [A Beginners Guide to Conference Speaking] (https://www.youtube.com/watch?v=jCz9QPrJ6Eo)with Paula Kennedy. She showed a friendly way to prepare proposals for CFPs and advice that will help us through the process.

Another good lightning session was about the Power-Aware Scheduling in Kubernetes with Yuan Chen from Apple. In this talk, Yan gave an overview of a new scheduler feature to support power-aware scheduling in Kubernetes and how it can help safely increase server hardware and data center infrastructure size and improve resource utilization and workload reliability for Kubernetes clusters.

Another of my favorites was Talking to Kubernetes with Rust with James Laverack, where he showed how to interact with Kubernetes in Rust.

Kubernetes Operators Panel Discussion

I also attended a Panel Discussion about Kubernetes Operators. In this panel discussion, Xing Yang, Melissa Logan, Sergey Pronin, and Alvaro Hernandez talked about the challenges that final users have when running data workloads in Kubernetes Operators. They also shared about the need to fulfill the process of data workloads. Check out this fantastic talk and learn more about Kubernetes Operators. A very curious and interesting fact is that most of the assistants use data workload with Kubernetes Operators

eBPF

The last talk I attended was an eBPF talk with Liz Rice (Chief Open Source Officer, Savant) In this talk, Liz shows a demo about how Cilium and its ClusterMesh feature can take care of many aspects of connectivity across multiple clusters in a cloud-agnostic way. Check this talk in CNCF YouTube Channel.

Clossign Kubecon Amsterdam

Finally, in Percona, we closed the event with a raffle to take home an Atari’; the expectations were relatively high and very fun, and many Percona lovers came to participate in the raffle.

You can check in which events Percona is going to be in the next months

PostgreSQL: Query Optimization With Python and PgBouncer

Mario García — Tue, 25 Apr 2023 00:00:00 UTC

Database application by Nick Youngson CC BY-SA 3.0 Pix4free

A few months ago I wrote a few blog posts on how to generate test data for your database project using Python, which you can find on the Percona blog and the Community blog:

The basic idea is to create a script that uses Faker, a Python library for generating fake data, and what the script does is

Divide the whole process into every CPU core available by implementing multiprocessing
The script will generate a total of 60 thousand records, divided by the number of CPU cores minus one
Each set of records is stored in a Pandas DataFrame, then concatenated into a single DataFrame
The DataFrame is inserted into the database using Pandas’ to_sql method, and pymongo’s insert_many method

How can the script be optimized? Instead of generating the data, storing it in a DataFrame, and then inserting it into the database, you can make every CPU core insert the data while generating it without storing it elsewhere before running the corresponding SQL statements. Multiprocessing is implemented to use every CPU core available but you also need to configure a connection pool for your PostgreSQL server.

Through this blog post, you will learn how to install and configure PgBouncer with Python to implement a connection pool for your application.

PgBouncer

PgBouncer is a PostgreSQL connection pooler. Any target application can be connected to PgBouncer as if it were a PostgreSQL server, and PgBouncer will create a connection to the actual server, or it will reuse one of its existing connections.

The aim of PgBouncer is to lower the performance impact of opening new connections to PostgreSQL.

Installation

Ir you’re an Ubuntu user, you can install PgBouncer from the repositories:

$ sudo apt install pgbouncer -y

If not available in the repositories, you can follow the instructions below for both Debian and Ubuntu as mentioned in the Scaleway documentation

Create the apt repository configuration file

$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

Import the repository signing key

$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

Update the apt package manager

$ sudo apt update

Install PgBouncer using apt

$ sudo apt install pgbouncer -y

Configuration

After installing PgBouncer, edit the configuration files, as stated in the Scaleway documentation.

Set up the PostgreSQL server details in /etc/pgbouncer/pgbouncer.ini

database_name = host=localhost port=5432 dbname=database_name

You may also want to set listen_addr to * if you want to to listen to TCP connections on all addresses or set a list of IP addresses.

Default listen_port is 6432

From this article by Abdullah Alger, the settings max_client_conn and default_pool_size, the former refers to the number of applications that will make connections and the latter is how many server connections per database. The defaults are set at 100 and 20, respectively.

Edit the /etc/pgbouncer/userlist.txt file and add your PostgreSQL credentials

“username” “password”

Add the IP address of the PgBouncer server to the PostgreSQL pg_hba.conf file

host all all PGBOUNCER_IP/NETMASK trust

By default, PgBouncer comes with trust authentication method. The trust method can be used in a development environment but is not recommended for production. For production, hba authentication is recommended.

After configuring PgBouncer, restart both the PostgreSQL and PgBouncer services

sudo systemctl reload postgresql
sudo systemctl reload pgbouncer

For more information about additional configuration options, check the PgBouncer documentation.

Python

Requirements

Dependencies

Make sure all the dependencies are installed before creating the Python script that will generate the data for your project.

You can create a requirements.txt file with the following content:

tqdm
faker
psycopg2

Or if you’re using Anaconda, create an environment.yml file:

name: percona
dependencies:
 - python=3.10
 - tqdm
 - faker
 - psycopg2

You can change the Python version as this script has been proven to work with these versions of Python: 3.7, 3.8, 3.9, 3.10, and 3.11.

Run the following command if you’re using pip:

pip install -r requirements.txt

Or run the following statement to configure the project environment when using Anaconda:

conda env create -f environment.yml

Database

Now that you have the dependencies installed, you must create a database named company.

Log into PostgreSQL:

$ sudo su postgres
$ psql

Create the company database:

create database company;

And create the employees table:

create table employees(
 id serial primary key,
 fist_name varchar(50) not null,
 last_name varchar(50) not null,
 job varchar(100) not null,
 address varchar(200) not null,
 city varchar(100) not null,
 email varchar(50) not null
);

Inserting Data

Now it’s time to create the Python script that will generate the data and insert it into the database.

from multiprocessing import Pool, cpu_count
import psycopg2
from tqdm import tqdm
from faker import Faker

fake = Faker()
num_cores = cpu_count() - 1

def insert_data(arg):
 x = int(60000/num_cores)
 print(x)
 with psycopg2.connect(database="database_name", user="user", password="password", host="localhost", port="6432") as conn:
 with conn.cursor() as cursor:
 for i in tqdm(range(x), desc="Inserting Data"):
 sql = "INSERT INTO employees (first_name, last_name, job, address, city, email) VALUES (%s, %s, %s, %s, %s, %s)"
 val = (fake.first_name(), fake.last_name(), fake.job(), fake.address(), fake.city(), fake.email())
 cursor.execute(sql, val)

if __name__=="__main__":
 with Pool() as pool:
 pool.map(insert_data, range(num_cores))

At first, the multiprocessing pool is created, and configured to use all available CPU cores minus one. Each core will call the insert_data() function.

On each call to the function, a connection to the database will be established through the default port (6432) of PgBouncer, meaning that the application will open a number of connections equal to num_cores, a variable that contains the number of CPU cores being used.

Then, the data will be generated with Faker and inserted into the database by executing the corresponding SQL statements.

In a CPU with 16 cores, the number of records inserted into the database on each call to the function will be equal to 60 thousand divided by 15, that is 4 thousand SQL statements executed.

This way you can modify the script and optimize it by configuring a connection pool with PgBouncer.

Percona Monitoring and Management 2.37 preview release

Taras Kozub — Thu, 20 Apr 2023 00:00:00 UTC

Percona Monitoring and Management 2.37 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.37 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.37.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variable when starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.37.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.37 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.37.0.ova

AMI

ami-013c92f3d0c727b8f

FerretDB - A Quick Look

David Stokes — Wed, 12 Apr 2023 00:00:00 UTC

There is an old saying that what looks like a duck and quacks like a duck is probably a duck. But what looks like MongoDB and acts like MongoDB could be FerretDB! To greatly simplify the technology behind this project, FerretDB speaks, or quacks, MongoDB but stores the data in PostgreSQL. PostgreSQL has had a rich JSON data environment for years and FerrtDB takes advantage of this capability. This is a truly Open Source MongoDB alternative and was released under the Apache 2.0 license.

FerretDB has been in development for a while, but they announced the first Generally Available Release of their product recently.

In the announcement is a quick “How To Get Started” section which details how to get FerretDB running with the help of Docker. As can be seen below, this is a very simple process.

$ docker run -d --rm --name ferretdb -p 27017:27017 ghcr.io/ferretdb/all-in-one
Unable to find image 'ghcr.io/ferretdb/all-in-one:latest' locally
latest: Pulling from ferretdb/all-in-one
f1f26f570256: Pull complete
1c04f8741265: Pull complete
dffc353b86eb: Pull complete
18c4a9e6c414: Pull complete
81f47e7b3852: Pull complete
5e26c947960d: Pull complete
a2c3dc85e8c3: Pull complete
17df73636f01: Pull complete
713535cdf17c: Pull complete
52278a39eea2: Pull complete
4ded87da67f6: Pull complete
05fae4678312: Pull complete
56b4f4aeea2d: Pull complete
68c486387c4f: Pull complete
5eb3eee800a9: Pull complete
8e5dd809e820: Pull complete
d3e85fce5b45: Pull complete
e6810cdbd43b: Pull complete
Digest: sha256:072312577c1daf469ac77d09284a638dea98b63f4f4334fd54959324847b93aa
Status: Downloaded newer image for ghcr.io/ferretdb/all-in-one:latest
58f00a86bad172674479f3663563af274e0dd3d15249029a403d0c85039b7ab5

Now that FerretDB is ready, we can use the MondoDB shell to speak to it.

$ docker exec -it ferretdb mongosh

Current Mongosh Log ID: 6435963392d12db06bdb7ecc
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.8.0
Using MongoDB: 6.0.42
Using Mongosh: 1.8.0

(the remaining output was omitted for brevity)

Entering some very basic MongoDB commands work as expected. Well, for the most part.

test> db
test
test> show collections;

test> db.createCollection('test');
{ ok: 1 }
test> show collections;
test

test> db.test.insert({name: "Dave", state: "Texas"});
DeprecationWarning: Collection.insert() is deprecated. Use insertOne, insertMany, or bulkWrite.
{
 acknowledged: true,
 insertedIds: { '0': ObjectId("6435ac52c4a22ac27f30e2a2") }
}
test> db.test.insertOne({name: "Dave", state: "Texas"});
{
 acknowledged: true,
 insertedId: ObjectId("6435ac5dc4a22ac27f30e2a3")
}
test> db.test.find();
[
 {
 _id: ObjectId("6435ac52c4a22ac27f30e2a2"),
 name: 'Dave',
 state: 'Texas'
 },
 {
 _id: ObjectId("6435ac5dc4a22ac27f30e2a3"),
 name: 'Dave',
 state: 'Texas'
 }
]
test> 

I expected that the inset on the deprecated ’insert’ command would not create a document but I was wrong. It took me a moment to realize that the ‘insert’ and ‘insertOne’ commands both worked after looking at the different ObjectIds. But what do we know about the server itself? Issuing a serverStatus commands confirms we are talking to the FerretDB server.

test> db.runCommand({serverStatus: 1});
{
 host: '58f00a86bad1',
 version: '6.0.42',
 process: 'ferretdb',
 pid: Long("10"),
 uptime: 6277.435694035,
 uptimeMillis: Long("6277435"),
 uptimeEstimate: Long("6277"),
 localTime: ISODate("2023-04-11T18:59:46.488Z"),
 freeMonitoring: { state: 'undecided' },
 metrics: {
 commands: {
 ping: { total: Long("1"), failed: Long("0") },
 getFreeMonitoringStatus: { total: Long("1"), failed: Long("0") },
 create: { total: Long("1"), failed: Long("0") },
 insert: { total: Long("3"), failed: Long("0") },
 atlasVersion: { total: Long("1"), failed: Long("1") },
 getLog: { total: Long("1"), failed: Long("0") },
 buildInfo: { total: Long("1"), failed: Long("0") },
 getCmdLineOpts: { total: Long("1"), failed: Long("0") },
 listCollections: { total: Long("2"), failed: Long("0") },
 ismaster: { total: Long("611"), failed: Long("0") },
 find: { total: Long("4"), failed: Long("0") },
 getParameter: { total: Long("1"), failed: Long("1") },
 hello: { total: Long("1"), failed: Long("0") },
 unknown: { total: Long("5"), failed: Long("0") }
 }
 },
 ok: 1,
 catalogStats: {
 collections: 210,
 capped: 0,
 timeseries: 0,
 views: 0,
 internalCollections: 0,
 internalViews: 0
 }
}
test> 

Summary

FerretDB is a MongoDB protocol server built upon PostgreSQL. Those unhappy with the change in MongoDB’s license change away from open source now have another path they can follow. I will have a full session at Percona Live on FerretDB where I will delve into how complete of an option this is for those desiring an open solution.

Using the JSON data type with MySQL 8 - Part II

Edith Puclla — Tue, 11 Apr 2023 00:00:00 UTC

If you read - Using the JSON data type with MySQL 8 - Part I, you will see that inserting data into MySQL of JSON type is a very common and effective practice. Now we’ll see how to do it with a Python project, using SQLAlchemy and Docker Compose, which further automates this example. You can run this example using a single command: docker-compose up

Before getting down to work, we will review some important concepts:

Percona Server for MySQL is an open source, drop-in replacement for MySQL Community that provides better performance, more scalability, and enhanced security features.
SQLAlchemy is a library that allows us to communicate between Python programs and databases.
Docker Compose is a tool for defining and running multi-container Docker applications.

Let’s start with the structure of this project:

We have a folder called app which contains the db.py file, and this is where we create the library database and establish the connection with this database.

bash
db_user = os.environ['DB_USER']
db_password = os.environ['DB_PASSWORD']

engine = create_engine(f"mysql+pymysql://{db_user}:{db_password}@db:3306/library")

In this file, we also create the class transactions. This will create the fields for the library databases with SQLAlchemy; we define the attributes, and they will be the database fields. We have four attributes: book_id, tittle, publishes, and labels. The last one (labels) of JSON data type.

bash
class transactions(base):
 __tablename__ = 'book'

 book_id = Column(Integer, primary_key=True)
 title = Column(String(50))
 publisher = Column(String(50))
 labels = Column(JSON)

 def __init__(self, book_id, title, publisher, labels):
 self.book_id = book_id
 self.title = title
 self.publisher = publisher
 self.labels = labels

base.metadata.create_all(engine)

Now let’s review the Python script called insert.py, where we use the transactions class to insert data into the database.

bash
import db
from sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=db.engine)
session = Session()

tr1 = db.transactions(1,'Green House', 'Joe Monter', '{"about" : {"gender": "action", "cool": true, "notes": "labeled"}}')

session.add(tr1)
session.commit()

Now let’s explore the docker-compose.yaml file, we have two services, the db and the api

bash
version: "3.8"
services:
 api:
 build: .
 container_name: api
 depends_on:
 db:
 condition: service_healthy
 db:
 image: percona/percona-server:8.0
 container_name: db
 restart: always
 environment:
 MYSQL_USER: root
 MYSQL_ROOT_PASSWORD: root
 MYSQL_DATABASE: library
 healthcheck:
 test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
 timeout: 20s
 retries: 10
 volumes:
 - my-db:/var/lib/mysql
 ports:
 - "3306:3306"
 expose:
 - "3306"

# Names for volume
volumes:
 my-db:

The db service uses the Percona Server for MySQL image (percona/percona-server:8.0) for the database and has a healthcheck that allows you to confirm when the database is started and ready to receive requests. The api service depends on the db service to start. The api service will build a Dockerfile, it does a build of the Python applications (of db.py and insert.py), so in this way, we can insert data into the database when it is ready.

It’s time to see the example in action; let’s locate it inside the json-mysql project and run docker-compose ps -d

Once this is done, we can connect to the database and query the table without needing to go inside the container with the following command:

bash
docker exec -i db mysql -uroot -proot <<< "use library;show tables;select \* from book;describe book;"

We can check the data types of our fields and the inserted data. You will also see the JSON data type “labels” data type.

bash
book_id title publisher labels
1 Green House Joe Monter "{\\"about\\" : {\\"gender\\": \\"action\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"
2 El camino Daniil Zotl "{\\"about\\" : {\\"gender\\": \\"documental\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"
3 London Bridge Mario Mesa "{\\"about\\" : {\\"gender\\": \\"drama\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"

bash
Field Type Null Key Default Extra
book_id int NO PRI NULL auto_increment
title varchar(50) YES NULL
publisher varchar(50) YES NULL
labels json YES NULL

Use “docker compose ps” to see your services running. In this case, we have the “db” service running, which is for the database, and we have “api” with the state “exited,” which means that the scripts to create the database and insert the data into the database “library” was created.

bash
NAME COMMAND SERVICE STATUS PORTS
api "/bin/sh -c 'bash -C…" api exited (0)
db "/docker-entrypoint.…" db running (healthy) 0.0.0.0:3306->3306/tcp, 33060/tcp

It was an example of inserting JSON data into MySQL using SQLAlchemy in Python and docker-compose for deployment.

You can find the project on GitHub. If there is any other way to make it better happy to hear it so I can improve this project.

You can explore more about Percona Server for MySQL, and if you want to see how this project start check Using the JSON data type with MySQL 8 - Part I

How a Database Monitoring Tool Can Help a Developer. The Story of One Mistake.

Daniil Bazhenov — Fri, 07 Apr 2023 00:00:00 UTC

I will tell you the real story of using database monitoring tools when developing an application. I will show you an example of how I managed to detect and fix a problem in the application.

A small clarification, the real story from my development practice happened a little more than a week ago, but for the article I took graphs of final debugging, so that the graphs show the correct sequence and fit into the available for explanation and demonstration. It’s just that in reality, I went out for coffee several times and thought for a long time about what was reflected in the graphs of monitoring:)

About the app and the process

I am developing a PHP application using MongoDB as a database. The application is lightweight, and most load falls on the database. I have implemented functions at the application level to adjust the number of queries, as the application can quickly load the database to 100%.

For development, I use several small dev instances in AWS, use Percona Server for MongoDB with three nodes as a database, and have Percona Monitoring and Management (PMM) installed for monitoring the databases.

My development process consists of the following steps:

I developed a new feature and ran it on the dev server for testing.
I check the prefiling on the PHP side, and there is no memory leak, and I am happy with the speed.
I check the database monitoring to ensure everything works fine.
I debug the feature, setting the number and types of queries in the function to balance the number of queries and the load on the database, if necessary.

Adding new functionality to the application

So I started the application and got ready to run the new feature. The feature was getting information from open sources, processing it, and saving it to the database. The second part of the functionality went through all the saved documents and did some additional processing.

At this point, the application already had a lot of features that loaded the CPU of the Primary Node by 25-40%, and everything was running stably. I decided to have a performance reserve, as I planned to add new features.

I checked several dashboards, and there were no anomalies or changes. PMM has many dashboards and charts, and I will only show a few, just some.

I saved the changes with the new feature and pushed it to the dev server to make it work. Then I checked that the function started without errors, and the result was visible in the database. I use MongoDB Compass to check the result of a database entry.

Something has gone differently than planned.

I waited a few minutes and rechecked the dashboard. At first glance, the main screen was fine. However, I was alarmed by the speed of processing. The number of operations has mostly stayed the same.

I scrolled down through the various charts on the dashboard and saw an anomaly.

The latency increased, and the app loaded the instance to 100% CPU.

I have made a test run on the application side and checked the profiler there, too. The app worked poorly, and queries were slow.

Finding the cause of the problem

I knew the reason was the new feature and immediately rolled back the last changes.

I had a rough idea of where the problem might be, made a few changes, and started again.

I did it several times, but the result was the same (the CPU was loaded at 100%).

I selected a period with a load and used the Query Analytics function built into the monitoring. Query Analytics shows a list of queries sorted by load or execution speed. Some of the queries to the Pages collection gave 90% load, and the Query Time was more than 3 minutes.

In Query Analytics, you can find slow queries, see their details, and then debug them in the application.

Fixing the problem

I made a few changes that fixed the problem.

The first problem was the indexes. I create indexes from within the application using the command.

$app['db']->CollectionName->createIndex(['index_key' => 1]);

Since the application uses many different collections and queries with conditions on various fields and with or without sorting, I have a lot of indexes.

I made a typo in this case, and the index was not created correctly.

After the indexes were created correctly, I needed quick runs to debug the number of queries to adjust the CPU load to around 50%.

You can see the final chart after debugging and fixing the problem.

Conclusion

Don’t forget to add indexes and make sure they work.

I am a simple developer who can make mistakes and do different experiments. Installing the monitoring was one of the experiments, and previously I just focused on the speed of the PHP script. From time to time, I have looked at the monitoring dashboard in the AWS control panel, but it gives less information, only about the instance itself, without being able to investigate in detail.

So, PMM have a great tools for debugging and searching “bottlenecks” in the databases. And I recommend installing and trying database monitoring with PMM if your application uses MySQL, PostgreSQL, or MongoDB.

How to prevent unauthorized users from connecting to ProxySQL

Valentin TRAËN — Thu, 30 Mar 2023 00:00:00 UTC

ProxySQL is a great load balancer which however suffers from some shortcomings concerning the management of MySQL users. ProxySQL provides a firewall which, in my case, is not complete enough to properly manage users and secure their access. Indeed, this firewall does not accept subnets and keeps unauthorized connections in ProxySQL. We cannot then be sure of not suffering a DDOS attack on our ProxySQL instance. In this article, I will explain how I managed to overcome this problem.

Reminder of the principle of connection through ProxySQL

To understand what follows, you have to bear in mind how ProxySQL connects to MySQL. The user connects to Proxysql which then establishes the connection to MySQL. For this, ProxySQL maintains MySQL users in its internal database. The names of MySQL users, their passwords as well as the MySQL destination server are entered in the mysql_users table. At each connection request to a MySQL server, ProxySQL checks the presence of the user in the mysql_users table to connect itself to MySQL with this same user.

Something is missing, isn’t it?

Yes, the host associated with each MySQL user is missing!

In MySQL, users are configured to only be able to connect from ProxySQL. In ProxySQL we don’t have this information. By default, all users can therefore connect to ProxySQL from any IP address and ProxySQL will open connections to MySQL for them. As I specified in the introduction, ProxySQL provides a Firewall to overcome this problem, but this one is not really satisfactory.

Prevent connection to ProxySQL with an unauthorized user

In this part, our ProxySQL instance will allow user bob to connect to the MySQL (mysql_server) instance. Bob is allowed to connect from IP_1 but cannot from IP_2. The ProxySQL instance is running on IP_PROXYSQL.

In MySQL, user bob was created like this:

sql
CREATE USER 'bob'@'IP_PROXYSQL' IDENTIFIED BY 'PASSWORD';

In ProxySQL, let’s create bob like this:

sql
INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('bob','PASSWORD',0);
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

and declare the MySQL server like this:

sql
INSERT INTO mysql_servers(hostgroup_id,hostname) VALUES (1,'mysql_server');
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

As you may have noticed, I didn’t declare the same hostgroup when creating the user and the server. Hostgroup 0 does not correspond to any MySQL server. By default, our user bob will therefore be able to connect to ProxySQL but his queries will not be redirected to any MySQL server. Let’s move on to host management. I will declare each authorized host in the mysql_query_rules table. In ProxySQL, this table is used, among other things, to assign different parameters to a connection. You see what I mean? Let’s declare our rule!

sql
INSERT INTO mysql_query_rules (rule_id,active,username,client_addr,destination_hostgroup,apply) VALUES (1,1,'bob','IP_1',1,1);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

I have just declared a rule indicating that all requests coming from user bob connected from IP_1 must be played on host 1. And icing on the cake, IP_1 can be a subnet (IP_1%), which would not have could not be possible with the firewall. From now on, bob will be able to perform queries from IP_1 and get results from MySQL. If bob plays a request from IP_2, he will not be able to obtain a result since the hostgroup queried will be 0 which does not correspond to any MySQL server. However, this is not satisfactory. Nothing prevents bob from creating a very large number of connections from IP_2. It won’t reach any MySQL servers but may be able to crash my ProxySQL instance. It’s time to deal with those unauthorized connections!

ProxySQL provides a scheduler which will be very useful here. This scheduler will allow us to play a bash script every x ms. I created this script in the ProxySQL datadir:

kill_connections.sh

bash
#!/bin/bash

PROXYSQL_USERNAME="${1}"
PROXYSQL_PASSWORD="${2}"
PROXYSQL_HOSTNAME="127.0.0.1"
PROXYSQL_PORT="6032"

mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -e "SELECT SessionID,user,cli_host FROM stats_mysql_processlist WHERE hostgroup = 0" | while read SessionID user cli_host; do
 if [ $SessionID != "SessionID" ]; then
 enabled_account=$(mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -se"SELECT count(*) FROM mysql_query_rules WHERE username = '$user' and '$cli_host' LIKE client_addr;")
 if [[ "$enabled_account" -eq 0 ]]; then
 mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -e "KILL CONNECTION $SessionID"
 fi

 fi
done

This script lists all the connections opened in ProxySQL on hostgroup 0. It then checks whether the connected user/host pair is authorized using the mysql_query_rules table. If not, the connection is killed. Let’s activate the scheduler in ProxySQL:

sql
INSERT INTO scheduler(filename, arg1, arg2, interval_ms) VALUES ('kill_connections.sh','proxysql_admin_user','proxysql_admin_password', 1000);
LOAD SCHEDULER TO RUNTIME;
SAVE SCHEDULER TO DISK;

Now, any connection opened in ProxySQL but not authorized will be automatically killed!

WARNING: unfortunately, the ProxySQL scheduler does not work like the MySQL scheduler. It is necessary to open the connection from a .sh file and therefore to indicate the ProxySQL administration credentials. These identifiers will then be visible by monitoring the list of server processes. To avoid this problem, I advise you to indicate the identifiers directly in the .sh file and to protect this file correctly on your server.

Additional Information

When I deploy ProxySQL, I always create a rule with a very high rule_id to block unauthorized connections; this is an additional barrier in case I forget something:

sql
INSERT INTO mysql_query_rules (rule_id,active,error_msg,destination_hostgroup) VALUES (999999999,1,'ProxySQL : Access denied',0);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

This rule redirects unauthorized connections to hostgroup 0 (if ever a user was declared in mysql_users with a hostgroup leading to a MySQL server) and displays an error message for each request. I create all my rules to manage hosts with a rule_id > or = 10000. This allows me to have 9999 empty slots if I ever want to create other priority rules in mysql_query_rules.

sql
INSERT INTO mysql_query_rules (rule_id,active,username,client_addr,apply) VALUES ((SELECT IFNULL(MAX(rule_id)+1,10000) FROM mysql_query_rules WHERE rule_id != (SELECT MAX(rule_id) FROM mysql_query_rules) AND rule_id > 9999),1,'USERNAME','HOST',1);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

Don’t hesitate to ask me questions, I’ll be happy to answer them.

Percona Monitoring and Management 2.36 preview release

Taras Kozub — Mon, 20 Mar 2023 00:00:00 UTC

Percona Monitoring and Management 2.36 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.36 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.36.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.36.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.36 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.36.0.ova

AMI

ami-0ce04c507ec1187b1

How to Develop a Simple Web Application Using Docker, Nginx, PHP, and Percona Server for MongoDB

Daniil Bazhenov — Fri, 17 Mar 2023 00:00:00 UTC

I’m developing an application that takes data from different sources, processes it, and prepares reports. In this series of articles, I will explain how to install and configure the tools, application, and database to develop and run the application.

About the application and choice of tools

The application I develop gets data from GitHub, Jira, and websites via API, processes it and creates reports according to the desired requirements.

The application is developed with PHP version 8+ and Nginx as a web server, and Percona Server for MongoDB as a database. For local development, I use Docker and Docker-compose.

I use PHP and Nginx because I’m familiar with them, and it’s a popular stack with lots of documentation and examples. Docker was chosen for the same reason. I used to install Nginx/Apache + PHP + Database in the same container, but over time I found Docker-compose and separate containers more convenient, so now I use docker-compose.

My application includes the following:

Web application to run in a browser and display reports.
Сonsole scripts in PHP for bulk data updates in the background on the server.

As a database for this application, I use MongoDB. There are objective reasons for that:

The API I’m querying for data from gives it back to me page by page in JSON format. I need to do a lot of queries, so my script gets all data and stores it in MongoDB beforehand to create reports without needing to go to API.
MongoDB is suitable for storing data in JSON and queries with different conditions.
The data schema from the API can be very different and flexible depending on the service and query. MongoDB allows me to save responses from the API to the database as it is, without complicated processing or preconfiguring the database schema. I didn’t want to spend much time setting up the database table schema.
Installing and configuring MongoDB for development is easy and does not require great skills to work with it.

I use Percona Server for MongoDB because it’s free and open source. I once thought about backups and monitoring, and Percona has ready-made solutions for that.

First, I will talk about my development configuration. I am starting from scratch using a minimal PHP application as an example.

Preparing Docker and Docker-compose

Dockerfile for PHM + MongoDB

For PHP to work with MongoDB, we need to install PHP with the required extensions. I prepared a Dockerfile for PHP 8.2 and used php-fpm because I use Nginx as a web server.

Dockerfile

FROM php:8.2-fpm

RUN apt-get -y update \
 && apt-get install -y libssl-dev pkg-config libzip-dev unzip git

RUN pecl install zlib zip mongodb \
 && docker-php-ext-enable zip \
 && docker-php-ext-enable mongodb

# Install composer (updated via entry point)
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

I also used Composer and installed it immediately in the container with Dockerfile.

Now I run the Image build command from the Dockerfile to use it in Docker Compose

docker build -t php8.2-fpm-mongo .

Where php8.2-fpm-mongo - is the name of the image to be used in docker-compose

Docker-compose.yml to test the web app

The next step is to create the docker-compose.yml file. I will add the Nginx web server and my Image with PHP and MongoDB to the Docker-compose file.

docker-compose.yml

version: '3.9'

services:
 web:
 image: nginx:latest
 ports:
 - '80:80'
 volumes:
 - ./app:/var/www/html
 - ./config/default.conf:/etc/nginx/conf.d/default.conf

 php-fpm:
 image: php8.2-fpm-mongo
 volumes:
 - ./app:/var/www/html

The application is located in the app/ folder in the same directory as docker-compose.yml.

Now it will be a very simple index.php script that prints out information about itself.

Create an app directory and an index.php file.

app/index.php

phpinfo();

Structure of files and folders

.
├── app
│ └── index.php
├── config
│ └── default.conf
├── Dockerfile
└── docker-compose.yml

You will also notice /config/default.conf. This is the configuration of Nginx for handling requests and running PHP. Here is my example of a default.conf file. Let’s create it too.

/config/default.conf

server {
 listen 80;
 server_name localhost;

 index index.php index.html;

 error_log /var/log/nginx/error.log;
 access_log /var/log/nginx/access.log;
 root /var/www/html;

 rewrite ^/(.*)/$ /$1 permanent;

 location / {
 try_files $uri $uri/ /index.php?$query_string;
 }

 location ~ \.php$ {
 try_files $uri =404;
 fastcgi_split_path_info ^(.+\.php)(/.+)$;
 fastcgi_pass php-fpm:9000;
 fastcgi_index index.php;
 include fastcgi_params;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 fastcgi_param PATH_INFO $fastcgi_path_info;
 fastcgi_buffering off;
 }
}

If we run docker-compose now, we can open localhost in the browser and see the running php from the app/index.php file.

Run docker-compose

docker-compose up -d

Open localhost in the browser.

Stop docker-compose to continue setting up. We haven’t connected MongoDB yet.

docker-compose down

Connect MongoDB to our docker-compose

Add the new db service to our docker-compose.yml.

version: '3.9'

services:
 web:
 image: nginx:latest
 ports:
 - '80:80'
 volumes:
 - ./app:/var/www/html
 - ./config/default.conf:/etc/nginx/conf.d/default.conf

 php-fpm:
 image: php8.2-fpm-mongo
 volumes:
 - ./app:/var/www/html
 environment:
 DB_USERNAME: root
 DB_PASSWORD: secret
 DB_HOST: mongodb # matches the service with mongodb

 mongodb:
 image: "percona/percona-server-mongodb:6.0.4"
 # image: "percona/percona-server-mongodb:6.0.4-3-arm64" # For Apple M1/M2
 volumes:
 - ./data:/data/db
 restart: always
 environment:
 MONGO_INITDB_ROOT_USERNAME: root
 MONGO_INITDB_ROOT_PASSWORD: secret
 MONGO_INITDB_DATABASE: tutorial
 ports:
 - "27017:27017"

If you examine the changes in the docker-compose.yml file carefully, you will notice:

I add image (db) with Percona Server MongoDB 6.0.4
I use data/ folder in the same directory as volumes. It’s convenient for me to easily access DB files, transfer them, and examine them locally.
I pass environment variables to create a MongoDB root user.
I also added environment variables in php-fpm to use them to connect to the database in the application.
And the volumes parameter will link our local app directory directly to the container, this will allow us to modify the code and immediately check the result in the browser without restarting the container.

 volumes:
 - ./app:/var/www/html

Let’s modify our PHP script to check the operation of the database.

Connecting to MongoDB in the application

Install required PHP packages to work with MongoDB

Create an app/composer.json file to install and use the required MongoDB libraries and extensions for PHP.

app/composer.json

{
 "require": {
 "mongodb/mongodb": "^1.6",
 "ext-mongodb": "^1.6"
 }
}

Connect to the php-fpm container and install the Composer packages

Run docker-compose

docker-compose up -d

Look up the name of the container with php-fpm, in my case it is github-php-fpm-1.

Run the command to connect to the container:

docker exec -it [php-fpm-container] bash

Run the installation of the Composer packages described in our composer.json file with

composer install

Now we can connect to MongoDB in our PHP application.

Connecting to MongoDB in a PHP application.

Now we slightly modify the index.php script to connect to the database and test data recording.

/app/index.php

// Enabling Composer Packages
require __DIR__ . '/vendor/autoload.php';

// Get environment variables
$local_conf = getenv();
define('DB_USERNAME', $local_conf['DB_USERNAME']);
define('DB_PASSWORD', $local_conf['DB_PASSWORD']);
define('DB_HOST', $local_conf['DB_HOST']);

// Connect to MongoDB
$db_client = new \MongoDB\Client('mongodb://'. DB_USERNAME .':' . DB_PASSWORD . '@'. DB_HOST . ':27017/');

$db = $db_client->selectDatabase('tutorial');

// Test insert data
for ($page = 1; $page <= 1000; $page++) {

 $data = [
 'page_id' => $page,
 'title' => "Page " . $page,
 'date' => date("m.d.y H:i:s"),
 'timestamp' => time(),
 'mongodb_time' => new MongoDB\BSON\UTCDateTime(time() * 1000)
 ];

 $updateResult = $db->pages->updateOne(
 [
 'page_id' => $page // query
 ],
 ['$set' => $data],
 ['upsert' => true]
 );

 echo $page . " " ;
}
echo '
Finish';
exit;

If we run localhost in the browser, our application will write 1,000 documents from the for loop into the database. It will also display the sequential numbers of the documents being written.

Connecting to MongoDB via MongoDB Compass

MongoDB Compass is a handy desktop application to work with MongoDB. I use it to browse databases and collections and create indexes.

This is a quick way to conveniently look through written data and check errors.

Let’s connect to the database using MongoDB Compass to check that the data is actually written.

You need to use Localhost as host and the user/password from docker-compose.

After connecting you will see 1000 documents written to the database and you can make test queries or add an index.

Don’t forget indexes MongoDB

If you write and read data on certain fields, make sure to create indexes on those fields.

For example, if you make 10,000 records with the script we developed above, you will notice a slow writing speed. It could be 20 seconds. But if you create an index on the page field, the write speed will be reduced by a factor of 10 to 2 seconds.

Always create indexes.

This is not hard to do through MongoDB Compass in the Indexes section of the collection.

This is also easy to do in our PHP app, using the method

$db->pages->createIndex(['page_id' => 1]);

This will create an index on page_id, because we do insert/upsert with a condition on this field and it is a unique key.

Add it before the for loop, and increase the number of pages to 10k to compare.

// Create an index
$db->pages->createIndex(['page_id' => 1]);

// Test insert data
for ($page = 1; $page <= 10000; $page++) {

This will greatly increase the speed at which the script runs.

Conclusion

We set up an environment and developed a PHP script to work with MongoDB.

In my opinion, it was simple. All the source code you can see and use from my GitHub

To summarize:

We have now installed standalone Percona Server for MongoDB using Docker-compose locally. However, it is recommended to use ReplicaSet with at least one node for production. We will definitely try this on a separate server using AWS as an example.
For production applications, it is recommended to use ReplicaSet with several nodes. We will definitely do that too.
We will install PMM to monitor database, see how our script loads the database, and see database queries with QAN and other PMM features.

In the next posts, I will work on improving the application. I will focus on database customization. You will learn how to improve the application so that it brings practical use, gets data from the GitHub API, and writes to the database. We’ll divide the application into console scripts and Web.

If you are interested in learning more about the PHP application, write in a comment or on the forum.

Some Notable Bugfixes in MySQL 8.0.32

Aleksandra Abramova — Wed, 15 Mar 2023 00:00:00 UTC

MySQL 8.0.32 came out recently and had some important bugfixes contributed by Perconians. Here is a brief overview of the work done.

Inconsistent data and GTIDs with mysqldump

Marcelo Altmann (Senior Software Engineer) fixed the bug when data and GTIDs backed up by mysqldump were inconsistent. It happened when the options –single-transaction and –set-gtid-purged=ON were both used because GTIDs on the server could have already increased between the start of the transaction by mysqldump and the fetching of GTID_EXECUTED. Marcelo developed a patch, and it was partially included in the release. Now, in MySQL 8.0.32, a FLUSH TABLES WITH READ LOCK is performed before fetching GTID_EXECUTED, to ensure its value is consistent with the snapshot taken by mysqldump. However, Percona Server for MySQL includes the entire patch, which does not require FLUSH TABLE WITH READ LOCK to work.

Marcelo also corrected the issue when the MySQL server exits on ALTER TABLE created an assertion failure: dict0mem.h:2498:pos < n_def.

Fixing garbled UTF characters

Kamil Holubicki (Senior Software Engineer) proposed a patch to fix garbled UTF characters in SHOW ENGINE INNODB STATUS. It happened because the string was truncated, and UTF characters (which are multibyte) were cut in the middle which caused garbage at the end of the string.

Duplicate table space objects in 5.6 to 8.0 upgrade

Rahul Malik (Software Engineer) investigated and fixed an issue when an 8.0 upgrade from MySQL 5.6 crashed with Assertion failure. It happened due to a duplicate table space object. All SYS_* tables are loaded, and then their table IDs are changed. Some SYS tables like SYS_ZIP_DICT, VIRTUAL can have ids > 1024 (say, 1028).

Changing table_ids of SYS_FIELDS from 4 to 1028 will conflict with the table_ids of those existing SYS_ZIP_DICT/VIRTUAL which haven’t been shifted by 1024 yet and are currently loaded with 1028. Hence, we need to change the IDs of that SYS tables in reverse order to fix it. So in the example above, SYS_FIELD is the first shift to 1028+1024, and then SYS_FIELD changes to 1028 to avoid conflicts.

Why open source databases matter

Great work by Marcelo, Kamil, Rahul, and everybody else who contributed to the MySQL 8.0.32 release.

This is why open source databases are so important. We can all help improve MySQL, and those improvements benefit all users of MySQL.

Percona is proud to be part of the MySQL community, and we hope you’ll join us in improving MySQL and its surrounding software. Check out our contributing page to find ways to contribute!

Using the JSON data type with MySQL 8

Edith Puclla — Mon, 13 Mar 2023 00:00:00 UTC

If you are a mobile app, frontend, backend, or game developer, you use data types such as string, numeric, or DateTime. You also know that since the advent of non-relational databases (NoSQL) such as MongoDB, which, by not being tied to a traditional SQL schema, do reading and writing on databases much faster. But MySQL showed that storing the JSON (JavaScript Object Notation) data type could also improve the speed of reading and writing relational databases.

This post will explore the JSON Data type in Percona Server for MySQL.

One of the key features of Percona Server is support for JSON data type, which allows for the storage of JSON documents within MySQL. It allows for more flexible and efficient storage of semi-structured data (which is more human-readable ) within a relational database.

We will install Percona Server for MySQL in a Docker container to make basic operations for inserting, modifying, and removing JSON data types.

To start, we will bring version 8.0 of Percona Server for MySQL; the name of this image in Docker Hub is percona-server. You will need Docker; if you don’t have it installed, follow the official Docker documentation.

docker pull percona/percona-server:8.0

We will run the container for Percona Server for MySQL, call our container percona-server and pass in an environment variable called MYSQL_ROOT_PASSWORD; This variable specifies a password that is set for the MySQL root account.

bash
docker run -d \
 --name percona-server \
 -e MYSQL_ROOT_PASSWORD=root \
 percona/percona-server:8.0

After confirming that our container is running with “docker ps,” we can enter our Percona Server for MySQL container to start executing commands.

docker exec -it percona-server /bin/bash

The Percona Server for MySQL database is already running, and we will proceed to connect to it:

mysql -uroot -p

Use root as a password.

Create the database called cinema

bash
CREATE DATABASE library;
USE library;

Create a table called books

bash
CREATE TABLE books (
 book_id BIGINT PRIMARY KEY AUTO_INCREMENT,
 title VARCHAR(100) UNIQUE NOT NULL,
 publisher VARCHAR(100) NOT NULL,
 labels JSON NOT NULL
) ENGINE = InnoDB;

Insert JSON type into books table

bash
INSERT INTO books(title,publisher, labels)
VALUES('Green House', 'Joe Monter', '{"about" : {"gender": "action", "cool": true, "notes": "labeled"}}');

INSERT INTO books(title,publisher, labels)
VALUES('El camino', 'Daniil Zotl', '{"about" : {"gender": "documental", "cool": true, "notes": "labeled"}}');

bash
select * from books;

As you can see, JSON is a more flexible data type than what you might be used to when working with data in MySQL.

Select with JSON_EXTRACT

bash
SELECT title, JSON_EXTRACT(labels, '$.about.notes') AS Notes FROM books;
A shortcut of JSON_EXTRACT is “->”
SELECT title, labels->'$.about.notes' AS Notes FROM books;

The short operator -> provides the same functionality as JSON_EXTRACT

bash
SELECT titulo, etiquetas->'$.acerca.genero' AS Genero FROM books;

Updating JSON type records

bash
UPDATE books SET labels = JSON_REPLACE(labels, '$.about.gender', 'romance') WHERE title = 'the roses';

UPDATE books SET labels = JSON_REPLACE(labels, '$.about.notes', 'not labeled') WHERE title = 'the roses';

bash
select * from books;

Deleting a JSON record

bash
DELETE FROM books WHERE book_id = 1 AND JSON_EXTRACT(labels, '$.about.gender') = "documental";

Deleting a value inside a JSON structure

bash
UPDATE books SET labels = JSON_REMOVE(labels, '$.about.notes') WHERE book_id = 2;

You can use these fundamental operations to manage JSON data types in Percona Server MySQL. This allows for more flexible and efficient data modeling and querying for applications that work with JSON data. How will that work in an application? Keep an eye out, I’ll be following this up with a blog about an application using JSON data in MySQL very soon.

Get more about Percona Server for MySQL documentation in our official documentation. And if you want to know why JSON is the preferred format for many developers and why it’s so popular, check out David Stokes’ blog: JSON and Relational Databases – Part One

Backups for MySQL With mysqldump

Mario García — Fri, 10 Mar 2023 00:00:00 UTC

Basic Usage

mysqldump is a client utility that can be used for doing logical backups. It will generate the necessary SQL statements to reproduce the original database.

Backup by Nick Youngson CC BY-SA 3.0 Pix4free

The following statements are some common uses of mysqldump:

mysqldump -u username -p database_name [table_name] > dump.sql
mysqldump -u username -p --databases db1_name db2_name > dump.sql
mysqldump -u username -p --all-databases > dump.sql

The first example is for backing up a single database. If you need to back up some specific tables instead of the whole database, write their names, space-separated.

With the --databases option, you can back up two or more databases, their names must be space separated.

To back up all the databases in your MySQL server, just append the --all-databases option.

The dump.sql file doesn’t contain the create database SQL statement. If you need it, add it with the -B option. This is unnecessary if you’re running mysqldump with the --databases and --all-databases options.

Ignoring tables when backing up a database is also possible with the --ignore-tables option.

$ mysqldump -u username -p database_name --ignore-tables=database_name.table1 > database_name.sql

If you need to ignore more than one database, just use the option as many times as needed.

$ mysqldump -u root -p database_name --ignore-table=database_name.table1 --ignore-table=database_name.table2 > database_name.sql

Schema Backup

In case you need to backup only the schema of your database with no data, run mysqldump with the --no-data option:

$ mysqldump -u username -p database_name --no-data > dump.sql

You can also backup the schema while running mysqldump with the --databases and --all-databases options.

$ mysqldump -u username -p --all-databases --no-data > dump.sql

$ mysqldump -u username -p --databases db1_name db2_name --no-data > dump.sql

Data Restore

To restore the databases in your dump.sql file, run the following command:

$ mysqldump -u root -p < dump.sql

If you need to restore a single database from the complete backup, you can do it by running any of the following statements:

$ mysqldump -u root -p -o database_name < dump.sql

$ mysqldump -u root -p --one-database database_name < dump.sql

In both cases, the database must exist in your MySQL server, as it only will restore the schema and the data.

Conditional Backup

If you need to create a backup that contains data that matches a condition, you can use a WHERE clause with mysqldump.

You can use a single where condition:

$ mysqldump database_name table_name --where="id > 500" > dump.sql

Or multiple conditions:

$ mysqldump database_name users --where="id > 500 and disabled = 0" > dump.sql

As explained here in the mysqldump.guru website.

For example, in a database with the following schema, built from the Movienet dataset:

Movienet Database

If you want to back up the movies produced in a specific country, like Mexico, a way to do it is by running mysqldump with a WHERE clause.

$ mysqldump -u root -p movienet movies --where=”country = 22” > dump.sql

22 is the country_id of Mexico in this particular database, created using this Python script.

You can also get those values by executing the following SQL statement:

select movies.movie_id, movies.title, countries.name as country from movies inner join countries on movies.country = countrie
s.country_id and movies.country = '22';

+-----------+-----------------------------------------------------------+---------+
| movie_id | title | country |
+-----------+-----------------------------------------------------------+---------+
| tt0047501 | Sitting Bull (1954) | Mexico |
| tt0049046 | Canasta de cuentos mexicanos (1956) | Mexico |
| tt0076336 | Hell Without Limits (1978) | Mexico |
| tt0082048 | El barrendero (1982) | Mexico |
| tt0082080 | Blanca Nieves y sus 7 amantes (1980) | Mexico |
| tt0083057 | El sexo de los pobres (1983) | Mexico |
| tt0110185 | El jardín del Edén (1994) | Mexico |
| tt0116043 | De jazmín en flor (1996) | Mexico |
| tt0121322 | El giro, el pinto, y el Colorado (1979) | Mexico |
| tt0133354 | Algunas nubes (1995) | Mexico |
| tt0207055 | La risa en vacaciones 4 (TV Movie 1994) | Mexico |
| tt0208889 | To and Fro (2000) | Mexico |
| tt0211878 | La usurpadora (TV Series 1998– ) | Mexico |
| tt0220306 | El amarrador 3 (1995) | Mexico |
| tt0229008 | El vampiro teporocho (1989) | Mexico |

Skipping Databases

There’s no option for mysqldump to skip databases when generating the backup, but here’s a solution that could work for you:

DATABASES_TO_EXCLUDE="db1 db2 db3"
EXCLUSION_LIST="'information_schema','mysql'"
for DB in `echo "${DATABASES_TO_EXCLUDE}"`
do
 EXCLUSION_LIST="${EXCLUSION_LIST},'${DB}'"
done
SQLSTMT="SELECT schema_name FROM information_schema.schemata"
SQLSTMT="${SQLSTMT} WHERE schema_name NOT IN (${EXCLUSION_LIST})"
MYSQLDUMP_DATABASES="--databases"
for DB in `mysql -u username -p -ANe"${SQLSTMT}"`
do
 MYSQLDUMP_DATABASES="${MYSQLDUMP_DATABASES} ${DB}"
done
MYSQLDUMP_OPTIONS="--routines --triggers"
mysqldump -u username -p ${MYSQLDUMP_OPTIONS} ${MYSQLDUMP_DATABASES} > MySQLDatabases.sql

The above BASH script will generate the backup of your MySQL server excluding the information_schema and mysql databases, listed in the EXCLUSION_LIST variable, as well as the databases of your choice in the DATABASES_TO_EXCLUDE variable.

Don’t forget to add the databases you want to exclude to the DATABASES_TO_EXCLUDE variable, replace the username, in both mysql and mysqldump commands, and add the required options to the MYSQLDUMP_OPTIONS variable.

Security Considerations

Some of the common questions in our forum are about how to do a partial restoration from a complete backup. For example, when you back up a database with mysqldump, you will get the statements for creating the schema of the database and inserting the data from your backup.

If you only need the schema, you can run mysqldump with the –no-data option. But if you need to restore the schema of a specific database from a complete backup, I found an interesting solution:

cat dump.sql | grep -v ^INSERT | mysql -u username -p

The above command will restore the schema of your database, skipping the SQL statements for inserting the data. It works well when you backup a single database, but there’s no reason to use it as you can get the schema with the --no-data option, instead of removing the inserts.

What happens if you try to run this command with a backup that includes all the databases in your server? You must be careful as this will try to overwrite the system schema in the mysql database which is dangerous. This database store authentication details and overriding the data will make you lose access to your server.

If you don’t need to backup the mysql database, run mysqldump with the --databases option to specify which databases you require or use the script shared in the Skipping Databases section.

Conclusion

Through this blog post you learned how to use mysqldump for backing up the databases in your MySQL server as well as some recommendations while using this tool. For advanced usage of mysqldump you can check this article in our blog.

Monitor your databases with Open Source tools like PMM

Edith Puclla — Mon, 06 Mar 2023 00:00:00 UTC

In this post, we will cover the value of database monitoring and how we can use Open Source tools like PMM Percona Monitoring and Management to monitor and manage databases effectively.

Why should I care about database monitoring?

Once you have passed the installation and configuration of your databases and it is well underway, you have to start monitoring it, and not only the database but the elements related to it.

Questions like these will begin to arise:

Is my database performing well?
- Are query response times consistently slow?
Is my database available and accepting connections?
- Connections to the database close to the maximum limit
Is my system stable?
- How about CPU, memory, and disk?
Am I experiencing avoidable downtime?
- Hardware failures, network outages.

Am I experiencing data loss?
- Disk crashes
- Human errors
Am I minimizing performance issues that can impact my business?
Can I quickly identify and resolve issues before they become more significant problems?

To answer these questions, you will need to find tools that let you keep your database monitored, and you can opt for free tools for monitoring. PMM is one of them, which is entirely open source.

Percona Monitoring and Management (PMM)

PMM is an open source database observability, monitoring, and management tool for:

MySQL
MariaDB
PostgreSQL
MongoDB
And others

It can also help to improve the performance of your databases, simplify their management and strengthen their security.

PMM is built on top of open source software

Grafana
VictoriaMetrics/Prometheus
ClickHouse
PostgreSQL
Docker

PMM Interface

There are three levels of depth:

Dashboards
Graphs
Metrics

Metrics & Database Monitoring

Important database metrics you should monitor:

It will depend on your specific database and use case
Monitor the metrics that are relevant to your database and your business
You should have alerts and monitoring processes to ensure you are aware of any problems as they occur or before

Some important metrics that could indicate potential database issues:

Query performance
High CPU utilization
High Memory usage
High Disk I/O
User Connection
Data growth
Others

Let’s analyze each of them, and they will also answer your questions at the beginning.

Long Query Response Times

PMM helps you monitor the performance of individual queries and identify slow-performing queries that need to be optimized. We can use Query Analytics in PMM to visualize all the queries running in our database; we can inspect each of them and see which is the one sending more queries per second and much longer it takes to execute it. Also, PMM will show you suggestions to fix or improve queries.

High CPU Utilization

PMM helps you monitor the number of CPU resources the database uses and identify performance bottlenecks.

In the section on CPU utilization, you will see how much of your CPU is being used in a period of time. This is very useful when you need to increase your resources.

High Memory usage

PMM helps you monitor the amount of memory being used by the database and determine if you need to add more memory or optimize your database configuration.

Disk I/O

PMM helps you monitor the number of disk I/O operations performed by the database and identify any potential performance bottlenecks. See here the panel of Disk IO Latency!

User connections

PMM helps you monitor the number of active database connections and determine if your user connection is sized appropriately. If you limit the number of users that should connect to your database, this panel will show you when you are reaching that limit so that you can increase the number.

Data growth

PMM helps you monitor your database growth over time and plan for capacity and performance needs. This dashboard helps to see the time period in which your database is growing and to be able to learn about performance issues or issues as they occur.

Summary

We see the importance of monitoring databases and how to explore PMM for some essential metrics to detect issues and prevent them on time.

Want to try PMM? We have a test environment to try PMM without having to install it first. Feel free to play with it and see how PMM works. If you like it, you can install PMM quickly and start using it in your own environment.

How to test code blocks in documentation

Denys Kondratenko — Tue, 28 Feb 2023 00:00:00 UTC

As any developer, I don’t like to write documentation. But if I am writing it, I would like to test that what I wrote works. I often found myself copy-pasting something from documentation and trying to run it in the terminal (commands, files, etc.), and it didn’t work.

There are usually some environment, typos, or even wrong commands in the doc (that people copy-pasted from the wrong place).

I know that issue, and after writing the documentation, I usually try to clean up everything in my environment and test the doc. I am reading it and executing commands as they wrote. Sometimes I find something needs to be fixed. So I needed to find a way to test it quickly.

For example, recent Podman doc has both code and files:

 You can override the environment variables by defining them in the file `~/.config/pmm-server/env`. For example, to override the path to a custom registry `~/.config/pmm-server/env`:

 ```sh
 mkdir -p ~/.config/pmm-server/
 cat << "EOF" > ~/.config/pmm-server/env
 PMM_TAG=2.35.0
 PMM_IMAGE=docker.io/percona/pmm-server
 PMM_PUBLIC_PORT=8443
 EOF
 ```

 !!! caution alert alert-warning "Important"
 Ensure that you modify PMM_TAG in `~/.config/pmm-server/env` and update it regularly as Percona cannot update it. It needs to be done by you.

1. Enable and Start.

 ```sh
 systemctl --user enable --now pmm-server
 ```

Documentation ages and there could be new images that wouldn’t work with this documentation anymore. Another issue is making changes to the existing documentation - how to know that there are no regressions with something new added or with some fixes?

Usually, to mitigate those issues, developers and/or tech writers are [re]checking everything manually.

There are many different automatic approaches to mitigate that issue. The ultimate solution for this is probably GNU Emacs Org Mode. I dream that one day I will learn Emacs and Org Mode.

But I need a solution now, and the team process is to have documentation in a Markdown format and track everything in GitHub - PMM Documentation. GitHub supports quite good code formating, and it is easy to develop and review Markdown there.

There are probably frameworks that could help with that, but I needed something quick and didn’t want to introduce yet another testing framework to the batch.

So I was looking for something that would allow me to quickly cut code snippets from the documentation and run them in a GitHub action. While searching, I found this doc blog: Test codeblocks in markdown documents. Which was exactly what I needed :)

The only problem I found out quickly is that I need something else. Pandoc builds AST tree just fine:

json
 {
 "t": "CodeBlock",
 "c": [
 [
 "",
 [
 "sh"
 ],
 []
 ],
 "mkdir -p ~/.config/pmm-server/\ncat << \"EOF\" > ~/.config/pmm-server/env\nPMM_TAG=2.35.0\nPMM_IMAGE=docker.io/percona/pmm-server\nPMM_PUBLIC_PORT=8443\nEOF"
 ]
 },
...
 {
 "t": "CodeBlock",
 "c": [
 [
 "",
 [
 "sh"
 ],
 []
 ],
 "systemctl --user enable --now pmm-server"
 ]
 }
...
 [
 {
 "t": "CodeBlock",
 "c": [
 [
 "",
 [
 "sh"
 ],
 []
 ],
 "#first pull can take time\nsleep 80\ntimeout 60 podman wait --condition=running pmm-server"
 ]
 }
 ]

As you see, the 3rd CodeBlock is not on the same level. So when using jq approach (from the blog post):

pandoc -i podman.md -t json | jq -r -c '.blocks[] | select(.t | contains("CodeBlock"))? | .c'

[["",[],[]],"```sh\npodman exec -it pmm-server \\\ncurl -ku admin:admin https://localhost/v1/version\n```"]p'

It returns only the block from that level that jq program specifies. My first reaction was to try to advance that filter, but there were so many levels of code block that could be found, and there was so much I needed to learn and do to create the number of filters that I abandoned the idea.

Still, Pandoc is a very powerful tool, so I started digging to find out if any embedded filters could help me filter only CodeBlocks. And apparently, there are Pandoc Lua Filters. After some experiments, I came up with the following:

lua
traverse = 'topdown'
function CodeBlock(block)
 if block.classes[1] == "sh" then
 print("#-----CodeBlock-----")
 io.stdout:write(block.text,"\n\n")
 end
 return nil
end

This gives me a sequence of blocks that are marked as sh:

sh
 pandoc -i podman.md --lua-filter ../../../_resources/bin/CodeBlock.lua -t html -o /dev/null

...
#-----CodeBlock-----
mkdir -p ~/.config/pmm-server/
cat << "EOF" > ~/.config/pmm-server/env
PMM_TAG=2.31.0
PMM_IMAGE=docker.io/percona/pmm-server
PMM_PUBLIC_PORT=8443
EOF

#-----CodeBlock-----
systemctl --user enable --now pmm-server

So that is easy to wrap up in a shell script and execute - locally or in a GitHub Action:

 - name: Copy test template
 run: cp _resources/bin/doc_test_template.sh ./docs_test_podman.sh

 - name: Get CodeBlocks and push them to test template
 run: pandoc -i docs/setting-up/server/podman.md --lua-filter _resources/bin/CodeBlock.lua -t html -o /dev/null >> docs_test_podman.sh

 - name: Run podman tests
 run: ./docs_test_podman.sh

Sometimes, you will find yourself in a situation where you need to execute something (env, infra, cleanup) that should not be shown in the documentation. For example, wait for the previous action before the next one:

sh
sleep 30
timeout 60 podman wait --condition=running pmm-server

I use html to hide CodeBlocks from the rendered document.

You could solve documentation testing at least for some or most of the cases with these simple conventions:

sh language identifier for the fenced code blocks for examples
Bash Heredoc for files

This approach is easy to use locally to test documentation you just wrote and integrate into the CI pipeline.

Exploring Databases on Containers with Percona Server for MySQL

Edith Puclla — Thu, 23 Feb 2023 00:00:00 UTC

In this blog, we will explore databases on containers. We will use Docker as a container engine tool and Percona Server for MySQL as a database administration tool. Both are open source tools.

MySQL is a relational database management system that stores data on disk. Percona Server for MySQL is a fork of MYSQL, providing much more advanced features. To run it correctly, we need to know volumes because we want to “persist” the data, the most important thing in databases.

Running a single Percona Server for MySQL container

First, let’s create a container without volumes:

Figure 1: From Percona Server for MySQL image to a running container in Docker

The following command will create a container called percona-server-1, where we can create databases and add some data.

bash
docker run -d --name percona-server-1 -e MYSQL_ROOT_PASSWORD=root percona/percona-server:8.0

Listing the image and the container:

After the container is created:

We have our base image, which is percona/percona-server:8.0
The base image in Docker is read-only. We can’t modify it. It allows you to spin up multiple containers from the same image with the same immutable base.
We can add data to our image. This new layer is readable and writable. If we create our database and populate it:

Accessing the detached container:

docker exec -it percona-server-1 /bin/bash

Connecting to the database

mysql -uroot -proot

Create a Database “cinema” and use it

bash
CREATE DATABASE cinema;
USE cinema;

Create table movies in Database “cinema”

bash
CREATE TABLE movies (
book_id BIGINT PRIMARY KEY AUTO_INCREMENT,
title VARCHAR(100) UNIQUE NOT NULL,
publisher VARCHAR(100) NOT NULL,
labels JSON NOT NULL
) ENGINE = InnoDB;

Insert data into Database “cinema”

sql
INSERT INTO movies(title,publisher, labels)
VALUES(‘Green House’, ‘Joe Monter’, ’{“about” : {“gender”: “action”, “cool”: true, “notes”: “labeled”}}’);

Checking table

sql
select \* from movies;

If you delete this container, everything will be deleted, too, your databases and your data because containers are temporary.

Figure 2: View of the layers that are generated when we create the container. Source: Severalnines AB

Running Multiple MySQL Containers

Now let’s see how the layers of two different containers work together.

bash
docker run -d --name percona-server-1 -e MYSQL_ROOT_PASSWORD=root percona/percona-server:8.0

bash
docker run -d --name percona-server-2 -e MYSQL_ROOT_PASSWORD=root percona/percona-server:8.0

Multiple containers can share the same base image, which is read-only. Each container can have its data state for reading and writing (Which is built on the top of the base image), but this state will be lost if we don’t create persistent volumes that can ve saved after the container is shut down.

Figure 3: View the layers generated when we create two different containers. Source: Severalnines AB

As we said before, “Volumes open the door for stateful applications to run efficiently in Docker.”

Running containers with Persistent Volumes

Now we will create a container with a persistent volume in Docker.

Figure 4: From Percona Server for MySQL image to a running container in Docker with volumes

percona-server is the base of the image. On top of that, we have all the changes we will make in the database. When we create the volume, we link a directory in the container with a directory on your local machine or in the machine where you want to persist the data. When you delete the container, you can attach another container to this volume to have the same data on a different container.

bash
docker run -d --name percona-server -e MYSQL_ROOT_PASSWORD=root -v local-datadir:/var/lib/mysql percona/percona-server:8.0

Figure 4: View of the layers that are generated when we create the container with volume.

Backing up and restroring databases

There are two kinds of backups in databases, logical and physical backups. We can use mysqldump to make logical backups and Percona XtraBackup, for physical backups. If we want to restore, we can use mysqldump and Percona XtraBackup, which offer much more advanced features.

Back up

bash
docker exec -it percona-server-backup mysqldump -uroot --password=root --single-transaction > /path/in/physical/host/dump.sql

Restore

bash
docker exec -it percona-server-restore mysql -u root --password=root < /path/in/physical/host/dump.sql

Now let’s share some tips to run databases on containers:

Constantly monitor your database and host system
Store data in a persistent volume outside the container
Limit resource utilization, e.g., Memory, CPU
Regularly back up the database and store the backup in a secure and separate location.
Have a plan for database migrations and disaster recovery.

We explored how databases work on containers. Volumes are the important thing to persist the data.

What is next? Watch this fantastic talk by Peter Zaitsev Open Source Databases on Kubernetes

Thanks for reading this! You can install Percona Server for MySQL from our Docker Repository and if you have doubts write us in our Percona community forum.

Percona Monitoring and Management 2.35 preview release

Taras Kozub — Tue, 14 Feb 2023 00:00:00 UTC

Percona Monitoring and Management 2.35 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.35 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.35.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.35.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.35 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.35.0.ova

AMI

ami-09d19be2cfb10a60c

Node metrics available inside of a container

Denys Kondratenko — Mon, 06 Feb 2023 00:00:00 UTC

Several people asked me this question: Could we get Node metrics inside of a container?

Usually, this comes from the fact that PMM or standalone people run node_exporter inside a container. PMM does it as a sidecar along with many other exporters to monitor DBs, and node_exporter comes out of the box as a default one. So people could see accurate data on dashboards, like Memory and CPU, that node_exporter reads inside the container.

My first reaction to this - the data is inaccurate, and if you need Node metrics, you need to run node_exporter on the Node so it has proper access to the host system (VM or HW). By inaccurate, I mean - not all data is there, and this data could be accurate only sometimes in some environments.

But once a pretty technical person asked me, I needed to respond to this person with some tech details. There was correct data from PMM client coming from the Kubernetes container about the host it was running.

One of the things I came up with that was true - you wouldn’t see the host process inside a container, and thus you wouldn’t see who and how is consuming memory and CPU.

I need help understanding why Memory information, CPU, and others are correct.

So I performed a small investigation to refresh my memory and learn more about namespaces, cgroups, and containers.

node_exporter

node_exporter documentation says:

The node_exporter is designed to monitor the host system. It’s not recommended to deploy it as a Docker container because it requires access to the host system.

For situations where Docker deployment is needed, some extra flags must be used to allow the node_exporter access to the host namespaces.

Be aware that any non-root mount points you want to monitor will need to be bind-mounted into the container.

If you start container for host monitoring, specify path.rootfs argument. This argument must match path in bind-mount of host root. The node_exporter will use path.rootfs as prefix to access host filesystem.

bash
docker run -d \
 --net="host" \
 --pid="host" \
 -v "/:/host:ro,rslave" \
 quay.io/prometheus/node-exporter:latest \
 --path.rootfs=/host

On some systems, the timex collector requires an additional Docker flag, –cap-add=SYS_TIME, in order to access the required syscalls.

So right away, we can see that additional privileges are needed. More interestingly, not only access to the /proc and /sys is required, but to the whole /. Also, some additional capabilities are needed.

If we will look briefly at the node_exporter code, we will indeed find different technics it uses to gather data:

procfs data
sysfs data
D-Bus socket (systemd data)
system calls (timex)
and probably more (udev, device data and etc)

Container

Containers and their ecosystem is quite a big topic that is described many times. Please check out “Demystifying Containers” by Sascha Grunert and “Building containers by hand” by Steve Ovens. You can find them in Links section.

What is related to my investigation is isolation from the host, and that is mostly namespaces and cgroups.

Other systems that are limiting access to the different files and calls inside a container:

capabilities
seccomp
selinux/apparmor
additional security options

`namespaces` and `cgroup`

Let us focus only on procfs, where a lot of needed monitoring information comes from. I aim to understand why we have some data in /proc that corresponds to the host data and some that do not.

First, /proc is a special filesystem that acts as an interface to internal data structures in the kernel. It can obtain information about the system and change certain kernel parameters at runtime (sysctl).

It is also quite an old interface that was created before any namespaces and cgroup. Many different applications expect the data there, and thus it can’t be easily namespaced.

Here is what was namespaced:

net
uts
ipc
pid
user
cgroups
time

Same in code:

So it means that a lot of data under, for example, /proc/net would be container specific. Same for other subsystems.

But the biggest difference for monitoring when namespaces and cgroup are used is that container has the access only to its own PID namespace. That means that even if we see all available memory or CPU, we can’t tell what processes from the host system could consume that. We could only see our namespace processes.

It is tough to tell what exactly namespaced under /proc. It looks like all the files (not dirs) under /proc are directly from the host kernel.

Thus we could see many files (they aren’t real files) from the host/root namespace and many that are specific to the container namespace.

For example, here you see uts (hostname) and network namespaces differences between root and container:

sh
#container
[root@0d514d31c0a3 opt]$ cat /proc/net/dev
Inter-| Receive | Transmit
 face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
 lo: 250045720 467744 0 0 0 0 0 0 250045720 467744 0 0 0 0 0 0
 tap0: 37944098 3043 0 0 0 0 0 0 173264 2426 0 0 0 0 0 0

#host
[dkondratenko@denlen ~]$ cat /proc/net/dev
Inter-| Receive | Transmit
 face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
 lo: 1674394721 1671478 0 0 0 0 0 0 1674394721 1671478 0 0 0 0 0 0
enp2s0f0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 wwan0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
wlp3s0: 3869584887 6162652 0 88037 0 0 0 0 1639861087 3732759 0 0 0 0 0 0
cni-podman0: 3688 53 0 0 0 0 0 53 22094 163 0 0 0 0 0 0

As you can see, network interfaces are different (net) and hostnames (uts namespace, 0d514d31c0a3 in the container and denlen in the host).

Linux Capabilities and seccomp

Linux Capabilities allow access for the unprivileged processes to perform some actions/call in the system. In node_exporter section, we have seen an example of the CAP_SYS_TIME that is needed to gather some of the data.

seccomp is a computer security facility in the Linux kernel. seccomp allows a process to make a one-way transition into a “secure” state where it cannot make any system calls except exit(), sigreturn(), read(), and write() to already-open file descriptors.

So both systems further restrict access to the data that might be needed to gather monitoring information.

Docker, Podman have default seccomp filters:

Linux Security Modules

Security-Enhanced Linux, SELinux is a Linux kernel security module that provides a mechanism for supporting access control security policies, including mandatory access controls (MAC).

AppArmor (“Application Armor”) is a Linux kernel security module that allows the system administrator to restrict programs’ capabilities with per-program profiles.

Both of those could further restrict access inside the container. For example, here is the part of the apparmor profile:

 deny @{PROC}/* w, # deny write for all files directly in /proc (not in a subdir)
 # deny write to files not in /proc//** or /proc/sys/**
 deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
 deny @{PROC}/sys/[^k]** w, # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
 deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, # deny everything except shm* in /proc/sys/kernel/
 deny @{PROC}/sysrq-trigger rwklx,
 deny @{PROC}/mem rwklx,
 deny @{PROC}/kmem rwklx,
 deny @{PROC}/kcore rwklx,

So it is possible to restrict access even for those root /proc files that provide memory and CPU information.

Additional security options

Container runtimes and tools could further harden security.

One example is masking mount points:

bash
[root@0d514d31c0a3 opt]$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c11,c680",size=0k,uid=1000,gid=1000,inode64)
devtmpfs on /proc/kcore type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=1048576,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=1048576,mode=755,inode64)
devtmpfs on /proc/latency_stats type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=1048576,mode=755,inode64)
devtmpfs on /proc/timer_list type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=1048576,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,context="system_u:object_r:container_file_t:s0:c11,c680",size=0k,uid=1000,gid=1000,inode64)
proc on /proc/asound type proc (ro,relatime)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)

The default masked paths are /proc/acpi, /proc/kcore, /proc/keys, /proc/latency_stats, /proc/sched_debug, /proc/scsi, /proc/timer_list, /proc/timer_stats, /sys/firmware, and /sys/fs/selinux. The default paths that are read-only are /proc/asound, /proc/bus, /proc/fs, /proc/irq, /proc/sys, /proc/sysrq-trigger, /sys/fs/cgroup.

And indeed, masking memory information is straightforward:

bash
podman run --detach --rm --replace=true --name=pmm-server -p 4443:443/tcp --security-opt=mask=/proc/meminfo:/proc/vmstat docker.io/percona/pmm-server:2

Summary

Kubernetes support most of the above technics as well. And different Kubernetes platforms have different security hardness.

My knowledge at the beginning of this road needed to be deeper, but the conclusion stays the same - don’t assume that any data inside the container is related to the host.

Looking at the technics, I didn’t know before and an overall trend of hardening security for the container, my conclusion would be - it is incorrect to assume that node_exporter could read and provide any meaningful data about the host within the container.

Container runtimes, tools, systems, and platforms provide the full capability to shut down, fake, and abstract any data or access that node_exporter needs. And we couldn’t control those - assume you have incorrect data.

Binding your application to the database in the Kubernetes cluster

Denys Kondratenko — Tue, 24 Jan 2023 00:00:00 UTC

dbaas-operator is Yet Another DBaaS Kubernetes Operator (need to suggest yadbko as a name) that tries to simplify and unify Database Cluster deployments by building a higher abstraction layer on top of Percona Kubernetes Operators.

So it becomes much easier to deploy the DB cluster with dbaas-operator and PMM DBaaS on top of it.

But another part of the picture is applications and their workloads to connect to the deployed DB Clusters.

Services and Applications

On Kubernetes, application deployment could be done in many ways, either manually or as part of automatic deployments.

PMM DBaaS offers both - UI to create DB Clusters and get credentials and API to automate those actions.

dbaas-operator adds Kubernetes native API to that batch as well.

But both require additional automation to join the application and the database in one deployment and provide a service to the end user.

And that operation is a challenging task, as every application could expect credentials in some specific format: secrets with hardcoded structures, environment variables with custom names, mount point secrets in particular locations, etc.

Database services add their complexity to that picture by exposing their connections and secrets in a format that is convenient or makes sense for them.

Usually, some Continues Delivery system or deployment package (helm, etc.) ensures all components’ correct deployment sequence and health. So many custom pipelines and packages exist to connect a specific application to a database service.

But for simplicity and scalability, it would be nice to have some standard for connection or software that automates such a connection.

Service Binding

Connecting services is a known pattern: Service Discovery (broker, registry, repository) for the Service-Oriented Architecture.

servicebinding.io is another pattern to bind applications and workloads to the services (REST APIs, databases, event buses, etc.). This specification aims to create a Kubernetes-wide specification for communicating service secrets to workloads in a consistent way.

Service Binding Operator glues services and Kubernetes workflows together. It does so for the services and applications that support ServiceBinding specifications and those that don’t.

Out of the box Service Binding Operator supports Percona Operator for MySQL based on Percona XtraDB Cluster (PXC), so we will deploy Database Cluster with dbaas-operator and connect it to the simple Java application. We will use Spring PetClinic application that supports Spring Cloud Bindings.

Create an environment

We need to have Kubernetes cluster, Operator Lifecycle Manager (OLM) to install operators, and all required operators installed. In this blog, I would use minikube and assume that operator-sdk is installed on the system

Here is a link to the script that:

setups multi-node Kubernetes cluster
installs OLM
installs needed operators with the help of OLM

As a result we get cluster with all needed operators:

sh
$ kubectl get sub -A
NAMESPACE NAME PACKAGE SOURCE CHANNEL
default dbaas-operator dbaas-operator dbaas-catalog stable-v0
default percona-server-mongodb-operator percona-server-mongodb-operator dbaas-catalog stable-v1
default percona-xtradb-cluster-operator percona-xtradb-cluster-operator dbaas-catalog stable-v1
operators my-service-binding-operator service-binding-operator operatorhubio-catalog stable

Create Database Cluster

We will use dbaas-operator to demonstrate how easy it is to create DB Cluster with it:

sh
$ cat <apiVersion: dbaas.percona.com/v1
kind: DatabaseCluster
metadata:
 name: test-pxc-cluster
spec:
 databaseType: pxc
 databaseImage: percona/percona-xtradb-cluster:8.0.27-18.1
 databaseConfig: |
 [mysqld]
 wsrep_provider_options="debug=1;gcache.size=1G"
 wsrep_debug=1
 wsrep_trx_fragment_unit='bytes'
 wsrep_trx_fragment_size=3670016
 secretsName: pxc-sample-secrets
 clusterSize: 1
 loadBalancer:
 type: haproxy
 exposeType: ClusterIP
 size: 1
 image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
 dbInstance:
 cpu: "1"
 memory: 1G
 diskSize: 15G
EOF

$ kubectl get db
NAME SIZE READY STATUS ENDPOINT AGE
test-pxc-cluster 2 2 ready test-pxc-cluster-haproxy.default 5m

Create Spring PetClinic app and bind it to the database

sh
$ kubectl apply -f https://raw.githubusercontent.com/redhat-developer/service-binding-operator/master/samples/apps/spring-petclinic/petclinic-mysql-deployment.yaml
deployment.apps/spring-petclinic created
service/spring-petclinic created

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
spring-petclinic-f7f587c5c-rvq2v 0/1 CrashLoopBackOff 2 (17s ago) 67s

As we didn’t create a binding yet, the application can’t connect to the database and thus fails.

Let us bind application to the database and verify it is working:

sh
$ cat <apiVersion: binding.operators.coreos.com/v1alpha1
kind: ServiceBinding
metadata:
 name: spring-petclinic
 namespace: default
spec:
 services:
 - group: pxc.percona.com
 version: v1
 kind: PerconaXtraDBCluster
 name: test-pxc-cluster
 application:
 name: spring-petclinic
 group: apps
 version: v1
 resource: deployments
EOF

$ kubectl get servicebindings
NAME READY REASON AGE
spring-petclinic True ApplicationsBound 4m47s

$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
spring-petclinic 1/1 1 1 17m

$ minikube service spring-petclinic --url
http://192.168.39.215:31181

What we have done above:

Created kind: ServiceBinding, which takes PXC secrets and maps them to the application as mount points.
As PetClinic supports ServiceBinding spec with Spring framework, it understands those mount points and connects to the database.

Here is what mount point by ServiceBinding specification that Spring Cloud Bindings library parsed and connected to the database:

sh
$ kubectl exec deployment/spring-petclinic -- ls -la /bindings/spring-petclinic/..2023_01_20_21_33_47.4121788695

total 56
drwxr-xr-x 2 root root 320 Jan 20 21:33 .
drwxrwxrwt 3 root root 360 Jan 20 21:33 ..
-rw-r--r-- 1 root root 18 Jan 20 21:33 clustercheck
-rw-r--r-- 1 root root 5 Jan 20 21:33 database
-rw-r--r-- 1 root root 32 Jan 20 21:33 host
-rw-r--r-- 1 root root 17 Jan 20 21:33 monitor
-rw-r--r-- 1 root root 17 Jan 20 21:33 operator
-rw-r--r-- 1 root root 18 Jan 20 21:33 password
-rw-r--r-- 1 root root 4 Jan 20 21:33 port
-rw-r--r-- 1 root root 7 Jan 20 21:33 provider
-rw-r--r-- 1 root root 17 Jan 20 21:33 proxyadmin
-rw-r--r-- 1 root root 18 Jan 20 21:33 replication
-rw-r--r-- 1 root root 18 Jan 20 21:33 root
-rw-r--r-- 1 root root 5 Jan 20 21:33 type
-rw-r--r-- 1 root root 4 Jan 20 21:33 username
-rw-r--r-- 1 root root 17 Jan 20 21:33 xtrabackup
$ kubectl exec deployment/spring-petclinic -- cat /bindings/spring-petclinic/..2023_01_20_21_33_47.4121788695/database
mysql
$ kubectl exec deployment/spring-petclinic -- cat /bindings/spring-petclinic/..2023_01_20_21_33_47.4121788695/host
test-pxc-cluster-haproxy.default

Check the url that was exposed by minikube:

Summary

There are many ways to deploy applications and services and connect them.

I am trying to collect some of them under my personal repo to understand the problem deeper. Please suggest other ways by commenting under this blog or in repo.

ServiceBinding specification is a standardized way that scales easily and allows you to connect Kubernetes workloads to the database services.

I will propose to dbaas-operator to implement that specification so that it could expose different Database engines (mysql, mongo, pg) in a standard way.

Percona Monitoring and Management 2.34 preview release

Taras Kozub — Tue, 17 Jan 2023 00:00:00 UTC

Percona Monitoring and Management 2.34 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.34 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.34.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.34.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.34 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.34.0.ova

AMI

ami-08c09a75c3dd22956

Setting Up PMM For Monitoring Your Databases on Windows

Mario García — Mon, 16 Jan 2023 00:00:00 UTC

Before deploying Percona Monitoring and Management (PMM) in production, you might want to test it or set up a development instance locally. Since many developers and DBAs have Windows desktops, I wanted to demonstrate how to set up PMM on Windows for an easy test environment. In this post, I’ll walk you through setting up PMM with Docker and WSL.

If you’re a Linux user, check the blog post I wrote on Setting up PMM for monitoring MySQL in a local environment. There you can find instructions for installing Percona Monitoring and Management (PMM) on Linux and how to set it up for monitoring a MySQL instance. Otherwise, continue reading to get PMM up and running on Windows.

Getting Started With PMM on Windows

If you’re a Windows user and want to try PMM, the recommended way for installing both the server and client would be to use the official Docker images and follow these guides:

Before running the commands in those guides, you should install Docker Desktop and Windows Subsystem for Linux (WSL). These instructions should work for users who are on current versions of Windows 10 and Windows 11. For installing WSL, follow the instructions on the Microsoft Learn website. Then, get the Docker Desktop installer. Now you’re ready to install and configure PMM.

PMM Server

As stated in the documentation, you can store data from your PMM in:

Docker volume (Preferred method)
Data container
Host directory

The preferred method is also recommended for Windows. Open PowerShell and execute the instructions in the Run Docker with volume section. I’ve reproduced the steps here to save you time:

Get the Docker image:

$ docker pull percona/pmm-server:2

Create a volume:

$ docker volume create pmm-data

Run the image:

$ docker run --detach --restart always \
--publish 443:443 \
-v pmm-data:/srv \
--name pmm-server \
percona/pmm-server:2

Change the password for the default admin user:

$ docker exec -t pmm-server change-admin-password

Once PMM Server is installed, open the browser and visit https://localhost. You will see the PMM login screen.

PMM Login Screen

Now that the server is up and running, you need to get its IP address before connecting the client to the server. To get the IP address, you need the name or container ID. You can get it by running:

$ docker ps

The docker ps command will give you a list of the containers running on your system, as follows:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fd988ad761aa percona/pmm-server:2 "/opt/entrypoint.sh" 2 months ago Up 11 minutes (healthy) 80/tcp, 0.0.0.0:443->443/tcp pmm-server

Take note of the container ID or name of the PMM Server container. Then, execute this command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' your_container

Replacing your_container with the container ID or name copied previously.

PMM Client

Go to the Set Up PMM Client section in the documentation and follow the first two steps. All of these commands are executed from PowerShell.

In Step 3, you need to specify the IP address of the PMM Server by setting up the PMM_SERVER environment variable:

bash
$ $env:PMM_SERVER=’X.X.X.X:443’

Replacing X.X.X.X with the server’s IP address. Then, initialize the container:

$ docker run \
--rm \
--name pmm-client \
-e PMM_AGENT_SERVER_ADDRESS=$env:PMM_SERVER \
-e PMM_AGENT_SERVER_USERNAME=admin \
-e PMM_AGENT_SERVER_PASSWORD=admin \
-e PMM_AGENT_SERVER_INSECURE_TLS=1 \
-e PMM_AGENT_SETUP=1 \
-e PMM_AGENT_CONFIG_FILE=config/pmm-agent.yaml \
--volumes-from pmm-client-data \
percona/pmm-client:2

PMM_AGENT_SERVER_PASSWORD default value is admin. Replace this value with the password assigned when the server was configured.

Configure Your Database

Now that the client is connected to the server, you must configure PMM for monitoring your database. Follow the instructions below.

MySQL
- Installation
- Configuration
PostgreSQL
- Installation
- Configuration
MongoDB
- Installation
- Configuration

Once PMM is configured, the Home Dashboard will show the databases that are being monitored. For more information and advanced configuration, check the documentation.

PMM Home Dashboard

MySQL Installation

If you already have a MySQL instance running, skip the installation process and continue with the configuration.

On Windows, you can install Percona Server for MySQL on Ubuntu running under WSL, but it’s better to use the official Docker image, as the MySQL server would be running on the same network as PMM.

For installing MySQL using Docker, follow the instructions in the Percona Server for MySQL documentation.

You need to start the container with the latest version of Percona Server for MySQL 8.0:

$ docker run -d \
 --name ps \
 -e MYSQL_ROOT_PASSWORD=root \
 percona/percona-server:8.0

Where ps is the name of the container, and the default password for the root user is root. You can change these values according to your needs.

MySQL Configuration

Once Percona Server for MySQL is running, you need to get its IP address by running:

$ docker ps

bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fb4ddb35e48 percona/pmm-client:2 "/usr/local/percona/…" 2 minutes ago Up 2 minutes pmm-client

Copy the container ID or name of the Percona Server for MySQL container. Then, execute this command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' your_container

Replacing your_container with the value you copied previously.

The IP address of the PMM Client container is also needed.

For configuring PMM for monitoring MySQL, we need to create a PMM user. First, log into MySQL:

$ docker run -it --rm percona mysql -h MYSQL_SERVER -uroot -p

Where MYSQL_SERVER is the IP address of the Percona Server for MySQL container

Then, execute the following SQL statements

sql
CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;

sql
GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';

Replacing pass with your desired password, and localhost with the IP address of the PMM Client container.

And finally, register the MySQL server for monitoring:

bash
$ docker exec pmm-client pmm-admin add mysql --username=pmm --password=pass --host MYSQL_SERVER --query-source=perfschema

Where MYSQL_SERVER is the IP address of the Percona Server for MySQL container. Replace this value with the IP address and replace pass with the password of your pmm user.

PostgreSQL Installation

If you already have a PostgreSQL instance running, skip the installation process and continue with the configuration.

On Windows, you can install PostgreSQL using the Windows installer or install it on Ubuntu running under WSL, but it’s better to install it using the official image provided by the PostgreSQL project.

You need to start the container with the latest version of PostgreSQL:

bash
$ docker run --name postgres -e POSTGRES_PASSWORD=password -d postgres

Where password is the password for the default postgres user. Replace this value according to your needs.

PostgreSQL Configuration

Once PostgreSQL is running, you need to get its IP address by running:

$ docker ps

bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0460c671db12 postgres "docker-entrypoint.s…" 6 days ago Up 48 seconds 5432/tcp postgres

Copy the container ID or name of the PostgreSQL container. Then, execute this command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' your_container

Replacing your_container with the value you copied previously.

The IP address of the PMM Client container is also needed.

For configuring PMM for monitoring PostgreSQL, we need to create a PMM user. First, log into PostgreSQL:

$ docker exec -it postgres psql --user postgres

Then, execute the following SQL statement:

sql
CREATE USER pmm WITH SUPERUSER ENCRYPTED PASSWORD '';

Replacing with your desired password.

And finally, register the PostgreSQL server for monitoring:

bash
$ docker exec pmm-client pmm-admin add postgresql --username=pmm --password=pass --host POSTGRESQL_SERVER --query-source=perfschema

Where POSTGRESQL_SERVER is the IP address of the PostgreSQL container. Replace this value with the IP address and replace pass with the password of your pmm user.

MongoDB Installation

If you already have a MongoDB instance running, skip the installation process and continue with the configuration.

On Windows, you can install Percona Server for MongoDB on Ubuntu running under WSL, but it’s better to use the official Docker image, as the MongoDB server would be running on the same network as PMM.

For installing MongoDB using Docker, follow the instructions in the Percona Server for MongoDB documentation.

You need to start the container with the latest version of Percona Server for MongoDB 6.0:

$ docker run --name psmdb -d percona/percona-server-mongodb:6.0

MongoDB Configuration

Once Percona Server for MongoDB is running, you need to get its IP address by running.

$ docker ps

bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2c3d291535b3 percona/percona-server-mongodb:6.0 "/entrypoint.sh mong…" 6 days ago Up 27 minutes 27017/tcp psmdb

Copy the container ID or name of the Percona Server for MongoDB container. Then, execute this command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' your_container

Replacing your_container with the value you copied previously.

The IP address of the PMM Client container is also needed.

For configuring PMM for monitoring MongoDB, we need to create a PMM user. First, connect to the admin database in MongoDB using the MongoDB Shell (mongosh):

$ docker run -it --link psmdb --rm percona/percona-server-mongodb:6.0 mongosh mongodb://MONGODB_SERVER:27017/admin

Where MONGODB_SERVER is the IP address of your MongoDB server.

Then, create the user for PMM, executing the following instructions:

db.createRole({
 "role":"explainRole",
 "privileges":[
 {
 "resource":{
 "db":"",
 "collection":""
 },
 "actions":[
 "collStats",
 "dbHash",
 "dbStats",
 "find",
 "listIndexes",
 "listCollections"
 ]
 }
 ],
 "roles":[]
})

db.getSiblingDB("admin").createUser({
 "user":"pmm",
 "pwd":"",
 "roles":[
 {
 "role":"explainRole",
 "db":"admin"
 },
 {
 "role":"clusterMonitor",
 "db":"admin"
 },
 {
 "role":"read",
 "db":"local"
 }
 ]
})

Replacing with the password you want to assign to the pmm user.

And finally, register the MongoDB server for monitoring:

bash
$ docker exec pmm-client pmm-admin add mongodb --username=pmm --password=pass --host MONGODB_SERVER

Where MONGODB_SERVER is the IP address of the MongoDB container. Replace this value with the IP address and replace pass with the password of your pmm user.

How To Generate Test Data for Your Database Project With Python

Mario García — Mon, 09 Jan 2023 00:00:00 UTC

If you need test data for the database of your project, you can get a dataset from Kaggle or use a data generator. In the first case, if you need to process the data before inserting it into the database, you can use Pandas, a widely used Python library for data analysis. This library supports different formats, including CSV and JSON, and it also provides a method for inserting data into a SQL database.

If you choose a data generator instead, you can find one for MySQL in one of the repositories on our Percona Lab GitHub account. Are you using other database technologies? You can follow the guides I already published where I explain how to create your own data generator for MySQL (it could work for PostgreSQL) and MongoDB.

If you create you’re own data generator, this is the process you must follow:

Generate fake data using Faker
Store generated data in a Pandas DataFrame
Establish a connection to your database
Insert the content of the DataFrame into the database

Requirements

Dependencies

Make sure all the dependencies are installed before creating the Python script that will generate the data for your project.

You can create a requirements.txt file with the following content:

pandas
tqdm
faker

Or if you’re using Anaconda, create an environment.yml file:

yaml
name: percona
dependencies:
 - python=3.10
 - pandas
 - tqdm
 - faker

You can change the Python version as this script has been proven to work with these versions of Python: 3.7, 3.8, 3.9, 3.10, and 3.11.

Depending on the database technology you’re using, you must add the corresponding package to your requirements.txt or environment.yml file:

MySQL → PyMySQL
PostgreSQL → psycopg2
MongoDB → pymongo

Run the following command if you’re using pip:

pip install -r requirements.txt

Or run the following statement to configure the project environment when using Anaconda:

conda env create -f environment.yml

Database

Now that you have the dependencies installed, you must create a database named company, for MySQL or PostgreSQL.

Log into MySQL:

$ mysql -u root -p

Replace root with your username, if necessary, and replace localhost with the IP address or URL of your MySQL server instance if needed.

Or log into PostgreSQL:

$ sudo su postgres
$ psql

and create the company database:

sql
create database company;

You don’t need to create the MongoDB database previously.

Creating a Pandas DataFrame

Before creating the script, it’s important to know that we need to implement multiprocessing for optimizing the execution time of the script.

Multiprocessing is a way to take advantage of the CPU cores available in the computer where the script is running. In Python, single-CPU use is caused by the global interpreter lock, which allows only one thread to carry the Python interpreter at any given time. With multiprocessing, all the workload is divided into every CPU core available. For more information see this blog post.

Now, let’s start creating our own data generator. First, a modules directory needs to be created, and inside the directory, we will create a module named dataframe.py. This module will be imported later into our main script, and this is where we define the method that will generate the data.

You need to import the required libraries and methods:

python
from multiprocessing import cpu_count
import pandas as pd
from tqdm import tqdm
from faker import Faker

pandas. Data generated with Faker will be stored in a Pandas DataFrame before being imported into the database.
tqdm(). This method is required for adding a progress bar to show the progress of the DataFrame creation.
Faker(). It’s the generator from the faker library.
cpu_count(). This is a method from the multiprocessing module that will return the number of cores available.

Then, a faker generator will be created and initialized, by calling the Faker() method. This is required to generate data by accessing the properties in the Faker library.

And we determine the number of cores of the CPU available, by calling the cpu_count() method and assigning this value to the num_cores variable.

python
fake = Faker()
num_cores = cpu_count() - 1

num_cores is a variable that stores the value returned after calling the cpu_count() method. We use all the cores minus one to avoid freezing the computer.

python
def create_dataframe(arg):
 x = int(60000/num_cores)
 data = pd.DataFrame()
 for i in tqdm(range(x), desc='Creating DataFrame'):
 data.loc[i, 'first_name'] = fake.first_name()
 data.loc[i, 'last_name'] = fake.last_name()
 data.loc[i, 'job'] = fake.job()
 data.loc[i, 'company'] = fake.company()
 data.loc[i, 'address'] = fake.address()
 data.loc[i, 'city'] = fake.city()
 data.loc[i, 'country'] = fake.country()
 data.loc[i, 'email'] = fake.email()
 return data

Then, we define the create_dataframe() function, where:

x is the variable that will determine the number of iterations of the for loop where the DataFrame is created.
data is an empty DataFrame that will later be fulfilled with data generated with Faker.
Pandas DataFrame.loc attribute provides access to a group of rows and columns by their label(s). In each iteration, a row of data is added to the DataFrame and this attribute allows assigning values to each column.

The DataFrame that is created after calling this function will have the following columns:

bash
 # Column Non-Null Count Dtype
--- ------ -------------- -----
first_name 60000 non-null object
last_name 60000 non-null object
job 60000 non-null object
company 60000 non-null object
address 60000 non-null object
country 60000 non-null object
city 60000 non-null object
email 60000 non-null object

Note: The script is generating 60 thousand records but it can be adapted to your project, you can modify this value in the x variable.

Connection to the Database

MySQL and PostgreSQL

Before inserting the data previously generated with Faker, we need to establish a connection to the database, and for doing this the SQLAlchemy library will be used.

SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.

python
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

engine = create_engine("mysql+pymysql://user:password@localhost/company")
Session = sessionmaker(bind=engine)

From SQLAlchemy, we import the create_engine() and the sessionmaker() methods. The first one is for connecting to the database, and the second one is for creating a session bond to the engine object.

Don’t forget to replace user, password, and localhost with your authentication details. Save this code in the modules directory and name it as base.py.

For PostgreSQL, replace:

python
engine = create_engine("mysql+pymysql://user:password@localhost/company")

With:

python
engine = create_engine("postgresql+psycopg2://user:password@localhost:5432/company")

Database Schema Definition

For MySQL and PostgreSQL, the schema of the database can be defined through the Schema Definition Language provided by SQLAlchemy, but as we’re only creating one table and importing the DataFrame by calling Pandas to_sql() method, this is not necessary.

When calling Pandas to_sql() method, we define the schema as follows:

python
from sqlalchemy.types import *
schema = {
 "first_name": String(50),
 "last_name": String(50),
 "job": String(100),
 "company": String(100),
 "address": String(200),
 "city": String(100),
 "country" String(100),
 "email": String(50)
}

Then we pass the schema variable as a parameter to this method.

Save this code in the modules directory with the name schema.py.

MongoDB

Before inserting the data previously generated with Faker, we need to establish a connection to the database, and for doing this the PyMongo library will be used.

python
from pymongo import MongoClient

uri = "mongodb://user:password@localhost:27017/"
client = MongoClient(uri)

From PyMongo, we import the MongoClient() method.

Don’t forget to replace user, password, localhost, and port (27017) with your authentication details. Save this code in the modules directory and name it base.py.

Generating Your Data

MySQL and PostgreSQL

All the required modules are now ready to be imported into the main script, now it’s time to create the sql.py script. First, import the required libraries:

python
from multiprocessing import Pool
from multiprocessing import cpu_count
import pandas as pd

From multiprocessing, Pool() and cpu_count() are required. The Python Multiprocessing Pool class allows you to create and manage process pools in Python.

Then, import the modules previously created:

python
from modules.dataframe import create_dataframe
from modules.schema import schema
from modules.base import Session, engine

Now we create the multiprocessing pool, configured to use all available CPU cores minus one. Each core will call the create_dataframe() function and create a DataFrame with 4 thousand records. After each call to the function has finished, all the DataFrames created will be concatenated into a single one.

python
if __name__ == "__main__":
 num_cores = cpu_count() - 1
 with Pool() as pool:
 data = pd.concat(pool.map(create_dataframe, range(num_cores)))
 data.to_sql(name='employees', con=engine, if_exists = 'append', index=False, dtype=schema)

And finally, we will insert the DataFrame into the MySQL database by calling the to_sql() method. All the data will be stored in a table named employees.

The table employees is created without a primary key, so we execute the following SQL statement to add an id column that is set to be the primary key of the table.

python
with engine.connect() as conn:
 conn.execute("ALTER TABLE employees ADD id INT NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST;")

For PostgreSQL, replace this line:

python
conn.execute("ALTER TABLE employees ADD id INT NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST;")

With:

python
conn.execute("ALTER TABLE employees ADD COLUMN id SERIAL PRIMARY KEY;")

MongoDB

All the required modules are now ready to be imported into the main script, now it’s time to create the mongodb.py script. First, import the required libraries:

python
from multiprocessing import Pool
from multiprocessing import cpu_count
import pandas as pd

From multiprocessing, Pool() and cpu_count() are required. The Python Multiprocessing Pool class allows you to create and manage process pools in Python.

Then, import the modules previously created:

python
from modules.dataframe import create_dataframe
from modules.base import client

python
if __name__ == "__main__":
 num_cores = cpu_count() - 1
 with Pool() as pool:
 data = pd.concat(pool.map(create_dataframe, range(num_cores)))
 data_dict = data.to_dict('records')
 db = client["company"]
 collection = db["employees"]
 collection.insert_many(data_dict)

After logging into the MongoDB server, we specify the database and the collection where the data will be stored.

And finally, we will insert the DataFrame into MongoDB by calling the insert_many() method. All the data will be stored in a collection named employees.

Running the script

Run the following statement to populate the table:

$ python sql.py

Or:

$ python mongodb.py

Execution time depends on the CPU cores available on your machine. I’m running this script on an Intel i7 1260P that has 16 cores, but using 15.

Query Your Data

Once the script finishes, you can check the data in the database.

MySQL and PostgreSQL

Connect to the company database.

MySQL:

sql
use company;

PostgreSQL:

sql
\c company

Then, get the number of records.

sql
select count(*) from employees;

The count() function returns the number of records in the employees table.

bash
+----------+
| count(*) |
+----------+
| 60000 |
+----------+
1 row in set (0.22 sec)

MongoDB

use company;
db.employees.count()

The count() function returns the number of records in the employees table.

Or you can display the records in the employees table:

db.employees.find().pretty()

The code shown in this blog post can be found on my GitHub account in the data-generator repository.

Automating Percona's XtraBackup

Phil Plachta — Wed, 04 Jan 2023 00:00:00 UTC

Percona’s XtraBackup is a beautiful tool that allows for the backup and restoration of MySQL databases.

From the documentation:

The Percona XtraBackup tools provide a method of performing a hot backup of your MySQL data while the system is running. Percona XtraBackup is a free, online, open source, complete database backups solution for all versions of Percona Server for MySQL and MySQL®. Percona XtraBackup performs online non-blocking, tightly compressed, highly secure full backups on transactional systems so that applications remain fully available during planned maintenance windows.

It is great but it quickly becomes difficult to wield when using it multiple times per day across multiple environments. XtraBackup Automator attempts to make this easier by providing the ability to:

Schedule when we should create backups
- Times of day, when to make a base backup vs incremental
Archive old backups
- Decide what to do with the base backup and its increments when we are ready to create a new base
Maintain x days of archives
- Define how many archived backup groups should we keep before removing them from the file system

XtraBackup Automator automates away the management of MySQL backups.

Considerations Before Installing

I strongly recommend testing this on some sort of preproduction environment first. The thing I’ve seen most likely to cause trouble is the archival process. By default, this tool uses the gztar ’tarball’ as its compression method, which can be resource intensive if you are working on a large database backup. For instance, one of our servers (a Google Cloud Platform virtual machine with 8 vCPU, 32gb RAM, 1000GB SSD persistent disk, running Debian 10) with a ~140GB base backup currently jumps in CPU usage by ~13% for 4 hours, with a handful of 5%-15% jumps in RAM usage, while creating this archive. Another downside of this compression method is that it can take 10-20 minutes to unzip, depending on settings. The benefit of the tarball is that we are able to take these large backups from 140GB to < 10GB and this is worth all that other trouble for us as we want to have two weeks of daily backups. If these down sides are not acceptable, I recommend playing with the archive type as described in the config. I have not personally tested any other methods.

I am assuming that you have administrative access to the server this will run on as installing systemd services and timers requires root access. I see no reason why Cron Jobs could not be used to run this program, but I have never tested that and all documentation references systemd and its tools.

Info & Requirements

Developed On

OS:
- Debian GNU/Linux 10 (buster)
Python Version:
- Python 3.10.4
Python Packages
- Name: pexpect, Version: 4.8.0
Percona XtraBackup Version:
- XtraBackup version 8.0.28-21 based on MySQL server 8.0.28 Linux (x86_64)
MySQL
- MySql Ver 8.0.28 for Linux on x86_64 (MySQL Community Server - GPL)

Required Python Libraries

pexpect

Required Files

Installing

Below is a general explanation of how to install and start running this program. I would suggest running the program manually via command line a couple times, in a preproduction environment, to verify things are working as you expect.

Download The Files

Download the Required Files from https://github.com/phildoesdev/xtrabackupautomator

Review Your Config Settings

Review the Configuration section of this readme and alter these settings to your liking.
Any altered folder paths may affect the create folder instructions below. At minimum you must include database login information, but alter as necessary. I suggest reading through all the config options to see what might be interesting to tweak.

Edit your systemd service and timer

If you change the location that the script should run from you must alter the file path in the xtrabackupautomator.service file. I will not explain much else here as there is a lot that might go into these settings. I have given some default settings that hopefully make sense.

I have also included several links that describe what is possible in the Sources & Links section. If there are specific questions in the future I will address them here.

Install the required dependencies

$ python3 -m pip install pexpect

Create the directory for our code to live in

bash
$ sudo mkdir /lib/xtrabackupautomator
$ sudo chmod 700 /lib/xtrabackupautomator

Create the directories for our backups to save to

bash
$ sudo mkdir -p /data/backups/mysql
$ sudo mkdir -p /data/backups/archive
$ sudo mkdir -p /data/backups/archive/archive_restore

$ sudo chmod 760 /data/backups/mysql
$ sudo chmod 700 /data/backups/archive
$ sudo chmod 700 /data/backups/archive/archive_restore
$ sudo chown -R root:root /data/backups/

Move your downloaded files

bash
$ sudo mv xtrabackupautomator.py /lib/xtrabackupautomator/.
$ sudo mv xtrabackupautomator.service /etc/systemd/system/.
$ sudo mv xtrabackupautomator.timer /etc/systemd/system/.

Enable your service and timer

bash
$ sudo systemctl daemon-reload

$ sudo systemctl enable xtrabackupautomator.timer

$ sudo systemctl start xtrabackupautomator.timer
$ sudo systemctl status xtrabackupautomator.timer

Congrats, you are now installed!

You have now installed XtraBackup Automator, it will begin running automatically according to your xtrabackupautomator.timer file. If you wish to run it manually, you can run the python file, or use my preferred method, the systemd start command to start it and journalctl to view its output:

bash
$ systemctl start xtrabackupautomator.service
$ journalctl -f -n 100 -u xtrabackupautomator

Unzipping and Restoring your Backup

I have included a link in the Sources & Links section on unzipping tar gz files.

I strongly suggest reading the official Percona documentation on restoring backups.

For a point of a reference, I will describe my generic unzip and restore process. I am using the directory /data/backups/archive/archive_restore/ as a place to unzip and restore from.

Executing any of the below commands can obviously be very dangerous as we must stop mysql, wipe the current data, and restore with our prepared backup. Read the documentation and come up with your own plan! The code below is only meant as a reference and may change greatly with time and environments. Always test your plans in a preprod environment!!

bash
# Always verify our version of Percona's XtraBackup and MySQL match before performing a backup... these differences can make restores fail or behave oddly.
$ sudo mysql --version
$ sudo xtrabackup --version

# Clean our restore folder, just to be safe
$ rm -r /data/backups/archive/archive_restore/*

# Unzip our archived backup to an empty folder. Always verify we have enough disk space to unzip before unzipping
$ sudo tar -xvf database_backup_12_23_2022__06_25_10.tar.gz -C /data/backups/archive/archive_restore/

# Sanity check, verify we are looking at the backup we think we are (This command checks the base folder, check the latest incremental folder we may have)
$ cd /data/backups/archive/archive_restore/data/backups/mysql/
$ sudo cat base/xtrabackup_info | grep 'tool_version\|server_version\|start_time\|end_time\|partial\|incremental'

# Prepare the base
$ sudo xtrabackup --prepare --apply-log-only --no-server-version-check --target-dir=/data/backups/archive/archive_restore/data/backups/mysql/base

# Prepare each incremental folder. This must be done for each incremental folder we wish to back up to.
$ sudo xtrabackup --prepare --apply-log-only --no-server-version-check --target-dir=/data/backups/archive/archive_restore/data/backups/mysql/base --incremental-dir=/data/backups/archive/archive_restore/data/backups/mysql/inc_0

# Repeat as necssary....
...

# Stop SQL and our backup script as we do not want it running mid restore
$ sudo systemctl stop mysql
$ sudo systemctl stop xtrabackupautomator.timer

# Wipe bad/corrupted sql data from current instance
$ sudo rm -rv /var/lib/mysql/*

# Verify our mysql data is wiped
$ sudo ls /var/lib/mysql

# I use this method to restore my base backup, there are other options but they did not work correctly in my environment
$ sudo xtrabackup --copy-back --target-dir=/data/backups/archive/archive_restore/data/backups/mysql/base

# Verify the contents are the size we expect as a sanity check and apply the correct ownership to the files
$ du -hs /var/lib/mysql/
$ sudo chown -R mysql:mysql /var/lib/mysql

# Restart mysql and xtrabackupautomator. Verify MySQL's status
$ sudo systemctl start mysql
$ sudo systemctl xtrabackupautomator.timer
$ sudo systemctl status mysql

Configuration

In an attempt to make this a one file, easy to install piece of software, I included the configuration struct in the xtrabackupautomator.py file, in the __init__ method of the XtraBackupAutomator class, on line ~60 (as of this writing). I will describe that struct, its default values, and other relevant information below. Most of this information can also be found in comments throughout, or in the getter methods for each variable.

== db ==
 -un
 [DEFAULT_VALUE: ""]
 XtraBackup user you set up during your initial configuration of Percona's XtraBackup

 -pw
 [DEFAULT_VALUE: ""]
 This user's password

 -host
 [DEFAULT_VALUE: "localhost"]
 The IP of your database

 -port
 [DEFAULT_VALUE: 3306]
 The port to access database


== folder_names ==
 -base_dir
 [DEFAULT_VALUE: "/data/backups/"]
 The root directory for all backup related things. Holds current backup and any archived backups.
 This is the default location and is reflected in the setup as we request you create this folder.
 If you change this directory in the config this change must be reflected in the setup.

 -datadir
 [DEFAULT_VALUE: "mysql/"]
 Folder that current backups will be saved to. This would be the folder that holds the base backup and any
 incremental backups before they are archived
 If you change this directory in the config this change must be reflected in the setup.
 *** XtraBackupAutomator WILL ARCHIVE AND DELETE ANYTHING IN HERE. THIS SHOULD BE AN EMPTY FOLDER, NOT UTILIZED BY ANYTHING ELSE.

 -archivedir
 [DEFAULT_VALUE: "archive/"]
 Folder that a group of backups will be archived to.
 If you change this directory in the config this change must be reflected in the setup.
 *** XtraBackupAutomator COULD POTENTIALLY DELETE ANY NON-DIRECTORY IN HERE.

== file_names ==
 -basefolder_name
 [DEFAULT_VALUE: "base"]
 Foldername for the base backup

 -incrementalfolder_perfix
 [DEFAULT_VALUE: "inc_"]
 Folder name prefix for incremental backups.
 Suffixed with the current number of incremental backups minus one
 e.g., 'inc_0'

 -archive_name_prefix
 [DEFAULT_VALUE: "database_backup_"]
 Prefix for the archive files.
 Suffixed by the datetime of the archive
 e.g., 'database_backup_11_28_2022__06_25_03.tar.gz'


== archive_settings ==
 -allow_archive
 [DEFAULT_VALUE: True]
 An override to enable/disable all archive settings.
 Currently, disabling this will cause the program to do a base backup and then incremental backups forever.

 -archive_zip_format
 [DEFAULT_VALUE: "gztar"]
 The default archive file type. I like tarballs because they zip our large database into a manageable file.
 However, tarballs can take a long time to create and require a fair amount of resources if your DB is large.
 This setting will depend on your system and the size of your DB. I recommend playing around with this.
 Other zip options: [Shutil Man Page](https://docs.python.org/3/library/shutil.html#shutil.make_archive)

 -archived_bu_count
 [DEFAULT_VALUE: 7]
 Keep x archived backups, once this threshold is reached the oldest archive will be deleted.
 Archiving daily, this is a week of archives.

 -enforce_max_num_bu_before_archive
 [DEFAULT_VALUE: True]
 One of two ways to 'force archive' of backups.
 This counts the # of incremental backup folders and initiates the archives once that number is reached.
 A sample use case is that in your systemd timer file is scheduled to do 5 backups throughout the day, so setting this to
 true and max_num_bu_before_archive_count set to 4 (because we do not count the base) would give you a 'daily archive'

 -max_num_bu_before_archive_count
 [DEFAULT_VALUE: 4]
 The max number of incremental backups to do before we archive (does not count the base).
 Set to 0 to archive after each base

 -enforce_archive_at_time
 [DEFAULT_VALUE: False]
 One of two ways to 'force archive' of backups.
 This will archive what ever base or incremental folders exist if a backup is happening within the
 archive_at_utc_24_hour hour. This is intended to make it easier to schedule when your archive and base backup occur.
 These can be resource intensive and so it is nice to do at off hours.
 *If this program is scheduled to run more than once during the 'archive_at_utc_24_hour' hour each run will cause an archive.

 -archive_at_utc_24_hour
 [DEFAULT_VALUE: 6]
 If a backup happens within this hour we will archive w/e was previously there and create a new base.
 Matching this with a time setup in your xtrabackupautomator.timer allows you to choose when your backups will
 occur.
 No explicit consideration for daylight savings time.
 Defaults to the hour of 1:00am EST, 6:00am UTC.


== general_settings ==
 -backup_command_timeout_seconds
 [DEFAULT_VALUE: 30]
 Give us 'backup_command_timeout_seconds' seconds for the command to respond.
 This is not the same as saying 'a backup can only take this long'.

 -max_time_between_backups_seconds
 [DEFAULT_VALUE: 60*60*24]
 Max number of seconds between this backup and the last.
 If the last backup is older than this we will archive and create a new base.
 This is in an attempt to prevent an incremental backup that might span days or weeks due to this service being
 turned off or some such.
 Defaults (arbitrarily) to 24 hours

 -additional_bu_command_params
 [DEFAULT_VALUE: ["no-server-version-check"]]
 Any additional parameters that you wish to pass along to your backup commands.
 We loop this list, put a '--' before each element and append it to the end of our backup commands.
 This gets applied to the base and incremental backup commands.
 These are params that I have found useful.


== log_settings ==
 -is_enabled
 [DEFAULT_VALUE: True]
 Enables/Disables all logging type settings. This was useful in testing, so I kept it around.

 -log_child_process_to_screen
 [DEFAULT_VALUE: True]
 If this is set to true the child process's output will be dumped to screen but not actually logged anywhere

 -is_log_to_file
 [DEFAULT_VALUE: True]
 If set to True we will try to log to the 'default_log_file' in the 'default_log_path' directory

 -default_log_path
 [DEFAULT_VALUE: "/var/log/"]
 The path that we will try to place our log file ('default_log_file')

 -default_log_file
 [DEFAULT_VALUE: "xtrabackupautomator.log"]
 The file name we will try to log to

Sources & Links

Official Percona XtraBackup Documentation
- https://docs.percona.com/percona-xtrabackup/8.0/index.html
Systemctl Overveiw
Systemctl Timers Overview
- https://linuxconfig.org/how-to-schedule-tasks-with-systemd-timers-in-linux
- https://opensource.com/article/20/7/systemd-timers
Systemctl Services Details
- https://www.freedesktop.org/software/systemd/man/systemd.service.html
Systemctl Timers Details
- https://www.freedesktop.org/software/systemd/man/systemd.timer.html
OnCalendar Expected Formats
- https://www.freedesktop.org/software/systemd/man/systemd.time.html#
Archive Zip Options
- https://docs.python.org/3/library/shutil.html#shutil.make_archive
How to Extract (Unzip) Tar Gz File
- https://linuxize.com/post/how-to-extract-unzip-tar-gz-file/
Restoring Xtrabackup Incremental Backups
- https://docs.percona.com/percona-xtrabackup/8.0/backup_scenarios/incremental_backup.html

Dashboard Story: How We Created PMM Dashboard for Highload

Anton Bystrov — Thu, 22 Dec 2022 00:00:00 UTC

Let’s say you have highload instances. How do you monitor them? There are a lot of servers with 100, 200… 500+ nodes. How can we collect, check, and analyze metrics from all these servers? How can we understand what and where something happened? Scroll, scroll, scroll… down? That was the task that we faced at Percona and successfully resolved.

Issue With Home Dashboard

Percona Monitoring and Management dashboards are based on Grafana. So, when you opened PMM, you can see Grafana’s dashboards. Home Dashboard on the main page of PMM contains different metrics from all environments, databases and other resources. You can see here panels with current resources’ utilization - CPU, Memory, Disk Space, I/O Operations, Network. Certainly, these are very important metrics, which can help us quickly catch some issues… But after some years, we noticed that instances have more and more nodes. And we caught an issue with our Home Dashboard. At big instances (more than 100…200..etc nodes), there were performance issues. It was tough to understand what happened, and a user needed a lot of time for checking. From this point, we started our way.

Searching the Root Cause of Our Issues

Before starting an investigation and searching for “bottleneck”, we defined some questions to answer first:

What’s happened?
Where to catch performance issues?
How can we fix it?

Let’s answer these questions! To find out what’s happened, we need to check the response time for our Home Dashboard. So we created a PMM instance with 200 nodes for testing.

What can we see here? Loading time is more than two minutes! Let’s dive deeper and check the longest requests.

The longest time takes to get a request to VictoriaMetrics storage. If we try to scroll down, we can see “Lazy load” of our page and slower and slower working of our browser. Why? Because we request tons and tons of metrics.

And it seems that we caught here our main problem - too much data, too many requests, and too many responses. What can we do? We decided to create a new Home Dashboard!

Strategies for Creating a Dashboard

There are a lot of strategies for creating dashboards. But we need a short, informative, user-friendly one. Our final goal is to provide a simple answer for the questions: “Is it all good? I want to drink my morning coffee” or “Is something bad? We need to repair it ASAP!”

Let’s investigate what we can do here.

In the O’Reilly Site Reliability Engineering book, we can read about four golden signals strategy: Latency, Traffic, Errors and Saturation. Let’s meet each of these signals.

Latency

Latency means the time it takes to service a request. One important moment — differences between successful and unsuccessful requests.

For example, an HTTP 500 error means that the connection was lost and this error served very quickly, however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors.

Traffic

Traffic is a measure of how much demand is being placed on your system. For web service, it is usually HTTP requests per second, for audio may be network I/O rate, for key-value storage systems — transactions and retrievals per second.

Errors

Errors are the rate of requests that fail, either explicitly (e.g., HTTP 500s), or implicitly (for example, an HTTP 200 success response, but coupled with the wrong content).

Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content.

Saturations

It is a measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential.

If you measure all four golden signals and call for a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring.

There are also USE and RED strategies that we took into consideration.

R — Rate, request per second E — Errors, how many request return error D — Duration, latency, the time it takes to service a request

U — utilization, how fully resource working S — saturation, how long queue at this resources E — errors, how many errors do we have?

POC of Home Dashboard

All these strategies are interesting and helpful. But we want to compile the best for our dashboard. Our Tech Lead Dani Guzmán Burgos created a Proof-of-Concept (POC) of our new Home Dashboard. Our main idea is a simple answer to the questions “all good” or “is something bad.”

When you open this dashboard, you can see simple color panels — green or red. How do we measure this?

Here we can see common information about our environment: how many nodes we have, disk operations, DB and node uptime, and advisors’ checks. There is also a very interesting panel with the name “Environment Health,” which is our secret feature.

For anomaly detection, we use CPU and disk metrics. And here, we also answer questions about how fully our resources are working and what duration we have. On the right panels, we can see data with 15 minutes relative time (to prevent peaks and performance issues). On the left side, we can compare current metrics with metrics from a week ago to measure trends.

In the Command center, we can find more details about what’s wrong. There are three kinds of panels: current usage, anomalies, and metrics for one week ago.

As main metrics, we use CPU, disk queue, write latency, read latency and used memory. These metrics can very quickly help us understand what’s happened in our system.

And finally, the panel Service Summary shows detailed information about each service (node, server) in our system: number of connections to DB, QPS at each of them, and uptime.

Polishing the Dashboard - Feedback Matters

When we discussed the POC with other teams, we got the question, “what does it mean — No anomalies?” Then we added a detailed description “No alerts because CPU less than xx percent.” Sounds better, doesn’t it?

Our previous dashboard looked good, but we wanted more! What could we improve? We already have CPU, Disk anomalies, maybe we can add more metrics here? And we did! High memory? Perfect! Also, to prevent paying a lot for unused hardware, we implemented “Low CPU Servers” where we get alerts when using less than 30 CPUs.

When we have red statuses for nodes in the Anomaly Detection section, we can explore it and drill down. We can jump to a more detailed level and check what happened - CPU, Disk, and Memory for each metric.

The first version of Overview was changed too. We added more details about different databases. Some panels were removed after feedback. And the main feature is filtering.

Here we tried to create a view where a user can choose the environment and see only its nodes.

That’s how we achieved our final goal - you can open the dashboard, check it, and then drink your morning cup of coffee with a calm mind!

Try this out if you’re already using Percona PMM. If you’re not? You can set up and try out PMM in just a few minutes, start with the Quickstart."

Testing Kubernetes with KUTTL

Andrew Minkin — Fri, 16 Dec 2022 00:00:00 UTC

Automated testing is the only way to be sure that your code works. Enabling automated testing can be hard and we say a lot of tools to write automated tests in the industry since the beginning. Some veterans in the industry may remember Selenium, Cucumber frameworks that help automate testing in the browser. However, testing in Kubernetes can be hard.

In Percona we deal with Kubernetes and have different operators to automate the management of databases. It requires testing. A lot of testing. We have different frameworks to help us with it

Codecept.js to write UI tests for PMM. Also, we use a playwright for some cases.
We have tools to help us with API testing as well as automating some routines by running bash commands during the test step.

However, those frameworks are not applicable to test Kubernetes workloads as well as Kubernetes operators. I’ve been working in the PMM integrations team for six months and saw different approaches to automate testing for PMM/DBaaS. We have a Go test library with wrappers around kubectl and codecept.js for end-to-end tests for the User Interface.

What challenges do we have?

Well, to be sure that a database cluster creation works we need to automate the following steps

Installation of operators to Kubernetes cluster
Test integration with version service to respect compatibility matrix.
Create a database cluster and wait once it’ll be available
Do some assertions against Kubernetes as well as UI.

The main pain point here is that we need to wait up to 10-15 minutes for each step and we can’t have different test cases to cover as many cases as we can. Yet we can achieve some performance benefits by paralleling workloads, still, it requires learning Javascript and testing framework to work with it. We had some architectural changes recently and moved from our custom gRPC API to create and manage database clusters to an operator that runs on top of other operators and converts K8s’ Custom Resource from generic format to operator specific. We had a couple of options for this new project and after research, we chose kuttl as a framework for integration/e2e testing.

What is KUTTL anyway and why should I care?

KUTTL is the KUbernetes Test TooL. It’s written in Go and provides a declarative way to test Kubernetes operators using Kubernetes primitives. It’s easy to start kuttling. Let’s take a deeper look. I’ll use dbaas-operator as an example. DBaaS-operator is an operator that has a simple and generic Custom Resource Definition available to create Percona Server MongoDB or Percona XtraDB Cluster instances in kubernetes. It uses underlying operators as a dependencies. We have the following structure

e2e-tests
├── kind.yml
├── kuttl-eks.yml
├── kuttl.yml
└── tests
 └── pxc
 ├── 00-assert.yaml
 ├── 00-deploy-operators.yaml
 ├── 01-assert.yaml
 ├── 01-deploy-pxc.yaml
 ├── 02-assert.yaml
 ├── 02-upgrade-pxc.yaml
 ├── 03-assert.yaml
 ├── 03-restart-pxc.yaml
 ├── 04-delete-cluster.yaml
 ├── 05-assert.yaml
 ├── 05-create-cluster.yaml
 ├── 06-assert.yaml
 ├── 06-scale-up-pxc.yaml
 ├── 07-assert.yaml
 ├── 07-scale-down-pxc.yaml
 └── 08-delete-cluster.yaml

2 directories, 19 files

Let’s discuss these YAML files more

kind.yml contains settings to run Kind
kuttl.yml has all required settings for Kuttl framework and kuttl-eks.yml has some EKS specific configurations
tests folder has test steps and assertions

Kind and KUTTL settings

Let’s discuss Kind and kuttl settings and I’ll start with KUTTL first

yaml
apiVersion: kuttl.dev/v1beta1
kind: TestSuite
kindConfig: e2e-tests/kind.yml  # Path to Kind config that will be used to create Kind clusters
crdDir: config/crd/bases  # Path to a directory that contains CRD files. Kuttl will apply them before running tests
artifactsDir: /tmp/  # Path to a directory to store artifacts such as logs and other information
testDirs:
- e2e-tests/tests  # Path to directories that have test steps

Kind config is quite easy

yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
containerdConfigPatches:
- |-
 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
 endpoint = ["http://kind-registry:5000"]

The aforementioned config will use local registry and will create 3 k8s worker nodes controlled by control plane

Writing tests

At the first glance, kuttling can be easy because it uses Kubernetes primitives as a test step and assertion but I had a couple of problems to test my operator. Let’s take a look at a couple of examples. Since dbaas-operator depends on PXC operator we need to prepare our environment for testing. Let’s write first test that installs PXC operator and ensures that it was installed.

cat e2e-tests/tests/pxc/00-deploy-pxc-operator.yml

apiVersion: kuttl.dev/v1beta1
kind: TestStep
timeout: 10 # Timeout for the test step
commands:
 - command: kubectl apply -f https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v${PXC_OPERATOR_VERSION}/deploy/bundle.yaml -n "${NAMESPACE}"

KUTTL test steps easily extensible with commands. One can run even scripts as a prerequisite for a test case. PXC operator installs CRDs and creates a deployment and here’s an example of assertion.

cat e2e-tests/tests/pxc/00-assert.yml

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 120 # Timeout waiting for the state
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: perconaxtradbclusters.pxc.percona.com
spec:
 group: pxc.percona.com
 names:
 kind: PerconaXtraDBCluster
 listKind: PerconaXtraDBClusterList
 plural: perconaxtradbclusters
 shortNames:
 - pxc
 - pxcs
 singular: perconaxtradbcluster
 scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 name: databaseclusters.dbaas.percona.com
spec:
 group: dbaas.percona.com
 names:
 kind: DatabaseCluster
 listKind: DatabaseClusterList
 plural: databaseclusters
 shortNames:
 - db
 singular: databasecluster
 scope: Namespaced
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: percona-xtradb-cluster-operator
status:
 availableReplicas: 1
 observedGeneration: 1
 readyReplicas: 1
 replicas: 1
 updatedReplicas: 1

Our first test is ready and one needs to run kubectl kuttl test --config ./e2e-tests/kuttl.yml to run kuttl.

More advanced tests

We need to run our operator first to be able to work with resources and test it. KUTTL recomends configure it via TestSuite by the following example

yaml
apiVersion: kuttl.dev/v1beta1
kind: TestSuite
...
commands:
 - command: ./bin/manager
...

However, since dbaas-operator depends on underlying operators it needs to work correctly even if they are not present in a Kubernetes cluster. It has the following logic in the controller

// SetupWithManager sets up the controller with the Manager.
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
 fmt.Println(os.Getenv("WATCH_NAMESPACE"))
 unstructuredResource := &unstructured.Unstructured{}
 unstructuredResource.SetGroupVersionKind(schema.GroupVersionKind{
 Group: "apiextensions.k8s.io",
 Kind: "CustomResourceDefinition",
 Version: "v1",
 })
 controller := ctrl.NewControllerManagedBy(mgr).
 For(&dbaasv1.DatabaseCluster{})
 err := r.Get(context.Background(), types.NamespacedName{Name: pxcCRDName}, unstructuredResource)
 if err == nil {
 if err := r.addPXCToScheme(r.Scheme); err == nil {
 controller.Owns(&pxcv1.PerconaXtraDBCluster{})
 }
 }
 err = r.Get(context.Background(), types.NamespacedName{Name: psmdbCRDName}, unstructuredResource)
 if err == nil {
 if err := r.addPSMDBToScheme(r.Scheme); err == nil {
 controller.Owns(&psmdbv1.PerconaServerMongoDB{})
 }
 }
 return controller.Complete(r)
}

The controller.Owns sets up a controller to watch specified resources and once they were changed it’ll run a reconciliation loop to sync changes. Also, it checks that operator is present in the cluster by checking that deployment and CRDs are available. It means that to make the operator work correctly in tests we need to choose from the following options

Restart operator once upsteam operator was installed by sending HUP signal
Run operator only after underlying operator is present in a cluster

Hence, I moved command as the next step before creating a cluster. You can see it below

cat e2e-tests/tests/pxc/01-deploy-pxc.yml

apiVersion: kuttl.dev/v1beta1
kind: TestStep
timeout: 10
commands:
 - script: WATCH_NAMESPACE=$NAMESPACE ../../../bin/manager
 background: true
---
apiVersion: dbaas.percona.com/v1
kind: DatabaseCluster
metadata:
 name: test-cluster
spec:
 databaseType: pxc
 databaseImage: percona/percona-xtradb-cluster:8.0.23-14.1
 databaseConfig: |
 [mysqld]
 wsrep_provider_options="debug=1;gcache.size=1G"
 secretsName: pxc-sample-secrets
 clusterSize: 1
 loadBalancer:
 type: haproxy
 exposeType: ClusterIP
 size: 1
 image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
 dbInstance:
 cpu: "1"
 memory: 1G
 diskSize: 15G

Note: command supports only simple commands and does not fully support env variables. It supports only $NAMESPACE, $PATH and $HOME. However, script solves the problem of setting WATCH_NAMESPACE environment variable.

In nutshell, the test step above does two things:

Runs the operator
Creates a database cluster

The assertion checks that kubernetes cluster has the DatabaseCluster object with ready status as well as PXC cluster with the same status.

yaml

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 600
---
apiVersion: dbaas.percona.com/v1
kind: DatabaseCluster
metadata:
 name: test-cluster
spec:
 databaseType: pxc
 databaseImage: percona/percona-xtradb-cluster:8.0.23-14.1
 databaseConfig: |
 [mysqld]
 wsrep_provider_options="debug=1;gcache.size=1G"
 secretsName: pxc-sample-secrets
 clusterSize: 1
 loadBalancer:
 type: haproxy
 exposeType: ClusterIP
 size: 1
 image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
 dbInstance:
 cpu: "1"
 memory: 1G
 diskSize: 15G
---
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
 name: test-cluster
spec:
 allowUnsafeConfigurations: true
 crVersion: 1.11.0
 haproxy:
 enabled: true
 image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
 serviceType: ClusterIP
 size: 1
 pxc:
 configuration: |
 [mysqld]
 wsrep_provider_options="debug=1;gcache.size=1G"
 expose: {}
 image: percona/percona-xtradb-cluster:8.0.23-14.1
 livenessProbes: {}
 readinessProbes: {}
 resources:
 requests:
 cpu: "1"
 memory: 1G
 serviceType: ClusterIP
 sidecarResources: {}
 size: 1
 volumeSpec:
 persistentVolumeClaim:
 resources:
 requests:
 storage: 15G
 secretsName: pxc-sample-secrets
 updateStrategy: SmartUpdate
 upgradeOptions:
 apply: 8.0-recommended
 schedule: 0 4 * * *
status:
 ready: 2
 size: 2
 state: ready

Caveats and notes

I had problems running tests in Kind. They were flaky because PXC operator can’t expose metrics and had problems with liveness probe. I haven’t figured out how to fix it but as a workaround I use minikube to run tests

 minikube start --nodes=4 --cpus=2 --memory=4g --apiserver-names host.docker.internal --kubernetes-version=v1.23.6
 minikube kubectl -- config view --flatten --minify > ~/.kube/test-minikube
 KUBECONFIG=~/.kube/test-minikube kubectl kuttl test --config ./e2e-tests/kuttl.yml

Further steps

There’s always room for improvement and I have these steps in mind

Use docker images and OLM bundles as a way to run the operator for tests. This will be the best way to simulate a production like environment.
Add more advanced tests for database clusters such as running queries, try loading data as well as capacity testing. It’s easily achivable with kuttl

Percona Monitoring and Management 2.33 preview release

Taras Kozub — Thu, 08 Dec 2022 00:00:00 UTC

Percona Monitoring and Management 2.33 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.33 is now available as a preview release.

Release notes can be found in here

Percona Monitoring and Management server docker

docker tag:

perconalab/pmm-server:2.33.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.33.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.33 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.33.0.ova

AMI

ami-005acacf35adcfa57

Percona Monitoring and Management 2.32 preview release

Taras Kozub — Fri, 04 Nov 2022 00:00:00 UTC

Percona Monitoring and Management 2.32 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.32 is now available as a preview release.

Release notes can be found in here

Percona Monitoring and Management server docker

docker tag:

perconalab/pmm-server:2.32.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.32.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.32 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.32.0.ova

AMI

ami-02cfe7580e77fb5fa

AWS Summit Mexico City: Back to In-Person Events

Mario García — Wed, 19 Oct 2022 00:00:00 UTC

In March 2020, I presented my latest talk at an in-person event. This event was organized by a local university, Then when events started to happen online, during the pandemic, I was a speaker at about 41 events, where I presented 38 talks and 7 workshops (until September this year), in both Spanish and English.

Having the opportunity to join so many events and collaborate with communities around the world was one of the advantages that events were organized in virtual spaces. I met awesome people doing amazing things, and I learned a lot from them. But I really was missing attending in-person events, especially those being held in other cities.

On July 12, I joined Percona as a Technical Evangelist, and two months later I was at the airport, in my city, waiting for my flight to my first in-person event after 2 and a half years, and the first one as Perconian. AWS Summit Mexico City was held on September 21 and September 22 at Expo Santa Fe Mexico.

The last time I visited the Expo I was attending Campus Party Mexico 2013, with the Firefox OS launch team, for a pre-launch event, and my last trip to Mexico City was in October 2019, before leaving for London to speak at GitLab Commit London.

I was at AWS Summit Mexico City just as an attendee, looking forward to learning more about AWS, doing some networking, and meeting some friends I hadn’t seen in a long time or never seen in person.

The Adventure Had Begun

I booked my flight for September 20 at 9 PM, the only non-stop flight available. I arrived early at the airport, just in time for boarding. It was raining, so take-off was delayed until 10 PM. The plane landed in Mexico City at 11:30 PM.

My hotel was in Santa Fe, near the location of the event. Once I arrived I took an Uber and after 35 minutes I was at the hotel. But to my surprise my reservation was cancelled, and there were no rooms available. There was a problem with their payment system and my debit card was declined.

Had to call other hotels nearby, and finally found one with rooms available, 8 minutes from the Expo. I got a better deal: a lower cost, and a bigger room. Slept just four hours but I was ready for the event.

Visiting Booth

On the first day, I arrived early to the Expo for registering and getting my badge. The expo zone would open at 8 AM, so I had to wait. There were booth by sponsors and AWS.

During the first day these were the booths I visited:

VMWare: When I was in college I tested VMWare for learning about virtualization, and how to install Linux on other operating systems. I haven’t used it since then, but it was interesting to know that VMWare now has some other tools and cloud services like Cloud Health.
Datadog / Dynatrace: While passing by Datadog and Dynatrace booths I had the opportunity to watch a demo of their platforms, specifically those monitoring and observability features they provide for databases, and a wide range of different technologies.

On Datadog integration with AWS you can get:
- Infrastructure Monitoring
- App Performance Monitoring (APM)
- Log Management
- Security Monitoring
  
  More information available in the documentation.
Dynatrace provides monitoring features for the following areas:
- Applications and Microservices
- Infrastructure Monitoring
- Digital Experience
- Database Monitoring
  
  More information available in the documentation.
HashiCorp: I’ve been a HashiCorp Ambassador since 2021 and as part of the program I’ve been creating content related to Vagrant and Packer, including blog posts and a few talks I’ve presented at virtual events. These days I’m learning about Terraform. That’s why I passed by the booth to say hi.
MongoDB: MongoDB is on my list of technologies I would like to learn more about. The resources they shared with me were so helpful for starting my learning journey.
Nextgen.io: Nexgen.io is a platform that provides support in the following areas:
- Transform the traditional monolithic application to micro services
- Provide out-of-the box DevOps features for any development project
- Provide Solid and highly scalable container based platform
- Provide built-in application integration capability with efficient data mapping tool
- Provide out-of-the box B2B capability.
- Managed Services with free upgrade
I was interested in knowing more about the DevOps solutions they provide and what I learned is that through the platform you can get help on setting up the CI/CD pipelines of your project.
AWS Community Builders: Being a member of the GitLab Heroes, GitKraken Ambassadors and HashiCorp Ambassadors programs let me learn more about DevOps and know some tools I use regularly, as well as improving my writing and public speaking skills. Knowing that there’s a AWS Community Builders program where you can be recognized for your contributions to the community, and know more about AWS, the tools and services you have access to when registering on the platform, is an opportunity for anyone looking to be an expert on AWS and expand their network. Applications are not open right now.

Not only was so informative to visit the booths, but allowed me to introduce myself as a Technical Evangelist and be aware of what people know about Percona.

Attending Talks

These are the talks I attended during AWS Summit Mexico City:

Keynote - Kevin Miller / AWS VP - Simple Storage Service
Should I use Serverles? Myths and realities for developers (Spanish) - David Victoria / Emite - Director of Operations
Learn about AWS Global Infrastructure extended to the border - Leonardo Solano / AWS Senior Hybrid Cloud Solutions Architect
How to accelerate containers creation process on AWS with AWS App2Container (A2C) - Oscar Ramírez Vital / AWS Solutions Architect
Accelerating IT modernization in government agencies - Rosendo Martinez, José Luis Vallín
Introduction to security in the cloud with IAM - Uriel Enrique Arellano / Cloud Engineer - Bootcamp Institute
Modernization of education at various levels - Alex Luna / AWS Sr. Solutions Architect, Juan Manuel Zenil / Escuela Bancaria y Comercial - CIO

During the past weeks I’ve been using AWS, had to remember how to create and launch an EC2 instance, I’m learning how to create Kubernetes clusters (with no previous knowledge on k8s) on Amazon Elastic Kubernetes Service, I’ve had to read about AWS Identity and Access Management (IAM), and I’m also learning about eksctl and Terraform.

What I got after attending these talks is having a better understanding of the services and tools available on AWS, knowing about the global infrastructure of the platform, learning about what Serverless is and getting an overview of IAM.

Containerizing applications is one of the topics I’ve been learning about, but never heard of AWS App2Container, that is a command line tool for containerizing Java and .NET applications. While I don’t use any of those technologies, it was interesting to know this tool.

It was also good to hear testimonials of companies and government agencies on how they use AWS, including Nu, CONTPAQi and the government of Querétaro who shared their experience.

Back in March, it was announced that new AWS Local Zones would be launched across Latin America. The new locations included Bogotá, Colombia; Buenos Aires, Argentina; Lima, Peru; Queretaro, Mexico; Rio de Janeiro, Brazil; and Santiago, Chile. This was one of the announcements made in the talks presented by AWS.

While the recordings are not available on Amazon Web Services YouTube channel, here’s a list of videos I recommend to watch:

Networking

Attending AWS Summit Mexico City was also an opportunity for meeting people I hadn’t seen in a long time, like a friend that works at Accenture who I saw the last time back in 2013 at the Firefox OS launch event in Mexico City, and another friend that is also a GitLab Hero who now works at Dynatrace, and I never met in person before.

Having conversations with sponsors and some attendees gave me an overview of what people know about Percona. Most of the people I talked to never heard of Percona before and some were interested to know more about what we do.

And I also was able to meet and spend time with Edith and Donnie, Perconians who were also attending the event.

How Was the Whole Experience?

The good:

Getting an overview of AWS: Services and tools, and global infrastructure
Understanding serverless
Learning about AWS Identity and Access Management (IAM)
Hearing testimonials of how AWS is being used
Simultaneous sessions
Live translation for talks in English

Expectations for future AWS events:

More technical sessions, especially workshops.
Different networking and after-event activities (by organizers and sponsors, and local AWS communities).
A venue with Internet connection

In general, attending AWS Summit Mexico City or any AWS event in the future is something I would recommend to anyone who is starting to use AWS or already have some experience, as this is a place to learn from experts and practitioners, get a better understanding of important concepts, an overview of the platform and the services and tools available, as well as expand the network of contacts and meet local community members, and AWS users.

After the Event

After two days at AWS Summit Mexico City, Edith and I traveled to Querétaro, a city located two and a half hours from Mexico City, for meeting other Perconians. Once there we gathered together for having breakfast and later worked at a local coworking space. We met Mauricio, Eduardo, and David from the Managed Services team.

Resources

Some good resources that were shared during the event:

MySQL: Tracing a single query with PERFORMANCE_SCHEMA

Kristian Köhntopp — Tue, 18 Oct 2022 00:00:00 UTC

My task is to collect performance data about a single query, using PERFORMANCE_SCHEMA (P_S for short) in MySQL, to ship it elsewhere for integration with other data.

In a grander scheme of things, I will need to define what performance data from a query I am actually interested in. I will also need to find a way to attribute the query (as seen on the server) to a point in the codebase of the client, which is not always easy when an ORM or other SQL generator is being used. And finally I will need to find a way to view the query execution in the context of the client code execution, because the data access is only a part of the system performance.

But this is about query execution in the server, and the instrumentation available to me in MySQL 8, at least to get things started. So we take the tour of performance schema, and then run one example query (a simple join) and see what we can find out about this query.

First published on https://blog.koehntopp.info/ and syndicated here with permission of the author.

Performance Schema, the 10.000 m view

The Manual has a major chapter that covers P_S in details. The original idea of P_S is to have a bunch of preallocated memory areas without locks, presented to the database itself as tables.

P_S is unusual in the way that P_S “tables” are never locked while you work with them. That means the values in a “table” can change while you read them. That is important - if you for example calculate percentages, they may not add up to 100%. If you ORDER BY, the sort may or may not be stable.

These are good properties: P_S will not freeze the server, and you won’t kill the server by working with P_S tables.

It is a good idea to make a copy of P_S tables while you work with them, by turning off subquery merging with select /*+ NO_MERGE(t) */ , and then materializing P_S tables in subqueries.

Originally, P_S also had no secondary indexes, so joining P_S tables against other P_S tables did not work efficiently. That was probably a good idea, because joining against a table that is changing while you execute the join is probably generating random results anyway. But because it is so common, and because MySQL itself does this now internally in sys.*, secondary indexes to join efficiently now exist. That does not make the joins more correct, but at least you get the result faster.

I wrote about all this in an earlier article.

Instruments, Objects, Actors, Threads and Consumers

The data P_S collects is centered around three major things:

Time consumed. In database servers, that is mostly wait time - waiting on I/O or locks.
Data transferred. In database servers, that is mostly pages read or written. In a way, this related to I/O wait.
Memory used. In database servers, that is buffers allocated - how large, and how often, and peak usage.

P_S collects this in the form of “events” (and takes care to note that P_S events are not binlog events or other any events). The collection points are in the database server code, which is instrumented, so the collectors are instruments.

The thing that the server code works on may be of a certain kind, for example a table or another object in the server, but have a variable identity (that would be different tables, with different names). Instruments can be filtered by using object names.

The activity done in the server is done on behalf of a database user, in the form of user@host or, new in MySQL 8, using roles. The entity on which behalf the server is working on is called the actor.

The activity done in the server is also done in the context of a thread, some of which are background threads, while the majority in a busy server are usually connection threads.

And finally, the data collected is put into the in-memory tables of P_S. These come in various groups, and are called consumers.

Data is collected from objects using instruments. Instruments can be turned on and off. Their collected data is then filtered by Objects, Actors and Threads, and finally dropped into consumers. Many consumers are aggregates, some collect information specific to one query execution.

For each of these things there is a setup_... table that controls how event data is collected by the instrumentation, filtered and the consumed in result tables. In parallel, object identities are collected in ..._instances tables, which are needed to resolve object identities.

sql
mysql [localhost:8025] {msandbox} (performance_schema) > show tables like "setup_%";
+----------------------------------------+
| Tables_in_performance_schema (setup_%) |
+----------------------------------------+
| setup_actors |
| setup_consumers |
| setup_instruments |
| setup_objects |
| setup_threads |
+----------------------------------------+
5 rows in set (0.00 sec)

mysql [localhost:8025] {msandbox} (performance_schema) > show tables like "%_instances";
+--------------------------------------------+
| Tables_in_performance_schema (%_instances) |
+--------------------------------------------+
| cond_instances |
| file_instances |
| mutex_instances |
| prepared_statements_instances |
| rwlock_instances |
| socket_instances |
+--------------------------------------------+
6 rows in set (0.00 sec)

mysql [localhost:8025] {msandbox} (performance_schema) > select * from socket_instances;
+----------------------------------------+-----------------------+-----------+-----------+-----------+-------+--------+
| EVENT_NAME | OBJECT_INSTANCE_BEGIN | THREAD_ID | SOCKET_ID | IP | PORT | STATE |
+----------------------------------------+-----------------------+-----------+-----------+-----------+-------+--------+
| wait/io/socket/mysqlx/tcpip_socket | 106328376 | 44 | 21 | :: | 18025 | ACTIVE |
| wait/io/socket/mysqlx/unix_socket | 106328688 | 44 | 22 | | 0 | ACTIVE |
| wait/io/socket/sql/server_tcpip_socket | 106329000 | 1 | 26 | 127.0.0.1 | 8025 | ACTIVE |
| wait/io/socket/sql/server_unix_socket | 106329312 | 1 | 28 | | 0 | ACTIVE |
| wait/io/socket/sql/client_connection | 106330560 | 50 | 41 | | 0 | ACTIVE |
+----------------------------------------+-----------------------+-----------+-----------+-----------+-------+--------+
5 rows in set (0.00 sec)

Current, History and History Long Tables vs. Summaries

P_S collects data in a lot of summary table, which should not interest us that much here. Our task is to look at the performance data of a single, individual query to better understand what happened when it ran.

These unaggregated tables are events_transactions, events_statements, events_stages and events_waits. For each of them, we have _current, _history or _history_long tables.

The _current tables contain one entry for the currently running thread. The _history tables contain a configurable number of entries for each thread, for example 10 per thread. And the _history_long tables contain a configurable number of entries, shared across all threads, for example 10.000 in total. As the server continues to execute statements and produce events, old entries are discarded and new entries are added, automatically. Additionally, each query execution is aggregated along several dimensions in summary tables. Summary tables state these dimension(s) using by_, for example _by_user_by_eventname or similar.

In current MySQL, P_S is enabled by default. But not all instruments and consumers are enabled, because some instrumentation slows query execution down, and some consumers can use a lot of memory. To be fast and safe, all memory is statically allocated at config change so the memory resource usage of P_S is constant and no allocations are made during execution.

We can enable all instrumentation completely with this SQL:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > UPDATE
 -> performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES';
Query OK, 494 rows affected (0.00 sec)
Rows matched: 1216 Changed: 494 Warnings: 0

mysql [localhost:8025] {msandbox} (performance_schema) > UPDATE
 -> performance_schema.setup_consumers SET ENABLED = 'YES';
Query OK, 0 rows affected (0.00 sec)
Rows matched: 15 Changed: 0 Warnings: 0

When we look at one of these tables, for example events_statements_current, we see a structure like this:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > show create table events_statements_current\G
*************************** 1. row ***************************
 Table: events_statements_current
Create Table: CREATE TABLE `events_statements_current` (
 `THREAD_ID` bigint unsigned NOT NULL,
 `EVENT_ID` bigint unsigned NOT NULL,
 `END_EVENT_ID` bigint unsigned DEFAULT NULL,
 `EVENT_NAME` varchar(128) NOT NULL,
 `SOURCE` varchar(64) DEFAULT NULL,
 `TIMER_START` bigint unsigned DEFAULT NULL,
 `TIMER_END` bigint unsigned DEFAULT NULL,
 `TIMER_WAIT` bigint unsigned DEFAULT NULL,
...
 `NESTING_EVENT_ID` bigint unsigned DEFAULT NULL,
 `NESTING_EVENT_TYPE` enum('TRANSACTION','STATEMENT','STAGE','WAIT') DEFAULT NULL,
 `NESTING_EVENT_LEVEL` int DEFAULT NULL,
 `STATEMENT_ID` bigint unsigned DEFAULT NULL,
 PRIMARY KEY (`THREAD_ID`,`EVENT_ID`)
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
1 row in set (0.00 sec)

That is, events are tagged with a THREAD_ID (which is not a CONNECTION_ID() as seen in processlist), an EVENT_ID/END_EVENT_ID bracket, various source and timer values and for further dissection, a NESTING_EVENT_ID and _TYPE.

We can translate processlist ids into thread ids using the P_S.THREADS table, and then use this to limit our view on the events_statements_current table:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select thread_id from threads
 -> where processlist_id = connection_id();
+-----------+
| thread_id |
+-----------+
| 50 |
+-----------+
1 row in set (0.00 sec)

mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_id,
 -> event_name,
 -> source,
 -> sys.format_time(timer_wait),
 -> sql_text
 -> from events_statements_current
 -> where thread_id = 50\G
*************************** 1. row ***************************
 thread_id: 50
 event_id: 88463
 event_name: statement/sql/select
 source: init_net_server_extension.cc:94
sys.format_time(timer_wait): 341.04 us
 sql_text: select thread_id, event_id, event_name, source, sys.format_time(timer_wait), sql_text from events_statements_current where thread_id = 50
1 row in set (0.00 sec)

So running this query took 341 Microseconds, or 0.341 ms. And sources are named after their location in the server sourcecode, filename and line number.

Events exist in a hierarchy: wait events nest within stage events, which nest within statement events, which nest within transaction events. Some nested events refer to their own type, for example, statement events can point to other statement events they are nested in. Other events refer to their enclosing context in the hierarchy. Nesting Event ID, Type and Level make this clear.

Instrument names

The manual explains Instrument Names.

They have path-like names that group instruments hierarchically, for example wait/io/file/innodb/innodb_data_file. This is a wait event, io related, specifically file I/O, more specific innodb and even more specific innodb_data_file. Looking at other fields in the events_waits_history table, we would see the file name as part of the OBJECT_SCHEMA.OBJECT_NAME designator for this event. That means, we can see how long we waited for I/O coming from this specific file or going to the file.

Further up in the nesting we would see, at the statement level, the actual SQL_TEXT, and also the number of rows scanned. That means we can get a rough estimate why this particular statement instance was slow - for example, the plan was good, the number of rows was low, but we see a lot of actual file IO waits, so probably the buffer pool was cold.

The manual page above discusses the instrument names at length, and it is important to get an overview of what exists and what is measured. Specifically, for statement level entries the instruments vary during query execution and become more detailed, as they reflect the progress in understanding of the server about the nature of the statement as it is executed.

An example run

In a freshly restarted idle server, we log in to a shell and use world for the world sample database. This is a tiny database, but because the server has been just restarted, nothing of it is cached. We run a simple query:

sql
mysql [localhost:8025] {msandbox} (world) > select
 -> co.continent,
 -> co.name as country,
 -> co.population as copop,
 -> ci.name as capital,
 -> ci.population as cipop
 -> from country as co join city as ci
 -> on co.capital = ci.id
 -> where continent = 'Europe';
+-----------+-------------------------------+-----------+------------------------------------+---------+
| continent | country | copop | capital | cipop |
+-----------+-------------------------------+-----------+------------------------------------+---------+
| Europe | Albania | 3401200 | Tirana | 270000 |
...
| Europe | Yugoslavia | 10640000 | Beograd | 1204000 |
+-----------+-------------------------------+-----------+------------------------------------+---------+
46 rows in set (0.00 sec)

Now let’s check what we can find out, using a second session:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select * from threads
 -> where processlist_db = 'world'\G
*************************** 1. row ***************************
 THREAD_ID: 48
 NAME: thread/sql/one_connection
 TYPE: FOREGROUND
 PROCESSLIST_ID: 9
 PROCESSLIST_USER: msandbox
 PROCESSLIST_HOST: localhost
 PROCESSLIST_DB: world
PROCESSLIST_COMMAND: Sleep
 PROCESSLIST_TIME: 4
 PROCESSLIST_STATE: NULL
 PROCESSLIST_INFO: NULL
 PARENT_THREAD_ID: 1
 ROLE: NULL
 INSTRUMENTED: YES
 HISTORY: YES
 CONNECTION_TYPE: Socket
 THREAD_OS_ID: 346752
 RESOURCE_GROUP: USR_default
1 row in set (0.00 sec)

In our case we are only interested in the fact that our thread/sql/one_connection in the processlist is shown as connection 9, but internally has a thread_id of 48. The Linux Operating System PID is 346752.

We can use this to check events_transactions_wait, and find

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_id,
 -> state,
 -> source,
 -> sys.format_time(timer_wait) as timer,
 -> nesting_event_id,
 -> nesting_event_type
 - from events_transactions_history
 -> where thread_id = 48;
+-----------+----------+-----------+-----------------+-----------+------------------+--------------------+
| thread_id | event_id | state | source | timer | nesting_event_id | nesting_event_type |
+-----------+----------+-----------+-----------------+-----------+------------------+--------------------+
| 48 | 179 | COMMITTED | handler.cc:1328 | 496.05 us | 85 | STATEMENT |
| 48 | 401 | COMMITTED | handler.cc:1328 | 559.99 us | 278 | STATEMENT |
| 48 | 5606 | COMMITTED | handler.cc:1328 | 3.03 ms | 5574 | STATEMENT |
+-----------+----------+-----------+-----------------+-----------+------------------+--------------------+
3 rows in set (0.00 sec)

Why are there three statement events? We can check:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_id,
 -> sql_text
 -> from events_statements_history
 -> where thread_id = 48 and event_id in (85, 278, 5574);
+-----------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| thread_id | event_id | sql_text |
+-----------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 48 | 85 | show databases |
| 48 | 278 | show tables |
| 48 | 5574 | select co.continent, co.name as country, co.population as copop, ci.name as capital, ci.population as cipop from country as co join city as ci on co.capital = ci.id where continent = 'Europe' |
+-----------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)

The mysql command line client is running from the sandbox with /home/kris/opt/mysql/8.0.25/bin/mysql --defaults-file=/home/kris/sandboxes/msb_8_0_25/my.sandbox.cnf world. Autocompletion for names is not disabled. So on client startup, the client invisibly runs show databases to learn the names of all databases for autocompletion. It then enters the world database as requested and runs show tables to learn the names of all tables in the world database.

Only then we come to the prompt and can paste our query.

We are only interested in thread_id = 48 AND event_id = 5574, our query.

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select *
 -> from events_statements_history
 -> where thread_id = 48 AND event_id = 5574\G
*************************** 1. row ***************************
 THREAD_ID: 48
 EVENT_ID: 5574
 END_EVENT_ID: 6624
 EVENT_NAME: statement/sql/select
 SOURCE: init_net_server_extension.cc:94
 TIMER_START: 61628458852000
 TIMER_END: 61631769329000
 TIMER_WAIT: 3310477000
 LOCK_TIME: 224000000
 SQL_TEXT: select co.continent, co.name as country, co.population as copop, ci.name as capital, ci.population as cipop from country as co join city as ci on co.capital = ci.id where continent = 'Europe'
 DIGEST: 409c336982f0d3d45c4b29da77fe83aed12c6043e8ce9771c11ec82ff347e647
 DIGEST_TEXT: SELECT `co` . `continent` , `co` . `name` AS `country` , `co` . `population` AS `copop` , `ci` . `name` AS `capital` , `ci` . `population` AS `cipop` FROM `country` AS `co` JOIN `city` AS `ci` ON `co` . `capital` = `ci` . `id` WHERE `continent` = ?
 CURRENT_SCHEMA: world
 OBJECT_TYPE: NULL
 OBJECT_SCHEMA: NULL
 OBJECT_NAME: NULL
 OBJECT_INSTANCE_BEGIN: NULL
 MYSQL_ERRNO: 0
 RETURNED_SQLSTATE: NULL
 MESSAGE_TEXT: NULL
 ERRORS: 0
 WARNINGS: 0
 ROWS_AFFECTED: 0
 ROWS_SENT: 46
 ROWS_EXAMINED: 92
CREATED_TMP_DISK_TABLES: 0
 CREATED_TMP_TABLES: 0
 SELECT_FULL_JOIN: 0
 SELECT_FULL_RANGE_JOIN: 0
 SELECT_RANGE: 0
 SELECT_RANGE_CHECK: 0
 SELECT_SCAN: 0
 SORT_MERGE_PASSES: 0
 SORT_RANGE: 0
 SORT_ROWS: 0
 SORT_SCAN: 0
 NO_INDEX_USED: 0
 NO_GOOD_INDEX_USED: 0
 NESTING_EVENT_ID: NULL
 NESTING_EVENT_TYPE: NULL
 NESTING_EVENT_LEVEL: 0
 STATEMENT_ID: 125
1 row in set (0.00 sec)

The statement is statement/sql/select. It took 3310477000 picoseconds (3.31ms) to run. The sql_text is the full text of the statement (up to a cutoff point, in order to manage memory consumption). The parsed statement is called digest_text - identifiers are quoted, whitespace is normalized, actual constants are replaced with placeholders, and (not shown here) variable length WHERE ... IN (...) clauses are shortened with ellipses. This normalized digest is then hashed and produces an identifier for this group of identically formed statements, the digest. We learn about the number of rows looked at, 92 and sent, 46. No special flags indicating specific execution modes were set.

We can use the event_id to look even deeper:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_id,
 -> event_name,
 -> source,
 -> sys.format_time(timer_wait) as timer
 -> from events_stages_history
 -> where thread_id = 48
 -> and nesting_event_id = 5574
 -> order by event_id;
+-----------+----------+--------------------------------------+----------------------+-----------+
| thread_id | event_id | event_name | source | timer |
+-----------+----------+--------------------------------------+----------------------+-----------+
| 48 | 5610 | stage/sql/optimizing | sql_optimizer.cc:270 | 15.88 us |
| 48 | 5611 | stage/sql/statistics | sql_optimizer.cc:534 | 703.58 us | <- !!
| 48 | 5710 | stage/sql/preparing | sql_optimizer.cc:618 | 31.93 us |
| 48 | 5712 | stage/sql/executing | sql_union.cc:1126 | 2.26 ms | <- !!
| 48 | 6593 | stage/sql/end | sql_select.cc:586 | 1.86 us |
| 48 | 6594 | stage/sql/query end | sql_parse.cc:4542 | 1.93 us |
| 48 | 6596 | stage/sql/waiting for handler commit | handler.cc:1594 | 9.05 us |
| 48 | 6600 | stage/sql/closing tables | sql_parse.cc:4593 | 14.02 us |
| 48 | 6621 | stage/sql/freeing items | sql_parse.cc:5042 | 29.4 us |
| 48 | 6623 | stage/sql/cleaning up | sql_parse.cc:2252 | 1.17 us |
+-----------+----------+--------------------------------------+----------------------+-----------+
10 rows in set (0.00 sec)

These are the various execution stages of our statement - we select by thread_id and with the event_id of the statement, 5574 as a nesting_id, ordered by event_id. Time was consumed by the stage/sql/statistics phase, looking up table stats for a good execution plan, and then by the actual query execution in stage/sql/executing. The former took 0.7ms (703.58us), the latter 2.26ms.

We are interested in what took so long, specifically, so we look into waits for event_ids 5611 and 5712 - finding nothing, and also nothing particularly time-consuming:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_id,
 -> event_name,
 -> sys.format_time(timer_wait) as timer,
 -> nesting_event_type,
 -> nesting_event_id,
 -> operation
 -> from events_waits_history
 -> where thread_id = 48
 -> order by event_id;
+-----------+----------+------------------------------------------+-----------+--------------------+------------------+-----------+
| thread_id | event_id | event_name | timer | nesting_event_type | nesting_event_id | operation |
+-----------+----------+------------------------------------------+-----------+--------------------+------------------+-----------+
| 48 | 6614 | wait/synch/mutex/innodb/trx_mutex | 56.26 ns | STAGE | 6600 | lock |
| 48 | 6615 | wait/synch/mutex/innodb/trx_mutex | 48.64 ns | STAGE | 6600 | lock |
| 48 | 6616 | wait/synch/mutex/innodb/trx_mutex | 50.4 ns | STAGE | 6600 | lock |
| 48 | 6617 | wait/synch/mutex/innodb/trx_mutex | 46.88 ns | STAGE | 6600 | lock |
| 48 | 6618 | wait/synch/mutex/innodb/trx_mutex | 43.36 ns | STAGE | 6600 | lock |
| 48 | 6619 | wait/synch/mutex/innodb/trx_mutex | 45.12 ns | STAGE | 6600 | lock |
| 48 | 6620 | wait/synch/mutex/sql/LOCK_table_cache | 66.8 ns | STAGE | 6600 | lock |
| 48 | 6622 | wait/io/socket/sql/client_connection | 18.91 us | STAGE | 6621 | send |
| 48 | 6624 | wait/synch/mutex/sql/THD::LOCK_thd_query | 124.23 ns | STAGE | 6623 | lock |
| 48 | 6626 | wait/io/socket/sql/client_connection | NULL | WAIT | 6625 | recv |
+-----------+----------+------------------------------------------+-----------+--------------------+------------------+-----------+
10 rows in set (0.00 sec)

We can see I/O in a global summary, and the timings make sense in the context of the experiment:

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> object_type,
 -> object_schema,
 -> object_name,
 -> count_star,
 -> sys.format_time(max_timer_read) as `read`,
 -> sys.format_time(max_timer_write) as `write`,
 -> sys.format_time(max_timer_fetch) as `fetch`
 -> from table_io_waits_summary_by_table
 -> where object_schema = 'world'\G
*************************** 1. row ***************************
 object_type: TABLE
object_schema: world
 object_name: city
 count_star: 46
 read: 569.17 us
 write: 0 ps
 fetch: 569.17 us
*************************** 2. row ***************************
 object_type: TABLE
object_schema: world
 object_name: country
 count_star: 47
 read: 703.4 us
 write: 0 ps
 fetch: 703.4 us
...

But why are the I/O times not visible to us? That’s a bit unclear. My theory was that the reads happen asynchronously by some background thread. But a quick query shows no time spend on reader threads.

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> t.thread_id,
 -> t.name,
 -> t.type
 -> from threads as t join events_waits_summary_by_thread_by_event_name as e
 -> on t.thread_id = e.thread_id
 -> where e.max_timer_wait > 0
 -> and e.event_name = 'wait/io/file/innodb/innodb_data_file';
+-----------+---------------------------------------------+------------+
| thread_id | name | type |
+-----------+---------------------------------------------+------------+
| 1 | thread/sql/main | BACKGROUND |
| 9 | thread/innodb/io_write_thread | BACKGROUND |
| 10 | thread/innodb/io_write_thread | BACKGROUND |
| 11 | thread/innodb/io_write_thread | BACKGROUND |
| 12 | thread/innodb/io_write_thread | BACKGROUND |
| 13 | thread/innodb/page_flush_coordinator_thread | BACKGROUND |
| 33 | thread/innodb/clone_gtid_thread | BACKGROUND |
| 47 | thread/sql/one_connection | FOREGROUND |
| 48 | thread/sql/one_connection | FOREGROUND |
+-----------+---------------------------------------------+------------+
9 rows in set (0.00 sec)

We seem to be unable to attribute time spent loading data from disk to a specific thread, and we seem to be unable to account for the runtime of certain stages by looking at waits. That’s unexpected.

Memory only as summary

Diverse memory_% tables exist to track memory usage in the server. All of these tables are summary tables, there are no memory events tables that could trace memory usage per query. That might be okay.

sql
mysql [localhost:8025] {msandbox} (performance_schema) > show tables like '%memory%';
+-----------------------------------------+
| Tables_in_performance_schema (%memory%) |
+-----------------------------------------+
| memory_summary_by_account_by_event_name |
| memory_summary_by_host_by_event_name |
| memory_summary_by_thread_by_event_name |
| memory_summary_by_user_by_event_name |
| memory_summary_global_by_event_name |
+-----------------------------------------+
5 rows in set (0.00 sec)

We can do interesting things with stuff such as memory_summary_by_thread_by_event_name, at least on our mostly idle server.

sql
mysql [localhost:8025] {msandbox} (performance_schema) > select
 -> thread_id,
 -> event_name,
 -> count_alloc,
 -> sum_number_of_bytes_alloc,
 -> high_number_of_bytes_used
 -> from memory_summary_by_thread_by_event_name
 -> where thread_id = 48
 -> and HIGH_NUMBER_OF_BYTES_USED > 0
 -> order by HIGH_NUMBER_OF_BYTES_USED desc limit 5;
+-----------+-------------------------------+-------------+---------------------------+---------------------------+
| thread_id | event_name | count_alloc | sum_number_of_bytes_alloc | high_number_of_bytes_used |
+-----------+-------------------------------+-------------+---------------------------+---------------------------+
| 48 | memory/sql/THD::main_mem_root | 22 | 1181008 | 613544 |
| 48 | memory/innodb/memory | 226 | 1059368 | 250032 |
| 48 | memory/sql/dd::objects | 205 | 47432 | 44648 |
| 48 | memory/innodb/ha_innodb | 26 | 35784 | 35784 |
| 48 | memory/innodb/fil0fil | 2 | 65600 | 32800 |
+-----------+-------------------------------+-------------+---------------------------+---------------------------+
5 rows in set (0.00 sec)

No EXPLAIN

Another thing that would be useful to collect from P_S is the actual execution plan of a query. But while we can explain a lot of statements by running EXPLAIN , and while we can EXPLAIN FOR CONNECT ..., the former is not the recorded execution plan, and the latter only works while the query is running. It’s the actual execution plan while the query executes, but it is not recorded.

Summary

A lot of information about query execution can be gathered from P_S. The query execution can be broken down in statements, stages and waits. Specifically, statements collect a lot of interesting quality flags. Stages can collect percentages of completion for long-running queries and give a general feel about where in the query execution time is spent. Waits should be able to attribute time to individual operations in the database server, but specifically for file I/O this seems to be more complicated, and I have not been able to solve it.

We can see waits for I/O summary tables, and we can see a lot of other statistical information in other summary tables. We can also use additional tables not covered here for debugging (for example DATA_LOCKS for locking behavior).

Memory instrumentation is interesting, but at this stage it is unclear to me if it is sufficient.

It seems to be really hard to record execution plans together with statements.

More experimentation with more complicated queries is necessary to see if it is possible to see things like sorting, temp files and similar operations, and attribute time to these operations.

The number of queries on P_S necessary to extract information about a single query is staggering, a 10:1 ratio. At least filters exist and are on by default, so that I do not have to hear my monitoring noise in my monitoring. That is good.

I could really use a single large JSON blob containing the entire package with performance data for a query, at once - one query to trace one query. That is, the information from transaction, statement, stages, waits, the execution plan and the memory consumption for a given transaction or statement, in one go.

First published on https://blog.koehntopp.info/ and syndicated here with permission of the author.

What is Open Source and why should you care

Teslim Balogun — Fri, 14 Oct 2022 00:00:00 UTC

The term Open Source Software reminds me of Abraham Lincoln’s widely accepted definition of Democracy. Lincoln said, “Democracy is the government of the people, by the people, and for the people”. Similarly, Open Source is software of the community, by the community, and for the community.

Formally, Open Source refers to projects or programs whose source code can be modified, shared, and/or commercialized by the public/community at will. A license that is attached to an Open Source project determines the extent to which the project can be consumed, modified, or utilized.

Open Source is everywhere, the browser you are using to view this page is likely an Open Sourced one. Android OS, Linux Kernel, and JavaScript are examples of OS projects that people like you and I improve daily. As a developer, technical writer, or designer; Open Source opens the doors to many opportunities; jobs, networking, communication skills, etc.

Note: The term Open Source in this article will mostly refer to Open Source Software and Open Source in general.

Benefits of open source
Getting started with Open Source
Clearing up some misconceptions
Where to find Open Source projects
- Finding projects to contribute to
- Knowing where to make changes or contributions in a project
What kind of contributions are legit
Some Interesting GitHub projects you can contribute to
Joining the Open Source community
- Notable Open Source Advocates
Bonus: Hacktoberfest
Conclusion

Benefits of open source

Being a member of the Open Source Community brings many benefits including job opportunities, sponsorships, networking, improved communication skills, software integrity, and much more. Job recruiters watch out for people who give back to the community. Contributing to OS will boost your chances of getting jobs because it shows that you are willing to foster new relationships and work with others. Remotely interacting with new people also improves your ability to communicate ideas effectively when working on projects.

In the OS community, you are free to break things and make honest mistakes. Others will fix them and ensure that the software is working correctly. With Open Source, even when you abandon a project. The community will keep it alive. This saves you a great deal of time to work on other projects. Some major benefits of contributing to Open Source include:

Getting free Swag.
Networking and meeting new people.
It makes you a better coder/writer - People will correct you whenever you break stuff or find errors in your work.

Getting started with Open Source

Are you new to the Open Source Community? If your answer is yes, then you are in the right place. Even if your answer is No, there are a few tips that you can still take away from this post.

Clearing up some misconceptions

Ability to code is necessary for Open Source: Open Source is not limited to software, there are many low code/ no-code OS projects that you can find on GitHub today.
High technical skill is required to start contributing: This is not true at all, simple changes like fixing typos, grammatical corrections, and spelling errors are all welcome in the OS community as long as it adds value or improvement.

Where to find Open Source projects

If you are looking for Open Source Projects to contribute to, look no further than GitHub - the defacto home of Open Source projects. GitHub is home to millions of Open Source projects. There are lots of projects that can align with your specialty if you know how to look properly.

Finding Open Source projects to contribute to can be difficult, especially for newcomers. One reason is that they are not looking in the right places. When it comes to contributing to Open Source, newcomers are often faced with two problems:

Finding projects to contribute to
Knowing where to make changes or contributions to a project

Finding projects to contribute to

Go to GitHub
Click issues in the navigation panel
Filter the results by entering a keyword into the search bar e.g good first issue, help-wanted, JavaScript, technical writing, documentation, React, etc
Search for something specific within your domain such as react, documentation, design, etc good first issue and first-timers only are labels for issues that are appropriate for newcomers to work on. Try these first if you are just starting out.
Find an issue you would like to work on. If it hasn’t been assigned to someone else, ask the maintainers if you can work on the issue and create a PR for it.
Many big projects often have abandoned or incomplete issues. You can find these by searching for “todo” or looking through much older issues.

Note: Alternatively, you can visit any of the websites below to find issues without going through the hassle of following the steps above. The following websites are tailor-made for finding issues easily:

If you are not familiar with Open Source etiquette, then I suggest that you read these briefly:

Knowing where to make changes or contributions to a project

Believe it or not, knowing how to find bugs, errors, mistakes, etc in files in a project is also a skill. If you go into a project’s repo, you’ll likely spot an issue somewhere if you look close enough. some of the most common issues you can find in repos include:

Broken links in websites, project documentation, etc
Grammatical and Spelling errors
Lacklustre designs
Lack of translation, a11y, ARIA etc.
Absence of comprehensive documentation
Absence of issue/PR templates in a repo
Lack of contribution guidelines

Note: Always make sure you go through the README.md and CONTRIBUTING.md files of a repository before creating Pull Requests. This will ensure that you adhere to a project’s guidelines for making contributions. Sometimes valuable PRs are rejected because contributor’s did not follow a Project’s rules. This could be something as trivial as not giving your PR an appropriate title or description. So be wary of this

These are just a few off the top of my head. There are many ways in which you can find issues with a project. Ensure that you do not open irrelevant or low-quality issues. Also, bear these in mind before opening issues.

A good way to know if your issue is a quality one is to ask yourself , “how would this change or suggestion help the users and maintainers of this project.” If your answer does not sound credible to you, do not open it.
Your issue should tell the maintainers of a project the relevance of your suggestion

What kind of contributions are legit?

Contributions to Open Source are not limited to Pull Requests only. Raising Issues, making Code Reviews, Financial Contributions, etc. are other ways of contributing to Open Source. Until recently, non-code stuff like blog posts, Figma designs, etc was not popular in the Open Source community. Things are beginning to change, however. This year’s Hacktoberfest which is accepting low-code and non-code contributions for the first time after 8 years is a good example of this new trend.

Some Interesting GitHub projects you can contribute to

Percona - Percona is an Open Source platform for monitoring, securing and optimizing database environments (MySQL, PostgreSQL, MongoDB etc) on any infrastructure.
LinkFree - Free Open Source alternative to Linktree.
defaang - Free Open Source alternative to LeetCode.
Code-Magic - A website for generating performant CSS with GUI. Plus it’s purely based on TypeScript, CSS, and HTML (No frameworks involved).
FreeCodeCamp Quiz Site - You can add new quizzes to the site by following the instructions in the repo.
Abbreve - A website for quickly checking the meaning of common abbreviations and slang used for communicating over social media.

Joining the Open Source community

Community, community, community… We keep mentioning community in the world of Open Source. This is because Open Source would not exist without the amazing group of individuals who are constantly working hard to make Open Source projects free and accessible to people like you and me.

Below is a list of arguably the most popular Open Source communities on Twitter (that I know of).

FreeCodeCamp
Percona
EddieHub Everything Open Source (contributions, hackathons, first timers, etc).
4C A large OS community for OS projects and networking.
defaang / dojo clan YK Dojo’s OS community.
GitHub - The largest Open Source community in the world, all the communities listed above have their repositories hosted on GitHub.

Note: You should have a good reason for joining any of these communities. If you want to benefit from these communities, you should engage with the members, interact and find people who are within your domain of interests, and do not spam. Also, ensure that you adhere to the rules of these communities too.

Notable Open Source Advocates

Follow these people on Twitter to get all the latest updates on the happenings in the Open Source community:

Eddie Jaoude - Eddie is a devoted member of the OS community, he hosts Twitter spaces regularly to help people who are just getting started in Open Source. He has a YouTube channel where he creates content beyond Open Source (freelancing tips, content creation tips, mini-tutorials, etc)
Dunsin - Dunsin is the creator of Code-Magic, which is a website for generating CSS code for different effects through GUI. She’s also an active member of the OS community on Twitter.
YK Dojo - YK is also a popular YouTuber and an avid member of the OS community. He often does live-coding Streams on Twitch too

These are just a few names in the OS community on Twitter. There are so many more Open Source advocates on Twitter apart from the 3 individuals listed above.

Bonus: Hacktoberfest

Hacktoberfest is a yearly celebration of Open Source and the Open Source community throughout October. Hacktoberfest is organized yearly by Digital Ocean.

A minimum of 4 accepted Pull Requests is required before the 25th of October to win a Hacktoberfest-themed shirt and sticker. Starting from this year, low-code and/or no-code contributions will also be accepted as valid contributions.

If you’d like to take part in the next Hacktoberfest, set a reminder for October now to avoid missing out on all the fun.

Conclusion

Imagine a world without Open Source. As you can see, there’s nothing to hate about Open Source. The world of Open Source exposes you to opportunities, meeting new friends, winning swag/money . Without Open Source, the world would certainly not be as advanced as it is today in terms of technology.

Learning Kubernetes Operators with Percona Operator for MongoDB

Edith Puclla — Thu, 13 Oct 2022 00:00:00 UTC

One of the topics that have resonated a lot for me since the first KubeCon I attended in 2018 is Kubernetes Operators.

The concept of Operators was introduced much earlier in 2016 by the CoreOS Linux development team. They were looking for a way to improve automated container management in Kubernetes.

What do we mean by a Kubernetes Operator?

We use the definition of CNCF. The Kubernetes project defines “Operator” simply: “Operators are software extensions that use custom resources to manage applications and their components“.

This means that among the applications that can be run on Kubernetes, there are applications that still require manual operations to manage them and complete the Kubernetes deployment cycle because Kubernetes itself can’t manage all these manual operations. It is what the Operators take care of, to automate those manual processes of the applications deployed in Kubernetes.

How can this be possible?

The Operators use/extend the Kubernetes API (this API has the basics needed for a user to interact with the Kubernetes cluster) and create custom resources to add new functionality according to the needs of an application to be flexible and scalable.

Once the creation of the custom resource is finished, it creates objects that can be managed using kubectl, as other default Kubernetes resources are managed, such as Deployments, Pods, etc.

Here we see the difference between the workflows with and without operators.

With Operators

Without Operators

The above illustration is based on a presentation by Sai Vennam.

It is time for an example! We will see how Percona Operator for MongoDB works.

Percona Operator for MongoDB contains everything we need to quickly and consistently deploy and scale Percona Server for MongoDB instances into a Kubernetes cluster on-premises or in the cloud.

You can find Percona Operator for MongoDB officially in:

Why does Percona Server for MongoDB (a database) need a Kubernetes Operator?

Kubernetes has been designed for stateless applications. Kubernetes in many cases doesn’t require operators for stateless applications because Kubernetes doesn’t need more automation logic. But stateful applications like databases do need operators because they cannot automate the entire process natively.

One of the main benefits of operators is the automation of repetitive tasks that are often handled by human operators, eliminating errors in application lifecycle management.

Installing MongoDB Percona Operator using GKE

This guide shows you how to deploy Percona Operator for MongoDB on Google Kubernetes Engine (GKE). We use GKE which takes less time to set up Kubernetes in Google Cloud just for the purpose of this demo. This demonstration assumes you have some experience with the platform. For more information on the GKE, see Kubernetes Engine Quickstart

As prerequisites, we need Google Cloud shell and Kubectl. You can find the installation guides for AWS and AZURE in the Percona documentation. Let´s start!

Creating a GKE cluster with three nodes.

bash
gcloud container clusters create my-cluster-name --project percona-product --zone us-central1-a --cluster-version 1.23 --machine-type n1-standard-4 --num-nodes=3

Now you should configure the command-line access to your newly created cluster to make kubectl able to use it.

gcloud container clusters get-credentials my-cluster-name --zone us-central1-a --project percona-product

Finally, use your Cloud Identity and Access Management [Cloud IAM] to control access to the cluster. The following command will give you the ability to create Roles and RoleBindings:

bash
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account)

Install the Operator and deploy your MongoDB cluster

Create a new namespace called percona-demo-namespace

kubectl create namespace percona-demo-namespace

Set the context for the namespace

bash
kubectl config set-context $(kubectl config current-context) --namespace=percona-demo-namespace

Deploy the Operator

kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.13.0/deploy/bundle.yaml

The operator has been started, and you can deploy your MongoDB cluster:

kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.13.0/deploy/cr.yaml

When the process is over, your cluster will obtain the ready status. Check with:

 kubectl get psmdb.

Note: “psmdb” stands for Percona Server for MongoDB

Verifying the cluster operation

You will need the login and password for the admin user to access the cluster. Use kubectl get secrets command to see the list of Secrets objects

kubectl get secret my-cluster-name-secrets -o yaml

Bring it back to a human-readable form to MONGODB_DATABASE_ADMIN_PASSWORD and MONGODB_DATABASE_ADMIN_USER

We check the details of the Services, before testing the connection to the cluster

Run a Docker container with a MongoDB client and connect its console output to your terminal. The following command will do this, naming the new Pod percona-client:

bash
kubectl run -i --rm --tty percona-client --image=percona/percona-server-mongodb:4.4.16-16 --restart=Never -- bash -il

Now run mongo tool in the percona-client command shell using the login (which is normally clusterAdmin)

mongo "mongodb://clusterAdmin:Dgqjc1HElUvvGnH9@my-cluster-name-mongos.percona-demo-namespace.svc.cluster.local/admin?ssl=false"

Woolaaa! We have deployed MongoDB in Kubernetes using Operator, It works! :)

Now that you have the MongoDB cluster, you have full control to configure and manage MongoDB deployment from a single Kubernetes control plane, which means that you can manage MongoDB instances in the same way you manage default objects in Kubernetes like Deployments, Pods, or Services. For advanced configuration, topics see our guide Percona Operator for MongoDB.

Conclusion

Kubernetes Operators extend the Kubernetes API to automate processes that cannot be achieved natively with Kubernetes. This is the case for stateful applications like MongoDB. Percona develops Percona Operator for MongoDB that contains everything you need to quickly and consistently deploy and scale Percona Server for MongoDB instances into a Kubernetes cluster on-premises or in the cloud. You can try it on different cloud providers and tutorials for more advanced configurations.

You can find Percona Operator for MongoDB in Hacktoberfest! If you’re looking to improve your Kubernetes skills, this is a great project to start contributing to.

We also have a public roadmap of Percona Kubernetes Operators. If you have any feedback or want to draw our attention to a particular feature, feel free to be part of it and vote for issues! :)

Resources:

Recap Monthly Percona Developer Meetup Hacktoberfest

Edith Puclla — Wed, 05 Oct 2022 00:00:00 UTC

The Monthly Percona Developer Meetup is an opportunity to get a behind-the-scenes view of different projects in Percona and directly interact with the experts to exchange ideas, ask questions, etc.

From now on, Monthly Percona Developer Meetups will take place monthly to have open discussions and in-person (online) communication. You can join our Developer Meetup via Restream, the Percona YouTube channel, and LinkedIn events.

The topic of the first Monthly Percona Developer Meetup was Hacktoberfest.

What is Hacktoberfest? Hacktoberfest is an annual event hosted by DigitalOcean that encourages people to contribute to Open Source throughout October. It is inclusive, everyone can join to work on Open Source projects, and you can participate by choosing your favorite project.

You need two essential things to participate. First, register on the Hacktoberfest website anytime between September 26 and October 31, 2022. Second, look through the participating projects, choose your favorites, and consult the documentation guiding you to send your first contribution.

Forty thousand participants who complete Hacktoberfest and have at least four pull/merge requests accepted between October 1 and October 31 can have a tree planted in their name or the Hacktoberfest T-shirt. In that case, you can pick a tree planted in your name or the Hacktoberfest 2022 T-shirt.

Let’s ask the Percona experts!

Which Percona projects joined Hacktoberfest?

All the Percona GitHub Projects with the label: “hacktoberfest” are ready to contribute.

How do I find issues?

On GitHub, the issues are tagged with the good-first-issue tag.
On Jira, tagged with newbie, hacktoberfest and onboarding
You can also pick any other issue you like or create new ones.

What types of contributions count?

You can contribute in several ways: coding, documentation, testing, design, discussions, Content Creator (Blog posts, videos), etc.; they all count.

How can you reach the Percona team?

The best way to do it is directly in the Percona projects, or you can join our Community Forum.

Look at some of the Percona projects added to Hacktoberfest this year.

Starting for percona/percona-docker. Supported by Evgeniy Patlan, Manager, Build & Release Engineering. There are images for basic scenarios. The idea is to create more images and improve the existing ones or extend them to Docker Compose. Also, contributions to improve the Docker setup are welcome. You can find issues in Jira or GitHub, where you can also participate in discussions about Percona Docker images.

Our next project is Percona Monitoring and Management (PMM). This project is supported by Artem Gavrilov, Backend Software Engineer, and Nurlan Moldomurov, Full-Stack Engineer. PMM is a great project to contribute to during Hacktoberfest. There are minor and easy-to-do issues in Github; it is not necessary to register them in Jira. These contributions are welcome if you have any good ideas, want to improve something or simplify some process. Send your PR; the maintainers will review it as soon as possible. We also have advanced issues if you want to go for more advanced tasks.

For our third project, we have percona/mongodb_exporter. Carlos Salguero is the maintainer for this project. He says it is a project easy to get started, and there is not much-complicated logic behind this. It is about running some MongoDB internal commands to get statistics like diagnostic data or replica status, passing JSON to produce metrics from these commands. You use a complete Makefile to start sandbox instances to test almost everything; you don’t have a virtual machine or different MongoDB instances. The issues are in GitHub and Jira. The primary programming language is Go.

It’s the turn of percona/percona-server-mongodb-operator; Denys Kondratenko is the maintainer of this project. It is an excellent opportunity to learn about Kubernetes and how to extend it and maintain stateful databases inside Kubernetes. Most of the things are tracked in GitHub and Jira.

Our next project is percona/pg_stat_monitor, a Query Performance Monitoring tool for PostgreSQL. The maintainer is Ibrar Ahmed, Sr. Software Engineer (PostgreSQL). The primary area to contribute is the releases. If there are contributions with ideas on improving the information provided for pg_stat_monitor, they are welcome. The issues are defined in Jira.

In summary, those are some of our projects that are in Hacktoberfest. This month is Open Source month; it’s Hacktoberfest month party!

Remember that you can interact with the maintainers of each project through GitHub/Jira or our Community Forum.

If you haven’t sent a PR, this is the time to do it. Write us at community-team@percona.com if you have any questions.

Happy Hacktoberfest with Percona!

Percona Monitoring and Management 2.31 preview release

Taras Kozub — Mon, 19 Sep 2022 00:00:00 UTC

Percona Monitoring and Management 2.31 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.31 is now available as a preview release.

Release notes can be found in here

Known issue

PMM-10735: OVF stopped working in a few minutes.

Percona Monitoring and Management server docker

docker tag:

perconalab/pmm-server:2.31.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.31.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.31 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

Creating a Kubernetes cluster on Amazon EKS with eksctl

Mario García — Tue, 13 Sep 2022 00:00:00 UTC

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service that makes it easy for you to run Kubernetes on AWS and on-premises. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Amazon EKS is certified Kubernetes-conformant, so existing applications that run on upstream Kubernetes are compatible with Amazon EKS.

Getting started guides available in the AWS documentation explain two different procedures for creating an EKS cluster. One using eksctl, a simple command line utility for creating and managing Kubernetes clusters on Amazon EKS, and the other one using the AWS Management Console and AWS CLI.

Through this article, you will learn how to use eksctl for creating a Kubernetes cluster on Amazon EKS.

eksctl is a simple CLI tool for creating and managing clusters on EKS - Amazon’s managed Kubernetes service for EC2. It is written in Go, uses CloudFormation, and was created by Weaveworks.

For using eksctl, you must:.

Install kubectl.
Install AWS IAM Authenticator for Kubernetes.
Install AWS CLI.
Create a user with minimal IAM policies.

After running eksctl, you will get a cluster with default configuration:

Exciting auto-generated name
Two m5.large worker nodes
Use the official AWS EKS AMI
Default us-west-2 region
A dedicated VPC

Creating IAM user

Go to console.aws.amazon.com/iamv2, create a user group, named EKS, and attach the policies described in the minimal IAM policies section from the eksctl documentation.

These policies already exist, and you must attach them as they are.

AmazonEC2FullAccess (AWS managed)
AWSCloudFormationFullAccess (AWS managed)

In addition to previous policies, you must create:

EksAllAccess (click to expand)

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": "eks:*",
 "Resource": "*"
 },
 {
 "Action": [
 "ssm:GetParameter",
 "ssm:GetParameters"
 ],
 "Resource": [
 "arn:aws:ssm:*::parameter/aws/*",
 "arn:aws:ssm:*::parameter/aws/*"
 ],
 "Effect": "Allow"
 },
 {
 "Action": [
 "kms:CreateGrant",
 "kms:DescribeKey"
 ],
 "Resource": "*",
 "Effect": "Allow"
 },
 {
 "Action": [
 "logs:PutRetentionPolicy"
 ],
 "Resource": "*",
 "Effect": "Allow"
 }
 ]
}

IAMLimitedAccess (click to expand)

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "iam:CreateInstanceProfile",
 "iam:DeleteInstanceProfile",
 "iam:GetInstanceProfile",
 "iam:RemoveRoleFromInstanceProfile",
 "iam:GetRole",
 "iam:CreateRole",
 "iam:DeleteRole",
 "iam:AttachRolePolicy",
 "iam:PutRolePolicy",
 "iam:ListInstanceProfiles",
 "iam:AddRoleToInstanceProfile",
 "iam:ListInstanceProfilesForRole",
 "iam:PassRole",
 "iam:DetachRolePolicy",
 "iam:DeleteRolePolicy",
 "iam:GetRolePolicy",
 "iam:GetOpenIDConnectProvider",
 "iam:CreateOpenIDConnectProvider",
 "iam:DeleteOpenIDConnectProvider",
 "iam:TagOpenIDConnectProvider",
 "iam:ListAttachedRolePolicies",
 "iam:TagRole",
 "iam:GetPolicy",
 "iam:CreatePolicy",
 "iam:DeletePolicy",
 "iam:ListPolicyVersions"
 ],
 "Resource": [
 "arn:aws:iam:::instance-profile/eksctl-*",
 "arn:aws:iam:::role/eksctl-*",
 "arn:aws:iam:::policy/eksctl-*",
 "arn:aws:iam:::oidc-provider/*",
 "arn:aws:iam:::role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup",
 "arn:aws:iam:::role/eksctl-managed-*"
 ]
 },
 {
 "Effect": "Allow",
 "Action": [
 "iam:GetRole"
 ],
 "Resource": [
 "arn:aws:iam:::role/*"
 ]
 },
 {
 "Effect": "Allow",
 "Action": [
 "iam:CreateServiceLinkedRole"
 ],
 "Resource": "*",
 "Condition": {
 "StringEquals": {
 "iam:AWSServiceName": [
 "eks.amazonaws.com",
 "eks-nodegroup.amazonaws.com",
 "eks-fargate.amazonaws.com"
 ]
 }
 }
 }
 ]
}

Replace , in both policies, with your AWS account ID, you can find it in the upper right corner, in the navigation bar. For other ways of getting your account ID, go to Your AWS account ID and its alias in the docs.

Add a new user, named eksctl, to the group previously created.

Don’t forget to download or copy your credentials, Access Key ID and Secret Access Key, as you will need them for setting up authentication.

Installing AWS CLI

On Linux, download the installer:

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

Unzip the installer:

$ unzip awscliv2.zip

And run the installer:

$ sudo ./aws/install

For instructions on how to install AWS CLI on other operating systems, go to Installing or updating the latest version of the AWS CLI in the documentation.

After installing AWS CLI, run the following command for setting up authentication locally:

$ aws configure --profile eksctl

It will ask you for your AWS credentials and default region.

Installing AWS IAM Authenticator

On Linux, run the following command:

$ curl -o aws-iam-authenticator https://s3.us-west-2.amazonaws.com/amazon-eks/1.21.2/2021-07-05/bin/linux/amd64/aws-iam-authenticator

Apply execute permissions to the binary:

$ chmod +x ./aws-iam-authenticator

Create a folder in your $HOME directory and add it to the $PATH variable:

$ mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin

Add $HOME/bin to your .bashrc:

$ echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc

For Mac and Windows, check Installing aws-iam-authenticator in the documentation.

Installing kubectl

NOTE: From the documentation - You must use a kubectl version that is within one minor version difference of your Amazon EKS cluster control plane. For example, a 1.22 kubectl client works with Kubernetes 1.21, 1.22, and 1.23 clusters.

As of today, the latest version of Kubernetes used by eksctl is 1.21. Run the following command for installing the corresponding version of kubectl:

$ curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.21.2/2021-07-05/bin/linux/amd64/kubectl

Apply execute permissions to the binary:

$ chmod +x ./kubectl

Copy the binary to $HOME/bin:

$ cp ./kubectl $HOME/bin/kubectl

If you’re using another version of Kubernetes, check Installing or updating kubectl in the documentation, where you can also find instructions for other operating systems.

Installing eksctl and creating a Kubernetes cluster

Download the binary and copy it to /usr/local/bin:

$ curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

$ sudo mv /tmp/eksctl /usr/local/bin

On Mac and Windows, you can install eksctl following the instructions in the GitHub repository.

Once installed, create a cluster with default configuration, and authenticate to AWS using IAM user created previously.

$ eksctl create cluster --profile eksctl

NOTE: From the documentation - That command will create an EKS cluster in your default region (as specified by your AWS CLI configuration) with one managed nodegroup containing two m5.large nodes.

For a cluster with custom configuration, create a config file, named cluster.yaml, with the following content:

yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
 name: basic-cluster
 region: eu-north-1

nodeGroups:
 - name: ng-1
 instanceType: m5.large
 desiredCapacity: 10
 volumeSize: 80
 ssh:
 allow: true # will use ~/.ssh/id_rsa.pub as the default ssh key
 - name: ng-2
 instanceType: m5.xlarge
 desiredCapacity: 2
 volumeSize: 100
 ssh:
 publicKeyPath: ~/.ssh/ec2_id_rsa.pub

Run eksctl to create the cluster as follows:

$ eksctl create cluster -f cluster.yaml --profile eksctl

While running, eksctl will create the cluster and all the necessary resources.

It will take a few minutes to complete. After the command is executed, you can go to us-east-1.console.aws.amazon.com/eks/home?region=us-east-1#/clusters to see the cluster.

Don’t forget to replace us-east-1 in the URL, if your default region is different.

Cluster credentials can be found in ~/.kube/config. Try kubectl get nodes to verify that this file is valid, as suggested by eksctl.

If, for any reason, you need to delete your cluster, just run:

$ eksctl delete cluster --name=ferocious-painting-1660755039 --profile eksctl

Replacing name with corresponding value.

You’ve created your first Kubernetes cluster using eksctl. Check the documentation for more information on how to create and manage clusters.

Running PMM with Docker on Ubuntu 20.04

Edith Puclla — Fri, 05 Aug 2022 00:00:00 UTC

I started at Percona a few weeks ago and was looking for a quick way to learn about PMM (Percona Monitoring and Management), which is one of my favorite technologies within Percona to monitor the health of our database infrastructure, explore new patterns in the database behavior, manage and improve the performance of our databases, all with customizable dashboards and real-time alerts using Grafana and VictoriaMetrics.

The best of all is that PMM is Open Source, you can check the PMM repository in case you want to contribute.

There are many flavors for PMM installation, here I will describe the steps to install PMM on Ubuntu 20.04, using Docker for PMM Server on an Amazon EC2 instance.

This image summarizes our goal.

Requirements

An Amazon EC2 instance with Ubuntu 20.04
- This instance is configured with a Security Group with TCP port 443 open.
Docker
- You can install Docker by following this guide.
- Manage Docker as a non-root user: sudo usermod -aG docker $USER
MySQL
- I am using Percona Server for MySQL from Percona apt repository

Installing PMM Server with Docker

Download PMM server Docker image

docker pull percona/pmm-server:2

Create the data volume container

docker create --volume /srv --name pmm-data percona/pmm-server:2 /bin/true

Run PMM server container

docker run --detach --restart always --publish 443:443 --volumes-from pmm-data --name pmm-server percona/pmm-server:2

Verify the creation of the container.

docker ps

Start a web browser and in the address bar enter the IP address of the PMM server host: https://:443/. For example, https://172.31.53.46. If you are running on your local machine use https://localhost:443/. Woohoo! We have a PMM Server running and we can see our dashboard!

Note: Some browsers may not trust the self-signed SSL certificate when you first open the URL. If this is the case, Chrome users may want to type thisisunsafe to bypass the warning.

The user and password are “admin” and “admin”, It will ask you to change the password after login in for the first time, for this demo I will use admin2020 as a password. We will use these credentials to register the node in PMM Server later.

Until now we have only PMM Server. To monitor a database, we need a PMM client.

Installing PMM client

PMM Client is a collection of agents and exporters that run on the host being monitored. Let´s install it using the repository package.

Download Percona Repo Package

bash
wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb

Install Percona Repo Package

bash
sudo apt install ./percona-release_latest.$(lsb_release -sc)_all.deb

Update apt cache

sudo apt update

Install Percona Monitoring and Management Client

sudo apt install pmm2-client

Checking the installation. We will use pmm-admin in the next steps.

sudo pmm-admin -v

Creating a user for monitoring

Let’s create a user in MySQL.

Login in MySQL for use the command-line: mysql -uroot -p
Create a “pmm” user with “welcOme1!” As a password

bash
CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'welcOme1!' WITH MAX_USER_CONNECTIONS 10;

Give “pmm” user with specific permission to monitor the database

bash
GRANT SELECT, PROCESS, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';

Checking if the user was created correctly with the respective permissions, use

bash
 show grants for 'pmm'@'localhost';

Connect Client to Server

sudo pmm-admin config --server-insecure-tls --server-url=https://admin:admin2020@172.17.0.1:443

Note: I am using 172.17.0.1 because this is the private IP where the PMM Server is running. You can get this IP by entering the docker container and typing “hostname -I”

After registering your client with the server you will see this information:

Check if the node was registered

pmm-admin inventory list nodes

A new node should appear in the list which is pmm-server

Adding a MySQL Database to monitoring

Use pmm-admin to register the database with the user we created in MySQL

bash
sudo pmm-admin add mysql --username=pmm --password=welcOme1! --query-source=perfschema

In the dashboard, we will see that our node and database are registered and ready to be monitored by PMM.

That’s it! :) We learned how to monitor our databases for free with Percona Monitoring Database (PMM). Additionally, you can go to the next level by registering a PMM instance with Percona Platform and receive more information.

I hope you’ve enjoyed this tutorial, and if you need help following it, feel free to contact the Percona team support. We will be happy to help.

Setting up PMM for monitoring MySQL on a local environment

Mario García — Fri, 05 Aug 2022 00:00:00 UTC

Percona Monitoring and Management (PMM) is an open source database monitoring, observability, and management tool that can be used for monitoring the health of your database infrastructure, exploring new patterns in database behavior, and managing and improving the performance of your databases no matter where they are located or deployed.

PMM is designed to work with MySQL (including Percona Server for MySQL, Percona XtraDB Cluster, Oracle MySQL Community Edition, Oracle MySQL Enterprise Edition, and MariaDB), PostgreSQL (including Percona Distribution for PostgreSQL), MongoDB (including Percona Server for MongoDB), Amazon RDS, Amazon Aurora, Proxy SQL, and Percona XtraDB Cluster.

Debian, Ubuntu, and Red Hat (AlmaLinux, Oracle Linux or Rocky Linux may also work) are supported. If you try installing on another distribution, might get the following message when trying to activate ps80, pdps-8.0 or pdpxc-8.0 repositories:

bash
$ sudo percona-release setup ps80
Specified repository is not supported for current operating system!

Meaning your OS is not supported yet.

While PMM, both the server and the client, can be installed on most operating systems, if you want to set up MySQL on an OS that is not supported, you must consider configuring a virtual machine for Percona Server for MySQL.

Check the documentation for more details about repositories maintained by Percona and supported platforms.

You can find system requirements for PMM in the Frequently Asked Questions. PMM Server and PMM clients communicate through the ports specified in the Terminology section.

Note: Instructions for installing PMM and Percona Server for MySQL, described in the following sections, are for Debian, Ubuntu and derivatives. For Red Hat and derivatives, check the Quickstart guide and Installing Percona Server for MySQL on Red Hat Enterprise Linux and CentOS from the documentation.

Configuring a virtual machine for MySQL

If you’re on Linux and using a distribution that is not supported, configure a virtual machine before installing Percona Server for MySQL, otherwise continue with the “Installing and Configuring MySQL” section.

Multipass

Multipass is an open source tool to generate cloud-style Ubuntu VMs quickly on Linux, macOS, and Windows.

It gives you a simple but powerful CLI that allows you to quickly access an Ubuntu command line or create your own local mini-cloud. On Linux, Multipass must be installed through a snap package. If Snap is not installed on your system, check the documentation for instructions on how to install.

$ sudo snap install multipass

Then, create your virtual machine:

$ multipass launch lts --name percona

By default, when running multipass launch lts –name percona, Multipass will create a virtual machine with 1 GB of RAM and a 4.7 GB disk. A new installation of MySQL only uses 2.4 GB, along with the operating system. A VM created with default configuration of Multipass would be enough for running MySQL.

If you need a virtual machine with more resources, you can create a custom one with desired memory, storage and CPUs.

$ multipass launch lts --name percona --mem 2G --disk 10G --cpus 2

The previous command will create a VM with 2 GB of RAM, a 10 GB disk and 2 CPUs.

Once your VM is created and launched, you can access it by running multipass shell percona.

No additional configuration is required. Ports will be open automatically, and you can connect to any service configured on your VM through the IP address assigned by Multipass.

multipass info percona will give you information about your VM, including IP address.

bash
Name: percona
State: Running
IPv4: 10.203.227.64
Release: Ubuntu 20.04.4 LTS
Image hash: 692406940d6a (Ubuntu 20.04 LTS)
Load: 0.09 0.09 0.10
Disk usage: 2.3G out of 9.5G
Memory usage: 550.4M out of 1.9G
Mounts: –

10.203.227.64 is the IP address of your virtual machine. You will need this value to set up PMM for monitoring MySQL.

Run ip route show to know the IP address that the host is identified by when logging into the VM. You will see a line similar to this:

10.203.227.0/24 dev mpqemubr0 proto kernel scope link src 10.203.227.1

10.203.227.1 is the IP address that Multipass uses to identify the host.

Both host and virtual machine IP addresses are required for configuring PMM.

Log into your virtual machine for continuing with installation of Percona Server for MySQL:

$ multipass shell percona

Installing required packages

Install curl and gnupg2

Before installing PMM or Percona Server for MySQL, make sure curl and gnupg2 are installed.

$ sudo apt install -y curl gnupg2

Install percona-release

The percona-release configuration tool allows users to automatically configure which Percona Software repositories are enabled or disabled. It supports both apt and yum repositories. Percona Server for MySQL will be installed from the ps80 repository and percona-release is necessary for activating this repository.

A good resource to learn more about this tool is this article from Percona blog.

Get the repository packages:

bash
$ wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb

Install the downloaded package with dpkg:

bash
$ sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb

Installing and Configuring MySQL

Install Percona Server for MySQL

Enable the ps80 repository:

$ sudo percona-release setup ps80

Install percona-server-server, the package that provides the Percona Server for MySQL:

$ sudo apt install percona-server-server

After installation, confirm that the service is running. You can check the service status by running:

$ service mysql status

If the server is running, you will get the following output:

bash
● mysql.service - Percona Server
 Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
 Active: active (running) since Mon 2022-08-01 08:20:59 CDT; 1h 20min ago
 Main PID: 15552 (mysqld)
 Status: "Server is operational"
 Tasks: 38 (limit: 2339)
 Memory: 362.7M
 CGroup: /system.slice/mysql.service
 └─15552 /usr/sbin/mysqld

Aug 01 08:20:57 percona systemd[1]: Starting Percona Server...
Aug 01 08:20:59 percona systemd[1]: Started Percona Server.

Otherwise, start the server:

$ sudo service mysql start

Install MySQL Shell

MySQL Shell is an advanced client and code editor for MySQL. This document describes the core features of MySQL Shell. In addition to the provided SQL functionality, similar to mysql, MySQL Shell provides scripting capabilities for JavaScript and Python and includes APIs for working with MySQL

Install MySQL Shell by running:

$ sudo apt install percona-mysql-shell

MySQL Shell will be used for configuring PMM to monitor MySQL. When necessary, just log into MySQL Shell as root:

$ mysqlsh root@localhost

It will ask you for the password you assigned to the root user during the installation of Percona Server for MySQL.

Installing and Configuring PMM

PMM runs from a container, so Docker must be installed if not already on your system. Percona has an easy-install script that would install Docker and any other required packages, as well as installing PMM Server.

The easy-install script provided by Percona checks if Docker is already on your system, otherwise it uses the get-docker script that will create a docker.list file inside the /etc/apt/sources.list.d directory, containing the official repository, and it will install and configure Docker on your system.

Run the following command to get PMM Server:

$ curl -fsSL https://www.percona.com/get/pmm | /bin/bash

Install PMM client:

$ sudo apt install pmm2-client

Connect client to server:

$ sudo pmm-admin config --server-insecure-tls --server-url=https://admin:@pmm.example.com

Replace with default password (admin) and pmm.example.com with localhost. Once you set up PMM and log into the dashboard from the browser, you will be required to change your password.

Go to https://localhost in the browser.

Note: If you’re running MySQL from a virtual machine, log into your VM before running the following instructions.

Log into MySQL Shell as root and change to SQL mode:

bash
$ mysqlsh root@localhost
 \sql

Create a PMM user for monitoring MySQL:

sql
CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';

Replacing 'pass' with your desired password.

Note: Replace 'localhost' with the IP address of the host, if you installed Percona Server for MySQL on a virtual machine.

bash
$ sudo pmm-admin add mysql --username=pmm --password= --query-source=perfschema

Where is the password you assigned to the user created for monitoring MySQL.

Note: if you installed Percona Server for MySQL on a virtual machine, replace above command as follows: sudo pmm-admin add mysql --username=pmm --password= --host --query-source=perfschema.

PMM is now configured and monitoring MySQL.

Percona Monitoring and Management 2.29.0 Preview Release

Taras Kozub — Tue, 12 Jul 2022 00:00:00 UTC

Percona Monitoring and Management 2.29.0 Preview Release

Hello folks! Percona Monitoring and Management (PMM) 2.29.0 is now available as a Preview Release.

We encourage you to try this PMM Preview Release in testing environments only, as these packages and images are not fully production-ready. The final version is expected to be released through the standard channels in the coming week.

Release Notes can be found in here

Known issues

PMM-10312: Metrics are not displayed on Experimental Overview and Summary dashboards

Percona Monitoring and Management server docker

docker tag:

perconalab/pmm-server:2.29.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client Release Candidate tarball for 2.29.0 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.29.0.ova

AMI

ami-0e68224439dd6f200

Optimizing the Storage of Large Volumes of Metrics for a Long Time in VictoriaMetrics

Anton Bystrov — Thu, 02 Jun 2022 00:00:00 UTC

Introduction

Nowadays, the main tools for monitoring the operation of any application are metrics and logs. An important role is played by the time of their storage. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time. In the case, when the project is just starting, their volume is relatively small, but over time it becomes larger and larger and there is a need for optimization. In this article, I will touch upon the mechanisms for processing, storing and optimizing metrics during their long-term storage.

VictoriaMetrics at a Glance

The monitoring solution and the base for storing time series VictoriaMetrics was released relatively recently - in 2018, but has already gained popularity.

Initially, VictoriaMetrics was designed as a time series database, but over time it has grown into a full-fledged alternative to Prometheus with its own ecosystem.

VictoriaMetrics is currently a fast, cost-effective and scalable monitoring solution. You can deploy the application either from a binary file, a docker image or a snap package, or build it yourself from the source code. Single-node and cluster versions are available.

Why Choose VictoriaMetrics?

The main reasons for switching from Prometheus to VictoriaMetrics for us were significant savings in system requirements and the ability to work in Push mode.

Despite many external tests, we wanted to get our own data. Test bench configuration had 8 CPU, 32 GB RAM, SSD drive. The test lasted 24 hours. 25 virtual machines were the source, each of which emulated 10 MySQL instances. In terms of the volume of metrics, there were to 96100 metrics per second, the total volume was about 8.5 billion metrics per day.

The result of testing was about three times less disk space usage with VictoriaMetrics (8.44 GB against 23.11 with Prometheus), approximately twice less the amount of RAM. The CPU requirements were about the same.

As for the push mode, it works as follows: exporters work on the target host and collect metrics, then in the classical scheme of work, Prometheus polls the exporters and collects metrics at specified intervals. This scheme of operation has a significant disadvantage as we need to maintain several ports open. The new scheme uses the VMAgent component, which is installed on the client side and collects metrics from exporters, after which it pushes to the VictoriaMetrics server.

Also, important factors in favor of VictoriaMetrics were: ease of installation and subsequent support, the possibility of more flexible performance settings and the availability of features that are not available in other applications. For example, Prometheus lacks downsampling.

VictoriaMetrics Single-Node and Cluster Versions

VictoriaMetrics can work in two versions: single-node and cluster.

The single-node version is used for relatively small amounts of data (less than a million metrics per second) and does not provide scalability and fault tolerance, since all application components are connected into a monolith.

VictoriaMetrics consists of the following components:

vmstorage - the storage itself;
vminsert - endpoint for receiving metrics based on the Prometheus remote_write API;
vmselect - a component that allows you to make queries using the Prometheus querying API.

In the official documentation, developers recommend using the single-node version and use clustered version only if there is a real need, and you understand the consequences of such a decision.

General Principles of TSDB Work

TSDB (a time series database) is used as a database for storing metrics. TSDB has many differences compared to relational databases - write operations prevail over read operations, there are no relationships between data. Since the metric has one value at a certain point in time, there is no need for nested structures. Typically, an amount of data is large.

The unit of data in such a database is a time point. The data structure of such a point consists of:

timestamp - time in Unix format
The name field, from which the name of the metric is taken. This field can be missing, but this is an antipattern, because in any case we need to know the name of the metric we are tracking.
Additional Label fields that are needed for any actions with metrics (aggregation by some attribute, filtering, etc.)
Field with metric value.

When data is recorded, time series are formed. A time series is a sequence of strictly monotonically increasing data points over time that can be accessed using a metric.

Thus, we can say that the database is relatively “static”, because it contains a certain amount of data (metrics) that do not change over time. That means that, the data in these time series increases over time, but their number remains the same. This is the basis for optimization examples that will be discussed later.

Optimization of Large Queries

If the number of metrics and their storage time increases, the amount of required resources for the application inevitably increases too. VictoriaMetrics has mechanisms for adjusting consumed resources.

The memory.allowedPercent and memory.allowedBytes keys allow you to limit the amount of memory for external buffers and query caching data.

The search.maxUniqueTimeseries key prevents excessive resource consumption when executing large queries, and in some cases can prevent the application from crashing with an out of memory error when executing large queries. This parameter is set to 300000 by default and reflects the number of unique series returned in response to the request to /api/v1/query and /api/v1/query_range endpoints.

Also, the search.maxSamplesPerQuery key can be very useful, which limits the number of returned metrics in one query.

The search.maxQueueDuration key is responsible for the time to wait for a response to a request.

In general, there are a fairly large number of keys that affect performance. In this post, I mention only whose that we use in our practice.

What Downsampling Is and How It Works

An important feature of VictoriaMetrics is downsampling - the ability to delete data as it becomes obsolete. This functionality is available only in the Enterprise version. But it is also built into PMM - Percona Monitoring and Management.

As I mentioned above (in the paragraph describing the features of the TSDB work), time series must have a large amount of data and be unchanged in order to obtain maximum sampling efficiency.

The -downsampling.period key is responsible for the work. Example: the frequency of collecting metrics in our case is once every 5 seconds, but in this case, the volume of the database will grow very quickly if we have a large amount of metrics. So we define a policy for storing metrics - after one hour we store metrics with an interval of 10 seconds, every other day with an interval of 30 seconds, after a week - with an interval of 1 minute, after a month - with 5 minutes, after a year - 1 hour. So it will look like this: -downsampling.period=1h:10s,1d:30s,1w:1m,30d:5m,360d:1h

What Deduplication Is and How It Works

Deduplication is a technology that allows you to analyze duplicate data and replace it with an appropriate reference. The use of deduplication can significantly reduce the amount of data. It is used when Prometheus or vmagent are working in HA mode and write to one VictoriaMetrics instance.

In this case, we definitely need deduplication, since the database stores overlapping data, which significantly increases its volume and, in case of large volumes, the data request time. The dedup.minScrapeInterval key is responsible for the operation of deduplication.

For example, -dedup.minScrapeInterval=60s means that within the same time series, all data will be collapsed and only the first point within 60 seconds will be saved. Since version 1.77 leave the last raw sample per each -dedup.minScrapeInterval discrete interval.

It is recommended to set this parameter to scrape_interval for metrics. According to best practice, scrape_interval should be the same for all metrics, but this is a topic is for a separate post.

Example of Deduplication

As an example, let’s consider the case where scrape_interval=10s and minScrapeInterval=15s.

Before deduplication: 05, 10, 15, 25, 35, 45, 55

interval: [00…15] [15…30] [30…45] [45…60] timestamp: [05 10] [15 25] [35 ] [45 55]

Thus, after deduplication, only those points will remain: 05, 15, 35, 45.

Rotation of Metrics

Rotation of metrics is their removal when they become obsolete. The retentionPeriod key is responsible for rotation in the VM. By default, this period is 30 days. Therefore, you should immediately set the required storage period when launching VictoriaMetrics. Let’s dive a little deeper into the features of data storage. Example: we set the metrics rotation time to 1 year, 4 months, 2 weeks, 3 days and 5 hours -retentionPeriod=1.3y2w3d5h. The year is a fractional number here to avoid the confusion with “m” which can mean both the month and the minute.

When writing, data is stored in directories like ../data/{small,big}. This directory contains data like rowsCount_blocksCount_minTimestamp_maxTimestamp. The directories are rotated as follows: in the first unit of time of the selected period (day, week, month, year), metrics for the period preceding the previous one are deleted. Example: metric rotation is set to 1 month by default. On March 1, the directory containing the data for January is deleted. An important feature is that it is possible to increase the rotation time without any data loss of the running instance. If the rotation time is reduced accordingly, the data that goes beyond this time will be deleted.

Based on personal experience, a sufficient data retention period for metrics will be one and a half years.

It is also worth emphasizing that if you plan to store data indefinitely, then you still need to set the data retention period, in this case it is set to a very large number, for example, 900 years.

Conclusion

In this post, we looked into the possibilities that VictoriaMetrics can offer to optimize the storage of metrics and reduce the usage of disk space and RAM. It can help you to reduce your costs significantly, especially if you need to store a lot of metrics. But be mindful regarding the parameters to be set, with clear understanding of your goals.

Reduce Replication Lag

Wayne Leutwyler — Wed, 01 Jun 2022 00:00:00 UTC

Replication Lag is just a fact of life with async-replication. We can’t stop lag, but we can help to reduce it. Many times the Seconds_Behind_Source can be very deciving, I have seen it go from 1 hour behind to 0 lag in the blink of an eye. There are many factors that can add to replica lag. Some of these are:

Network IO
Disk IO
Database Workload
Database settings

In this blog we will look a few database settings to help reduce lag. The settings we will look at are listed below.

blinlog_transaction_dependency_tracking
binlog_group_commit_sync_delay
replica_parallel_type
replica_parallel_workers

Hardware:

Two Raspberry Pi 4 with 8GB of RAM.
Sandisk 128GB Extreme microSDXC card.

Software:

OS Raspbian Bullseye 64bit.
Percona Server version 8.0.26.
Sysbench 1.0.20.

Testing Setup

Using Sysbench I set up 10 tables with 250,000 rows of data. If interested here is the command I used:

text
sysbench /usr/share/sysbench/oltp_read_write.lua \
--mysql-db=YOUR-DB --threads=4 --mysql-host=YOUR-HOST \
--mysql-user=YOUR-USER --mysql-password=YOUR-PASSWORD --tables=10 \
--table-size=250000 prepare

Test one with default settings

If you have not changed any of the default setting you can skip the blow changes. If your not sure verify and change as needed.

The initial testing was done with the following default settings. Make sure you make these settings on both the primary and the replica. You will need to stop replication on the replica before making the changes. Once changes are complete restart replication.

text
 mysql >set global binlog_transaction_dependency_tracking = 'COMMIT_ORDER';
 mysql >set global binlog_group_commit_sync_delay = 0;
 mysql >set global replica_parallel_type = 'DATABASE';
 mysql >set global replica_parallel_workers = 0;

Using sysbench I ran the OLTP read/write test. I used the following setting for the test.

text
 sysbench --db-driver=mysql --report-interval=2 \
 --threads=4 --time=300 --mysql-host=YOUR-HOST \
 --mysql-user=YOUR-USER --mysql-password=PASSWD \
 --mysql-db=YOUR-DB /usr/share/sysbench/oltp_read_write.lua run

At the end of this test, replicacation lag was 20 minutes behind the primary.

Test two with adjusted settings

In the second test we will apply the new setting to help reduce replication lag time. Just like in test one you will want to make these changes on both primary and replica. Make sure to stop replication on the replica before applying the changes. Once changes are applied restart replication on the repliica.

Make the following settings on your primary:

text
 mysql >set global binlog_transaction_dependency_tracking = 'writeset';
 mysql >set global replica_parallel_type = 'LOGICAL_CLOCK';
 mysql >set global replica_parallel_workers = 4;

Make the following changes on your replica:

text
 mysql >set global binlog_group_commit_sync_delay = 3000;
 mysql >set global replica_parallel_type = 'LOGICAL_CLOCK';
 mysql >set global replica_parallel_workers = 4;

I repeated the test from above. At the end of this test, replicacation lag was 6 minutes behind the primary.

blinlog_transaction_dependency_tracking = writeset

I repeated the test from above. At the end of this test, replication lag was 6 minutes behind the primary.

Setting Details:

blinlog_transaction_dependency_tracking = writeset

This allows for transactions that are marked as indipendent to be applied in parallel on the replica. Note to take advantage of this you need to set replica_parallel_workers to a non 0 value.

binlog_group_commit_sync_delay = 3000

This controls how many microseconds the binary log commit waits before syncing the binlog to disk. Change this to a non 0 value will enable more transactions to be synced at one time to disk. This will reduce the overall time to commit.

replica_parallel_type = logical_clock

As of version 8.0.27 the default value is logical_clock. Transaction will be applied in parrallel on the replica based on the source timestamp in the binlog.

replica_parallel_workers = 4

As of 8.0.27 this is a defaul value of 4. This allows for multithreading on the replica, and set the number of applier threads.

Conclusion

As we look at the results of both tests we saw a very big difference in lag. The Sysbench workload might not reflect a real world database uses, but it does provide us with baseline numbers to compare.

In the first test we saw a lag of 20 minutes at the end of the sysbench test. In our second test we say just 6 minutes of lag at the end of the sysbench test.

Dropping lag from 20 minutes down to 6 minutes is a decrease of more than 50%. That is a huge decrease.

As I high lighted above two of these variables will become default values with 8.0.27. I did my testing on 8.0.26 so I could demo these changes on a version what did not have the new standards.

How and Why Contribute to Communities

Denys Kondratenko — Mon, 30 May 2022 00:00:00 UTC

Why

Lets start with a simple question “Why to contribute?”.

In our day to day development’s and user’s life we use tons of OSS (open source) software. Ppl develop that software together to have ability to use them in more standard and open way, so they spend less time negotiating on interfaces and tools (and that is not the main reason, one of the reasons for OSS).

As any sustainable process, OSS development also needs not only users but contributors to be able to move project forward as well as to sustain bugs, time and new tech trends. As users we have different use cases that might not be yet implemented but could be very valuable for other users.

As an example I use minikube for the development and testing of PMM DBaaS solution. That tool allows me to run Kubernetes (k8s) locally and run Percona operators with help of DBaaS.

One of the great minikube features is to run real multi-node k8s clusters (see this blog post for details):

sh
$ minikube start --nodes=4 --cpus=4 --memory=8G
...
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 2d22h v1.22.3
minikube-m02 Ready  2d22h v1.22.3
minikube-m03 Ready  2d22h v1.22.3
minikube-m04 Ready  2d22h v1.22.3

I usually run integration test with --driver=kvm and some simple sanity tests with --driver=podman.

During my testing I found out that I can’t deploy operators with DBaaS on minikube multi-node cluster and I found similar Jira issue about it:

console
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
percona-server-mongodb-operator-fcc5c8d6-rphcs 1/1 Running 0 3h11m
percona-xtradb-cluster-operator-566848cf48-zm28g 1/1 Running 0 3h11m
pmm-0 1/1 Running 0 8m19s
test-haproxy-0 2/3 Running 0 9s
test-pxc-0 0/2 Init:CrashLoopBackOff 1 (5s ago) 9s
$ kubectl logs test-pxc-0 pxc-init
++ id -u
++ id -g
+ install -o 2 -g 2 -m 0755 -D /pxc-entrypoint.sh /var/lib/mysql/pxc-entrypoint.sh
install: cannot create regular file '/var/lib/mysql/pxc-entrypoint.sh': Permission denied

So that is Why - ability to use minikube to test operator’s DB deployments.

Community Hackdays

Percona engineering management came with idea of dedicating a Focus day (we have those in Percona :) to community contributions. That was a great initiative, even if community contribution is our routine (we do it day to day when needed), having dedicated day is a nice way to educate others on how to do it on a good set of examples.

I run with my minikube multi-node issue as an example of both day to day work and what could be achieved during one community hackday.

Day to day community hacking

minikube issue affects me as a developer so I spent a day to investigate it and half a day to find out workaround and next steps.

First I spent quite a time to understand what is going on and if that issue of minikube or DBaaS, or maybe operator’s issue. It was interesting detective work and I found out that it is indeed minikube related issue and similar issue already exists in GitHub: kubernetes/minikube #12360.

I have described my findings in this comment and later found workaround that enables me and my colleagues to continue to use minikube in multi-node setup.

That was day to day community hacking, I also spent a little time to find out how to fix it correctly and joined Minikube Triage party to discuss the issue (sorry folks, still need to find time to join it regularly and help with triaging).

And there I left it to the next opportunity to contribute.

Hackday

Opportunity presented itself quite quickly with new Community Hackday initiative and I decided that it would be a great time to fix part of the issue as the complete fix would take longer than a day.

First step in fixing kubernetes/minikube #12360 is to fix kubernetes-csi/csi-driver-host-path to support unprivileged containers.

So I took it for the day and here describe my progress…

Contributing to the community project

So your first help on how to contribute usually are README.md and CONTRIBUTING.md.

I started with forking the repo on GH (GitHub) UI and cloning it locally:

$ git clone git@github.com:denisok/csi-driver-host-path.git

First what I would like to do is to compile the code, create container and reproduce the issue.

sh
$ cd csi-driver-host-path

$ make container

./release-tools/verify-go-version.sh "go"

======================================================
 WARNING

 This projects is tested with Go v1.18.
 Your current Go version is v1.16.
 This may or may not be close enough.

 In particular test-gofmt and test-vendor
 are known to be sensitive to the version of
 Go.
======================================================

mkdir -p bin
# os_arch_seen captures all of the $os-$arch-$buildx_platform seen for the current binary
# that we want to build, if we've seen an $os-$arch-$buildx_platform before it means that
# we don't need to build it again, this is done to avoid building
# the windows binary multiple times (see the default value of $BUILD_PLATFORMS)
export os_arch_seen="" && echo '' | tr ';' '\n' | while read -r os arch buildx_platform suffix base_image addon_image; do \
 os_arch_seen_pre=${os_arch_seen%%$os-$arch-$buildx_platform*}; \
 if ! [ ${#os_arch_seen_pre} = ${#os_arch_seen} ]; then \
 continue; \
 fi; \
 if ! (set -x; cd ./cmd/hostpathplugin && CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" go build -a -ldflags ' -X main.version=v1.8.0-6-g50b99a39 -extldflags "-static"' -o "/home/dkondratenko/Workspace/github/csi-driver-host-path/bin/hostpathplugin$suffix" .); then \
 echo "Building hostpathplugin for GOOS=$os GOARCH=$arch failed, see error(s) above."; \
 exit 1; \
 fi; \
 os_arch_seen+=";$os-$arch-$buildx_platform"; \
done
+ cd ./cmd/hostpathplugin
+ CGO_ENABLED=0
+ GOOS=
+ GOARCH=
+ go build -a -ldflags ' -X main.version=v1.8.0-6-g50b99a39 -extldflags "-static"' -o /home/dkondratenko/Workspace/github/csi-driver-host-path/bin/hostpathplugin .
docker build -t hostpathplugin:latest -f Dockerfile --label revision=v1.8.0-6-g50b99a39 .
STEP 1/7: FROM alpine
STEP 2/7: LABEL maintainers="Kubernetes Authors"
--> Using cache 9172a5d022e2a2550bcb0f6f7faa0b6a2126dcf7c1a0266924f4989370fbf80e
--> 9172a5d022e
STEP 3/7: LABEL description="HostPath Driver"
--> Using cache 532cdc0c943df037d70368de6b7e90adb39dda3c6f9d7645c7ca6a9bd8d50abd
--> 532cdc0c943
STEP 4/7: ARG binary=./bin/hostpathplugin
--> Using cache 762a2b09549d02f9cd3d1dd8220c1b6890ae48efc155ae7aff276ae53bf7836b
--> 762a2b09549
STEP 5/7: RUN apk add util-linux coreutils && apk update && apk upgrade
--> Using cache 4bd7cf3998cc06cfdc780d3abdf6cedc452170ad93cf46cd3f4d12a8f5f97f09
--> 4bd7cf3998c
STEP 6/7: COPY ${binary} /hostpathplugin
--> a8e75bbeab1
STEP 7/7: ENTRYPOINT ["/hostpathplugin"]
COMMIT hostpathplugin:latest
--> b0014a637af
Successfully tagged localhost/hostpathplugin:latest
b0014a637af31632b48f39def813637ad0d83d11d008d5b89edb52f28498b805

$ podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
  1ec47f8d8558 46 seconds ago 35.6 MB
localhost/hostpathplugin latest f36f889fb57b 2 minutes ago 35.6 MB

It appears to be super easy, I had Go 1.18 and podman already setup on my machine.

So I have an image and now need to reproduce the issue. I need k8s cluster, setup CSI driver and upload my custom container:

sh
$ minikube start --nodes=2 --cpus=2 --memory=2G

$ minikube addons disable storage-provisioner
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌑 "The 'storage-provisioner' addon is disabled"

$ kubectl delete storageclass standard
storageclass.storage.k8s.io "standard" deleted

$ cd deploy/kubernetes-distributed/

[kubernetes-distributed]$ ./deploy.sh
applying RBAC rules
curl https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v3.1.0/deploy/kubernetes/rbac.yaml --output /tmp/tmp.yXGWmlOXv9/rbac.yaml --silent --location
kubectl apply --kustomize /tmp/tmp.yXGWmlOXv9
serviceaccount/csi-provisioner created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
csistoragecapacities.v1beta1.storage.k8s.io:
 No resources found in default namespace.
deploying with CSIStorageCapacity v1beta1: true
deploying hostpath components
 ./hostpath/csi-hostpath-driverinfo.yaml
csidriver.storage.k8s.io/hostpath.csi.k8s.io created
 ./hostpath/csi-hostpath-plugin.yaml
 using image: k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
 using image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
 using image: k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3
 using image: k8s.gcr.io/sig-storage/livenessprobe:v2.6.0
daemonset.apps/csi-hostpathplugin created
 ./hostpath/csi-hostpath-storageclass-fast.yaml
storageclass.storage.k8s.io/csi-hostpath-fast created
 ./hostpath/csi-hostpath-storageclass-slow.yaml
storageclass.storage.k8s.io/csi-hostpath-slow created
 ./hostpath/csi-hostpath-testing.yaml
 using image: docker.io/alpine/socat:1.7.4.3-r0
service/hostpath-service created
statefulset.apps/csi-hostpath-socat created

$ kubectl patch storageclass csi-hostpath-fast -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/csi-hostpath-fast patched

There I have k8s cluster with 2 nodes, disabled standard minikube storage-provisioned (which doesn’t support multi-node) deleted storageclass that was working with that storage-provisioner and setup CSI hostpathplugin. Also enabled default flag on the storageclass for hostpathplugin so it would provision PVCs for me.

Lets create test manifest perm_test.yaml to reproduce the issue:

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
 labels:
 app: perm-test
 name: perm-test
spec:
 replicas: 1
 serviceName: perm-test
 selector:
 matchLabels:
 app: perm-test
 template:
 metadata:
 labels:
 app: perm-test
 spec:
 securityContext:
 fsGroup: 65534
 runAsGroup: 65534
 runAsNonRoot: true
 runAsUser: 65534
 containers:
 - image: busybox
 name: perm-test
 command: ["/bin/sh"]
 args:
 - "-c"
 - |
 touch /mnt/perm_test/file_test && echo passed && sleep 3600 && exit 0
 echo failed
 exit 1
 volumeMounts:
 - mountPath: /mnt/perm_test
 name: perm-test
 volumeClaimTemplates:
 - metadata:
 name: perm-test
 spec:
 accessModes: [ "ReadWriteOnce" ]
 resources:
 requests:
 storage: 1G

And test it to see that we really have a problem with unprivileged container:

sh
$ kubectl apply -f perm_test.yaml
statefulset.apps/perm-test created

$ kubectl logs perm-test-0

touch: /mnt/perm_test/file_test: Permission denied
failed

$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-hostpath-socat-0 1/1 Running 0 24h 10.244.1.13 minikube-m02  
csi-hostpathplugin-fnhvr 4/4 Running 0 2m27s 10.244.0.24 minikube  
csi-hostpathplugin-w5rxt 4/4 Running 0 2m30s 10.244.1.55 minikube-m02  
perm-test-0 0/1 Error 0 2m18s 10.244.1.56 minikube-m02  

If we put sleep 3600 before exit 1 we actually could jump into the container and inspect the permissions:

sh
$ kubectl exec --stdin --tty perm-test-0 -- sh

$ id
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

$ stat /mnt/perm_test
File: /mnt/perm_test
Size: 40 Blocks: 0 IO Block: 4096 directory
Device: 10h/16d Inode: 82570 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-05-27 13:21:56.905860356 +0000
Modify: 2022-05-27 13:21:56.905860356 +0000
Change: 2022-05-27 13:21:56.905860356 +0000

As we see that directory has Access: (0755/drwxr-xr-x) and when we would like to write to it we have not enough permissions for nobody user and file creation fails. We also could see that there are couple of pods running for the CSI plugin that actually provision PV/Cs.

Clean up:

$ kubectl delete -f perm_test.yaml
$ kubectl delete pvc perm-test-perm-test-0

I did code changes to add more logging to understand the program flow better and to see when the permissions would change if they actually. During changes I learned a little bit on glog and that it has -v=5 in arguments for containers, so Info level by default.

Lets create new image with those changes which we upload to the minikube and modify DeamonSet (csi driver):

sh
$ make container
$ rm hostpath.tar
$ podman save --format docker-archive -o hostpath.tar localhost/hostpathplugin
Copying blob 4fc242d58285 done
Copying blob 89f8b151f422 done
Copying blob 57a9469e70ba done
Copying config 29ba4a1533 done
Writing manifest to image destination
Storing signatures

$ minikube image load ./hostpath.tar

$ minikube image ls
...
docker.io/localhost/hostpathplugin:latest
...

$ kubectl set image ds/csi-hostpathplugin hostpath=localhost/hostpathplugin:latest

Another way to modify DeamonSet is to run edit $ kubectl edit ds csi-hostpathplugin, and change something. For example I was changing -v=6 and back to -v=5 so it would restart all containers with new image (that I uploaded).

stuck: I actually spent 2h trying to understand why I don’t see logs that I have added, and that actually led me to learn glog, but it was quite simple. By default kubectl logs csi-hostpathplugin-w5rxt shows logs for default container, not for hostpath. So I just needed to path right parameters kubectl logs csi-hostpathplugin-w5rxt -c hostpath

Adding volume to the pod happens in couple of stages, hostpath.go creates directory on a needed node and nodeserver.go publishes this volume to the pod by bind mounting target pod mount directory to the volume directory created by hostpath.go. Please check Spec.

It actually showed me that permission didn’t change from stage to stage but weren’t setup correctly on dir creation:

go
 case state.MountAccess:
 err := os.MkdirAll(path, 0777)
 if err != nil {
 return nil, err
 }

I have mode log before and after it, as it looked 0777 should be right one (allowing everyone to rwx on the directory):

console
I0527 19:09:38.234437 1 hostpath.go:177] VolumePath: /csi-data-dir/8dc9889d-ddf0-11ec-b319-7e80679203b2 AccessType: 0
I0528 07:07:57.543195 1 hostpath.go:187] mode info: -rwxr-xr-x for user: 0 group: 0

So actually mode is 0755 instead of 0777 as requested in MkdirAll, and documentation clarifies:

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error.
The permission bits perm (before umask) are used for all directories that MkdirAll creates.
If path is already a directory, MkdirAll does nothing and returns nil.

Lets check umask for the root user (minikube ssh -n minikube-m02):

sh
$ umask
0022

$ getfacl --default /tmp/hostpath-provisioner/
getfacl: Removing leading '/' from absolute path names
# file: tmp/hostpath-provisioner/
# owner: root
# group: root

$ getfacl /tmp/hostpath-provisioner/
getfacl: Removing leading '/' from absolute path names
# file: tmp/hostpath-provisioner/
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

mkdir syscall actually accounts mask, which is 022. Or even mask is ignored as ACL from parent dir could be propagated:

In my case there are no default ACLs but umask is set to 022 so: (0777 & ~0022 & 0777) actually gives us 0755.

sh
$ umask
0022

$ getfacl --default /tmp/hostpath-provisioner/
getfacl: Removing leading '/' from absolute path names
# file: tmp/hostpath-provisioner/
# owner: root
# group: root

$ getfacl /tmp/hostpath-provisioner/
getfacl: Removing leading '/' from absolute path names
# file: tmp/hostpath-provisioner/
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

So that was it, we need to get rid of a mask and proposed fix is:

go
 if err = os.Chmod(path, 0777); err != nil {
 glog.V(4).Infof("Couldn't change volume permissions: %w", err)
 }

Cleaned up once again, compiled, created and pushed container. Tested it - It works!

I created the branch on my fork, pushed it to my repo and followed PR procedure to create kubernetes-csi/csi-driver-host-path #356.

That was the end of my Hackday and one step in solving issue in more general way.

Value

The excersise has a lot of value for me and Percona. I learned a lot of new things about k8s PV/PVC provisioning and CSI. For Percona we enabled development (devs and ci/cd) to run deployments on multi-node k8s local clusters.

And hopefully for everyone else who needs to run unprivilege containers in multi-node with PVC.

All together ppl developing OSS projects to benefit from each other and use better inovating Open-Source Software as well as to have a lot of fun :) .

Percona Monitoring and Management 2.28.0 Preview Release

Taras Kozub — Thu, 05 May 2022 00:00:00 UTC

Percona Monitoring and Management 2.28.0 Preview Release

Hello folks! Percona Monitoring and Management (PMM) 2.28.0 is now available as a Preview Release.

Release Notes can be found in here

Percona Monitoring and Management server docker

docker tag:

perconalab/pmm-server:2.28.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client Release Candidate tarball for 2.28.0 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

Artifact: PMM2-Server-2.28.0.ova

AMI

Artifact: ami-09ce0dc58b2f81889

The MySQL Workshop Book Review

David Stokes — Tue, 03 May 2022 00:00:00 UTC

Good books on MySQL for beginners are rare and excellent ones are even rarer. I often get requests from novices starting with MySQL or intermediates looking to level up on recommendations on books targeted at their level. The MySQL Workshop (Amazon link) by Thomas Pettit and Scott Cosentino is a must buy for those two groups, or those of us who would like a handy reference.

This is a great book and I recommend getting a copy regardless of your MySQL expertise.

The Basics

At seven hundred pages, this book has a wide scope that starts with background concepts like data normalization, proceeds into creating databases, SQL, and administration. And there are sections on programming with Node.js, working with Microsoft applications, loading data, DBA tasks, and logical backups. There are exercises at the end of the chapters with solutions at the end of the book.

Writing such a book is a tremendous task and the authors need to be applauded as they have produced a great book.

The Nitty-Gritty

MySQL is a complex product and introducing concepts with a fresh approach is hard to do but this book does it consistently. Complex topics like creating functions are explained thoroughly without being bogged down in minute details.

Does it cover everything? Nope, and no book under a few thousand pages will ever do that (while keeping pace with product development). There are minor omissions like constraint checks which a still fairly new but I would like to point you to the section on triggers that is the clearest explanation on the subject I have found.

The writing style is concise, the formatting easy on the eyes, and I am sure the book will be very popular.

Liquibase Data is Git for Databases

Robert Reeves — Mon, 25 Apr 2022 00:00:00 UTC

Author’s Note: Robert will be demoing Liquibase Data at Percona Live 2022 on Wednesday, May 18 at 11:50am. Add this presentation to your schedule.

Git is an amazing tool for collaboration — developers can work together to build better software faster. However, the usual Git workflow neglects the database. With Liquibase Data we’re bringing git to the database so you can easily version containerized databases, share changes with team members, store versions in remote locations, and tag versions.

The Vanilla Git Workflow

The standard Git workflow is simple. A developer can git init to create a local repository. Next, after making changes, git commit creates a local version. Then, the developer pushes to a remote branch using git push. Finally, another developer can git pull to see the new code updates.

Liquibase Data Workflow

We created the same Git workflow in Liquibase Data. Using the Liquibase Data extension, Liquibase users can initialize a new database in a Docker container using liquibase data run. Which databases? ALL of them. All it requires is a database Docker image that has a volume mount for the data. Liquibase takes it from there. If you already run your development databases via Docker, you will find that Liquibase Data parallels the docker run command.

Here’s what you’ll be able to do:

Clone from remote repositories
Make changes to the database
Commit and push your changes to share with team members
Tag commits
Easily view the difference between two database commits to identify changes

Our team thinks this will be useful for test data management and supporting developer database workflows.

Just like you commit after changing your code, you can do the same with Liquibase Data. After you add data to your database or change the schema, run liquibase data commit. Commands such as push, remote, and log are also available.

Easily Compare Databases

Determining what has changed in your database schema can be very difficult. Liquibase Data makes it simple to find schema differences between commits using the diff command. With Liquibase Data, the required database starts automatically for you to create the diff.

Watch Liquibase Data Demos

Robert Reeves, CTO of Liquibase, demonstrates how to quickly provision a developer instance of MongoDB, make changes to MongoDB, and then commit the change. You’ll see how easy it is to roll your changes backward and forward.

Check out our other Liquibase Data demos for Oracle and SQL Server! Liquibase Data works with ANY database in a Docker Container.

Try Liquibase Data We think Liquibase Data will be helpful for developers sharing databases among team members. Just imagine — you’ll be able to share datasets you’re working on early in the process and share a separate one later in the process. The distribution of valid test data amongst Dev and QA will speed testing cycles and help find bugs sooner.

Of course, we want to hear from you! Tell us what you would like to see in Liquibase Data and share with us how you are using it. Our Open Beta program is a great way to experience the benefits and give us input to make it work even better. We have a tutorial that will walk you through, step by step, how to use Liquibase Data. Along the way, you will have an opportunity to provide your thoughts.

Finally, all of us at Liquibase thank you for your support over the past 15 years of open source greatness. We could not have done it with you. And, the best is yet to come!

A Quick Guide To Austin For Percona Live 2022 Attendees

David Stokes — Mon, 11 Apr 2022 00:00:00 UTC

Percona Live returns again to Austin May 16th through the 18th will find the city vibrant, charming, and weird. The semi-official motto for the city is ‘Keep Austin Weird’ and during your visit you will indeed see many of the residents working hard to do just that. Not in a bad way. Austin is at the intersection of so many cultural, artistic, and lifestyle modes that there are a fair amount of many different things happening at the same time to ensure that any dull moments you have will have to be an active choice on your part.

The following is a quick guide for those new to Austin or looking for activities for the days before or after the show

What To Wear?

Austin in May averages 86F/30C (which is better than the August 96F/35C) so shorts, t-shirts, and comfortable shoes are a must. Bring sunscreen and water if you plan to spend time outdoors.

What to Eat?

The two main choices are barbeque and Tex-Mex. But you will find any type of cuisine you desire either in the restaurants or the food trucks (mostly on South Congress Street but found throughout the city). You may see many celebrities but remember in Austin that Matthew McConaughey is just another professor at the University of Texas and that Elon Musk builds pickup trucks.

BBQ?

Barbecue is almost considered a religion in Texas and you will find many recommendations on where to go (see https://austin.eater.com/maps/best-barbecue-austin-restaurants) but everyone has their favorite. Major competitions are run each year to determine who is the best. My personal favorite is the Salt Lick has two locations that are sadly out of Austin proper and they are known for moderating the heat of their barbeque pits by using pecans which adds a unique flavor. And they do have a location at the airport too; the food is good but the ambience is lacking with all the flight announcements.

But the other places are pretty good too. Stubb’s, Franklin, Black’s are all excellent. If you do see a line at another place where salivating people are somewhat impatiently waiting to order, then join the queue. And it is okay to salivate too.

Big hint: If you are not used to Texas Sweet Tea start with half sweetened and half unsweetened until your gums and your dentist have time to adjust.

Tex-Mex

This style of food is a tasty combination of Chihuahuan Mexican food and frontier based ingredients with lots of cheese and chili. What started as simple staple foods for settlers on the frontier made from the available commodities has evolved into a tasty treat.
Chuy’s original restaurant is a top pick and features theme rooms. Sadly we have already missed the birthday of Elvis Presley where all who dress like the King or his wife Priscilla dine free in their Elvis room. For the less decor oriented try Matt’s El Rancho.

Big tip: You will generally get big portions, especially if you order fajitas or margaritas.

For breakfast, go to Snooze AM for the pineapple upside pancakes or an omelet.

Museums

The Bob Bullock Museum is at the state capital building and is the state’s official history museum. The Museum of the Weird is just as the name implies and, while not official, provides a look into the odder parts of Austin. Not too far away is the Alamo in San Antonio (nearby the Alamo is the Buckhorn Saloon which actually has two floors of oddities that are weirder than the Museum of the Weird). The Museum of the Pacific War in Fredericksburg is a must for history fans. The Contemporary Austin is great for art fans while the Texas Toy museum will appeal to your inner child.

Outdoors Activity

Swim in spring fed Barton Springs, ride the bike trails, and hike your feet off before you rent a paddleboard to tour Lake Austin. Lots of things for the physically active with appealing hikes and you may actually run into an Armadillo.

You can rent inner tubes to float the nearby Guadalupe or Comal Rivers. Rent another inner tube for your cooler of drinks. Or visit the Schlitterbahn water park. All three are a short drive away and worth an extra day on your trip for time to spend with family or friends.

Big hint: Stay hydrated as the heat is deceiving.

Other Activities

Gruene (pronounced ‘green’) Hall is the oldest dancehall in Texas and is the place where many top stars got their starts. Currently there are no acts scheduled during the time of Percona Live (well, I expect they will be at Percona Live learning about databases!) but fans of ZZ Top, George Strait, Willy Nelson, or Greg Allman will relish the history of the place before heading to the Grist Mill for a meal. The dance hall itself has not changed much since being built in 1878 and they will open the side flaps when the dancers need fresh air.

Sixth Street is the live music capital of Texas and you will find any genre there. This is where Stevie Ray Vaughn rose to fame and where Willie Nelson rebuilt his career after leaving Nashville.Ear plugs recommended but optional.

Yes, the bats do fly out from under the Congress Street bridge at sunset which is amazing when millions of them fly out. The last physical Percona Live they were shy and only a few appeared. I assume that they were intimidated by having so many DBAs nearby.

Austin is an awesome town and not just for Percona Live itself. I have only touched the proverbial iceberg tip on things to see and do there. If you have questions, find me at Percona Live or email me at david.stokes@percona.com and hopefully we can try one of the local craft brews together.

Percona Monitoring and Management 2.27.0 Preview Release

Taras Kozub — Fri, 08 Apr 2022 00:00:00 UTC

Percona Monitoring and Management 2.27.0 Preview Release

Percona Monitoring and Management 2.27.0 is now available as a Preview Release.

PMM team really appreciates your feedback!

Known issues:

PMM-9797 - Wrong Plot on Stat Panels for DB Conns and Disk Reads at Home Dashboard
PMM-9820 - QAN page disappeared after upgrade via UI

Release Notes can be found here

PMM server docker

docker tag:

perconalab/pmm-server:2.27.0-rc

PMM client package installation

Download the latest pmm2-client Release Candidate tarball for 2.27.0 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

Artifact: PMM2-Server-2.27.0.ova

AMI

Artifact: ami-05592e370cca655b9

Please also check out our Engineering Monthly Meetings https://percona.community/contribute/engineeringmeetings/ and join us on our journey in OpenSource! Contact us in https://forums.percona.com/.

Raspberry Pi Bullseye Percona Server 64bit

Wayne Leutwyler — Tue, 05 Apr 2022 00:00:00 UTC

I love the Raspberry Pi, and I love Percona server. The combination of the two can provide a nice home database. I have been running a Percona Server database since 2019 to hold all the weather information, that I collect from several of my Weather Stations.

I did a my first blog post on installing Percona Server 5.7 on the Raspberry Pi 3+.

You can read that blog post here: How to Build a Percona Server “Stack” on a Raspberry Pi 3+

Fast forward to 2022 and we now have the resources to build Percona Server 8.0 64-bit on the Raspberry Pi. In this post I will cover building and installing Percona Server 8.0.29 and Percona XtraBackup 8.0.29.

Prereqs:

Raspberry Pi 3B+, 4 or 400 (any memory size will work).
128GB or 256GB microSD card. Of course you can go bigger.

When installing the Raspberry Pi OS on a Pi 4 or 400 make sure to choose the 64-bit image. Install Raspberry Pi OS Bullseye on Raspberry Pi

The Builds

One step I found which will help to increase the speed and success of your build is to add a larger swap file.

Create a new swap file. A 4GB swap file worked just fine. I created the swap file on the / partition.

$ sudo dd if=/dev/zero of=/swapfile4GB bs=1M count=4096
$ sudo mkswap /swapfile4GB
$ sudo swapon /swapfile4GB
$ sudo chmod 0600 /swapfile4GB

You will need to install these additional packages listed below:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install build-essential pkg-config cmake devscripts debconf debhelper automake bison ca-certificates \
libcurl4-gnutls-dev libaio-dev libncurses-dev libssl-dev libtool libgcrypt20-dev zlib1g-dev lsb-release \
python3-docutils build-essential rsync libdbd-mysql-perl libnuma1 socat librtmp-dev libtinfo5 liblz4-tool \
liblz4-1 liblz4-dev libldap2-dev libsasl2-dev libsasl2-modules-gssapi-mit libkrb5-dev apt-get \
libreadline-dev libudev-dev libev-dev libev4 libprocps-dev

Let’s download Percona Server and some additional tools.

$ wget https://downloads.percona.com/downloads/Percona-Server-LATEST/Percona-Server-8.0.29-21/source/tarball/percona-server-8.0.29-21.tar.gz
$ tar -zxvf percona-server-8.0.29-21.tar.gz
$ wget https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.gz
$ tar -zxvf boost_1_77_0.tar.gz
$ wget https://downloads.percona.com/downloads/Percona-XtraBackup-LATEST/Percona-XtraBackup-8.0.29-22/source/tarball/percona-xtrabackup-8.0.29-22.tar.gz
$ tar -zxvf percona-xtrabackup-8.0.29-22.tar.gz

Build Percona Server

At the time of writing 8.0.29-21 is the current version. If you have a USB 3 external drive, you might find the build will perform better from that device. In my build I used a 500GB SSD drive.

$ cd percona-server-8.0.29-21
$ mkdir arm64-build
$ cd arm64-build
$ cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BOOST=/home/pi/boost_1_77_0 -DCMAKE_INSTALL_PREFIX=/usr/local/mysql
$ sudo make -j2
$ sudo make install

With the 4GB swap file you created above you can use make -j2 for the compile. Depending on which Pi you are using build time should be around 3 hours.

Build XtraBackup

At the time of writing 8.0.29-22 is the current version.

$ cd percona-xtrabackup-8.0.29-22
$ mkdir arm64-build
$ cd arm64-build
$ cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BOOST=$HOME/boost_1_77_0 -DCMAKE_INSTALL_PREFIX=/usr/local/xtrabackup
$ sudo make -j3
$ sudo make install

The builds are now complete. Since we created everything from source they are a few last things that need to be done.

We need to create the mysql user and set its home directory. We need to update the /usr/local/mysql to be owned by mysql.

$ sudo useradd mysql -d /usr/local/mysql
$ sudo chown -R mysql:mysql /usr/local/mysql
$ sudo mkdir -p /var/log/mysql
$ sudo chown -R mysql:mysql /var/log/mysql

One last thing we need before to start MySQL for the 1st time is an /etc/my.cnf.

Here is a sample you can work with.

$ sudo vi /etc/my.cnf

Copy and paste the contents below into your my.cnf.

[mysqld]
character-set-server = utf8mb4
port = 3306
socket = /usr/local/mysql/mysql.sock
pid-file = /usr/local/mysql/mysqld.pid
basedir = /usr/local/mysql
datadir = /data0/mysql/data
tmpdir = /data0/mysql/tmp
general_log_file = /var/log/mysql/mysql-general.log
log-error = /var/log/mysql/mysqld.log
slow_query_log_file =/var/log/mysql/slow_query.log
slow_query_log = 0 # Slow query log off
expire_logs_days = 5
log_error_verbosity = 1
lower_case_table_names = 1
max_allowed_packet = 32M
max_connections = 50
max_user_connections = 40
skip-external-locking
skip-name-resolve
table_open_cache=500
thread_cache_size=16
thread_pool_size=16

innodb_data_home_dir = /data0/mysql/data
innodb_log_group_home_dir = /data0/mysql/data
innodb_buffer_pool_size = 2048M
innodb_log_files_in_group = 2
innodb_log_file_size = 128M
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1

You will want to set the following setting to match your needs.

datadir = /your/data/location/
innodb_data_home_dir = this should match your datadir
innodb_log_group_home_dir = this should match your datadir

Now you will want to create a mysqld.server service file in /lib/systemd/system

$ sudo vi /lib/systemd/system/mysqld.service

Add the below contents to your mysqld.service.

[Unit]
Description=Percona Server 8.0
After=syslog.target
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf
TimeoutSec=300
WorkingDirectory=/usr/local/mysql
Restart=on-failure
#RestartPreventExitStatus=1
PrivateTmp=true

Let’s setup Percona Server to stop and start with the OS.

$ sudo systemctl enable mysqld.Service

Finish your build.

Once you have completed all the above steps. You can follow this blog post The lost art of Database Server Initialization.. Start at step 4.

Thats it. You have a new Percona Server 8.0 running on your Raspberry Pi 4.

This process does take some patience, but if you like the Raspberry Pi and

Percona Server this is well worth the time it takes.

Now for some screen shots.

Percona Server:
Command Line Interface:
XtraBackup complete:

The Ins and Outs of PostgreSQL Default Configuration Tuning

Aleksandra Abramova — Thu, 31 Mar 2022 00:00:00 UTC

If you’re wondering what the optimal settings for a newly installed Postgres database are, here are some simple steps to take to tune it right from the start. Matt Yonkovit discussed them with Charly Batista, Postgres Tech Lead at Percona during the live-streamed meetup. Watch the recording to see how Charly tunes a default installation of Percona Distribution for PostgreSQL 13.

Most of the default settings have been defined long, long time ago, where one gigabyte of RAM was very expensive. So, they are not optimal. There are lots of things that we can change, but let’s have a look at the basic things to make your box more reliable and raise both in speed and performance. They can be divided into 2 groups: OS settings (Linux kernel) and database settings.

OS (Linux) Settings

No matter that is your workload, these things you want to make sure you have set from the operating system perspective out of the gate to make your box healthier.

Swap and Swappiness

Allocate swap to prevent the kernel from killing the database. But keep in mind that swappines should not be too high. What is this swappiness? The swappiness tells the kernel how likely it should use the swap. Change it to 1 to allow the kernel to use the swap only when it is really necessary. For swap, we need to create a file and then allocate it. For the swappiness, we can tell the systemctl to change the swappiness of our box.

Transparent Huge Pages

Transparent huge pages are enabled by default on the Linux kernel. And it’s not a good thing for databases like Postgres. They can cause a lot of memory fragmentation. It can slow down your database and also cause memory problems. For example, you need one gigabyte of memory for one activity, and even though you have one gigabyte available, they are split into small pieces. You cannot allocate that one gigabyte of memory. The first thing that the kernel will try to do is swap. It will just kill the database. So the transparent huge page can lead to performance issues, because we just don’t have memory, even though the memory is there, but the memory is not able to allocate.

CPU Speed

Make sure the CPU runs at its max speed. Find the CPU Governor file and disable the on-demand utility. For the database, we don’t want to adjust on-demand, we always want it as fast as we can.

Postgres Settings

Here are some database settings that you can change to optimize your database regardless of the workload you have.

Shared Buffers Value

Change the value for shared buffers to 8 GB. You can ask - why? When we talk about MySQL, a good value for the shared buffer is 50% to 70% of your memory because that will give you the ability to grow. Typically, you want your hot data all in shared memory, all data that is access at a high frequency. But unlike MySQL, Postgres relies a lot on OS buffers. In case of Postgres, if you write intensive workload, it might want to get your shared buffer much smaller, like around 5% of memory that you have, because most of the things are going to go for the kernel buffer.

Random Page Cost

Make sure you get the random page cost right. The random page cost is one that we think could be a big win for us, just because the default is so high compared to sequential. Lowering that by default is probably a good thing. Random page cost is the cost optimizer change. So, it’s going to push random pages to be a bit more costly and favor some sequential.

We need to understand how Postgres stores data, and how Postgres stores the indexes. MySQL uses cluster storage here. The data that is stored on Postgres is not a cluster, it doesn’t organize the data. So it just keeps it. Random page cost is going to improve the index usage because it will prefer indexes. It changes the cost optimizer to prefer random pages or index scans over sequential. And it can improve or decrease performance a lot.

Note that to be able to get the random page cost right, you need to understand what kind of disks you have. If you are using AWS, we suppose you have SSDs and NVMe. They are really fast. And the cost for the random page is almost the cost of the sequential page, which is why the change is not so high.

Synchronous Commit

So one thing that we can change on Postgres is the synchronous commit. The synchronous commit will force the database to commit every time to the cache, to the kernel, every time that you do a commit or transaction. It is a trade off that can improve performance. But you lose a little on reliability.

But here is one setting on Postgres that you should never change even trying to improve performance - fsync. Just never change it. By default, it is on, and it is on the top of the synchronous commit. The fsync instructs the kernel to write flash data to the disk for crash safety. If you disable the fsync, you might have some performance benefits, but your writes to the disk become not safe enough. You can have disk corruption. It’s really based on the disk having its own cache and its own systems going on, and you’re basically relying on it to do everything for you, instead of forcing that right to be consistent. It’s fine to work and tune and play around the synchronous commit, but not with the fsync.

Conclusion

Everything above are things that are independent of your workload. But there are no strict rules that you should, for example, use eight gigabytes of shared buffer if you have 32 gigabytes of memory. After you do all of those things, come back again, run the load test to check your performance. You can’t get worse performance instead of better performance.

How long do you keep the metrics in PMM?

Anton Bystrov — Fri, 11 Feb 2022 00:00:00 UTC

Hello everyone! We are indeed excited to announce that the new release of VictoriaMetrics has many exciting features, one of them being downsampling.

Downsampling helps to reduce disk space usage and improves query performance in a big and long time series if applied independently per each time series. However, this feature works only with a large number of samples per series.

As we are keen on implementing downsampling in our future releases, we would like to understand how long you keep your metrics in PMM. Please go to the Poll page and provide your inputs.

We appreciate your help. Thank You!

Please also check out our Engineering Monthly Meetings https://percona.community/contribute/engineeringmeetings/ and join us on our journey in OpenSource! Contact us at community-team@percona.com .

5 Steps to Improve Performance of Default MySQL Installation

Thu, 27 Jan 2022 00:00:00 UTC

Let’s say you have a fresh MySQL installation. Are there any possible steps to improve performance right away? Yes, there are!

Recently, Marcos Albe (Principal Support Engineer, Percona) did an online tuning on the MySQL Meetup hosted by Matt Yonkovit (Head of Open Source Strategy, Percona). Here are some steps you can consider to make your fresh MySQL installation to run better right from the start.

So, we have a very basic default MySQL installation with some workload. It is connected to PMM, the slow query log is turned on. But it is largely unconfigured. Here is what actions Marcos considered to take to set up a new system to make sure that it’s actually set up from the beginning to reasonable defaults. Do some reactive configuration - go through the workload, observe bottlenecks and then configure to avoid bottlenecks

Step 1. Rate Limit for Slow Queries

Go to MySQL Summary Dashboard In PMM, find your instance and set a rate limit instead of setting low query time to zero. Once we get to the 3000-4000 queries per second, these might start impacting performance in a way that is going to show in the latency papers. The thing is that while it is important to collect as many query details as possible, you don’t want to collect too many because it can impact performance and have the opposite effect of what you’re trying to do.

Step 2. Spikes

Go down and turn the metrics off and on from data series from each of the graphs to be able to see the spikes on each magnitude. It allows you to find bad query patterns, under-dimensional or over-dimensional things.

Think of it as workload being the light, MySQL being the prism and the metrics being the reflection of the light.

Doing this, you can suggest different changes, like trying a slightly larger buffer size or increasing the thread cache. And as you go through the metrics, look at the values in the configuration, at the workload, and the actual work to find out if your hypotheses is correct.

Step 3. Buffer Pool Size

Increase the buffer pool size. It is probably the most used and most recommended setting.

Step 4. Redo Log Size

Increase the redo log size and restart the instance. Make the redo log file as large as reasonably possible. What is reasonable? Reasonable is an amount of time that will allow you to write at that rate for the duration of your big workload. The purpose is to allow more pages during the heavy write periods. The only thing you could fear here is the recovery time. You should do some testing to see if the recovery times are acceptable, just like you do for backups. And then if the time of recovery is unacceptable, you should consider having a HA setup, semi-synchronous or virtually synchronous setup, where you can failover to the next instance when this one crashes. Also, you could get faster drives, or you could try to convince your developers to write less

Step 5. InnoDB IO Capacity

Set InnoDB IO capacity to 200 unless you have proof you need more. Otherwise, you’re just forcing the flushing to happen too early. The thing is that you want to keep dirty pages. Dirty pages are the performance optimization. Imagine that you update the views counter for a popular video 100 times per second. If you have a very high capacity, you will probably write that road 50 times per second to disk. If you have a smaller capacity, you will probably write it once every few seconds. And then you’re actually only doing one write for hundreds of updates because all the rest were in memory and on the redo log.

Conclusion

If you want to watch the video of the meetup and see how Marcos tuned the installation, it is always available on the Community Website. The meetups for MySQL, PostgreSQL, PMM, and MongoDB are regularly live-streamed. Stay tuned to announcements and feel free to join!

PMM 2.26.0 Preview Release

Rasika Chivate — Thu, 27 Jan 2022 00:00:00 UTC

PMM 2.26.0 Preview Release

Percona Monitoring and Management 2.26.0 is now available as a Preview Release.

PMM team really appreciates your feedback!

We encourage you to try this PMM Preview Release in testing environments only, as these packages and images are not fully production-ready. The final version is expected to be released through the standard channels in the coming week.

Release Notes can be found here

PMM server docker

docker tag: perconalab/pmm-server:2.26.0-rc

PMM client package installation

Download the latest pmm2-client Release Candidate tarball for 2.26.0 from this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable original testing

install pmm2-client package for your OS via package manager.

OVA

Artifact: PMM2-Server-2022-01-27-1524.ova