Introduction

Running stateful database workloads on Kubernetes has always been a challenge. Many organizations running OpenShift choose to host their MySQL databases inside virtual machines managed by OpenShift Virtualization (formerly KubeVirt) — benefiting from the operational familiarity of VMs while staying within the Kubernetes ecosystem.

However, your application pods still need a reliable, topology-aware way to connect to that MySQL cluster. That’s where MySQL Router comes in.

In this article, we’ll walk through how to deploy MySQL Router as a Kubernetes Deployment on OpenShift, connecting it to a 3-node MySQL InnoDB Cluster (Group Replication) running on OpenShift Virtualization VMs. We'll also cover how to use a NetworkAttachmentDefinition with OVN-Kubernetes to give MySQL Router a dedicated L2 overlay network interface for reaching the database VMs — and how to validate everything is working correctly.

Architecture Overview

┌─────────────────────────────────────────────────────┐
│              OpenShift Cluster                      │
│                                                     │
│  ┌──────────────────────┐                           │
│  │   App Pods           │                           │
│  │  (connects via SVC)  │                           │
│  └──────────┬───────────┘                           │
│             │ ClusterIP Service                     │
│             │ :6446 (RW) / :6447 (RO)               │
│  ┌──────────▼───────────┐                           │
│  │   MySQL Router Pod   │◄── L2 Overlay Network     │
│  │   (Deployment)       │    (OVN-Kubernetes)       │
│  └──────────┬───────────┘                           │
│             │                                       │
│  ┌──────────▼────────────────────────────────────┐  │
│  │         OpenShift Virtualization              │  │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐       │  │
│  │  │ dbnodea  │ │ dbnodeb  │ │ dbnodec  │       │  │
│  │  │ PRIMARY  │ │SECONDARY │ │SECONDARY │       │  │
│  │  │ :3306    │ │ :3306    │ │ :3306    │       │  │
│  │  └──────────┘ └──────────┘ └──────────┘       │  │
│  └───────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘

The key components are:

• MySQL InnoDB Cluster — a 3-node Group Replication cluster running inside OpenShift Virtualization VMs
• MySQL Router — deployed as a Kubernetes Deployment with an init container that bootstraps the router configuration automatically
• OVN-Kubernetes L2 Overlay — a NetworkAttachmentDefinition that gives the router pod a secondary network interface on the same L2 segment as the database VMs
• Kubernetes Service — a ClusterIP service that exposes the router's RW/RO ports to application pods

Prerequisites

Before diving in, make sure you have:

• An OpenShift cluster with the OpenShift Virtualization operator installed
• A working 3-node MySQL InnoDB Cluster (Group Replication) running on VMs managed by OpenShift Virtualization
• The Multus CNI plugin enabled (it ships with OpenShift by default)
• A NetworkAttachmentDefinition configured for L2 overlay networking (covered below)
• A mysql-router-secret Kubernetes Secret containing the MySQL Router bootstrap password

Step 1: Setting Up the L2 Overlay Network

MySQL Router needs to reach the database VMs on their internal IPs. The cleanest way to achieve this on OpenShift with OVN-Kubernetes is to create a Layer 2 overlay network using a NetworkAttachmentDefinition.

yaml

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: abc-production-l2-overlay-nad
  namespace: abc-system-production
spec:
  config: |-
    {
        "cniVersion": "0.3.1",
        "name": "abc-production-l2-overlay-nad",
        "type": "ovn-k8s-cni-overlay",
        "netAttachDefName": "abc-system-production/abc-production-l2-overlay-nad",
        "topology": "layer2",
        "subnets": "192.168.100.0/24",
        "excludeSubnets": "192.168.100.10/32,192.168.100.11/32,192.168.100.12/32"
    }

The excludeSubnets field is important — it prevents OVN from auto-assigning IPs that are already in use by your MySQL VM nodes. With this configuration in place, you no longer need to hard-code a specific IP in the pod annotation. OVN will automatically assign a free IP from the subnet. The pod annotation becomes simply:

yaml

annotations:
  k8s.v1.cni.cncf.io/networks: |
    [
      {
        "name": "abc-production-l2-overlay-nad",
        "namespace": "abc-system-production"
      }
    ]

This is much cleaner than pinning a specific IP, which would cause pod scheduling conflicts if the pod is rescheduled.

Step 2: Creating the Secret

MySQL Router needs credentials to bootstrap against the MySQL cluster. Store the password in a Kubernetes Secret:

yaml

apiVersion: v1
kind: Secret
metadata:
  name: mysql-router-secret
  namespace: abc-system-production
type: Opaque
data:
  router-password: <base64-encoded-password>

Generate the base64 value with:

bash

echo -n 'your_password_here' | base64

Step 3: Deploying MySQL Router

The Deployment has two key parts: an init container that bootstraps the router configuration, and a main container that runs the router itself. Both share an emptyDir volume mounted at /etc/mysqlrouter.

Why an Init Container?

MySQL Router’s bootstrap process (mysqlrouter --bootstrap) connects to the cluster, queries the metadata, and writes a complete mysqlrouter.conf including all member IPs. This cannot be done at image build time because the cluster topology is dynamic. The init container solves this elegantly — it runs once, writes the config to the shared volume, and exits. The main container then starts the router using that generated config.

URL-Encoding the Password

A subtle but important detail: MySQL Router’s bootstrap command embeds credentials in a connection URI. If your password contains special characters (like @, #, or /), the connection string will break unless the password is URL-encoded. The init container script handles this with a pure-bash urlencode() function before passing the credentials to mysqlrouter --bootstrap.

Here is the complete Deployment:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysqlrouter-dep
  namespace: abc-system-production
  labels:
    app: mysqlrouter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysqlrouter
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  template:
    metadata:
      labels:
        app: mysqlrouter
      annotations:
        k8s.v1.cni.cncf.io/networks: |
          [
            {
              "name": "abc-production-l2-overlay-nad",
              "namespace": "abc-system-production"
            }
          ]
    spec:
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        seLinuxOptions:
          level: 's0:c37,c14'
        fsGroup: 1001360000
        seccompProfile:
          type: RuntimeDefault
      tolerations:
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        - key: node.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
      volumes:
        - name: mysqlrouter-config
          emptyDir: {}
      initContainers:
        - name: mysqlrouter-bootstrap
          image: mysql/mysql-router:8.0
          imagePullPolicy: IfNotPresent
          command:
            - /bin/bash
            - '-c'
            - |
              set -e
              urlencode() {
                local string="${1}"
                local encoded=""
                local pos c o
                for (( pos=0; pos<${#string}; pos++ )); do
                  c=${string:$pos:1}
                  case "$c" in
                    [-_.~a-zA-Z0-9]) o="${c}" ;;
                    *) printf -v o '%%%02X' "'$c" ;;
                  esac
                  encoded+="${o}"
                done
                echo "${encoded}"
              }
              echo "=== Port check ==="
              timeout 5 bash -c "cat < /dev/null > /dev/tcp/${MYSQL_HOST}/${MYSQL_PORT}" \
                && echo "PORT REACHABLE" \
                || { echo "PORT NOT REACHABLE - aborting"; exit 1; }
              echo "=== Starting bootstrap ==="
              ENCODED_PASSWORD=$(urlencode "${MYSQL_ROUTER_PASSWORD}")
              mysqlrouter \
                --bootstrap ${MYSQL_USER}:${ENCODED_PASSWORD}@${MYSQL_HOST}:${MYSQL_PORT} \
                --directory /etc/mysqlrouter \
                --conf-bind-address 0.0.0.0 \
                --force \
                2>&1
              echo "=== Bootstrap completed ==="
          env:
            - name: MYSQL_HOST
              value: "<mysql-primary-node-ip>"
            - name: MYSQL_PORT
              value: "3306"
            - name: MYSQL_USER
              value: root
            - name: MYSQL_ROUTER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-router-secret
                  key: router-password
          securityContext:
            capabilities:
              drop: [ALL]
            runAsUser: 1001360000
            runAsNonRoot: true
            allowPrivilegeEscalation: false
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 128Mi
          volumeMounts:
            - name: mysqlrouter-config
              mountPath: /etc/mysqlrouter
      containers:
        - name: mysqlrouter
          image: mysql/mysql-router:8.0
          imagePullPolicy: IfNotPresent
          command:
            - /bin/bash
            - '-c'
            - |
              set -e
              exec mysqlrouter -c /etc/mysqlrouter/mysqlrouter.conf
          ports:
            - name: rw-classic
              containerPort: 6446
              protocol: TCP
            - name: ro-classic
              containerPort: 6447
              protocol: TCP
            - name: rw-x
              containerPort: 6448
              protocol: TCP
            - name: ro-x
              containerPort: 6449
              protocol: TCP
            - name: http
              containerPort: 8443
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: 6446
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:
            tcpSocket:
              port: 6446
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            failureThreshold: 3
          securityContext:
            capabilities:
              drop: [ALL]
            runAsUser: 1001360000
            runAsNonRoot: true
            allowPrivilegeEscalation: false
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 128Mi
          volumeMounts:
            - name: mysqlrouter-config
              mountPath: /etc/mysqlrouter

A Note on OpenShift Security Context Constraints (SCC)

OpenShift enforces SCCs, which is why we explicitly set runAsUser: 1001360000 (a UID in the range assigned to the namespace), runAsNonRoot: true, allowPrivilegeEscalation: false, and drop all Linux capabilities. The seLinuxOptions and seccompProfile are similarly required for a restricted or restricted-v2 SCC. Without these, your pod will fail to schedule.

Step 4: Exposing MySQL Router via a Service

Create a ClusterIP service so that application pods can connect to MySQL Router using a stable DNS name (mysql-router.abc-system-production.svc.cluster.local) rather than pod IPs:

yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-router
  namespace: abc-system-production
  labels:
    app: mysqlrouter
spec:
  type: ClusterIP
  selector:
    app: mysqlrouter
  ports:
    - name: rw-classic
      protocol: TCP
      port: 6446
      targetPort: 6446
    - name: ro-classic
      protocol: TCP
      port: 6447
      targetPort: 6447
    - name: rw-x
      protocol: TCP
      port: 6448
      targetPort: 6448
    - name: ro-x
      protocol: TCP
      port: 6449
      targetPort: 6449

Your application should connect reads to port 6447 (round-robin across secondaries with fallback to primary) and writes to port 6446 (always routed to the current primary).

Step 5: Validating the Deployment

Use the following Job to run 5 parallel test connections and verify that routing is working correctly:

yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: mysql-router-test
  namespace: abc-system-production
spec:
  ttlSecondsAfterFinished: 120
  completions: 5
  parallelism: 5
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: mysql-test
        image: mysql:8.0
        command:
        - bash
        - -c
        - |
          echo "=== Testing R/W Port 6446 (Expected: PRIMARY) ==="
          mysql -h mysql-router -P 6446 -u root -p"${MYSQL_ROOT_PASSWORD}" \
            -e "SELECT @@hostname AS hostname, @@read_only AS read_only, @@port AS port;"

          echo "=== Testing Read-Only Port 6447 (Expected: REPLICA) ==="
          mysql -h mysql-router -P 6447 -u root -p"${MYSQL_ROOT_PASSWORD}" \
            -e "SELECT @@hostname AS hostname, @@read_only AS read_only, @@port AS port;"

          echo "=== Cluster Member Status ==="
          mysql -h mysql-router -P 6446 -u root -p"${MYSQL_ROOT_PASSWORD}" \
            -e "SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE, MEMBER_ROLE
                FROM performance_schema.replication_group_members;"
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-router-secret
              key: router-password
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"

A successful output will look like this:

=== Testing R/W Port 6446 (Expected: PRIMARY) ===
hostname    read_only   port
dbnodea     0           3306

=== Testing Read-Only Port 6447 (Expected: REPLICA) ===
hostname    read_only   port
dbnodec     1           3306

=== Cluster Member Status ===
MEMBER_HOST         MEMBER_PORT   MEMBER_STATE   MEMBER_ROLE
192.168.100.10      3306          ONLINE         SECONDARY
192.168.100.11      3306          ONLINE         PRIMARY
192.168.100.12      3306          ONLINE         SECONDARY

Port 6446 returns read_only=0, confirming it is the PRIMARY. Port 6447 returns read_only=1, confirming it is a SECONDARY. All three cluster members are ONLINE.

How MySQL Router Stays in Sync

You might wonder: what happens when the primary fails over? MySQL Router’s metadata cache is the key. The generated mysqlrouter.conf contains a [metadata_cache:mycluster] section that lists all three bootstrap server addresses and sets use_gr_notifications = 1. This means the router subscribes to Group Replication notifications and instantly knows when a new node becomes primary — without waiting for a TTL to expire.

The relevant section from the bootstrapped config:

ini

[metadata_cache:mycluster]
cluster_type = gr
bootstrap_server_addresses = IP1:3306,IP2:3306,IP3:3306
user = mysqlrouter
ttl = 0.5
use_gr_notifications = 1

[routing:primary]
bind_port = 6446
destinations = metadata-cache://mycluster/?role=PRIMARY
routing_strategy = first-available
protocol = classic

[routing:secondary]
bind_port = 6447
destinations = metadata-cache://mycluster/?role=SECONDARY
routing_strategy = round-robin-with-fallback
protocol = classic

The round-robin-with-fallback strategy on the secondary port means that if all secondaries go offline, reads will automatically fall back to the primary — a sensible default for most workloads.

Key Takeaways

Deploying MySQL Router on OpenShift to front a VM-hosted InnoDB Cluster is well worth the setup effort. A few things worth remembering from this walkthrough:

Use an init container for bootstrapping. The bootstrap process is stateful — it writes a config file based on the live cluster topology. An init container with a shared emptyDir volume is the idiomatic Kubernetes way to handle this.

URL-encode your password. If your MySQL Router password contains special characters, the bootstrap URI will silently fail or connect to the wrong host. The pure-bash urlencode() function in the init container script solves this robustly.

Use the subnets + excludeSubnets approach in your NAD. Hard-coding pod IPs creates operational headaches. Let OVN-Kubernetes assign IPs automatically while protecting the IPs reserved for your database VMs.

Set OpenShift-compatible security contexts. Don’t fight the SCC — set the right runAsUser, drop all capabilities, and use seccompProfile: RuntimeDefault. It only takes a few lines and prevents hours of debugging.

Validate with a parallel Job. Running 5 completions in parallel quickly verifies that both the RW and RO routing paths are working, and shows you the real cluster topology from the router’s perspective.

Conclusion

MySQL Router is a lightweight but powerful component that transforms a raw MySQL Group Replication cluster into a topology-aware, highly-available data tier. Running it as a Kubernetes Deployment on OpenShift — with an OVN-Kubernetes L2 overlay connecting it to your OpenShift Virtualization database VMs — gives you the best of both worlds: the simplicity of VM-based MySQL operations combined with cloud-native connectivity and observability.

The patterns shown here — init containers for bootstrapping, secondary network interfaces for DB connectivity, parallel validation Jobs — are broadly applicable beyond MySQL Router to any scenario where containerized workloads need to reach VM-hosted databases on OpenShift.

There’s a particular kind of dread that comes with staring at a database cluster where every node shows OFFLINE.

No reads. No writes. Just silence where your production data used to be.

That’s exactly where we found ourselves with a MySQL InnoDB Cluster — three nodes, all down, all stubbornly refusing to cooperate. This is the complete story of how we diagnosed the failure, untangled a mess of GTID inconsistencies and duplicate UUIDs, and brought the cluster fully back online.

If you’re facing something similar, this guide is for you.

The Setup

Our cluster — ABCCLUSTER — runs three nodes:

• Node A — <IP 1> (intended PRIMARY)
• Node B — <IP 2> (SECONDARY)
• Node C — <IP 3> (SECONDARY)

Group Replication is configured with communication_stack=MYSQL, meaning it communicates over port 3306 — not the commonly assumed 33061. This turned out to be a critical early discovery.

Phase 1 — Diagnosis: Understanding What Actually Broke

The first instinct when a cluster goes dark is to start restarting services. Resist that urge. Diagnose first.

Step 1 — Check cluster member visibility

SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE, MEMBER_ROLE 
FROM performance_schema.replication_group_members;

Only Node A was visible — and it showed OFFLINE. Nodes B and C were completely invisible to the group.

Step 2 — Read the error logs

tail -100 /var/log/mysqld.log

The logs revealed connection failures between nodes on port 3306. Not 33061–3306. This matters.

Step 3 — Test port connectivity

nc -zv abc-prod-mysql-db-cluster-data-node-a 33061
nc -zv abc-prod-mysql-db-cluster-data-node-b 33061
nc -zv abc-prod-mysql-db-cluster-data-node-c 33061

Port 33061 was blocked — but as the next step revealed, that wasn’t our actual problem.

Step 4 — Check persisted Group Replication variables

SELECT * FROM performance_schema.persisted_variables 
WHERE VARIABLE_NAME LIKE 'group_replication%';

This confirmed communication_stack=MYSQL. Our cluster communicates on port 3306, not 33061. The blocked port was a red herring.

Step 5 — Note the Group Name

SHOW VARIABLES LIKE 'group_replication_group_name';

Result: 91527b32-58f3-11ee-b8bf-fa163ec0acaf — we'll need this later.

Phase 2 — Fixing Privileges Before Rebooting

This phase trips up a lot of people. MySQL Shell’s dba.rebootClusterFromCompleteOutage() requires very specific privileges — and they need WITH GRANT OPTION.

Step 6 — Launch MySQL Shell

mysqlsh root@localhost

Step 7 — Attempt cluster reboot (it failed)

dba.rebootClusterFromCompleteOutage('ABCCLUSTER');

Error: root@'%' was missing dynamic privileges with WITH GRANT OPTION.

Step 8 — Verify what’s missing

SHOW GRANTS FOR 'root'@'%';

Several dynamic privileges were present but without WITH GRANT OPTION. MySQL Shell needs to be able to delegate these grants to recovery accounts.

Step 9 — Disable read-only mode on all 3 nodes

Run this on each node before making any changes:

SET GLOBAL super_read_only = OFF;
SET GLOBAL read_only = OFF;

Step 10 — Grant the full required privilege set on all 3 nodes

GRANT CLONE_ADMIN, CONNECTION_ADMIN, CREATE USER, EXECUTE, FILE, 
GROUP_REPLICATION_ADMIN, PERSIST_RO_VARIABLES_ADMIN, PROCESS, RELOAD, 
REPLICATION CLIENT, REPLICATION SLAVE, REPLICATION_APPLIER, 
REPLICATION_SLAVE_ADMIN, ROLE_ADMIN, SELECT, SHUTDOWN, 
SYSTEM_VARIABLES_ADMIN ON *.* TO 'root'@'%' WITH GRANT OPTION;

FLUSH PRIVILEGES;

Why all three nodes? During recovery, MySQL Shell connects to every member. If any node is missing privileges, the reboot will fail partway through.

Phase 3 — Rebooting the Cluster

With privileges fixed, we can attempt the reboot — but first we need to know which node has the most recent data to elect it as PRIMARY.

Step 11 — Check GTID position on all nodes

SELECT @@global.gtid_executed;

Results:

• Node A: b3ce0ea4:1-372 ← most transactions
• Node B: b3ce0ea4:1-369
• Node C: b3ce0ea4:1-369

Node A is 3 transactions ahead. It must be the PRIMARY.

Step 12 — Reboot with force, nominating Node A

dba.rebootClusterFromCompleteOutage('ABCCLUSTER', {
    force: true,
    primary: 'abc-prod-mysql-db-cluster-data-node-a:3306'
});

Node A came online as PRIMARY. ✅

However, Nodes B and C could not rejoin — they had GTID sets incompatible with what Node A had already purged. This is a common outcome when one node is ahead of others and binary logs have been purged.

Phase 4 — Recovering Node B

Because Node B’s GTID history conflicted with the cluster’s current state, a standard rejoin wasn’t possible. We needed to wipe its state and clone fresh data from Node A.

Step 13 — Reset GTID state on Node B

SET GLOBAL super_read_only = OFF;
SET GLOBAL read_only = OFF;
STOP GROUP_REPLICATION;
RESET MASTER;

Step 14 — Attempt rejoin (failed — purged transactions)

cluster.rejoinInstance('root@abc-prod-mysql-db-cluster-data-node-b:3306');

Error: Transactions required for recovery have been purged from all cluster members. We need a full clone.

Step 15 — Attempt add with clone (failed — duplicate UUID)

cluster.addInstance('root@abc-prod-mysql-db-cluster-data-node-b:3306', 
    {recoveryMethod: 'clone'});

Error: Duplicate server UUID 8a6e08dc-a1f5-11ed-8464-fa163ee55914

This happens when a node was previously cloned from another, or when auto.cnf was copied between servers.

Step 16 — Fix the duplicate UUID

systemctl stop mysql
rm /data/mysql/auto.cnf
systemctl start mysql

MySQL regenerates a fresh UUID on startup when auto.cnf is absent.

Step 17 — Attempt add again (failed — duplicate server_id)

cluster.addInstance('root@abc-prod-mysql-db-cluster-data-node-b:3306', 
    {recoveryMethod: 'clone'});

Error: server_id=2 already used by an active cluster member.

Step 18 — Fix the duplicate server_id

SET GLOBAL super_read_only = OFF;
SET GLOBAL read_only = OFF;
SET PERSIST server_id = 22;

Then update the config file so it survives restarts:

vi /etc/my.cnf
# Change: server-id=2
# To:     server-id=22
systemctl restart mysql

Step 19 — Successfully add Node B via clone

cluster.addInstance('root@abc-prod-mysql-db-cluster-data-node-b:3306', 
    {recoveryMethod: 'clone'});

Clone initiated. Node A’s data was transferred to Node B. ✅

Step 20 — Rescan to update cluster metadata

cluster.rescan();

Node B’s new UUID was registered in the metadata. Node C appeared as unavailable — we answered N when prompted to remove it from metadata, since we planned to recover it next.

Phase 5 — Recovering Node C

The same pattern as Node B, applied consistently.

Step 21 — Prepare Node C

ssh root@abc-prod-mysql-db-cluster-data-node-c
systemctl stop mysql
rm /data/mysql/auto.cnf
systemctl start mysql

SET GLOBAL super_read_only = OFF;
SET GLOBAL read_only = OFF;
SET PERSIST server_id = 33;

vi /etc/my.cnf
# Change: server-id=3
# To:     server-id=33
systemctl restart mysql

Step 22 — Add Node C via clone

cluster.addInstance('root@abc-prod-mysql-db-cluster-data-node-c:3306', 
    {recoveryMethod: 'clone'});

Step 23 — Final status verification

var cluster = dba.getCluster('ABCCLUSTER');
cluster.status();

All three nodes online. Cluster fully operational. ✅

The Root Cause Fix: Hardening my.cnf

This outage happened partly because Group Replication settings weren’t persisted to disk. After a server restart, GR didn’t start automatically. To prevent that, add the following to /etc/my.cnf on all three nodes:

gtid_mode=ON
enforce_gtid_consistency=ON
binlog_checksum=NONE
plugin-load-add=group_replication.so
group_replication=FORCE_PLUS_PERMANENT
group_replication_group_name="91527b32-58f3-11ee-b8bf-fa163ec0acaf"
group_replication_start_on_boot=ON
group_replication_recovery_get_public_key=ON

The key lines to understand:

• group_replication_start_on_boot=ON — GR starts automatically after mysqld restarts
• group_replication=FORCE_PLUS_PERMANENT — prevents MySQL from starting without GR loaded
• group_replication_recovery_get_public_key=ON — avoids authentication failures during recovery

5 Lessons from This Outage

1. Know your communication stack. communication_stack=MYSQL means port 3306, not 33061. Chasing firewall rules for the wrong port wastes critical time during an outage.

2. GTID position determines your PRIMARY. Always query @@global.gtid_executed on every node before rebooting. Picking the wrong primary can cause data loss.

3. Duplicate UUIDs and server_ids are silent killers. If nodes were ever cloned from each other, check both auto.cnf and server_id before attempting to add them to a cluster.

4. Clone is your friend after GTID divergence. When purged transactions make incremental recovery impossible, recoveryMethod: 'clone' bypasses the GTID history entirely and gives you a clean slate.

5. Persist your config. SET PERSIST keeps variables across restarts in mysqld-auto.cnf, but your my.cnf should reflect the same values for transparency and manual recovery scenarios.

The Complete Recovery Path at a Glance

Phase What We Did Outcome Diagnosis Checked status, logs, ports, GR variables Identified communication stack and root errors Privileges Granted full dynamic privilege set with GRANT OPTION MySQL Shell able to proceed Reboot Force-rebooted with Node A as PRIMARY Node A online; B/C GTID incompatible Node B Cleared auto.cnf, fixed server_id, cloned from A Node B rejoined as SECONDARY Node C Same process as Node B Node C rejoined as SECONDARY Hardening Updated my.cnf on all nodes GR will survive future restarts

Database outages are stressful — but they’re survivable with a methodical approach. The key is resisting the urge to brute-force restarts and instead following the evidence from logs, GTID positions, and error messages. Each failure in this recovery pointed directly at the next step.

Hopefully this walkthrough saves someone else a few hours of pain.

Have questions or a different approach that worked for you? Drop a comment below.