Catalogue / linkedin-2022-mysql-ha

LinkedIn hardens automated MySQL primary failover in Openark Orchestrator with transaction checks, heartbeats and throttling

LinkedIn, 2022. Multi-tenant MySQL clusters and automated failover with Openark Orchestrator.

Organisation
LinkedIn
Evidence
Reported from production
Source
Addressing the last mile problem with MySQL high availability

LinkedIn Engineering Blog, published 2022-04-22

Records

Decision#d1

Adopt Openark Orchestrator for automated MySQL primary failover across GTID and non-GTID clusters

Failure#f1

Recurring huge rogue transactions stalled shared MySQL primaries and triggered unwanted failovers

Decision#d2

Measure absolute replication lag with heartbeat writes and guard promotions with FailMasterpromotionOnLagMinutes

Decision#d3

Register each cluster's dedicated backup/ETL replica as a prefer failover candidate rather than Must

Decision#d4

Throttle Orchestrator failovers within a two-minute window via a pre-failover hook, skipped during maintenance

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.