1 Star 0 Fork 9

kaluola / orchestrator

forked from Capulus / orchestrator 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

orchestrator

License

orchestrator is free and open sourced under the Apache 2.0 license.

二次开发

该版本基于《https://github.com/openark/orchestrator》 3.2.4版本开发

仓库同步更新于: 《https://github.com/baijiu1/orchestrator》

新增功能:

1、 生态建设: 一键precheck与switchover工具
2、 生态建设: VIP漂移工具
3、 改进: 从CMDB元数据中心取值,避免在集群中建表,避免使用sql_log_bin语句造成主备gtid不一致
4、 修复: 优雅切换时,无法选择到正确的datacenter实例,导致切换失败的情况
5、 新增: orchestrator-client DataCenter显示
6、 新增: 双主模式切换(该模块时根据我们实际环境单独开发,不影响其他切换函数或切换方式)
7、 重要功能: 在等待SQL线程应用的基础上,添加日志补齐功能,具体见下面具体描述

待新增功能:

1、 日志补齐: 
        1)将orchestrator节点作为binlog server,利用tcp协议拉取各个集群的binlog日志,当发生宕机,可以及时从orchestrator节点上解析。
        2)数据库宕机后,会从目标集群上的binlog目录中拿取最新的binlog文件到orchestrator节点上,进行比较,比较后,将应用最新的日志文件。

一、 一键工具使用方式:

构建:

cd ~/orchestrator/tool/
go build main.go

./main --OpType=[precheck|switchover] --ClusterName=[cluster name] [--force]

说明:
--OpType: 选择是进行预先检查还是进行切换,切换可选--force进行强制切换
--ClusterNmae: 选择要进行检查或切换的集群

由于该工具对CMDB有非常深的依赖,所以需要自己修改myflag下的flag.go文件做自定义修改

二、 vip漂移脚本使用:

cd ~/orchestrator/hook/

go build main.go

useage:

./main --ClusterName={failureClusterAlias} --DeadStatus={failureType} --OldMaster={failedHost} --OldMasterPort={failedPort} --NewMaster={successorHost} --NewMasterPort={successorPort} --VipAddr={failureClusterDomain} --InterFace="eth0"

说明:
该脚本是通过外部的orchestrator传入的环境变量来做到vip切换控制的。
{failureClusterAlias}: 失败的集群名称(别名)。
{failureType}: 探测到失败的类型,一般为deadmaster,还有其他类型,可以看源码logic/topology_recovery.go下的getCheckAndRecoverFunction()函数里面的switch case语句,里面记录了所有的失败类型。return 后面true和false来控制是否进入恢复过程。至于如何判断是否恢复的,可以看我csdn博客或者公众号:收获不止Oracle
{failedHost}: 老主
{failedPort}: 老主的端口
{successorHost}: 新主
{successorPort}: 新主的端口
{failureClusterDomain}: 通过CMDB查到的vip地址,对应DetectClusterDomainQuery查询语句,需要去元数据库查询的
InterFace:  网卡名称,可以自己填写自己环境的网口名称

当然还有很多环境变量,具体可以查看官网文档。

此VIP漂移脚本支持的是keepalived形式的vip挂载和卸载: ip addr add vip/32 dev eth0 这样的格式。

三: 以CMDB为中心取值: 新增三个配置项,示例配置文件:conf/orchestrator-simple.conf.json:

MetaDBHost: 该参数意思是到哪个元数据库去取值
MetaDBPort: 对应的端口
MetaDBName: 对应的数据库名称

比如,我要去元数据库取对等切换节点,让这两个对等切换节点处于同一个数据中心(datacenter)或取集群别名,或者取vip,那可以这样配置:

DetectDataCenterQuery: "select dc_vaild from dbinfo where hostname = 'dc_vaild_host_flag' and port = 'dc_vaild_port_flag'"
DetectClusterDomainQuery: "select vip as cluster_domain from dbinfo where hostname = 'dc_vaild_host_flag' and port = 'dc_vaild_port_flag'"
DetectClusterAliasQuery: "select cluster_name as cluster_alias from dbinfo where hostname = 'dc_vaild_host_flag' and port = 'dc_vaild_port_flag'"

这里是通过hostname和port字段去确定一个实例的,同理physical_ip和port去确定一个实例也一样。

其中, dc_vaild_host_flagdc_vaild_port_flag 是占位符,程序里去替换的东西,不要动。

这里你只需要替换dc_vaild 、dbinfo 、hostname、 port字段就好,改成你们表里自己定义的字段或表明。

字段说明:

1. dc_vaild: 标识某两个实例处于同一个数据中心的字段,比如A->B->C三个实例,我需要标识A和B处于同一数据中心,那么在表里只需要将这两个实例的dc_vaild设置为Y就好,或者设置为相同的值
2. dbinfo: 表名。也就是取数据的元数据表的表明。
3. hostname: 这个主要是实例的主机名或者IP地址。具体看HostnameResolveMethod设置。
4. port: 顾名思义就是端口字段。

把元数据放到配置文件中来做的一点是因为:如果要在提供服务的集群上做的话,就需要在除对等实例外的实例上用sql_log_bin去更新dc_vaild这个字段,是有侵入的。

当然也可以把这份元数据维护在orchestrator自身的那个数据库当中,我们这边是有一个总的cmdb,所以放在这里。

四、 BUG修复

1.优雅切换无法选择正确的数据中心

处于同一个IDC数据中心的实例需要相互切换时,设置: PreventCrossDataCenterMasterFailover = True,在进行切换时,选择不到正确的数据中心,导致切换失败

修复如下:

func chooseCandidateReplica(...){
    ...
    priorityMajorVersion, _ := getPriorityMajorVersionForCandidate(replicas)
	priorityBinlogFormat, _ := getPriorityBinlogFormatForCandidate(replicas)
    // 新增getPriorityDataCenterForCandidate函数,返回datacenter字段,具体代码请查看inst/instance_topology.go
    priorityDataCenter, _ := getPriorityDataCenterForCandidate(replicas)

    // 新增对参数:PreventCrossDataCenterMasterFailover 判断
    if config.Config.PreventCrossDataCenterMasterFailover {
        for _, replica := range replicas {
            replica := replica
            if isGenerallyValidAsCandidateReplica(replica) &&
                // 新增数据中心判断 没有改判断,所以切换失败
                IsDataCenterCandidateReplica(priorityDataCenter, replica) &&
                !IsBannedFromBeingCandidateReplica(replica) &&
                !IsSmallerMajorVersion(priorityMajorVersion, replica.MajorVersionString()) &&
                !IsSmallerBinlogFormat(priorityBinlogFormat, replica.Binlog_format) {
                // this is the one
                candidateReplica = replica
                break
            }
	    }
    } else {
        // 还是原来的切换方案
        for _, replica := range replicas {
            replica := replica
            if isGenerallyValidAsCandidateReplica(replica) &&
                !IsBannedFromBeingCandidateReplica(replica) &&
                !IsSmallerMajorVersion(priorityMajorVersion, replica.MajorVersionString()) &&
                !IsSmallerBinlogFormat(priorityBinlogFormat, replica.Binlog_format) {
                // this is the one
                candidateReplica = replica
                break
            }
	    }
    }
    ...
}

以上代码修复了在优雅切换中,配置了数据中心参数,但是实例选择不正确的问题,同时也不会影响不配置该参数的情况。该bug已提交至github官方。

五、 日志补齐

日志补齐系统的原理用到了 show slave status\G 里的 execute_master_position 这个位点来做的。

orchestrator默认是超过配置的 ReasonableReplicationLagSeconds 秒后,切换会直接退出,所以新增了一个配置: SlaveBinLogEnableMaxLagSeconds

日志补齐系统具体介入时机:

if (secondbehindmaster > ReasonableReplicationLagSeconds && secondbehindmaster < SlaveBinLogEnableMaxLagSeconds) {
    ...
    // 直接返回true
    return instance, true, nil
} else if (secondbehindmaster < ReasonableReplicationLagSeconds) {
    // sql线程等待代码
} else {
    return instance, false, nil
}

以上代码就是日志补齐系统的介入时机。也就是延迟大于ReasonableReplicationLagSeconds配置并且小于SlaveBinLogEnableMaxLagSeconds配置。

具体怎么做?

Exec_Master_Log_Pos记录的是SQL thread执行到master binlog的文件和位置,对应的master上binlog的文件和位置

所以,首先拿到Exec_Master_Log_Pos,通过它找到binlog日志文件,拉取到orchestrator主节点上,通过mysqlbinlog解析为sql文件,然后在新主库上应用。

应用日志的账号是什么?

这里就是用到了配置文件中配置的账号和密码: MySQLTopologyUser、 MySQLTopologyPassword,所以在原来权限的基础上,要把alter、insert、update、delete权限加上去。

六、 orchestrator-client改进

因为我们需要用到datacenter来配置同机房实例来互相切换,所以需要在orchestrator-client中显示datacenter的配置情况,以便切换前做最后检查

改进代码如下:

func (this *Instance) descriptionTokens() {
    ...
    tokens = append(tokens, this,DataCenter)
    ...
}

使用方式如下:

orchestrator-client -c topology-tabulated -a <cluster_name>

可以看到datacenter显示

七、 双主模式切换

useage:

`orchestrator-client -c cascade-master-takeover-auto -i slave_inst:port -d dest_inst:port`
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2014 Outbrain Inc Copyright 2015 Booking.com Copyright 2016 - Feb 2020 GitHub Copyright 2020 - Shlomi Noach Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

基于orchestrator 3.2.4 二次开发和生态建设 展开 收起
Go
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Go
1
https://gitee.com/corollaD/orchestrator.git
git@gitee.com:corollaD/orchestrator.git
corollaD
orchestrator
orchestrator
master

搜索帮助