2 Star 24 Fork 13

mjuyangyousong / MyRpc

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

MyRpc

EMAIL: mju_yangyousong@qq.com

POWERED BY Yousong Yang

编译环境

编译器需支持:C++17

如何开始

1.CLONE项目
2.mkdir build
3.cd build
4.cmake .. -DCMAKE_CXX_COMPILER={支持C++17的编译器}
5.make
6.初期运行 生成配置文件
7.关闭服务 修改配置文件

性能测试

测试平台: VMware虚拟机15 core:4 mem:8G 主机:thinpad E14 r5 4500 内存16gb

zeromq:
  muduo:50字节 单程延迟46us
        50000字节 单程延迟64us
  MyRpc:50字节 单程延迟43us
        50000字节 单程延迟62us
        150000字节 单程延迟112us

压力测试
  SERVER 100CPU 单线程 1500字节数据包 转发回Client QPS 90000+

Log:
  单线程 80字节信息 9341520+qps

Rpc:
  单对单序列RPC 解析RPC 调用RPC
    SERVER: 序列化RPC+发送包 一次调用一个数据包 连续发送 每次65.601ns
    Client: 接受包+解析RPC+调用 每次40.739ns

如何使用

RPC:
RPC消息包默认格式: 单位字节
|-------------------------------------------------------------------------------------------------------------|
| 消息包大小 4 | 消息类型(定义在dataformat) 4 | 调用PRC序号 4 | 消息数据大小 4 | 是否调用回调 1 | 消息数据 -- |
|-------------------------------------------------------------------------------------------------------------|
connect->Read 返回对端一次Write的内容 即:消息类型(定义在dataformat) 调用PRC序号 消息数据大小 是否调用回调 消息数据 or 任何数据
一次Write对应一次Read 需要口头规定消息包最大大小 以免一次Read无法读尽
Rpc调用过程:
对端注册一个RPC
Rpc.Init_fun(type(口头约定的类型),fun_address 可以是lambda 可以是函数地址,size 希望的接收到的参数数据包大小(消息数据大小),回调函数(可选))
发送端发送一个RPC
Message::Make_rpc(connnet,(口头约定的类型),缓冲区,是否调用回调,参数...)
    // Server:
      Main():
        myrpc::Message message = myprc::Message();  // 初始化messgae
        net_tools::Config config("Server");         // 加载配置
        net_tools::Log log;                         // 初始化LOG
        net_tools::Tcpserver tcp;                   // 创建TCP服务
        tcp.Set_connect_over_callback(conn_func);   // 设置回调
        tcp.Initserver(true,2,0);                   // 注册TCP服务
        tcp.Startserver();                          // 开始服务

      conn_func(net_tools::net::Connect* connect):
        connect->Set_read_callback(read_func);      // 设置收到消息包回调(可选)
        connect->Set_close_callback(close_func);    // 设置对端Close回调(可选)
        connect->Set_lose_callback(lose_func);      // 设置心跳包失效回调(可选)
        char buf[512] = {};                         // 用户缓冲区
        myrpc::Message::Make_rpc(connect,3,buf,true,12,13,14);  //调用 3号RPC调用 缓冲区为buf 调用回调函数 参数为 (int)12 13 14
        myrpc::Message::Make_rpc(connect,2,buf,true,15);        //调用 2号RPC调用 缓冲区为buf 调用回调函数 参数为 (int)15

    // Client:

      Main():
          myrpc::Message message = myprc::Message(); // 初始化messgae
          net_tools::Config config("Client");         // 加载配置
          net_tools::Log log;                         // 初始化LOG
          net_tools::Tcpclient tcp;                   // 创建TCP服务
          message.Get_rpc()->Init_fun(3,myrpc::base::Rpc::Test,sizeof_fun(myrpc::base::Rpc::Test),[](net_tools::net::Connect*,void*,unsigned int){NT_LOG_INFO << "rpc_test" << NT_LOG_ENDL;});
          message.Get_rpc()->Init_fun(2,Test,sizeof_fun(Test),[](net_tools::net::Connect*,void*,unsigned int){NT_LOG_INFO << "rpc" << NT_LOG_ENDL;});
          {
            int b = 0;
            auto fun = [=](int a)->void{NT_LOG_INFO << a << b << NT_LOG_ENDL;};
            message->Get_rpc()->Init_fun(1,fun); //接口会保存该lambda 延长生命周期
          }
          tcp.Set_connect_over_callback(conn_func);   // 设置回调
          tcp.Startclient(true);                      // 循环尝试连接
          pause();

          char buf[200] = {};                         // 缓冲区
      read_func(net_tools::net::Connect* connect)
          connect->Read(buf,200);                     // 读取消息 buf缓冲区地址 200 大小
          message->Consume_message(connect,buf,200);  // 消费消息

工程目录 OLD

├── base                         // 网络库 base
|   ├── base_buffer             // 能移动的buffer 零拷贝发送
│   ├── bplustree               // 模板B+树 Timebplustree调用
│   ├── channel                 // Channel 包装每一个event事件
│   ├── channelpool             // 管理一个线程内所有Channel
│   ├── condition               // 封装列posix的条件变量
│   ├── config                  // 解析配置 JSON
│   ├── count_down_cond         // 带计数器的条件变量
│   ├── cpuinfo                 // 分析平台CPU
│   ├── epoll                   // 封装列linux epoll
│   ├── eventloop               // 事件循环
│   ├── eventloopthread         // 包装了事件循环的线程
│   ├── eventloopthreadpool     // IO线程池
│   ├── function                // 引入std::function
│   ├── hash                    // 闭散列HASH
│   ├── json                    // 包装列JSONCPP
│   ├── logbuffer               // LOG的缓冲区
│   ├── log                     // LOG调用
│   ├── logfile                 // LOG写入的文件
│   ├── logstream               // LOG iostream风格调用
│   ├── logthread               // LOG线程
│   ├── mempool                 // 分配固定大小的内存池
│   ├── mutex                   // 封装posix 互斥量
│   ├── noncopyable             // 基类 不允许左值拷贝构造
│   ├── thread                  // 包装了posix线程
│   ├── threadpool              // 简易逻辑线程池
│   ├── timebplustree           // 管理单个线程单次定时任务
│   ├── timeevent               // 单词定时任务
│   ├── timeeventset            // 管理单个对象的单次定时任务
│   ├── timeevery               // 循环定时任务
│   ├── timeeverymap            // 管理单个对象打循环定时任务
│   ├── timequeue               // 管理单个线程打循环任务时间轮
│   ├── timer                   // 包装了timefd定时器
├── rpc                          // Rpc框架
│   ├── dataformat              // 信息包格式定义
│   ├── dmq                     // 分布式消息队列
│   ├── message                 // 信息包注册消费中心
│   ├── nodeformat              // 节点格式定义
│   ├── rpc                     // rpc框架
│   ├── raft                    // raft实现分布式(未实现)
│   ├── rpccall                 // 元编程Rpc序列 反序列
│   ├── serverfinder            // 服务发现
├── net                          // 网络库 net
│   ├── accept                  // 封装posix accpet
│   ├── buffer                  // TCP连接缓冲区
|   ├── buffer_reader           // base_buffer迭代器
│   ├── connect                 // 单个TCP连接
│   ├── connectpool             // 管理单个线程的CONNECT对象
│   ├── heartbeat               // 心跳系统 FOR SERVER
│   ├── socket                  // 封装posix socket调用
│   ├── tcpclient               // TCP client主体or单次连接
│   ├── tcpserver               // TCP server
└── user                         // 一些示例

Future:

eventloop已经过时 该网络库为过时的框架
网络库使用eventloop使得代码需要非阻塞 而很多的业务逻辑需要并行的rpc调用 过分的回调函数实现异步 所以说要使用应用层的线程来改造网络库
至少lock_free 追求wait_free 应用层的线程也容易实现负载均衡 避免一个调用卡住后面的调用导致超时 task_steal等
epoll的模型也能转换为io_uring 减少进入内核的次数 从而提高吞吐量 
现阶段的rpc是在eventloop上进行的 只能实现丑陋的回调异步 需要应用层线程来实现join 从而让业务逻辑能够简化并行处理多个rpc返回值的问题
而且可以方便的终止rpc 通过一个rpc的返回决定另外的rpc的处理方式
rpc也需要实现可扩展的格式 这在现基础上 可以使用数据包附带偏移量的指示 按照原来预定的设想 通过编译期的参数推导实现入参 框架集成pb 简化依赖项
为了可扩展性 protobuf可选支持
服务发现需要raft的支持 保证服务发现的高可用
集成管理的接口 不停机配置 维护

目标:

底层网络库io_uring+应用层线程+集成协议的rpc+raft支持的服务发现+raft接口拓展为集群功能+可选的解耦中间件(类似kafka的消息队列 减少整个大集群连接的拓扑复杂度)

下一步:

重构网络库
应用层线程支持
rpc协议更新
高可用的服务发现
raft接口拓展机器为集群功能
往上搭建DMQ 消息队列
实现最终的满足业务开发绝大多数需求的rpc架构 MyRpc

如何学习

推荐书籍:

C++

服务器

操作系统

分布式

数据库

杂项

工程简介

MyRpc

Rpc:

自研Rpc框架 两步实现Rpc调用

RPC: 使用元编程实现RPC序列反序列以及调用

自制function 实现类型擦除 相对于多态的类型擦除 调用速度提升7倍

faster_function 相对于多态实现的rpccall 速度提升15倍 qps达到 22m

不使用JSON等序列化工具 支持不定长 支持lambda(带捕获参数) 函数指针 自定义类型 回调函数

网络库:

参考muduo的事件驱动型网络库

相对于muduo添加了定时器树,心跳树功能

两种定时任务策略 定时器树,TIMERFD

适合作为接入,解析数据包,转发服务器,IO密集

LOG系统 c++ iostream风格

缓冲区系统 内部采用内存池 隔离用户与系统调用 简化用户业务逻辑 支持base_buffer零拷贝

内存池 仿ptmalloc 为缓冲区提供大块内存

逻辑线程池 任务队列 用以执行用户注册的业务函数

IO线程池 EPOLL监听注册IO描述符

定时器树 用B+树作为底层 维护收到连接注册的定时期任务或在用户调用的定时期任务

定时器循环任务map 以任务名为key 注册循环定时器任务 每个任务一个timefd 当定时任务多时 可用前缀树优化

心跳树 使用类B+树 通过划分心跳时间片 高效管理连接心跳

Config 使用json解析配置

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

c++高性能RPC框架 展开 收起
C++ 等 3 种语言
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
C++
1
https://gitee.com/mjuyangyousong/MyRpc.git
git@gitee.com:mjuyangyousong/MyRpc.git
mjuyangyousong
MyRpc
MyRpc
master

搜索帮助