建了一个6个osd的集群,创建一个6个pg的pool,使用block_bench写数据,然后kill 掉一个osd,导致pg重新分布,在pg成员变更过程中触发了snapshot recovery,发送snapshot时osd coredump。
报的错误:
==2004069==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250004aabe8 at pc 0x560f402a9a59 bp 0x7ffc4e927cb0 sp 0x7ffc4e927ca0
WRITE of size 8 at 0x6250004aabe8 thread T0 (reactor_5)
#0 0x560f402a9a58 in msg::rdma::transport_data::from_net_context(msg::rdma::memory_pool<ibv_recv_wr>::net_context*) (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2a3ba58)
#1 0x560f40337d35 in msg::rdma::server::handle_cqe_poll() (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2ac9d35)
#2 0x560f403392c6 in msg::rdma::server::cqe_poller(void*) (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2acb2c6)
#3 0x560f3fe7248d in thread_execute_poller /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:819
#4 0x560f3fe74399 in thread_poll /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:941
#5 0x560f3fe74809 in spdk_thread_poll /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:992
#6 0x560f3ff5e26d in _reactor_run /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:924
#7 0x560f3ff5ea29 in reactor_run /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:962
#8 0x560f3ff5f593 in spdk_reactors_start /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:1064
#9 0x560f3ff52b23 in spdk_app_start /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/app.c:645
#10 0x560f4023f159 in main /root/euler_workspace/fastblock/src/osd/osd.cc:703
#11 0x7f271304cfcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#12 0x7f271304d07c in __libc_start_main_impl ../csu/libc-start.c:409
#13 0x560f3f9cdba4 in _start (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x215fba4)
0x6250004aabe8 is located 8 bytes to the right of 8928-byte region [0x6250004a8900,0x6250004aabe0)
allocated by thread T0 (reactor_5) here:
#0 0x7f27146a4337 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:102
#1 0x560f403666b8 in std::_MakeUniq<msg::rdma::transport_data::remote_info []>::__array std::make_unique<msg::rdma::transport_data::remote_info []>(unsigned long) (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2af86b8)
#2 0x560f402a969d in msg::rdma::transport_data::from_net_context(msg::rdma::memory_pool<ibv_recv_wr>::net_context*) (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2a3b69d)
#3 0x560f40337d35 in msg::rdma::server::handle_cqe_poll() (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2ac9d35)
#4 0x560f403392c6 in msg::rdma::server::cqe_poller(void*) (/root/euler_workspace/fastblock/build/src/osd/fastblock-osd+0x2acb2c6)
#5 0x560f3fe7248d in thread_execute_poller /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:819
#6 0x560f3fe74399 in thread_poll /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:941
#7 0x560f3fe74809 in spdk_thread_poll /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/thread/thread.c:992
#8 0x560f3ff5e26d in _reactor_run /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:924
#9 0x560f3ff5ea29 in reactor_run /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:962
#10 0x560f3ff5f593 in spdk_reactors_start /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/reactor.c:1064
#11 0x560f3ff52b23 in spdk_app_start /root/euler_workspace/fastblock/build/deps_build/spdk-prefix/src/spdk/lib/event/app.c:645
#12 0x560f4023f159 in main /root/euler_workspace/fastblock/src/osd/osd.cc:703
#13 0x7f271304cfcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
通过日志得知成功发送了部分snapshot:
Hi liuhq189, welcome to the openEuler Community.
I'm the Bot here serving you. You can find the instructions on how to interact with me at Here.
If you have any questions, please contact the SIG: sig-SDS, and any of the maintainers: @chixinze , @liuzhiqiang , @liuqinfei , @luorixin , @李韶雄 , @吴兴义 , @孙逸方 , @刘闳全 , @szy , @openeuler-ci-bot
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
登录 后才可以发表评论