2.4K Star 8.2K Fork 4.4K

GVPMindSpore / mindspore

 / 详情

ops.eigvals算子卡死无法输出

DONE
Bug-Report
创建于  
2024-05-13 22:46
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

使用ops.eigvals()算子时,如果使用ascend 静态图推理计算,会出现卡死的情况,在较长时间等待后报错退出。

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :2.2.14
    -- Python version (e.g., Python 3.7.5) :3.7.5
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):Linux localhost.localdomain 4.18.0-193.el8.aarch64 #1 SMP Fri May 8 11:05:12 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
    -- GCC/Compiler version (if compiled from source):

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:

/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

`import mindspore.nn as nn
import mindspore.ops as ops
import mindspore as ms
import numpy as np
import time

class TestEigValNet(nn.Cell):
def init(self):
super().init()

def construct(self, x):
    x = ops.eigvals(x)
    return x

ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend")
x = ms.Tensor(np.loadtxt("tensor.txt"))
print(x.shape)
net = TestEigValNet()
start_time = time.time()
y = net(x)
end_time = time.time() # This gets the time when your code finished
execution_time = end_time - start_time # This calculates the difference in the two times

print(f"Execution time: {execution_time} seconds")
print(y.shape)`

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

  1. ascend环境执行

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

`(/home/weichong/env/nemo-2214) [root@localhost distributed]# python eigen_value_test.py
(2048, 2048)
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.412 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:605] GetDumpPath] Environment variable 'RANK_ID' is empty, using the default value: 0
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.469 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:613] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/node_dump/...
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.505 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:694] PrintDebugInfoAndDumpFailNode] Task fail infos, rt task_id: 2, rt stream_id: 4, tid: 1937592, device_id: 0, retcode: 507011 ( model execute failed)
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.537 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:703] PrintDebugInfoAndDumpFailNode] Task DebugString, AicpuTask: so_name: libmindspore_cpu_kernels.so, kernel_name: RunCpuKernel, mindspore stream_id: 0, dump_flag: 0, input_data_addr: [0]: 0x12483b800000, output_data_addr: [0]: 0x12483d800c00, [1]: 0x1240c0000000
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.755 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:712] PrintDebugInfoAndDumpFailNode] Dump task error infos (input/output's value) for node:[Default/Eig-op0], save path: ./rank_0/node_dump,
In file /home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/ops/function/linalg_func.py:166/ u, _ = _get_cache_prim(P.Eig)(compute_v=False)(A)/
In file eigen_value_test.py:14//
In file eigen_value_test.py:13/ return x/

[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.069 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:605] GetDumpPath] Environment variable 'RANK_ID' is empty, using the default value: 0
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.131 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:613] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/exec_order/...
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.456 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:781] DumpDebugInfoFile] Execute order has saved at /home/nemo/distributed/rank_0/exec_order/kernel_graph_0.csv
Traceback (most recent call last):
File "eigen_value_test.py", line 23, in

File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/nn/cell.py", line 680, in call
out = self.compile_and_run(*args, **kwargs)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/nn/cell.py", line 1023, in compile_and_run
return _cell_graph_executor(self, *new_args, phase=self.phase)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1589, in call
return self.run(obj, *args, phase=phase)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1628, in run
return self._exec_pip(obj, *args, phase=phase_real)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 121, in wrapper
results = fn(*arg, **kwargs)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1608, in _exec_pip
return self._graph_executor(args, phase)
RuntimeError: Run task for graph:kernel_graph_0 error! The details refer to 'Ascend Error Message'.


  • Ascend Error Message:

E39999: Inner Error!
E39999 Aicpu kernel execute failed, device_id=0, stream_id=4, task_id=2, errorCode=91.[FUNC:PrintAicpuErrorInfo][FILE:task_info.cc][LINE:1522]
TraceBack (most recent call last):
Aicpu kernel execute failed, device_id=0, stream_id=4, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1454]
rtStreamSynchronizeWithTimeout execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
synchronize stream failed, runtime result = 507011[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)


  • C++ Call Stack: (For framework developers)

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:297 RunGraph
`

Special notes for this issue/备注 (Optional / 选填)

评论 (4)

Nemo 创建了Bug-Report

Please assign maintainer to check this issue.
请为此issue分配处理人。
@Nemo

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review

载入的tensor可联系我获取,或随机生成2048*2048的tensor

Shawny 负责人设置为Henry Shi
Shawny 关联项目设置为MindSpore Issue Assistant
Shawny 计划开始日期设置为2024-05-15
Shawny 计划截止日期设置为2024-06-15
Shawny 添加了
 
mindspore-assistant
标签
Shawny 添加了
 
sig/ops
标签
Shawny 任务状态TODO 修改为WIP
Shawny 负责人Henry Shi 修改为wang_ziqi
Shawny 添加协作者Henry Shi

Eig AICPU算子性能较差,大shape场景计算耗时较久,由于CANN存在超时时间限制,过长的计算时间会导致任务执行超时,建议先缩小shape使用该AICPU算子

Nemo 任务状态WIP 修改为DONE

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(4)
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助