name | about | labels |
---|---|---|
Bug Report | Use this template for reporting a bug | kind/bug |
使用ops.eigvals()算子时,如果使用ascend 静态图推理计算,会出现卡死的情况,在较长时间等待后报错退出。
Ascend
/GPU
/CPU
) / 硬件环境:Please delete the backend not involved / 请删除不涉及的后端:
/device ascend
Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :2.2.14
-- Python version (e.g., Python 3.7.5) :3.7.5
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):Linux localhost.localdomain 4.18.0-193.el8.aarch64 #1 SMP Fri May 8 11:05:12 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
-- GCC/Compiler version (if compiled from source):
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative
/Graph
):
Please delete the mode not involved / 请删除不涉及的模式:
/mode graph
`import mindspore.nn as nn
import mindspore.ops as ops
import mindspore as ms
import numpy as np
import time
class TestEigValNet(nn.Cell):
def init(self):
super().init()
def construct(self, x):
x = ops.eigvals(x)
return x
ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend")
x = ms.Tensor(np.loadtxt("tensor.txt"))
print(x.shape)
net = TestEigValNet()
start_time = time.time()
y = net(x)
end_time = time.time() # This gets the time when your code finished
execution_time = end_time - start_time # This calculates the difference in the two times
print(f"Execution time: {execution_time} seconds")
print(y.shape)`
`(/home/weichong/env/nemo-2214) [root@localhost distributed]# python eigen_value_test.py
(2048, 2048)
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.412 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:605] GetDumpPath] Environment variable 'RANK_ID' is empty, using the default value: 0
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.469 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:613] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/node_dump/...
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.505 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:694] PrintDebugInfoAndDumpFailNode] Task fail infos, rt task_id: 2, rt stream_id: 4, tid: 1937592, device_id: 0, retcode: 507011 ( model execute failed)
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.537 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:703] PrintDebugInfoAndDumpFailNode] Task DebugString, AicpuTask: so_name: libmindspore_cpu_kernels.so, kernel_name: RunCpuKernel, mindspore stream_id: 0, dump_flag: 0, input_data_addr: [0]: 0x12483b800000, output_data_addr: [0]: 0x12483d800c00, [1]: 0x1240c0000000
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.226.755 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:712] PrintDebugInfoAndDumpFailNode] Dump task error infos (input/output's value) for node:[Default/Eig-op0], save path: ./rank_0/node_dump,
In file /home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/ops/function/linalg_func.py:166/ u, _ = _get_cache_prim(P.Eig)(compute_v=False)(A)/
In file eigen_value_test.py:14//
In file eigen_value_test.py:13/ return x/
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.069 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:605] GetDumpPath] Environment variable 'RANK_ID' is empty, using the default value: 0
[WARNING] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.131 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:613] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/exec_order/...
[ERROR] DEVICE(1937592,ffff00fbf1e0,python):2024-05-13-22:29:10.239.456 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:781] DumpDebugInfoFile] Execute order has saved at /home/nemo/distributed/rank_0/exec_order/kernel_graph_0.csv
Traceback (most recent call last):
File "eigen_value_test.py", line 23, in
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/nn/cell.py", line 680, in call
out = self.compile_and_run(*args, **kwargs)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/nn/cell.py", line 1023, in compile_and_run
return _cell_graph_executor(self, *new_args, phase=self.phase)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1589, in call
return self.run(obj, *args, phase=phase)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1628, in run
return self._exec_pip(obj, *args, phase=phase_real)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 121, in wrapper
results = fn(*arg, **kwargs)
File "/home/weichong/env/nemo-2214/lib/python3.7/site-packages/mindspore/common/api.py", line 1608, in _exec_pip
return self._graph_executor(args, phase)
RuntimeError: Run task for graph:kernel_graph_0 error! The details refer to 'Ascend Error Message'.
E39999: Inner Error!
E39999 Aicpu kernel execute failed, device_id=0, stream_id=4, task_id=2, errorCode=91.[FUNC:PrintAicpuErrorInfo][FILE:task_info.cc][LINE:1522]
TraceBack (most recent call last):
Aicpu kernel execute failed, device_id=0, stream_id=4, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1454]
rtStreamSynchronizeWithTimeout execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
synchronize stream failed, runtime result = 507011[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:297 RunGraph
`
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:
载入的tensor可联系我获取,或随机生成2048*2048的tensor
Eig AICPU算子性能较差,大shape场景计算耗时较久,由于CANN存在超时时间限制,过长的计算时间会导致任务执行超时,建议先缩小shape使用该AICPU算子
登录 后才可以发表评论