[ST][MS][MF][910B3]llama2/internlm/qwen/baichuan2/yi/codellama/qwen1.5/glm3/glm2/deepseek网络910b3 推理报ARuntimeError: The pointer[primitive] is null.

name	about	labels
Bug Report	Use this template for reporting a bug	kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

模型地址：https://gitee.com/mindspore/mindformers
llama2/internlm/qwen/baichuan2/yi/codellama/qwen1.5/glm3/glm2网络910b3 推理报ARuntimeError: The pointer[primitive] is null.

Environment / 环境信息 (Mandatory / 必填)

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device 910B3

Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g.,r1.6 commit_id=xxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):
ok版本
CANN版本：Milan_C17/20240414
MindSpore版本：master_20240516061516_ec3d4f083be7
MindFormers版本：dev_20240516121523_e1f10765325660601a8ed
失败版本
runpkg_version:Milan_C17/20240414
MindSpore:2.3.0.B521:master_20240517221756_2f410aa8a72d8e9b2cf3cd2fd05903cf307e3768
MindFormers版本：dev_20240517213214_956fb6a1711fcf9fee
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

测试仓库地址：MindFormers_Test/cases/
用例：
test_mf_llama2_70b_eval_squad_8p_0001
test_mf_yi_6b_infer_batch_incremental_1p_0001
test_mf_llama2_7b_infer_alpaca_batch_incremental_1p_0001
test_mf_baichuan2_13b_infer_belle_batch_incremental_1p_0001
test_mf_llama2_70b_infer_alpaca_batch_incremental_8p_0001
test_mf_codellama_34b_batch_4_infer_910_4p_0001
test_mf_glm2_6b_ptuning2_infer_batch_incremental_1p_0001
test_mf_chatglm3_6b_32k_kbk_infer_batch_incremental_1p_0001
test_mf_qwen1_5_14b_kbk_infer_batch_incremental_1p_0001
test_mf_baichuan2_13b_infer_belle_incremental_1p_0001
test_mf_yi_6b_infer_incremental_1p_0001
test_mf_llama2_7b_infer_alpaca_incremental_1p_0001
test_mf_baichuan2_13b_kbk_infer_batch_incremental_1p_0001
test_mf_llama2_7b_kbk_infer_alpaca_batch_incremental_1p_0001
test_mf_baichuan2_13b_kbk_infer_incremental_1p_0001
test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001
test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001
test_mf_llama2_7b_kbk_infer_alpaca_incremental_1p_0001
test_mf_yi_34b_kbk_infer_batch_incremental_4p_0001
test_mf_llama2_70b_infer_alpaca_incremental_8p_0001
test_mf_codellama_34b_incremental_infer_910_4p_0001
test_mf_qwen1_5_72b_kbk_infer_incremental_4p_0001
test_mf_yi_34b_kbk_infer_incremental_4p_0001
test_mf_llama2_70b_kbk_infer_alpaca_batch_incremental_4p_0001
test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001
test_mf_qwen_14b_infer_alpaca_batch_incremental_1p_0001
test_mf_baichuan2_7b_infer_belle_incremental_1p_0001
test_mf_qwen_14b_infer_alpaca_incremental_1p_0001
test_mf_qwen1_5_72b_kbk_infer_batch_incremental_4p_0001
test_mf_llama2_70b_kbk_infer_alpaca_incremental_4p_0001
test_mf_llama2_13b_infer_alpaca_batch_incremental_1p_0001
test_mf_qwen_14b_batch_4_dis_inferl_2p_0001
test_mf_llama2_13b_infer_alpaca_incremental_1p_0001
test_mf_internlm_20b_kbk_infer_batch_incremental_1p_0001
test_mf_qwen_7b_infer_incremental_1p_0001
test_mf_internlm_20b_kbk_infer_incremental_1p_0001
test_mf_deepseek_33b_kbk_infer_incremental_4p_0001.py
test_mf_deepseek_33b_kbk_infer_batch_incremental_4p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

get code from mindformers
set vocab_file path in config file
python llama2_generate.py --model_type llama2_7b --yaml_file ./configs/llama2/predict_llama2_7b.yaml --use_past True --checkpoint_path /home/workspace/large_model_ckpt//llama2/7b/llama2_7b.ckpt --batch_size 1
验证llama 7b网络增量推理成功

Describe the expected behavior / 预期结果 (Mandatory / 必填)

网络增量推理成功

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

[ERROR] RUNTIME_FRAMEWORK(1498435,fffce7fff120,python):2024-05-18-11:28:05.287.233 [mindspore/ccsrc/runtime/graph_scheduler/actor/actor_common.cc:274] WaitRuntimePipelineFinish] Wait runtime pipeline finish and an error occurred: The pointer[primitive] is null.

----------------------------------------------------
- Framework Unexpected Exception Raised:
----------------------------------------------------
This exception is caused by framework's unexpected error. Please create an issue at https://gitee.com/mindspore/mindspore/issues to get help.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/backend/common/optimizer/dynamic_shape/dynamic_shape_helper.cc:491 InferShape

Traceback (most recent call last):
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/../run_qwen_incremental.py", line 156, in <module>
    main(args)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/../run_qwen_incremental.py", line 100, in main
    result = task.predict(input_data=prompt,
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/_checkparam.py", line 1372, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/trainer/trainer.py", line 692, in predict
    output_result = self.trainer.predict(
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 343, in predict
    return self.predict_process(config=config,
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/trainer/base_trainer.py", line 937, in predict_process
    output_results = self.pipeline_task(input_data, top_k=top_k)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/pipeline/base_pipeline.py", line 151, in __call__
    outputs = self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/pipeline/base_pipeline.py", line 237, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/pipeline/base_pipeline.py", line 303, in forward
    return self._forward(model_inputs, **forward_params)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/pipeline/text_generation_pipeline.py", line 197, in _forward
    output_ids = self.network.generate(input_ids, **forward_params)
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/generation/text_generator.py", line 830, in generate
    target_list, is_finished = self.infer(input_ids=input_ids,
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/generation/text_generator.py", line 952, in infer
    res, current_index = self.forward(input_ids=input_ids,
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/generation/text_generator.py", line 1056, in forward
    res = self._incremental_infer(
  File "/home/jenkins0/MindFormers_Test/cases/qwen/7b/infer/test_mf_qwen_7b_infer_incremental_1p_0001/mindformers/generation/text_generator.py", line 306, in _incremental_infer
    res = self(
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/nn/cell.py", line 696, in __call__
    out = self.compile_and_run(*args, **kwargs)
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1017, in compile_and_run
    return _cell_graph_executor(self, *new_args, phase=self.phase)
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/common/api.py", line 1685, in __call__
    return self.run(obj, *args, phase=phase)
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/common/api.py", line 1724, in run
    return self._exec_pip(obj, *args, phase=phase_real)
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/common/api.py", line 132, in wrapper
    results = fn(*arg, **kwargs)
  File "/home/miniconda3/envs/large_model_39/lib/python3.9/site-packages/mindspore/common/api.py", line 1704, in _exec_pip
    return self._graph_executor(args, phase)
RuntimeError: The pointer[primitive] is null.

----------------------------------------------------
- Framework Unexpected Exception Raised:
----------------------------------------------------
This exception is caused by framework's unexpected error. Please create an issue at https://gitee.com/mindspore/mindspore/issues to get help.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/backend/common/optimizer/dynamic_shape/dynamic_shape_helper.cc:491 InferShape

Special notes for this issue/备注 (Optional / 选填)

走给黄生帅

Please assign maintainer to check this issue.
请为此issue分配处理人。
@zhongjicheng

感谢您的提问，您可以评论//mindspore-assistant更快获取帮助：

如果您刚刚接触MindSpore，或许您可以在教程找到答案
如果您是资深Pytorch用户，您或许需要：

如果您遇到动态图问题，可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
模型精度调优问题可参考官网调优指南
如果您反馈的是框架BUG，请确认您在ISSUE中提供了MindSpore版本、使用的后端类型（CPU、GPU、Ascend）、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
如果您已经定位出问题根因，欢迎提交PR参与MindSpore开源社区，我们会尽快review

二分由以下PR引入：
!69548:pick maskter-kbk-infer-opt commits

根因分析：
ReshapeExt算子从Internal移到了RTS实现，但是RTS之前动态shape改造不彻底，build时没有调用基类的init，导致primitive_为空

修复方案：
在ReshaeExt中进行相应初始化

相关PR：
!69640:fix reshapeext

是否新增ut/st:
否

自验结果：
本地执行llama2推理，OK

回归版本：
mindformers:dev_20240521202240_babf7ed140df1c7b5f4268d5b3c40a026dd06239
mindspore:r2.3.0.B522.master_20240521191216_1286b53153d66f0049e3b62c1e53071bfb7e9516
runpkg_version:Milan_C18/20240517

回归步骤：参考issue复现步骤
基本功能：问题已解决

INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - train.py:check_performence:279 - generate speed:34.815 is up to 31.18
STEP 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/yi/6b/infer/test_mf_yi_6b_infer_incremental_1p_0001/generate_infer.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/yi/6b/infer/test_mf_yi_6b_infer_incremental_1p_0001/generate_infer.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 16  tokens
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/yi/6b/infer/test_mf_yi_6b_infer_incremental_1p_0001/generate_infer.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 43.27 is less to 50.6
STEP 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - test_mf_yi_6b_infer_incremental_1p_0001.py:test_run:135 - Step5: Checking infer result
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/yi/6b/infer/test_mf_yi_6b_infer_incremental_1p_0001/question_answer_incremental.txt] success
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:check_infer_result:276 - upload success
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:get_log_divided:379 - 标准答案分词是：DNA分子具有双螺旋结构，这种结构使DNA能够存储大量遗传信息。DNA分子的两条链通过氢键连接在一起，形成两条反向平行的链。DNA分子的碱基对按照特定的顺序排列，这种顺序被称为遗传密码子。遗>传信息储存在DNA分子的两条链上，通过复制传递给下一代细胞。
INFO 2024-05-22 02:14:36 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:get_log_divided:383 - 实测答案分词是：[' DNA分子具有双螺旋结构，两条链上的碱基通过氢键连接形成碱基对．<|endoftext|>']
[' DNA分子具有双螺旋结构，两条链上的碱基通过氢键连接形成碱基对．<|endoftext|>']

INFO 2024-05-22 02:14:37 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:get_log_divided:400 - standard_cut_infer_ret_list是：['DNA', '分子', '具有', '双螺旋', '结构', '，', '这种', '结构', '使', 'DNA', '能够', '存储', '大量', '遗传信息', '。', 'DNA', '分子', '的', '两条', '链', '通过', '氢键', '连接', '在', '一起', '，', '形成', '两条', '反向', '平行', '的', '链', '。', 'DNA', '分子', '的', '碱基对', '按照', '特定', '的', '顺序排列', '，', '这种', '顺序', '被', '称为', '遗传', '密码子', '。', '遗', '>', '传', '信息', '储存', '在', 'DNA', '分子', '的', '两条', '链上', '，', '通过', '复制', '传递', '给', '下一代', '细胞', '。']
INFO 2024-05-22 02:14:37 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:get_log_divided:401 - test_cut_infer_ret_list是：['[', "'", ' ', 'DNA', '分子', '具有', '双螺旋', '结构', '，', '两条', '链上', '的', '碱基', '通过', '氢键', '连接', '形成', '碱基对', '．', '<', '|', 'endoftext', '|', '>', "'", ']', '\n', '[', "'", ' ', 'DNA', '分子', '具有', '双螺旋', '结构', '，', '两条', '链上', '的', '碱基', '通过', '氢键', '连接', '形成', '碱基对', '．', '<', '|', 'endoftext', '|', '>', "'", ']', '\n']
INFO 2024-05-22 02:14:37 - test_mf_yi_6b_infer_incremental_1p_0001 - infer.py:check_infer_result:287 - 两个文本余弦相似度是：0.47

INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - train.py:check_performence:279 - generate speed:42.127 is up to 22.23
STEP 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/glm2/glm2_6b_ptuning2/infer/test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001/glm2_6b_ptuning2_incremental_infer.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/glm2/glm2_6b_ptuning2/infer/test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001/glm2_6b_ptuning2_incremental_infer.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 61  tokens
INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/glm2/glm2_6b_ptuning2/infer/test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001/glm2_6b_ptuning2_incremental_infer.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:17:01 - test_mf_glm2_6b_ptuning2_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 60.03 is less to 188.1

INFO 2024-05-22 02:24:49 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [grep 'generate speed' /home/jenkins/workspace/TDT_deployment/MindFormers_Test/cases/llama2/70b/infer/test_mf_llama2_70b_infer_alpaca_incremental_8p_0001/output/msrun_log/worker_0.log | awk '{print $(NF-1)}' | sed -n '$p'] success
INFO 2024-05-22 02:24:49 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - train.py:check_performence:279 - generate speed:16.188 is up to 11.79
STEP 2024-05-22 02:24:49 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:24:49 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /home/jenkins/workspace/TDT_deployment/MindFormers_Test/cases/llama2/70b/infer/test_mf_llama2_70b_infer_alpaca_incremental_8p_0001/output/msrun_log/worker_0.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:24:50 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /home/jenkins/workspace/TDT_deployment/MindFormers_Test/cases/llama2/70b/infer/test_mf_llama2_70b_infer_alpaca_incremental_8p_0001/output/msrun_log/worker_0.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:24:50 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 504  tokens
INFO 2024-05-22 02:24:50 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /home/jenkins/workspace/TDT_deployment/MindFormers_Test/cases/llama2/70b/infer/test_mf_llama2_70b_infer_alpaca_incremental_8p_0001/output/msrun_log/worker_0.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:24:50 - test_mf_llama2_70b_infer_alpaca_incremental_8p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 233.57 is less to 530.2

INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - train.py:check_performence:279 - generate speed:574.022 is up to 405.0
STEP 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/baichuan2/7b/infer/test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001/sh_eval.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/baichuan2/7b/infer/test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001/sh_eval.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 1568  tokens
INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/baichuan2/7b/infer/test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001/sh_eval.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:32:32 - test_mf_baichuan2_7b_infer_belle_batch_incremental_1p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 46.68 is less to 104.5

INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - train.py:check_performence:279 - generate speed:54.29 is up to 26.22
STEP 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/internlm/7b/infer/test_mf_internlm_7b_infer_alpaca_incremental_1p_0001/interlm_7b_incremental_infer.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/internlm/7b/infer/test_mf_internlm_7b_infer_alpaca_incremental_1p_0001/interlm_7b_incremental_infer.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 7  tokens
INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/internlm/7b/infer/test_mf_internlm_7b_infer_alpaca_incremental_1p_0001/interlm_7b_incremental_infer.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:53:04 - test_mf_internlm_7b_infer_alpaca_incremental_1p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 38.04 is less to 144.1

INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - train.py:check_performence:279 - generate speed:32.6 is up to 29.54
STEP 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:177 - checking compile time
INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/qwen1_5/14B/infer/test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001/research/qwen1_5/output/log/rank_0/info.log | grep 'total time:'|awk -F'total time:' '{print $NF}'|awk -Fs '{print $1}'|head -n 2  | sed s/,//g] success
INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/qwen1_5/14B/infer/test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001/research/qwen1_5/output/log/rank_0/info.log | grep 'generated tokens:'|awk -F'generated tokens:' '{print $NF}'|awk -F'tokens' '{print $1}'|head -n 1  ] success
INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - infer.py:get_infer_compile_time:145 - first_tokens: 219  tokens
INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - local_cmd_actuator.py:exec_shell_cmd:47 - Exec [ cat /data/jenkins_workspace/TDT_deployment/MindFormers_Test/cases/qwen1_5/14B/infer/test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001/research/qwen1_5/output/log/rank_0/info.log | grep 'generated tokens:'|awk -F'generate speed:' '{print $NF}'|awk -F'tokens' '{print $1}'|tail -n 1  ] success
INFO 2024-05-22 02:31:47 - test_mf_qwen1_5_14b_kbk_infer_incremental_1p_0001 - infer.py:check_infer_compile_time:191 - compile_time: 56.78 is less to 363.0

测试结论：
回归人员：zhongjicheng
回归时间： 2024-05-22

GVP MindSpore / mindspore

内容风险标识