Environment

• Hardware Environment(Ascend/GPU/CPU):

Uncomment only one /device <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

device cpu

• Software Environment:
  -- MindSpore version (source or binary):
  -- Python version (e.g., Python 3.7.5):
  -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
  -- GCC/Compiler version (if compiled from source):

Related testcase

Steps to reproduce the issue

Simple execution of benchmark
benchmark_train --epochs=1 --modelFile=mobilenetv3_train.ms --inDataFile=train_io/mobilenetv3_input1.bin,train_io/mobilenetv3_input2.bin --expectedDataFile=train_io/mobilenetv3_output

This memory leak is unrelated to ToD. it is a bug in the scheduler reallocation (as can be seen below):
first by InferSubGraphShape
scheduler.cc:82
then by ScheduleSubGraphToKernels
scheduler.cc:87

It can be reproduced by executing lite benchmark application as well

Describe the current behavior

Parameter allocation is called twice, so leak created.

Describe the expected behavior

Allocation should be called once

Related log / screenshot

**first allocation: **
mindspore::kernel::PopulateSoftmaxCrossEntropyParameter (prim=0x7ffff5ac0994)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_populate_parameter.cc:144
#1 0x00007ffff7c1bde8 in mindspore::lite::Scheduler::InferNodeShape (this=0x7fffffffd540, node=0x555555684c30,
infer_shape_interrupt=0x7fffffffd32b) at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:153
#2 0x00007ffff7c1c60e in mindspore::lite::Scheduler::InferSubGraphShape (this=0x7fffffffd540, subgraph_index=0,
infer_shape_interrupt=0x7fffffffd32b) at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:204
#3 0x00007ffff7c1b3ab in mindspore::lite::Scheduler::Schedule (this=0x7fffffffd540, dst_kernels=0x5555556ca2c0)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:82
#4 0x00007ffff7c3008b in mindspore::lite::LiteSession::CompileGraph (this=0x5555556ca2b0, model=0x555555673c10)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/lite_session.cc:402
#5 0x00007ffff7c46935 in mindspore::lite::TrainSession::CompileTrainGraph (this=0x5555556ca0a0, model=0x555555673c10)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_session.cc:95
#6 0x00007ffff7c4a985 in mindspore::session::TrainSession::CreateSession (model=0x555555673c10, context=0x5555556728e0, train_mode=false)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_session.cc:488
#7 0x000055555558a0d6 in mindspore::lite::NetTrain::RunNetTrain (this=0x7fffffffda00)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/net_train.cc:582
#8 0x000055555558da31 in mindspore::lite::RunNetTrain (argc=9, argv=0x7fffffffdfa8)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/net_train.cc:920
#9 0x000055555558254e in main (argc=9, argv=0x7fffffffdfa8)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/main.cc:23

Second allocation for the same node:
mindspore::kernel::PopulateSoftmaxCrossEntropyParameter (prim=0x7ffff5ab9994)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_populate_parameter.cc:144
#1 0x00007ffff7c18a88 in mindspore::lite::Scheduler::InferNodeShape (this=0x7fffffffd530, node=0x555555684bb0,
infer_shape_interrupt=0x7fffffffd04b) at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:153
#2 0x00007ffff7c1a791 in mindspore::lite::Scheduler::FindBackendKernel (this=0x7fffffffd530,
in_tensors=std::vector of length 2, capacity 2 = {...}, out_tensors=std::vector of length 2, capacity 2 = {...}, node=0x555555684bb0,
prefer_data_type=mindspore::kTypeUnknown) at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:517
#3 0x00007ffff7c1afcb in mindspore::lite::Scheduler::ScheduleNodeToKernel (this=0x7fffffffd530, src_node=0x555555684bb0,
prefer_data_type=mindspore::kTypeUnknown) at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:579
#4 0x00007ffff7c1b4e1 in mindspore::lite::Scheduler::ScheduleSubGraphToKernels (this=0x7fffffffd530, subgraph_index=0,
dst_kernels=0x5555556ca240, in_tensors=0x0, out_tensors=0x0, prefer_data_type=mindspore::kTypeUnknown)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:609
#5 0x00007ffff7c1813a in mindspore::lite::Scheduler::Schedule (this=0x7fffffffd530, dst_kernels=0x5555556ca240)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/scheduler.cc:87
#6 0x00007ffff7c2cd2b in mindspore::lite::LiteSession::CompileGraph (this=0x5555556ca230, model=0x555555673a30)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/lite_session.cc:402
#7 0x00007ffff7c435d5 in mindspore::lite::TrainSession::CompileTrainGraph (this=0x5555556ca020, model=0x555555673a30)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_session.cc:95
#8 0x00007ffff7c47625 in mindspore::session::TrainSession::CreateSession (model=0x555555673a30, context=0x5555556729d0, train_mode=false)
at /home/yoni/git-proj/mindspore/mindspore/lite/src/train/train_session.cc:488
#9 0x000055555558a0d6 in mindspore::lite::NetTrain::RunNetTrain (this=0x7fffffffd9f0)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/net_train.cc:582
#10 0x000055555558da31 in mindspore::lite::RunNetTrain (argc=9, argv=0x7fffffffdf98)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/net_train.cc:920
#11 0x000055555558254e in main (argc=9, argv=0x7fffffffdf98)
at /home/yoni/git-proj/mindspore/mindspore/lite/tools/benchmark_train/main.cc:23

Special notes for this issue