Hi, I am trying to model a pressure regulator in simspace, but I could not manage to connect mechanical system to fluid system. I used single acting actuator and connect it to matlab function that has a function form all the "to workspace" blocks how can I fix the system I attached the picture of the systems.

% Compute spring force (negative due to opposing motion)
F_spring = -x * (k1 + k2) + F_preload;

% Compute pressure forces
F_pressure = P_out * A_outlet + P_atm * A_atm – P_sen * A_sen – P_in * A_inlet;

% Compute friction force
F_piston = mu * A_pis * P_sen;
F_poppet = mu * A_pop * P_in;
F_friction = sign(v) * (F_piston + F_poppet);

% Net force (sum of spring, pressure, and friction forces)
F_net = F_spring + F_pressure – F_friction;

% Note: m_tot * x_ddot = F_net should be solved externally (e.g., in Simulink)
endHi, I am trying to model a pressure regulator in simspace, but I could not manage to connect mechanical system to fluid system. I used single acting actuator and connect it to matlab function that has a function form all the "to workspace" blocks how can I fix the system I attached the picture of the systems.

% Compute spring force (negative due to opposing motion)
F_spring = -x * (k1 + k2) + F_preload;

% Compute pressure forces
F_pressure = P_out * A_outlet + P_atm * A_atm – P_sen * A_sen – P_in * A_inlet;

% Compute friction force
F_piston = mu * A_pis * P_sen;
F_poppet = mu * A_pop * P_in;
F_friction = sign(v) * (F_piston + F_poppet);

% Net force (sum of spring, pressure, and friction forces)
F_net = F_spring + F_pressure – F_friction;

% Note: m_tot * x_ddot = F_net should be solved externally (e.g., in Simulink)
end Hi, I am trying to model a pressure regulator in simspace, but I could not manage to connect mechanical system to fluid system. I used single acting actuator and connect it to matlab function that has a function form all the "to workspace" blocks how can I fix the system I attached the picture of the systems.

% Compute spring force (negative due to opposing motion)
F_spring = -x * (k1 + k2) + F_preload;

% Compute pressure forces
F_pressure = P_out * A_outlet + P_atm * A_atm – P_sen * A_sen – P_in * A_inlet;

% Compute friction force
F_piston = mu * A_pis * P_sen;
F_poppet = mu * A_pop * P_in;
F_friction = sign(v) * (F_piston + F_poppet);

% Net force (sum of spring, pressure, and friction forces)
F_net = F_spring + F_pressure – F_friction;

% Note: m_tot * x_ddot = F_net should be solved externally (e.g., in Simulink)
end simulink, simscape MATLAB Answers — New Questions

​

Hello MathWorks team,
I have a Simulink model named experiment_demo.slx that includes a MATLAB Function block with the following code:
function adjusted_setpoint = fcn(setpoint, cfg)
adjusted_setpoint = setpoint * cfg.gain * cfg.offset;
setpoint is provided via an Inport.
cfg is a tunable parameter defined as a Simulink.Parameter with a Bus type (ModelConfigBus) and configured via the Model Explorer.
Model configuration:
Solver: Fixed-step
Code Generation:
System target file: ert.tlc
Code interface packaging: Reusable function
Code is generated using Embedded Coder. In the generated code:
The ModelConfigBus struct is defined in experiment_demo_types.h as:
typedef struct {
real_T gain;
real_T offset;
} ModelConfigBus;
My goal is to pass the cfg parameter directly into the generated experiment_demo_step() function, so that it can be configured from an external source (e.g., another module or application). Ideally, I would like the struct to be declared in experiment_demo.h and used as an input argument in experiment_demo.c.
Questions:
What is the recommended way to expose a tunable struct parameter like cfg as an input to the step() function in the generated code?
Is there a configuration or customization (e.g., via code interface settings or TLC customization) that allows this struct to be passed explicitly to the step function?
Would moving the struct definition from experiment_demo_types.h to experiment_demo.h be supported or recommended for this use case?
Thank you!Hello MathWorks team,
I have a Simulink model named experiment_demo.slx that includes a MATLAB Function block with the following code:
function adjusted_setpoint = fcn(setpoint, cfg)
adjusted_setpoint = setpoint * cfg.gain * cfg.offset;
setpoint is provided via an Inport.
cfg is a tunable parameter defined as a Simulink.Parameter with a Bus type (ModelConfigBus) and configured via the Model Explorer.
Model configuration:
Solver: Fixed-step
Code Generation:
System target file: ert.tlc
Code interface packaging: Reusable function
Code is generated using Embedded Coder. In the generated code:
The ModelConfigBus struct is defined in experiment_demo_types.h as:
typedef struct {
real_T gain;
real_T offset;
} ModelConfigBus;
My goal is to pass the cfg parameter directly into the generated experiment_demo_step() function, so that it can be configured from an external source (e.g., another module or application). Ideally, I would like the struct to be declared in experiment_demo.h and used as an input argument in experiment_demo.c.
Questions:
What is the recommended way to expose a tunable struct parameter like cfg as an input to the step() function in the generated code?
Is there a configuration or customization (e.g., via code interface settings or TLC customization) that allows this struct to be passed explicitly to the step function?
Would moving the struct definition from experiment_demo_types.h to experiment_demo.h be supported or recommended for this use case?
Thank you! Hello MathWorks team,
I have a Simulink model named experiment_demo.slx that includes a MATLAB Function block with the following code:
function adjusted_setpoint = fcn(setpoint, cfg)
adjusted_setpoint = setpoint * cfg.gain * cfg.offset;
setpoint is provided via an Inport.
cfg is a tunable parameter defined as a Simulink.Parameter with a Bus type (ModelConfigBus) and configured via the Model Explorer.
Model configuration:
Solver: Fixed-step
Code Generation:
System target file: ert.tlc
Code interface packaging: Reusable function
Code is generated using Embedded Coder. In the generated code:
The ModelConfigBus struct is defined in experiment_demo_types.h as:
typedef struct {
real_T gain;
real_T offset;
} ModelConfigBus;
My goal is to pass the cfg parameter directly into the generated experiment_demo_step() function, so that it can be configured from an external source (e.g., another module or application). Ideally, I would like the struct to be declared in experiment_demo.h and used as an input argument in experiment_demo.c.
Questions:
What is the recommended way to expose a tunable struct parameter like cfg as an input to the step() function in the generated code?
Is there a configuration or customization (e.g., via code interface settings or TLC customization) that allows this struct to be passed explicitly to the step function?
Would moving the struct definition from experiment_demo_types.h to experiment_demo.h be supported or recommended for this use case?
Thank you! simulink, embeddedcoder, reusablefunction MATLAB Answers — New Questions

​

Hello,
Does anybody know a (undocumented) workaround to have multiple nodes selected in a uitree with checkboxes? I like to distinguish between checked and selected items.
If the uitree component is used without checkboxes it is possible. I am wondering why this is not implemented (yet)…
hF = uifigure();
hT = uitree(hF,’checkbox’);

hO1 = uitreenode(Parent=hT, Text=’object1′);
hO2 = uitreenode(Parent=hT, Text=’object2′);
hO3 = uitreenode(Parent=hT, Text=’object3′);
hO4 = uitreenode(Parent=hT, Text=’object4′);

hT.CheckedNodes = [hO1, hO2];
hT.SelectedNodes = [hO3, hO4];

%Error using matlab.ui.container.internal.model.TreeComponent/set.SelectedNodes (line 161)
%’SelectedNodes’ must be an empty array or a 1-by-1 TreeNode object that is a child in the CheckBoxTree.
Thanks in advance,
MartinHello,
Does anybody know a (undocumented) workaround to have multiple nodes selected in a uitree with checkboxes? I like to distinguish between checked and selected items.
If the uitree component is used without checkboxes it is possible. I am wondering why this is not implemented (yet)…
hF = uifigure();
hT = uitree(hF,’checkbox’);

hO1 = uitreenode(Parent=hT, Text=’object1′);
hO2 = uitreenode(Parent=hT, Text=’object2′);
hO3 = uitreenode(Parent=hT, Text=’object3′);
hO4 = uitreenode(Parent=hT, Text=’object4′);

hT.CheckedNodes = [hO1, hO2];
hT.SelectedNodes = [hO3, hO4];

%Error using matlab.ui.container.internal.model.TreeComponent/set.SelectedNodes (line 161)
%’SelectedNodes’ must be an empty array or a 1-by-1 TreeNode object that is a child in the CheckBoxTree.
Thanks in advance,
Martin Hello,
Does anybody know a (undocumented) workaround to have multiple nodes selected in a uitree with checkboxes? I like to distinguish between checked and selected items.
If the uitree component is used without checkboxes it is possible. I am wondering why this is not implemented (yet)…
hF = uifigure();
hT = uitree(hF,’checkbox’);

hO1 = uitreenode(Parent=hT, Text=’object1′);
hO2 = uitreenode(Parent=hT, Text=’object2′);
hO3 = uitreenode(Parent=hT, Text=’object3′);
hO4 = uitreenode(Parent=hT, Text=’object4′);

hT.CheckedNodes = [hO1, hO2];
hT.SelectedNodes = [hO3, hO4];

%Error using matlab.ui.container.internal.model.TreeComponent/set.SelectedNodes (line 161)
%’SelectedNodes’ must be an empty array or a 1-by-1 TreeNode object that is a child in the CheckBoxTree.
Thanks in advance,
Martin uifigure, uitree, selectednodes MATLAB Answers — New Questions

​

I used the method described here to save figures with a callback that would change their visibility to ‘on’ when opened.
fig = figure(‘Visible’,’off’);
set(fig, ‘CreateFcn’, ‘set(gcbo,”Visible”,”on”)’);
plot([1:10]);
savefig(fig,"C:temptest.fig")

close all
openfig("C:temptest.fig",’invisible’)
Behavior of sample code: figure opens and is visible
Desired Behavior: figure opens and is invisible.
Is this possible? Thank you.I used the method described here to save figures with a callback that would change their visibility to ‘on’ when opened.
fig = figure(‘Visible’,’off’);
set(fig, ‘CreateFcn’, ‘set(gcbo,”Visible”,”on”)’);
plot([1:10]);
savefig(fig,"C:temptest.fig")

close all
openfig("C:temptest.fig",’invisible’)
Behavior of sample code: figure opens and is visible
Desired Behavior: figure opens and is invisible.
Is this possible? Thank you. I used the method described here to save figures with a callback that would change their visibility to ‘on’ when opened.
fig = figure(‘Visible’,’off’);
set(fig, ‘CreateFcn’, ‘set(gcbo,”Visible”,”on”)’);
plot([1:10]);
savefig(fig,"C:temptest.fig")

close all
openfig("C:temptest.fig",’invisible’)
Behavior of sample code: figure opens and is visible
Desired Behavior: figure opens and is invisible.
Is this possible? Thank you. figure visibility MATLAB Answers — New Questions

​

The Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossingThe Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossing The Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossing enabled subsystem, zero-crossing MATLAB Answers — New Questions

​

So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months.So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months. So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months. simulink, matlab, reinforcement learning, custom td3 MATLAB Answers — New Questions

​

I have trained a ML model in regression learner app using optimizable GPR model using the default setting such as 5 k-fold validation and 30 iterations etc. Now I am traying to do the same using the Matlab script.using the following where X are the resgressors and Y is the response variable.
>> ML_mdl=fitrgp(X,Y,’OptimizeHyperparameters’,’all’,’HyperparameterOptimizationOptions’,struct(‘KFold’,5))
Are the two resulting models more or less equivalent? I know there will be some difference due to the probabilistic nature of the algorithm. When I test it on the entire training set, the R squared value is practically 1.0. Is it overfitting even with K-fold cross-correlation? The prediction on unseen testing set is not that good. Any suggestions?I have trained a ML model in regression learner app using optimizable GPR model using the default setting such as 5 k-fold validation and 30 iterations etc. Now I am traying to do the same using the Matlab script.using the following where X are the resgressors and Y is the response variable.
>> ML_mdl=fitrgp(X,Y,’OptimizeHyperparameters’,’all’,’HyperparameterOptimizationOptions’,struct(‘KFold’,5))
Are the two resulting models more or less equivalent? I know there will be some difference due to the probabilistic nature of the algorithm. When I test it on the entire training set, the R squared value is practically 1.0. Is it overfitting even with K-fold cross-correlation? The prediction on unseen testing set is not that good. Any suggestions? I have trained a ML model in regression learner app using optimizable GPR model using the default setting such as 5 k-fold validation and 30 iterations etc. Now I am traying to do the same using the Matlab script.using the following where X are the resgressors and Y is the response variable.
>> ML_mdl=fitrgp(X,Y,’OptimizeHyperparameters’,’all’,’HyperparameterOptimizationOptions’,struct(‘KFold’,5))
Are the two resulting models more or less equivalent? I know there will be some difference due to the probabilistic nature of the algorithm. When I test it on the entire training set, the R squared value is practically 1.0. Is it overfitting even with K-fold cross-correlation? The prediction on unseen testing set is not that good. Any suggestions? regression learner, script based ml model training MATLAB Answers — New Questions

​

I simulated a microgrid on matlab simulink. I want to run a load flow analysis of the system. However matlab does not recognize the solar PV array in the microgrid as a source. When I place a load flow bus after the inverter of the PV array, it produces an error message saying the model does not converge. Increasing the number of iterations does not help. Any advice would be greatly appreciatedI simulated a microgrid on matlab simulink. I want to run a load flow analysis of the system. However matlab does not recognize the solar PV array in the microgrid as a source. When I place a load flow bus after the inverter of the PV array, it produces an error message saying the model does not converge. Increasing the number of iterations does not help. Any advice would be greatly appreciated I simulated a microgrid on matlab simulink. I want to run a load flow analysis of the system. However matlab does not recognize the solar PV array in the microgrid as a source. When I place a load flow bus after the inverter of the PV array, it produces an error message saying the model does not converge. Increasing the number of iterations does not help. Any advice would be greatly appreciated simulink MATLAB Answers — New Questions

​

I can not add an example of my code below because it is in a closed space, however I’ve been able to isolate the issue.
While using matlab runtime 2023b with python, I am importing my own .ctf file. After about 2000 calls of the eigs() function in matlab I eventually receive a "double free or corruption (out) MATLAB is exiting because of the fatal error." I’ve never received this error when simply just using matlab with the same code base. So my theory is that there’s something wrong with MATLAB Runtime. Has anyone ruin into this? Does anyone have any advice on what I could possibily try next?I can not add an example of my code below because it is in a closed space, however I’ve been able to isolate the issue.
While using matlab runtime 2023b with python, I am importing my own .ctf file. After about 2000 calls of the eigs() function in matlab I eventually receive a "double free or corruption (out) MATLAB is exiting because of the fatal error." I’ve never received this error when simply just using matlab with the same code base. So my theory is that there’s something wrong with MATLAB Runtime. Has anyone ruin into this? Does anyone have any advice on what I could possibily try next? I can not add an example of my code below because it is in a closed space, however I’ve been able to isolate the issue.
While using matlab runtime 2023b with python, I am importing my own .ctf file. After about 2000 calls of the eigs() function in matlab I eventually receive a "double free or corruption (out) MATLAB is exiting because of the fatal error." I’ve never received this error when simply just using matlab with the same code base. So my theory is that there’s something wrong with MATLAB Runtime. Has anyone ruin into this? Does anyone have any advice on what I could possibily try next? matlab runtime, matlab compiler, python MATLAB Answers — New Questions

​

I have these two integrals and i want to find its analytical integral i tried in MATLAB symbolic tool box but i could not.

How to find this integral analytically.I have these two integrals and i want to find its analytical integral i tried in MATLAB symbolic tool box but i could not.

How to find this integral analytically. I have these two integrals and i want to find its analytical integral i tried in MATLAB symbolic tool box but i could not.

How to find this integral analytically. integration, bessel functions MATLAB Answers — New Questions

​