I have two undirected graphs that represent the same connectivity (isomorphic up to node relabeling). When I call cyclebasis on each, I get different sets of cycles. I understand a cycle basis can depend on the spanning tree/edge order, but I want a labeling-invariant notion of “minimal loop units.”
Code
clc; clear; close all

node = [
2 7
2 3
3 4
4 5
5 6
6 1
1 4
1 5
3 6
1 3
5 7
6 7
2 6
5 8];

G = graph(node(:,1), node(:,2), []);
cycles = cyclebasis(G)

figure(1); plot(G,’Layout’,’force’);

%%

node2 = [
1 2
2 3
3 4
4 1
1 5
5 6
2 4
6 7
3 6
2 5
3 7
4 7
2 6
6 8];

G2 = graph(node2(:,1), node2(:,2), []);
cycles2 = cyclebasis(G2)

figure(2); plot(G2,’Layout’,’force’);

Result:
cycles =
7×1 cell array
{[1 3 6 5]}
{[ 1 4 5]}
{[ 1 5 6]}
{[ 2 3 6]}
{[2 6 5 7]}
{[3 4 5 6]}
{[ 5 6 7]}

cycles2 =
7×1 cell array
{[ 1 2 4]}
{[ 1 2 5]}
{[ 2 3 4]}
{[ 2 3 6]}
{[2 3 7 6]}
{[2 4 7 6]}
{[ 2 5 6]}

Questions:
I know cyclebasis can vary with spanning tree/edge ordering. What’s the recommended way in MATLAB to obtain “minimal loop units” that do not depend on node labeling or edge input order? For example, in the above case, each cycle is a 3-node triangle, and there should be seven such cycles.I have two undirected graphs that represent the same connectivity (isomorphic up to node relabeling). When I call cyclebasis on each, I get different sets of cycles. I understand a cycle basis can depend on the spanning tree/edge order, but I want a labeling-invariant notion of “minimal loop units.”
Code
clc; clear; close all

node = [
2 7
2 3
3 4
4 5
5 6
6 1
1 4
1 5
3 6
1 3
5 7
6 7
2 6
5 8];

G = graph(node(:,1), node(:,2), []);
cycles = cyclebasis(G)

figure(1); plot(G,’Layout’,’force’);

%%

node2 = [
1 2
2 3
3 4
4 1
1 5
5 6
2 4
6 7
3 6
2 5
3 7
4 7
2 6
6 8];

G2 = graph(node2(:,1), node2(:,2), []);
cycles2 = cyclebasis(G2)

figure(2); plot(G2,’Layout’,’force’);

Result:
cycles =
7×1 cell array
{[1 3 6 5]}
{[ 1 4 5]}
{[ 1 5 6]}
{[ 2 3 6]}
{[2 6 5 7]}
{[3 4 5 6]}
{[ 5 6 7]}

cycles2 =
7×1 cell array
{[ 1 2 4]}
{[ 1 2 5]}
{[ 2 3 4]}
{[ 2 3 6]}
{[2 3 7 6]}
{[2 4 7 6]}
{[ 2 5 6]}

Questions:
I know cyclebasis can vary with spanning tree/edge ordering. What’s the recommended way in MATLAB to obtain “minimal loop units” that do not depend on node labeling or edge input order? For example, in the above case, each cycle is a 3-node triangle, and there should be seven such cycles. I have two undirected graphs that represent the same connectivity (isomorphic up to node relabeling). When I call cyclebasis on each, I get different sets of cycles. I understand a cycle basis can depend on the spanning tree/edge order, but I want a labeling-invariant notion of “minimal loop units.”
Code
clc; clear; close all

node = [
2 7
2 3
3 4
4 5
5 6
6 1
1 4
1 5
3 6
1 3
5 7
6 7
2 6
5 8];

G = graph(node(:,1), node(:,2), []);
cycles = cyclebasis(G)

figure(1); plot(G,’Layout’,’force’);

%%

node2 = [
1 2
2 3
3 4
4 1
1 5
5 6
2 4
6 7
3 6
2 5
3 7
4 7
2 6
6 8];

G2 = graph(node2(:,1), node2(:,2), []);
cycles2 = cyclebasis(G2)

figure(2); plot(G2,’Layout’,’force’);

Result:
cycles =
7×1 cell array
{[1 3 6 5]}
{[ 1 4 5]}
{[ 1 5 6]}
{[ 2 3 6]}
{[2 6 5 7]}
{[3 4 5 6]}
{[ 5 6 7]}

cycles2 =
7×1 cell array
{[ 1 2 4]}
{[ 1 2 5]}
{[ 2 3 4]}
{[ 2 3 6]}
{[2 3 7 6]}
{[2 4 7 6]}
{[ 2 5 6]}

Questions:
I know cyclebasis can vary with spanning tree/edge ordering. What’s the recommended way in MATLAB to obtain “minimal loop units” that do not depend on node labeling or edge input order? For example, in the above case, each cycle is a 3-node triangle, and there should be seven such cycles. graph, nodes, isomorphic, graph theory MATLAB Answers — New Questions

​

Hello, everyone,
I am working with spreadsheet tables, which I import by readtable method. I have several numeric columns, but my crucial point is the column with strings. The readtable function trim every string (removes leading and trailing whitespace). However, I need to avoid this trimming and leave the strings as they are. I tried to look into documentation, but found only the trimming related to tables originating from text files.
My current code related to this looks following:
mytable=readtable(filename,’Sheet’,sheetname,’NumHeaderLines’,0)

Do you have any suggestions?Hello, everyone,
I am working with spreadsheet tables, which I import by readtable method. I have several numeric columns, but my crucial point is the column with strings. The readtable function trim every string (removes leading and trailing whitespace). However, I need to avoid this trimming and leave the strings as they are. I tried to look into documentation, but found only the trimming related to tables originating from text files.
My current code related to this looks following:
mytable=readtable(filename,’Sheet’,sheetname,’NumHeaderLines’,0)

Do you have any suggestions? Hello, everyone,
I am working with spreadsheet tables, which I import by readtable method. I have several numeric columns, but my crucial point is the column with strings. The readtable function trim every string (removes leading and trailing whitespace). However, I need to avoid this trimming and leave the strings as they are. I tried to look into documentation, but found only the trimming related to tables originating from text files.
My current code related to this looks following:
mytable=readtable(filename,’Sheet’,sheetname,’NumHeaderLines’,0)

Do you have any suggestions? readtable, strtrim MATLAB Answers — New Questions

​

The Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossingThe Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossing The Simulink model shown in the screenshot does not behave as I expect, and I would like to understand the reason.

All blocks are on default settings, except:
Gain: 1/pi
Compare To Constant: operator >=, constant value 1.0
Logical Operator: XOR
(Scope: two inputs, line markers)
Note that the Compare To Constant block has Zero-crossing (ZC) detection enabled by default. The Enabled Subsystem consists of one input port directly connected to the output port.
All solver settings are at default values. The solver is auto(VariableStepDiscrete). I only change the StopTime of the simulation (see below).
Expected behavior
The enabled subsystem should be enabled for exactly one time point (in major time step) and "sample and hold" the value of its input (the scaled clock signal).
This should occur when the scaled clock signal reaches the threshold value of 1.0 (but not later than that, thanks to zero-crossing detection).
The sampled and held value (see Display block) should be 1.0 (sampled at time t=pi).
Observed behavior
Sometimes the result (= sampled and held value at the end of the simulation) is larger than expected. The scope screenshot (lower half) shows that the enabled subsystem seems to be enabled for two time points (instead of one). In the first time point (at t=pi) it copies a value of 1.0 to its output, in the subsequent second time point a value of approx. 1.025, which is then held until the end of the simulation.
Whether the problem occurs, weirdly depends on the StopTime. Some examples:
For these values of StopTime the problem occurs: 4.00001, 8, 9, 10.1, 11, 13.
For these values everything works fine: 4, 6, 7, 10, 12.

My analysis so far
When the problem occurs, Simulink has placed two time points at the zero-crossing, one just before and one just after the zero-crossing. When the problem doesn’t occur, there are three time points close to the zero-crossing (one just before, two just after the ZC).
Although the XOR block output is correctly true only for one time point (right hand side of the zero-crossing, see upper half of the scope screenshot), the final output of the enabled subsystem seems to always be equal to the value of its input sampled one time point later (that is: when the XOR has dropped to false again). That is not a problem if that time point is the third of 3 close to the ZC, but it produces a wrong result if that time point is much later than the ZC.
So I wonder why Simulink sometimes uses 3 time points at a ZC (good), but sometimes only 2 (bad). Any hints or explanations welcome why Simulink behaves like this and/or why it should(n’t).
Some more notes
I know the expected behavior could be implemented differently. That’s not the point. This is a minimal example. In my opinion it should not behave the way it does.
I’m unsure about the correct/official wording: What I mean by "time point" are the elements of "tout". Usually I refer to them as major time steps, but maybe that’s wrong as they’re points in time, not steps. (?)
My maybe related (and unfortunately still unanswered) question about zero-crossings and the number of time steps/points taken by Simulink: https://de.mathworks.com/matlabcentral/answers/553960-number-of-necessary-time-steps-to-handle-a-zero-crossing enabled subsystem, zero-crossing MATLAB Answers — New Questions

​

So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months.So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months. So I have tried to code up my custom TD3 agent to behave as much like the built in TD3 agent in the same simulink environment, the only difference between them is for the custom agent, I had to use a rate transition block to perform zero order hold between the states, rewards, done signal and the custom agent. I used the rate transition block specify mode for output port sample time options to set the custom agent sample time.

My code for my custom TD3 agent is below, I tried to make it as much like the built-in TD3 as possible, the ep_counter,num_of_ep properties are unused.
classdef test_TD3Agent_V2 < rl.agent.CustomAgent
properties
%neural networks
actor
critic1
critic2
%target networks
target_actor
target_critic1
target_critic2
%dimensions
statesize
actionsize
%optimizers
actor_optimizer
critic1_optimizer
critic2_optimizer
%buffer
statebuffer
nextstatebuffer
actionbuffer
rewardbuffer
donebuffer
counter %keeps count of number experiences encountered
index %keeps track of current available index in buffer
buffersize
batchsize
%episodes
num_of_ep
ep_counter
%keep count of critic number of updates
num_critic_update
end

methods
%constructor
function obj = test_TD3Agent_V2(actor,critic1,critic2,target_actor,target_critic1,target_critic2,actor_opt,critic1_opt,critic2_opt,statesize,actionsize,buffer_size,batchsize,num_of_ep)
%(required) call abstract class constructor
obj = obj@rl.agent.CustomAgent();
%define observation + action space
obj.ObservationInfo = rlNumericSpec([statesize 1]);
obj.ActionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
obj.SampleTime = -1; %determined by rate transition block
%define networks
obj.actor = actor;
obj.critic1 = critic1;
obj.critic2 = critic2;
%define target networks
obj.target_actor = target_actor;
obj.target_critic1 = target_critic1;
obj.target_critic2 = target_critic2;
%define optimizer
obj.actor_optimizer = actor_opt;
obj.critic1_optimizer = critic1_opt;
obj.critic2_optimizer = critic2_opt;
%record dimensions
obj.statesize = statesize;
obj.actionsize = actionsize;
%initialize buffer
obj.statebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.nextstatebuffer = dlarray(zeros(statesize,1,buffer_size));
obj.actionbuffer = dlarray(zeros(actionsize,1,buffer_size));
obj.rewardbuffer = dlarray(zeros(1,buffer_size));
obj.donebuffer = zeros(1,buffer_size);
obj.buffersize = buffer_size;
obj.batchsize = batchsize;
obj.counter = 0;
obj.index = 1;
%episodes (unused)
obj.num_of_ep = num_of_ep;
obj.ep_counter = 1;
%used for delay actor update and target network soft transfer
obj.num_critic_update = 0;
end
end

methods (Access = protected)
%Action method
function action = getActionImpl(obj,Observation)
% Given the current state of the system, return an action
action = getAction(obj.actor,Observation);
end

%Action with noise method
function action = getActionWithExplorationImpl(obj,Observation)
% Given the current observation, select an action
action = getAction(obj.actor,Observation);
% Add random noise to action
end

%Learn method
function action = learnImpl(obj,Experience)
%parse experience
state = Experience{1};
action_ = Experience{2};
reward = Experience{3};
next_state = Experience{4};
isdone = Experience{5};

%buffer operations
%check if index wraps around
if (obj.index > obj.buffersize)
obj.index = 1;
end
%record experience in buffer
obj.statebuffer(:,:,obj.index) = state{1};
obj.actionbuffer(:,:,obj.index) = action_{1};
obj.rewardbuffer(:,obj.index) = reward;
obj.nextstatebuffer(:,:,obj.index) = next_state{1};
obj.donebuffer(:,obj.index) = isdone;

%increment index and counter
obj.counter = obj.counter + 1;
obj.index = obj.index + 1;

%if non terminal state
if (isdone == false)
action = getAction(obj.actor,next_state); %select next action
noise = randn([6,1]).*0.1; %gaussian noise with standard dev of 0.1
action{1} = action{1} + noise; %add noise
action{1} = clip(action{1},-1,1); %clip action noise
else
%learning at the end of episode
if (obj.counter >= obj.batchsize)
max_index = min([obj.counter obj.buffersize]); %range of index 1 to max_index for buffer
%sample experience randomly from buffer
sample_index_vector = randsample(max_index,obj.batchsize); %vector of index experience to sample

%create buffer mini batch dlarrays
state_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
nextstate_batch = dlarray(zeros(obj.statesize,1,obj.batchsize));
action_batch = dlarray(zeros(obj.actionsize,1,obj.batchsize));
reward_batch = dlarray(zeros(1,obj.batchsize));
done_batch = zeros(1,obj.batchsize);

for i = 1:obj.batchsize %iterate through buffer and transfer experience over to mini batch
state_batch(:,:,i) = obj.statebuffer(:,:,sample_index_vector(i));
nextstate_batch(:,:,i) = obj.nextstatebuffer(:,:,sample_index_vector(i));
action_batch(:,:,i) = obj.actionbuffer(:,:,sample_index_vector(i));
reward_batch(:,i) = obj.rewardbuffer(:,sample_index_vector(i));
done_batch(:,i) = obj.donebuffer(:,sample_index_vector(i));
end

%update critic networks
criticgrad1 = dlfeval(@critic_gradient,obj.critic1,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic1,obj.critic1_optimizer] = update(obj.critic1_optimizer,obj.critic1,criticgrad1);
criticgrad2 = dlfeval(@critic_gradient,obj.critic2,obj.target_actor,obj.target_critic1,obj.target_critic2,{state_batch},{nextstate_batch},{action_batch},reward_batch,done_batch,obj.batchsize);
[obj.critic2,obj.critic2_optimizer] = update(obj.critic2_optimizer,obj.critic2,criticgrad2);
%update num of critic updates
obj.num_critic_update = obj.num_critic_update + 1;

%delayed actor update + target network transfer
if (mod(obj.num_critic_update,2) == 0)
actorgrad = dlfeval(@actor_gradient,obj.actor,obj.critic1,obj.critic2,{state_batch});
[obj.actor,obj.actor_optimizer] = update(obj.actor_optimizer,obj.actor,actorgrad);
target_soft_transfer(obj);
end
end

end
end

%function used to soft transfer over to target networks
function target_soft_transfer(obj)
smooth_factor = 0.005;

for i = 1:6
obj.target_actor.Learnables{i} = smooth_factor*obj.actor.Learnables{i} + (1 – smooth_factor)*obj.target_actor.Learnables{i};
obj.target_critic1.Learnables{i} = smooth_factor*obj.critic1.Learnables{i} + (1 – smooth_factor)*obj.target_critic1.Learnables{i};
obj.target_critic2.Learnables{i} = smooth_factor*obj.critic2.Learnables{i} + (1 – smooth_factor)*obj.target_critic2.Learnables{i};
end
end

end
end

%obtain gradient of Q value wrt actor
function actorgradient = actor_gradient(actorNet,critic1,critic2,states,batchsize)
actoraction = getAction(actorNet,states); %obtain actor action
%obtain Q values
Q1 = getValue(critic1,states,actoraction);
Q2 = getValue(critic2,states,actoraction);
%obtain min Q values + reverse sign for gradient ascent
Qmin = min(Q1,Q2);
Q = -1*mean(Qmin);
gradient = dlgradient(Q,actorNet.Learnables); %calculate gradient of Q value wrt NN learnables

actorgradient = gradient;
end

%obtain gradient of critic NN
function criticgradient = critic_gradient(critic,target_actor,target_critic_1,target_critic_2,states,nextstates,actions,rewards,dones,batchsize)
%obtain target action
target_actions = getAction(target_actor,nextstates);
%target policy smoothing
for i = 1:batchsize
target_noise = randn([6,1]).*sqrt(0.2);
target_noise = clip(target_noise,-0.5,0.5);
target_actions{1}(:,:,i) = target_actions{1}(:,:,i) + target_noise; %add noise to action for smoothing
end
target_actions{1}(:,:,:) = clip(target_actions{1}(:,:,:),-1,1); %clip btw -1 and 1
%obtain Q values
Qtarget1 = getValue(target_critic_1,nextstates,target_actions);
Qtarget2 = getValue(target_critic_2,nextstates,target_actions);
Qmin = min(Qtarget1,Qtarget2);
Qoptimal = rewards + 0.99*Qmin.*(1 – dones);
Qpred = getValue(critic,states,actions);
%obtain critic loss
criticLoss = 0.5*mean((Qoptimal – Qpred).^2);

criticgradient = dlgradient(criticLoss,critic.Learnables);
end
And here is my code when using the built in TD3 agent
clc
%define times
dt = 0.1; %time steps
Tf = 7; %simulation time
%create stateInfo and actionInfo objects
statesize = 38;
actionsize = 6;
stateInfo = rlNumericSpec([statesize 1]);
actionInfo = rlNumericSpec([actionsize 1],LowerLimit = -1,UpperLimit = 1);
mdl = ‘KUKA_EE_Controller_v18_disturbed’;
blk = ‘KUKA_EE_Controller_v18_disturbed/RL Agent’;
%create environment object
env = rlSimulinkEnv(mdl,blk,stateInfo,actionInfo);
%assign reset function
env.ResetFcn = @ResetFunction;
% %create actor network
actorlayers = [
featureInputLayer(statesize)
fullyConnectedLayer(800)
reluLayer
fullyConnectedLayer(600)
reluLayer
fullyConnectedLayer(actionsize)
tanhLayer
];
actorNet = dlnetwork;
actorNet = addLayers(actorNet, actorlayers);
actorNet = initialize(actorNet);
actor = rlContinuousDeterministicActor(actorNet, stateInfo, actionInfo);
%create critic networks
statelayers = [
featureInputLayer(statesize, Name=’states’)
concatenationLayer(1, 2, Name=’concat’)
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(400)
reluLayer
fullyConnectedLayer(1, Name=’Qvalue’)
];
actionlayers = featureInputLayer(actionsize, Name=’actions’);
criticNet = dlnetwork;
criticNet = addLayers(criticNet, statelayers);
criticNet = addLayers(criticNet, actionlayers);
criticNet = connectLayers(criticNet, ‘actions’, ‘concat/in2’);
criticNet = initialize(criticNet);
critic1 = rlQValueFunction(criticNet,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
criticNet2 = dlnetwork;
criticNet2 = addLayers(criticNet2, statelayers);
criticNet2 = addLayers(criticNet2, actionlayers);
criticNet2 = connectLayers(criticNet2, ‘actions’, ‘concat/in2’);
criticNet2 = initialize(criticNet2);
critic2 = rlQValueFunction(criticNet2,stateInfo,actionInfo,ObservationInputNames=’states’,ActionInputNames=’actions’);
%create options object for actor and critic
actoroptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.001);
criticoptions = rlOptimizerOptions(Optimizer=’adam’,LearnRate=0.003);
agentoptions = rlTD3AgentOptions;
agentoptions.SampleTime = dt;
agentoptions.ActorOptimizerOptions = actoroptions;
agentoptions.CriticOptimizerOptions = criticoptions;
agentoptions.DiscountFactor = 0.99;
agentoptions.TargetSmoothFactor = 0.005;
agentoptions.ExperienceBufferLength = 1000000;
agentoptions.MiniBatchSize = 250;
agentoptions.ExplorationModel.StandardDeviation = 0.1;
agentoptions.ExplorationModel.StandardDeviationDecayRate = 1e-4;
agent = rlTD3Agent(actor, [critic1 critic2], agentoptions);
%create training options object
trainOpts = rlTrainingOptions(MaxEpisodes=20,MaxStepsPerEpisode=floor(Tf/dt),StopTrainingCriteria=’none’,SimulationStorageType=’none’);
%train agent
trainresults = train(agent,env,trainOpts);
I made my custom TD3 agent with the same actor and critic structures, with the same hyperparameters, and with the same agent options. But it doesn’t seem to learn and I don’t know why. I don’t know if the rate transition block is having a negative impact on the training. One difference between my custom TD3 and the built-in TD3 is the actor gradient. In the matlab documentation on TD3 agent, it says the gradient is calculated for every sample in the mini batch then the gradient is accumulated and averaged out.
https://www.mathworks.com/help/reinforcement-learning/ug/td3-agents.html (TD3 documentation)

But how I calculated my actor gradient in my above code in the actorgradient function, I averaged the Q values over the minbatch first, then I performed only one gradient operation. So maybe that’s one possible reason why my built in TD3 agent isn’t learning. Here are my reward for
Built-TD3

Custom TD3, I stopped it early because it wasn’t learning

I would appreciate any help because I have been stuck for months. simulink, matlab, reinforcement learning, custom td3 MATLAB Answers — New Questions

​

I have opted for Differential Equations Toolbox, Math Symbolic Toolbox, Optimization and Global Optimization toolbox. Other than these I have no idea what I need and what I don’t. Can anyone from theoretical background help me pick the products.I have opted for Differential Equations Toolbox, Math Symbolic Toolbox, Optimization and Global Optimization toolbox. Other than these I have no idea what I need and what I don’t. Can anyone from theoretical background help me pick the products. I have opted for Differential Equations Toolbox, Math Symbolic Toolbox, Optimization and Global Optimization toolbox. Other than these I have no idea what I need and what I don’t. Can anyone from theoretical background help me pick the products. theoretical physics, newtoproduct MATLAB Answers — New Questions

​

I am using a machine with an intel processor with 14 physical cores (among those, 6 performance cores). If I run this code I get an error message.
numWorkers = 8;
p = parpool(numWorkers);

n = 10;
parfor i=1:n
disp(i)
end
The error message is the following:
Error using parpool (line 132)
Too many workers requested. The profile "Threads" has the NumWorkers property set to a
maximum of 6 workers but 8 workers were requested. Either request a number of workers
less than NumWorkers, or increase the value of the NumWorkers property for the profile
(up to a maximum of 512 for thread-based pools).

I don’t understand the error message. Does this mean that the maximum number of cores I can use on my machine is 6, even though it has 14 physical cores? Matlab cannot use cores that are not performance cores?

Thanks in advance for any feedback on this.I am using a machine with an intel processor with 14 physical cores (among those, 6 performance cores). If I run this code I get an error message.
numWorkers = 8;
p = parpool(numWorkers);

n = 10;
parfor i=1:n
disp(i)
end
The error message is the following:
Error using parpool (line 132)
Too many workers requested. The profile "Threads" has the NumWorkers property set to a
maximum of 6 workers but 8 workers were requested. Either request a number of workers
less than NumWorkers, or increase the value of the NumWorkers property for the profile
(up to a maximum of 512 for thread-based pools).

I don’t understand the error message. Does this mean that the maximum number of cores I can use on my machine is 6, even though it has 14 physical cores? Matlab cannot use cores that are not performance cores?

Thanks in advance for any feedback on this. I am using a machine with an intel processor with 14 physical cores (among those, 6 performance cores). If I run this code I get an error message.
numWorkers = 8;
p = parpool(numWorkers);

n = 10;
parfor i=1:n
disp(i)
end
The error message is the following:
Error using parpool (line 132)
Too many workers requested. The profile "Threads" has the NumWorkers property set to a
maximum of 6 workers but 8 workers were requested. Either request a number of workers
less than NumWorkers, or increase the value of the NumWorkers property for the profile
(up to a maximum of 512 for thread-based pools).

I don’t understand the error message. Does this mean that the maximum number of cores I can use on my machine is 6, even though it has 14 physical cores? Matlab cannot use cores that are not performance cores?

Thanks in advance for any feedback on this. parallel computing toolbox, threads, parfor MATLAB Answers — New Questions

​

Hi, all
I have a structural engineering book dated back at 1979 that presents a fairly odd equation to compute rotations at member joints.
This is:

Where C, V , h and K are constant properties. Theta is the joint rotation at the level considered, the level above and the level below (hence the subscript).
The book mentions a several ways of solving it. One being iteration procedures. It also refers to this equation as a difference equation, which I have no experience with at all. Although I have read that they are fairly similar to differential equations which i have some experience with.
My question is, is there a function or command in MATLAB that could be employed to solve this equation? Like there is for differential equations symbolically and numerically. And could somebody help me with this?

Many thanks,
ScottHi, all
I have a structural engineering book dated back at 1979 that presents a fairly odd equation to compute rotations at member joints.
This is:

Where C, V , h and K are constant properties. Theta is the joint rotation at the level considered, the level above and the level below (hence the subscript).
The book mentions a several ways of solving it. One being iteration procedures. It also refers to this equation as a difference equation, which I have no experience with at all. Although I have read that they are fairly similar to differential equations which i have some experience with.
My question is, is there a function or command in MATLAB that could be employed to solve this equation? Like there is for differential equations symbolically and numerically. And could somebody help me with this?

Many thanks,
Scott Hi, all
I have a structural engineering book dated back at 1979 that presents a fairly odd equation to compute rotations at member joints.
This is:

Where C, V , h and K are constant properties. Theta is the joint rotation at the level considered, the level above and the level below (hence the subscript).
The book mentions a several ways of solving it. One being iteration procedures. It also refers to this equation as a difference equation, which I have no experience with at all. Although I have read that they are fairly similar to differential equations which i have some experience with.
My question is, is there a function or command in MATLAB that could be employed to solve this equation? Like there is for differential equations symbolically and numerically. And could somebody help me with this?

Many thanks,
Scott difference equations, functions, commands MATLAB Answers — New Questions

​

The doc page Create Function Handle states:
"Keep the following in mind when creating handles to functions:
Scope — The function must be in scope at the time you create the handle. Therefore, the function must be on the MATLAB path or in the current folder."
Trying to verify that statement ….
Verify a proposed function name is not currently in scope
exist xyzzy
But we can create a handle to this non-existent function (seemingly contra to the doc page)
f = @xyzzy;
Error is generated if trying to use the handle, of course.
try
f()
catch ME
ME.message
end
After generating the handle we can create the function (typically through the editor, or renaming an existing file, or ….)
str = ["function a = xyzzy";"a=1;"];
writelines(str,"xyzzy.m");
type xyzzy.m
And now the function handle executes
f()
Am I misunderstanding something about that doc page, or is the the doc page wrong, or does the doc page reflect how Matlab is supposed to work and the preceding code indicates a bug of some sort?The doc page Create Function Handle states:
"Keep the following in mind when creating handles to functions:
Scope — The function must be in scope at the time you create the handle. Therefore, the function must be on the MATLAB path or in the current folder."
Trying to verify that statement ….
Verify a proposed function name is not currently in scope
exist xyzzy
But we can create a handle to this non-existent function (seemingly contra to the doc page)
f = @xyzzy;
Error is generated if trying to use the handle, of course.
try
f()
catch ME
ME.message
end
After generating the handle we can create the function (typically through the editor, or renaming an existing file, or ….)
str = ["function a = xyzzy";"a=1;"];
writelines(str,"xyzzy.m");
type xyzzy.m
And now the function handle executes
f()
Am I misunderstanding something about that doc page, or is the the doc page wrong, or does the doc page reflect how Matlab is supposed to work and the preceding code indicates a bug of some sort? The doc page Create Function Handle states:
"Keep the following in mind when creating handles to functions:
Scope — The function must be in scope at the time you create the handle. Therefore, the function must be on the MATLAB path or in the current folder."
Trying to verify that statement ….
Verify a proposed function name is not currently in scope
exist xyzzy
But we can create a handle to this non-existent function (seemingly contra to the doc page)
f = @xyzzy;
Error is generated if trying to use the handle, of course.
try
f()
catch ME
ME.message
end
After generating the handle we can create the function (typically through the editor, or renaming an existing file, or ….)
str = ["function a = xyzzy";"a=1;"];
writelines(str,"xyzzy.m");
type xyzzy.m
And now the function handle executes
f()
Am I misunderstanding something about that doc page, or is the the doc page wrong, or does the doc page reflect how Matlab is supposed to work and the preceding code indicates a bug of some sort? function handle MATLAB Answers — New Questions

​

With the change to deployment workflow in R2025a (Release Notes), I understand the Compiler apps have been updated to integrate with MATLAB® projects, represented by .prj files. The .prj files no longer represent deployment project like in previous releases.
How can I use my old .prj file from previous releases to compile code in R2025a?With the change to deployment workflow in R2025a (Release Notes), I understand the Compiler apps have been updated to integrate with MATLAB® projects, represented by .prj files. The .prj files no longer represent deployment project like in previous releases.
How can I use my old .prj file from previous releases to compile code in R2025a? With the change to deployment workflow in R2025a (Release Notes), I understand the Compiler apps have been updated to integrate with MATLAB® projects, represented by .prj files. The .prj files no longer represent deployment project like in previous releases.
How can I use my old .prj file from previous releases to compile code in R2025a?  MATLAB Answers — New Questions

​