Tracing a loss function to two outputs of the network in order to train a PINN model
Hello everybody,
I’m trying to recreate a machine learning code from this paper in matlab. In python it is somewhat straightforward, but MATLAB does not seem to have the functionality (that I have found) to perform the command:
torch.optim.Adam(list(Net_u.parameters())+list(Net_v.parameters()), lr=learning_rate)
that is, to update the learning parameters for two networks simultaneously. I’ve done some test with the dlgradient function and as I come to understand it, it is only capable of tracing the parameters regarding one function, e.g.;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY)
du_x = dlgradient(sum(U(1,:),"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(V(1,:),"all"),XY);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Will give completely different gradients then;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Having both higher derivatives traced on, will produce results identical to the first piece of code;
e.g;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
switched the order of dlgradient calls will produce results identical to the second piece of code;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
I only provide short snippets of code as the whole script is a little more then 300 lines, and not I believe is relevant.
At any rate, neither of those two options is able to solve the PINN problem in the example (gradient descent does not converge on the solution).
Has someone experienced and/or played around with these kinds of problems? Any insight would be greatly appreciated.Hello everybody,
I’m trying to recreate a machine learning code from this paper in matlab. In python it is somewhat straightforward, but MATLAB does not seem to have the functionality (that I have found) to perform the command:
torch.optim.Adam(list(Net_u.parameters())+list(Net_v.parameters()), lr=learning_rate)
that is, to update the learning parameters for two networks simultaneously. I’ve done some test with the dlgradient function and as I come to understand it, it is only capable of tracing the parameters regarding one function, e.g.;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY)
du_x = dlgradient(sum(U(1,:),"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(V(1,:),"all"),XY);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Will give completely different gradients then;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Having both higher derivatives traced on, will produce results identical to the first piece of code;
e.g;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
switched the order of dlgradient calls will produce results identical to the second piece of code;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
I only provide short snippets of code as the whole script is a little more then 300 lines, and not I believe is relevant.
At any rate, neither of those two options is able to solve the PINN problem in the example (gradient descent does not converge on the solution).
Has someone experienced and/or played around with these kinds of problems? Any insight would be greatly appreciated. Hello everybody,
I’m trying to recreate a machine learning code from this paper in matlab. In python it is somewhat straightforward, but MATLAB does not seem to have the functionality (that I have found) to perform the command:
torch.optim.Adam(list(Net_u.parameters())+list(Net_v.parameters()), lr=learning_rate)
that is, to update the learning parameters for two networks simultaneously. I’ve done some test with the dlgradient function and as I come to understand it, it is only capable of tracing the parameters regarding one function, e.g.;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY)
du_x = dlgradient(sum(U(1,:),"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(V(1,:),"all"),XY);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Will give completely different gradients then;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
Having both higher derivatives traced on, will produce results identical to the first piece of code;
e.g;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
switched the order of dlgradient calls will produce results identical to the second piece of code;
[U,~] = forward(net_u,XY);
[V,~] = forward(net_v,XY);
du_y = dlgradient(sum(uy,"all"),XY,"EnableHigherDerivatives",true);
du_x = dlgradient(sum(ux,"all"),XY,"EnableHigherDerivatives",true);
…. calculations of of loss function, exempted from code for brevity…
gradients_u = dlgradient(loss,net_u.Learnables);
gradients_v = dlgradient(loss,net_v.Learnables);
I only provide short snippets of code as the whole script is a little more then 300 lines, and not I believe is relevant.
At any rate, neither of those two options is able to solve the PINN problem in the example (gradient descent does not converge on the solution).
Has someone experienced and/or played around with these kinds of problems? Any insight would be greatly appreciated. machine learning, dlgradient, custom loss function MATLAB Answers — New Questions
