Tag Archives: matlab
Adjusting color limits on displayed color images
When displaying a color image is it possible to set the limits of the cData. The below dcoumentation states that if a double is given then the limits range from 0 to 1 with different values for int8 etc.
Is it possible to change what these limits are, ideally with different values for each channel. Currently my solution is to scale the origional data before senging it to image(), but this is slow to do each time the user wants to adjust these scale values to highlight different parts of the data. It also prevents tools like impixelinfo from giving correct values. Also if there are any other features that are plotted such as boxes then these also need to be redone. Zoom ranges and everything else also need to be preserved. This all becomes a big hassle and affects performance if the user is regulary changing the intensity scales.
How are these limits set? CLim seems to have no effect, neither do the colormaps.
Is it possible to set the black and white values of a displayed image through some undocumented feature?
Cheers Alaster
3-D array of RGB triplets — This format defines true color image data using RGB triplet values. Each RGB triplet defines a color for one pixel of the image. An RGB triplet is a three-element vector that specifies the intensities of the red, green, and blue components of the color. The first page of the 3-D array contains the red components, the second page contains the green components, and the third page contains the blue components. Since the image uses true colors instead of colormap colors, the CDataMapping property has no effect.
If CData is of type double, then an RGB triplet value of [0 0 0] corresponds to black and [1 1 1] corresponds to white.
If CData is an integer type, then the image uses the full range of data to determine the color. For example, if CData is of type uint8, then [0 0 0] corresponds to black and [255 255 255] corresponds to white. If CData is of type int8, then [-128 -128 -128] corresponds to black and [127 127 127] corresponds to white.
If CData is of type logical, then [0 0 0] corresponds to black and [1 1 1] corresponds to white.When displaying a color image is it possible to set the limits of the cData. The below dcoumentation states that if a double is given then the limits range from 0 to 1 with different values for int8 etc.
Is it possible to change what these limits are, ideally with different values for each channel. Currently my solution is to scale the origional data before senging it to image(), but this is slow to do each time the user wants to adjust these scale values to highlight different parts of the data. It also prevents tools like impixelinfo from giving correct values. Also if there are any other features that are plotted such as boxes then these also need to be redone. Zoom ranges and everything else also need to be preserved. This all becomes a big hassle and affects performance if the user is regulary changing the intensity scales.
How are these limits set? CLim seems to have no effect, neither do the colormaps.
Is it possible to set the black and white values of a displayed image through some undocumented feature?
Cheers Alaster
3-D array of RGB triplets — This format defines true color image data using RGB triplet values. Each RGB triplet defines a color for one pixel of the image. An RGB triplet is a three-element vector that specifies the intensities of the red, green, and blue components of the color. The first page of the 3-D array contains the red components, the second page contains the green components, and the third page contains the blue components. Since the image uses true colors instead of colormap colors, the CDataMapping property has no effect.
If CData is of type double, then an RGB triplet value of [0 0 0] corresponds to black and [1 1 1] corresponds to white.
If CData is an integer type, then the image uses the full range of data to determine the color. For example, if CData is of type uint8, then [0 0 0] corresponds to black and [255 255 255] corresponds to white. If CData is of type int8, then [-128 -128 -128] corresponds to black and [127 127 127] corresponds to white.
If CData is of type logical, then [0 0 0] corresponds to black and [1 1 1] corresponds to white. When displaying a color image is it possible to set the limits of the cData. The below dcoumentation states that if a double is given then the limits range from 0 to 1 with different values for int8 etc.
Is it possible to change what these limits are, ideally with different values for each channel. Currently my solution is to scale the origional data before senging it to image(), but this is slow to do each time the user wants to adjust these scale values to highlight different parts of the data. It also prevents tools like impixelinfo from giving correct values. Also if there are any other features that are plotted such as boxes then these also need to be redone. Zoom ranges and everything else also need to be preserved. This all becomes a big hassle and affects performance if the user is regulary changing the intensity scales.
How are these limits set? CLim seems to have no effect, neither do the colormaps.
Is it possible to set the black and white values of a displayed image through some undocumented feature?
Cheers Alaster
3-D array of RGB triplets — This format defines true color image data using RGB triplet values. Each RGB triplet defines a color for one pixel of the image. An RGB triplet is a three-element vector that specifies the intensities of the red, green, and blue components of the color. The first page of the 3-D array contains the red components, the second page contains the green components, and the third page contains the blue components. Since the image uses true colors instead of colormap colors, the CDataMapping property has no effect.
If CData is of type double, then an RGB triplet value of [0 0 0] corresponds to black and [1 1 1] corresponds to white.
If CData is an integer type, then the image uses the full range of data to determine the color. For example, if CData is of type uint8, then [0 0 0] corresponds to black and [255 255 255] corresponds to white. If CData is of type int8, then [-128 -128 -128] corresponds to black and [127 127 127] corresponds to white.
If CData is of type logical, then [0 0 0] corresponds to black and [1 1 1] corresponds to white. imagesc, clim, cdata, colormap, impixelinfo, image MATLAB Answers — New Questions
Error in displaying the whole matrix
Hello,
When I try to display the whole matrix I receive the data in the following shape:
-0.0008 0.0001 0.0000
-0.0005 0.0001 0.0000
-0.0004 0.0001 0.0000
-0.0002 0.0001 0.0000
-0.0000 0.0001 0.0000
However when I attempt to display the portion of matrix teh result is correct:
-823 8 1
-584 8 1
-409 8 1
-304.928530000000 8 1
-298.500000000000 8 1
-300.750000000000 8 1
The content of the workspace variable is correct.
The code responsible for this is as follows:
filename = {‘Alessandra’ ‘Alfredo’};
dataConditionDevice = [];
for ifile=1:1%length(filename)
filename{ifile}
filenameEditedTxt = fullfile(pathDataActivityDevice,[filename{ifile},’_trial_data.txt’])
dataConditionDevice = load(filenameEditedTxt);
end
RegardsHello,
When I try to display the whole matrix I receive the data in the following shape:
-0.0008 0.0001 0.0000
-0.0005 0.0001 0.0000
-0.0004 0.0001 0.0000
-0.0002 0.0001 0.0000
-0.0000 0.0001 0.0000
However when I attempt to display the portion of matrix teh result is correct:
-823 8 1
-584 8 1
-409 8 1
-304.928530000000 8 1
-298.500000000000 8 1
-300.750000000000 8 1
The content of the workspace variable is correct.
The code responsible for this is as follows:
filename = {‘Alessandra’ ‘Alfredo’};
dataConditionDevice = [];
for ifile=1:1%length(filename)
filename{ifile}
filenameEditedTxt = fullfile(pathDataActivityDevice,[filename{ifile},’_trial_data.txt’])
dataConditionDevice = load(filenameEditedTxt);
end
Regards Hello,
When I try to display the whole matrix I receive the data in the following shape:
-0.0008 0.0001 0.0000
-0.0005 0.0001 0.0000
-0.0004 0.0001 0.0000
-0.0002 0.0001 0.0000
-0.0000 0.0001 0.0000
However when I attempt to display the portion of matrix teh result is correct:
-823 8 1
-584 8 1
-409 8 1
-304.928530000000 8 1
-298.500000000000 8 1
-300.750000000000 8 1
The content of the workspace variable is correct.
The code responsible for this is as follows:
filename = {‘Alessandra’ ‘Alfredo’};
dataConditionDevice = [];
for ifile=1:1%length(filename)
filename{ifile}
filenameEditedTxt = fullfile(pathDataActivityDevice,[filename{ifile},’_trial_data.txt’])
dataConditionDevice = load(filenameEditedTxt);
end
Regards portion of matrix MATLAB Answers — New Questions
When does an ODE integrator reach the best numerical accuracy?
I’m using an ODE Matlab integrator (Bulirsch-Stoer) that is (supposedly, I never used it before) quite good for solving ODEs with high accuracy. I’m using it to solve highly non-linear diff. eqs. of which one does not know the true solution and which are highly senstivie to initial conditions and numerical accuracy. To be sure that it does the right thing I integrate the curves back and forth and see whether the back-integration returns to the intial conditions. It does. Of course, if the time interval (the ODE’s time-parameter "t") is large enough one sees it diverging (i.e., it doesn’t return to the intital conditions.) This is normal, and expected. However, since the integrator’s accuracy also depends on three independent parameters (the error tolerance, and two internal loops, midpoint and nr. of segmentation points) and it is difficult to find the optimal setting in a large 3D parameter space, I’m wondering whether the divergence from a true solution at some time, say t_max, beyond which the integrator no longer funrishes the correct values, is due to a non-optimal setting of the accuracy parameters (that is, I could still increase the accuracy), or wheteher it has reached the precision that is limited by the internal 15 digits number representation (that is, I reached the max. accuracy, and can’t do anything about it as long I work with double precision.)
So, my question is: Is there a method to know when one has effectively reached the best numerical evaluation in solving an ODE inside the limitation of a double precision integratioon? A method that essentially tells me: "Yes, that’s the best integration, beyond which you can’t go with a 15 digits internal representation, no matter how you fine-tune your solver."
I hope that I expressed clearly what my issue is. Feel free to ask for more information.I’m using an ODE Matlab integrator (Bulirsch-Stoer) that is (supposedly, I never used it before) quite good for solving ODEs with high accuracy. I’m using it to solve highly non-linear diff. eqs. of which one does not know the true solution and which are highly senstivie to initial conditions and numerical accuracy. To be sure that it does the right thing I integrate the curves back and forth and see whether the back-integration returns to the intial conditions. It does. Of course, if the time interval (the ODE’s time-parameter "t") is large enough one sees it diverging (i.e., it doesn’t return to the intital conditions.) This is normal, and expected. However, since the integrator’s accuracy also depends on three independent parameters (the error tolerance, and two internal loops, midpoint and nr. of segmentation points) and it is difficult to find the optimal setting in a large 3D parameter space, I’m wondering whether the divergence from a true solution at some time, say t_max, beyond which the integrator no longer funrishes the correct values, is due to a non-optimal setting of the accuracy parameters (that is, I could still increase the accuracy), or wheteher it has reached the precision that is limited by the internal 15 digits number representation (that is, I reached the max. accuracy, and can’t do anything about it as long I work with double precision.)
So, my question is: Is there a method to know when one has effectively reached the best numerical evaluation in solving an ODE inside the limitation of a double precision integratioon? A method that essentially tells me: "Yes, that’s the best integration, beyond which you can’t go with a 15 digits internal representation, no matter how you fine-tune your solver."
I hope that I expressed clearly what my issue is. Feel free to ask for more information. I’m using an ODE Matlab integrator (Bulirsch-Stoer) that is (supposedly, I never used it before) quite good for solving ODEs with high accuracy. I’m using it to solve highly non-linear diff. eqs. of which one does not know the true solution and which are highly senstivie to initial conditions and numerical accuracy. To be sure that it does the right thing I integrate the curves back and forth and see whether the back-integration returns to the intial conditions. It does. Of course, if the time interval (the ODE’s time-parameter "t") is large enough one sees it diverging (i.e., it doesn’t return to the intital conditions.) This is normal, and expected. However, since the integrator’s accuracy also depends on three independent parameters (the error tolerance, and two internal loops, midpoint and nr. of segmentation points) and it is difficult to find the optimal setting in a large 3D parameter space, I’m wondering whether the divergence from a true solution at some time, say t_max, beyond which the integrator no longer funrishes the correct values, is due to a non-optimal setting of the accuracy parameters (that is, I could still increase the accuracy), or wheteher it has reached the precision that is limited by the internal 15 digits number representation (that is, I reached the max. accuracy, and can’t do anything about it as long I work with double precision.)
So, my question is: Is there a method to know when one has effectively reached the best numerical evaluation in solving an ODE inside the limitation of a double precision integratioon? A method that essentially tells me: "Yes, that’s the best integration, beyond which you can’t go with a 15 digits internal representation, no matter how you fine-tune your solver."
I hope that I expressed clearly what my issue is. Feel free to ask for more information. ode, numerical integration MATLAB Answers — New Questions
Save Live Script automatically as PDF
Hello everyone,
I have a problem!
I wrote a program for the calculation of test data in the App Designer. The test results of the test person should be given to them in a report. Since I do not have access to the report generator, I have compiled the results in a live script and output them. I have set the button on the right to "hide code" and to save I press "save" and then "export to pdf".
Since I would like to compile the program as a standalone program, I cannot press the buttons myself and save the live script as a PDF.
I already tried the following:
matlab.internal.liveeditor.executeAndSave (which (‘Handout.mlx’));
matlab.internal.liveeditor.openAndConvert (‘Handout.mlx’, ‘test.pdf’);
Unfortunately, the code is always displayed there! But the design fits 🙂
Is there another way to automatically save the LiveScript as PDF without code?Hello everyone,
I have a problem!
I wrote a program for the calculation of test data in the App Designer. The test results of the test person should be given to them in a report. Since I do not have access to the report generator, I have compiled the results in a live script and output them. I have set the button on the right to "hide code" and to save I press "save" and then "export to pdf".
Since I would like to compile the program as a standalone program, I cannot press the buttons myself and save the live script as a PDF.
I already tried the following:
matlab.internal.liveeditor.executeAndSave (which (‘Handout.mlx’));
matlab.internal.liveeditor.openAndConvert (‘Handout.mlx’, ‘test.pdf’);
Unfortunately, the code is always displayed there! But the design fits 🙂
Is there another way to automatically save the LiveScript as PDF without code? Hello everyone,
I have a problem!
I wrote a program for the calculation of test data in the App Designer. The test results of the test person should be given to them in a report. Since I do not have access to the report generator, I have compiled the results in a live script and output them. I have set the button on the right to "hide code" and to save I press "save" and then "export to pdf".
Since I would like to compile the program as a standalone program, I cannot press the buttons myself and save the live script as a PDF.
I already tried the following:
matlab.internal.liveeditor.executeAndSave (which (‘Handout.mlx’));
matlab.internal.liveeditor.openAndConvert (‘Handout.mlx’, ‘test.pdf’);
Unfortunately, the code is always displayed there! But the design fits 🙂
Is there another way to automatically save the LiveScript as PDF without code? live script MATLAB Answers — New Questions
Why do I get ” Unable to resolve the name Mdl1b.Partition”
I keep getting "Unable to resolve the name Mdl1b.Partition" everytime I read all lines of the Excel file (1000 lines) but when I limit the lines to read to 50 or so it works perfectly. I do not know why doesn’t matlab read all the lines? is there a limit to the lines matlab reads?
%reading and defining X and Y
datmi=xlsread(‘Significant desc.xlsx’);
X=datmi(2:100,4:51);
Y=datmi (2:100,2);
%deviding the result data to train and test to draw graphs (which causes the error only when reading all the lines)
cvp=Mdl1b.Partition
The error I get
Error in ploting_ruselt (line 1)
cvp=Mdl1b.Partition
Error in ssvm (line 44)
ploting_ruseltI keep getting "Unable to resolve the name Mdl1b.Partition" everytime I read all lines of the Excel file (1000 lines) but when I limit the lines to read to 50 or so it works perfectly. I do not know why doesn’t matlab read all the lines? is there a limit to the lines matlab reads?
%reading and defining X and Y
datmi=xlsread(‘Significant desc.xlsx’);
X=datmi(2:100,4:51);
Y=datmi (2:100,2);
%deviding the result data to train and test to draw graphs (which causes the error only when reading all the lines)
cvp=Mdl1b.Partition
The error I get
Error in ploting_ruselt (line 1)
cvp=Mdl1b.Partition
Error in ssvm (line 44)
ploting_ruselt I keep getting "Unable to resolve the name Mdl1b.Partition" everytime I read all lines of the Excel file (1000 lines) but when I limit the lines to read to 50 or so it works perfectly. I do not know why doesn’t matlab read all the lines? is there a limit to the lines matlab reads?
%reading and defining X and Y
datmi=xlsread(‘Significant desc.xlsx’);
X=datmi(2:100,4:51);
Y=datmi (2:100,2);
%deviding the result data to train and test to draw graphs (which causes the error only when reading all the lines)
cvp=Mdl1b.Partition
The error I get
Error in ploting_ruselt (line 1)
cvp=Mdl1b.Partition
Error in ssvm (line 44)
ploting_ruselt partition, excel MATLAB Answers — New Questions
Whats the reason or any logical error so i am not getting the Peak values right as i calculated mathematically
Whats the reason or any logical error so i am not getting the Peak values right as i calculated mathematically
Ist phase Rpeak = 9.1 , 2nd phase Rpeak = 33.3. 3rd Phase = 90.9, 4th Phase = 50Whats the reason or any logical error so i am not getting the Peak values right as i calculated mathematically
Ist phase Rpeak = 9.1 , 2nd phase Rpeak = 33.3. 3rd Phase = 90.9, 4th Phase = 50 Whats the reason or any logical error so i am not getting the Peak values right as i calculated mathematically
Ist phase Rpeak = 9.1 , 2nd phase Rpeak = 33.3. 3rd Phase = 90.9, 4th Phase = 50 matlab code MATLAB Answers — New Questions
Saving randomly generated sequence into a binary file
Hello.
I wrote a simple logistics function that i use for my pesudo random number generator.I want to test it’s randomness so i downloaded a custom windows version of NIST STS.
the program rqeuires you to input the location of the binary file (.bin) of your data. i tried this code to save my sequence to .bin file but the program doesn’t recognize it.
please help.
the NIST-STS site : https://randomness-tests.fi.muni.cz/
i tried the demo .bin data that came with the software and it worked. only mine is being refused.
the code :
fileID = fopen(‘Sequence.bin’,’w’);
fwrite(fileID,(PRBG));
fclose(fileID);Hello.
I wrote a simple logistics function that i use for my pesudo random number generator.I want to test it’s randomness so i downloaded a custom windows version of NIST STS.
the program rqeuires you to input the location of the binary file (.bin) of your data. i tried this code to save my sequence to .bin file but the program doesn’t recognize it.
please help.
the NIST-STS site : https://randomness-tests.fi.muni.cz/
i tried the demo .bin data that came with the software and it worked. only mine is being refused.
the code :
fileID = fopen(‘Sequence.bin’,’w’);
fwrite(fileID,(PRBG));
fclose(fileID); Hello.
I wrote a simple logistics function that i use for my pesudo random number generator.I want to test it’s randomness so i downloaded a custom windows version of NIST STS.
the program rqeuires you to input the location of the binary file (.bin) of your data. i tried this code to save my sequence to .bin file but the program doesn’t recognize it.
please help.
the NIST-STS site : https://randomness-tests.fi.muni.cz/
i tried the demo .bin data that came with the software and it worked. only mine is being refused.
the code :
fileID = fopen(‘Sequence.bin’,’w’);
fwrite(fileID,(PRBG));
fclose(fileID); encryption, image processing, binary, output MATLAB Answers — New Questions
What is the difference between oobPredict and predict with ensemble of bagged decision trees?
1- I am using both fuctions to predict a response through random forest, but the predict function gives higher percentage of explained variance compared to oobPredict. Why is it so? – I think there is some fundamental thing that I have not yet fully grasped.
2- If there is something different between these methods in the way that they weigh trees how can I make these methods homogenous?
3- Can one use oobPredict in someway to make predictions with a new set of data?1- I am using both fuctions to predict a response through random forest, but the predict function gives higher percentage of explained variance compared to oobPredict. Why is it so? – I think there is some fundamental thing that I have not yet fully grasped.
2- If there is something different between these methods in the way that they weigh trees how can I make these methods homogenous?
3- Can one use oobPredict in someway to make predictions with a new set of data? 1- I am using both fuctions to predict a response through random forest, but the predict function gives higher percentage of explained variance compared to oobPredict. Why is it so? – I think there is some fundamental thing that I have not yet fully grasped.
2- If there is something different between these methods in the way that they weigh trees how can I make these methods homogenous?
3- Can one use oobPredict in someway to make predictions with a new set of data? random forest, regression, machine learning, curve fitting, decision trees, bagging, oob MATLAB Answers — New Questions
iteration newton raphson
i am having difficulty trying to find an iteration formula using newton-raphson for solving the equation x=cos(2x) and write an m-file for finding the solution whereby x matches cos(2x) to atleast 8 decimal places using format long. can anybody help me?i am having difficulty trying to find an iteration formula using newton-raphson for solving the equation x=cos(2x) and write an m-file for finding the solution whereby x matches cos(2x) to atleast 8 decimal places using format long. can anybody help me? i am having difficulty trying to find an iteration formula using newton-raphson for solving the equation x=cos(2x) and write an m-file for finding the solution whereby x matches cos(2x) to atleast 8 decimal places using format long. can anybody help me? iteration newton-raphson MATLAB Answers — New Questions
How to communicate with TCSPC via MATLAB?
Hello
I’m a graduate student in bioimaging lab.
In our lab, we have TCSPC device from company Becker & Hickl (BH) and it is connected through PCI bus.
It’s possible to use TCSPC via software developed from BH, but I want to use several functions of TCSPC via our imaging program written by MATLAB.
I tried to use several MATLAB functions (‘visadev’,’visadevfind’,etc.) to communicate with TCSPC, but I couldn’t connect to TCSPC via MATLAB.
Here’s the quetions to ask
1) Is there any reference code written in MATLAB which is used for communication with TCSPC via MATLAB?
2) If not, could I get some guidances for that? I downloaded .dll files from BH.Hello
I’m a graduate student in bioimaging lab.
In our lab, we have TCSPC device from company Becker & Hickl (BH) and it is connected through PCI bus.
It’s possible to use TCSPC via software developed from BH, but I want to use several functions of TCSPC via our imaging program written by MATLAB.
I tried to use several MATLAB functions (‘visadev’,’visadevfind’,etc.) to communicate with TCSPC, but I couldn’t connect to TCSPC via MATLAB.
Here’s the quetions to ask
1) Is there any reference code written in MATLAB which is used for communication with TCSPC via MATLAB?
2) If not, could I get some guidances for that? I downloaded .dll files from BH. Hello
I’m a graduate student in bioimaging lab.
In our lab, we have TCSPC device from company Becker & Hickl (BH) and it is connected through PCI bus.
It’s possible to use TCSPC via software developed from BH, but I want to use several functions of TCSPC via our imaging program written by MATLAB.
I tried to use several MATLAB functions (‘visadev’,’visadevfind’,etc.) to communicate with TCSPC, but I couldn’t connect to TCSPC via MATLAB.
Here’s the quetions to ask
1) Is there any reference code written in MATLAB which is used for communication with TCSPC via MATLAB?
2) If not, could I get some guidances for that? I downloaded .dll files from BH. communication MATLAB Answers — New Questions
GPS time extraction using GPSDO module for USRP N210
Hi
I am working on TDOA geolocation setup with 5 USRP N210, and GPSDO modules, I wish to extract the GPS time vector along with captured I/Q data on 5 USRPs installed at distant locations. This GPS time vector on each USRP will give me the time-stamp information which is necessary for aligning the captured data to work out the TDOA cross-correlation.
Can anybody provide me MATLAB commands to capture the GPS time vector?
Please help.
ThanksHi
I am working on TDOA geolocation setup with 5 USRP N210, and GPSDO modules, I wish to extract the GPS time vector along with captured I/Q data on 5 USRPs installed at distant locations. This GPS time vector on each USRP will give me the time-stamp information which is necessary for aligning the captured data to work out the TDOA cross-correlation.
Can anybody provide me MATLAB commands to capture the GPS time vector?
Please help.
Thanks Hi
I am working on TDOA geolocation setup with 5 USRP N210, and GPSDO modules, I wish to extract the GPS time vector along with captured I/Q data on 5 USRPs installed at distant locations. This GPS time vector on each USRP will give me the time-stamp information which is necessary for aligning the captured data to work out the TDOA cross-correlation.
Can anybody provide me MATLAB commands to capture the GPS time vector?
Please help.
Thanks gps synchronization, gpsdo, usrp n210, tdoa geolocation MATLAB Answers — New Questions
Issue in Reading serial data from the Arduino Uno in Simulink using Serial receive block
I’m using an Arduino Uno with Simulink and trying to read serial data using the Serial Receive block. The Arduino is printing voltage values from an analog sensor as follows:
int sensorValue = analogRead(A0);
float voltage = sensorValue * (5.0 / 1023.0);
Serial.println(voltage);
However, I’m facing a conflict because the Serial Receive block and External Mode are trying to use Serial Port 0, which is the only hardware serial port available on the Arduino Uno.
My Setup:
Hardware: Arduino Uno
Simulink Blocks:
Serial Receive block to capture the data sent from Arduino
Mode: I attempted to use both Connected I/O and External mode (Run on board), but as the simulation doesn’t allow to change the the serial port from 0 in either mode, nor can I change the pin number in the Serial Receive block.
Problem:
When I run the model, I receive the following errors:
Run with I/O
Caused by: Serial Port 0 used by the Serial Receive block, is also used for Connected I/O. Change the port number used by the block in your model.
Run on baord
Serial Port 0 used by the Serial Receive block is also used for External mode. Change the port number used by the block.
However, the Arduino Uno only has one hardware serial port, and there’s no option to change the serial port in the Serial Receive block or in the hardware setting in the Simulink.
How can I resolve this conflict and successfully read serial data from the Arduino Uno in Simulink?I’m using an Arduino Uno with Simulink and trying to read serial data using the Serial Receive block. The Arduino is printing voltage values from an analog sensor as follows:
int sensorValue = analogRead(A0);
float voltage = sensorValue * (5.0 / 1023.0);
Serial.println(voltage);
However, I’m facing a conflict because the Serial Receive block and External Mode are trying to use Serial Port 0, which is the only hardware serial port available on the Arduino Uno.
My Setup:
Hardware: Arduino Uno
Simulink Blocks:
Serial Receive block to capture the data sent from Arduino
Mode: I attempted to use both Connected I/O and External mode (Run on board), but as the simulation doesn’t allow to change the the serial port from 0 in either mode, nor can I change the pin number in the Serial Receive block.
Problem:
When I run the model, I receive the following errors:
Run with I/O
Caused by: Serial Port 0 used by the Serial Receive block, is also used for Connected I/O. Change the port number used by the block in your model.
Run on baord
Serial Port 0 used by the Serial Receive block is also used for External mode. Change the port number used by the block.
However, the Arduino Uno only has one hardware serial port, and there’s no option to change the serial port in the Serial Receive block or in the hardware setting in the Simulink.
How can I resolve this conflict and successfully read serial data from the Arduino Uno in Simulink? I’m using an Arduino Uno with Simulink and trying to read serial data using the Serial Receive block. The Arduino is printing voltage values from an analog sensor as follows:
int sensorValue = analogRead(A0);
float voltage = sensorValue * (5.0 / 1023.0);
Serial.println(voltage);
However, I’m facing a conflict because the Serial Receive block and External Mode are trying to use Serial Port 0, which is the only hardware serial port available on the Arduino Uno.
My Setup:
Hardware: Arduino Uno
Simulink Blocks:
Serial Receive block to capture the data sent from Arduino
Mode: I attempted to use both Connected I/O and External mode (Run on board), but as the simulation doesn’t allow to change the the serial port from 0 in either mode, nor can I change the pin number in the Serial Receive block.
Problem:
When I run the model, I receive the following errors:
Run with I/O
Caused by: Serial Port 0 used by the Serial Receive block, is also used for Connected I/O. Change the port number used by the block in your model.
Run on baord
Serial Port 0 used by the Serial Receive block is also used for External mode. Change the port number used by the block.
However, the Arduino Uno only has one hardware serial port, and there’s no option to change the serial port in the Serial Receive block or in the hardware setting in the Simulink.
How can I resolve this conflict and successfully read serial data from the Arduino Uno in Simulink? simulink, serial receiver, arduino MATLAB Answers — New Questions
Human Detection using YOLOV3 with own Weights
Hi everyone! I am trying to detect humans using yolov3 weights. Right now, I am trying to detect using an image but later I am going to use a camera to detect humans. I don’t know if this is possible using my own weights with my config file.Hi everyone! I am trying to detect humans using yolov3 weights. Right now, I am trying to detect using an image but later I am going to use a camera to detect humans. I don’t know if this is possible using my own weights with my config file. Hi everyone! I am trying to detect humans using yolov3 weights. Right now, I am trying to detect using an image but later I am going to use a camera to detect humans. I don’t know if this is possible using my own weights with my config file. human detection, yolov3, object detection MATLAB Answers — New Questions
Warning: Block diagram ‘XYZ’ contains disabled library links.
After deleting a model from library, I am trying to save that library, but I am coming accross a warning message as follows:
Warning: Block diagram ‘XYZ’ contains disabled library links. Use Model Advisor to find the disabled links in non-library models. The diagram has been saved but may not contain what you intended.
I can see that model advisor is able to help to resolve the Disabled library links but how can it be used to identify disabled links in non-library models.
Please suggest on this.
Regards,
Bhavnish.After deleting a model from library, I am trying to save that library, but I am coming accross a warning message as follows:
Warning: Block diagram ‘XYZ’ contains disabled library links. Use Model Advisor to find the disabled links in non-library models. The diagram has been saved but may not contain what you intended.
I can see that model advisor is able to help to resolve the Disabled library links but how can it be used to identify disabled links in non-library models.
Please suggest on this.
Regards,
Bhavnish. After deleting a model from library, I am trying to save that library, but I am coming accross a warning message as follows:
Warning: Block diagram ‘XYZ’ contains disabled library links. Use Model Advisor to find the disabled links in non-library models. The diagram has been saved but may not contain what you intended.
I can see that model advisor is able to help to resolve the Disabled library links but how can it be used to identify disabled links in non-library models.
Please suggest on this.
Regards,
Bhavnish. disabled library links MATLAB Answers — New Questions
deep learning architecture can explain how connected layers and filters
some one can explain how this connections fit each to previous stage
layers = [ imageInputLayer([28 28 1])
convolution2dLayer(5,20)
reluLayer
maxPooling2dLayer(2, ‘Stride’, 2)
fullyConnectedLayer(10)
softmaxLayer
classificationLayer() ]some one can explain how this connections fit each to previous stage
layers = [ imageInputLayer([28 28 1])
convolution2dLayer(5,20)
reluLayer
maxPooling2dLayer(2, ‘Stride’, 2)
fullyConnectedLayer(10)
softmaxLayer
classificationLayer() ] some one can explain how this connections fit each to previous stage
layers = [ imageInputLayer([28 28 1])
convolution2dLayer(5,20)
reluLayer
maxPooling2dLayer(2, ‘Stride’, 2)
fullyConnectedLayer(10)
softmaxLayer
classificationLayer() ] deep learning MATLAB Answers — New Questions
unable to incorporate own design Loss function in r2024a
Switching from r2023b to r2024 I made some changes in my Net (CNN). e.g. modified input/output and replace RegressionLayer with SoftmaxLayer, using trainnet function, etc.
I expected better performance, perspective compatibility (RegressionLayre is not more recommended) and have a vision of my Net optimization with use of Prune approach etc.
To the contrary to the previous version I am not able to involve my own Loss function (as it was done in previous version).
The (siplified) code is below, the used synthax was inspired by example:
https://www.mathworks.com/matlabcentral/answers/2100631-how-can-i-define-a-custom-loss-function-using-trainnet
The error message is:
Error using trainnet (line 46)
Error calling function during training.
Error in callMyLoss (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Caused by:
Error using myOwnLoss
The specified superclass ‘nnet.layer.softmaxLayer’ contains a parse error, cannot be found on MATLAB’s
search
path, or is shadowed by another file with the same name.
Error in callMyLoss>@(Y,target)myOwnLoss(name,Y,target) (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Error in nnet.internal.cnn.util.UserCodeException.fevalUserCode (line 11)
[varargout{1:nargout}] = feval(F, varargin{:});
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof=========================
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof=========================Switching from r2023b to r2024 I made some changes in my Net (CNN). e.g. modified input/output and replace RegressionLayer with SoftmaxLayer, using trainnet function, etc.
I expected better performance, perspective compatibility (RegressionLayre is not more recommended) and have a vision of my Net optimization with use of Prune approach etc.
To the contrary to the previous version I am not able to involve my own Loss function (as it was done in previous version).
The (siplified) code is below, the used synthax was inspired by example:
https://www.mathworks.com/matlabcentral/answers/2100631-how-can-i-define-a-custom-loss-function-using-trainnet
The error message is:
Error using trainnet (line 46)
Error calling function during training.
Error in callMyLoss (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Caused by:
Error using myOwnLoss
The specified superclass ‘nnet.layer.softmaxLayer’ contains a parse error, cannot be found on MATLAB’s
search
path, or is shadowed by another file with the same name.
Error in callMyLoss>@(Y,target)myOwnLoss(name,Y,target) (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Error in nnet.internal.cnn.util.UserCodeException.fevalUserCode (line 11)
[varargout{1:nargout}] = feval(F, varargin{:});
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof=========================
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof========================= Switching from r2023b to r2024 I made some changes in my Net (CNN). e.g. modified input/output and replace RegressionLayer with SoftmaxLayer, using trainnet function, etc.
I expected better performance, perspective compatibility (RegressionLayre is not more recommended) and have a vision of my Net optimization with use of Prune approach etc.
To the contrary to the previous version I am not able to involve my own Loss function (as it was done in previous version).
The (siplified) code is below, the used synthax was inspired by example:
https://www.mathworks.com/matlabcentral/answers/2100631-how-can-i-define-a-custom-loss-function-using-trainnet
The error message is:
Error using trainnet (line 46)
Error calling function during training.
Error in callMyLoss (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Caused by:
Error using myOwnLoss
The specified superclass ‘nnet.layer.softmaxLayer’ contains a parse error, cannot be found on MATLAB’s
search
path, or is shadowed by another file with the same name.
Error in callMyLoss>@(Y,target)myOwnLoss(name,Y,target) (line 55)
myTrainedNet = trainnet(Y,target,net, @(Y,target) myOwnLoss(name,Y,target),options);
Error in nnet.internal.cnn.util.UserCodeException.fevalUserCode (line 11)
[varargout{1:nargout}] = feval(F, varargin{:});
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof=========================
classdef myOwnLoss < nnet.layer.softmaxLayer
% own Loss
methods
%function layer = sseClassificationLayer(name)
function layer = myOwnLoss(name)
% layer = sseClassificationLayer(name) creates a sum of squares
% error classification layer and specifies the layer name.
% Set layer name.
layer.Name = name;
% Set layer description.
layer.Description = ‘my own Loss v.2024a’;
end
function loss = forwardLoss(layer, Y, T)
%%% function loss = forwardLoss(Yo, To)
% loss = forwardLoss(layer, Y, T) returns the Tdiff loss between
% the predictions Y and the training targets T.
disp("myLoss");
aa=1;
% just something very simple
loss = sum(Y-T,’all’);
end
% original backwardLoss
function dX = backwardLoss(layer, Y, T)
numObservations = size( Y, 3);
dX = (Y – T)./numObservations;
end
end
end
%=======================eof========================= loss function, trainnet MATLAB Answers — New Questions
HELP Canonical Huffman coding
Hi, can some one share the source code for Canonical Huffman?
Thank you…Hi, can some one share the source code for Canonical Huffman?
Thank you… Hi, can some one share the source code for Canonical Huffman?
Thank you… huffman, canonical, image compressing, entropy MATLAB Answers — New Questions
Simulink thermal coppling problem
I have this problem where i seem to have some thermal coppling of my Tank(G-TL) with my Constant Volume Chamber. Yet there shouldnt be anything that does effect the temperature in my Constant Volume Chamber i still have something definitely influencing it. When I only simulate the outflow of my Constant Volume Chamber (pink) filled with N2 with a start value of 300 bar and a starting temperature of 300 Kelvin, i get a temperature of 267 Kelvin at the point my Chamber pressure dropped down to 200 bar. But if I simulate the whole circuit (screenshot) i get a different temperature after i dropped down to 200 bar in my Constant Volume Chamber depending on the settings i set my Tank(G-TL) (pink+yellow) on. It would be very nice if someone has an idea what could cause this effect of the temperature behaviour of my Constant Volume Chamber in this Simulink modell?I have this problem where i seem to have some thermal coppling of my Tank(G-TL) with my Constant Volume Chamber. Yet there shouldnt be anything that does effect the temperature in my Constant Volume Chamber i still have something definitely influencing it. When I only simulate the outflow of my Constant Volume Chamber (pink) filled with N2 with a start value of 300 bar and a starting temperature of 300 Kelvin, i get a temperature of 267 Kelvin at the point my Chamber pressure dropped down to 200 bar. But if I simulate the whole circuit (screenshot) i get a different temperature after i dropped down to 200 bar in my Constant Volume Chamber depending on the settings i set my Tank(G-TL) (pink+yellow) on. It would be very nice if someone has an idea what could cause this effect of the temperature behaviour of my Constant Volume Chamber in this Simulink modell? I have this problem where i seem to have some thermal coppling of my Tank(G-TL) with my Constant Volume Chamber. Yet there shouldnt be anything that does effect the temperature in my Constant Volume Chamber i still have something definitely influencing it. When I only simulate the outflow of my Constant Volume Chamber (pink) filled with N2 with a start value of 300 bar and a starting temperature of 300 Kelvin, i get a temperature of 267 Kelvin at the point my Chamber pressure dropped down to 200 bar. But if I simulate the whole circuit (screenshot) i get a different temperature after i dropped down to 200 bar in my Constant Volume Chamber depending on the settings i set my Tank(G-TL) (pink+yellow) on. It would be very nice if someone has an idea what could cause this effect of the temperature behaviour of my Constant Volume Chamber in this Simulink modell? simulink, simscape MATLAB Answers — New Questions
Unique function to return last duplicate value
Hi there,
Is there a way I can use the ‘unique’ function to return the last duplicate value instead of the first one? Refer to the images attached for clarity. Essentially I’m using
"[Sp_Au,ix] = unique(Data.SetPoint, ‘stable’);
DataE = Data(ix,:);"
to retireve the unique values as per the SetPoint column and save it into ‘DataE’, but instead of it starting at rows that correspond to the first "10", I want it to retrieve the last "10" row just before it starts increasing. Is there a way to do that?
So from table Data, instead of it taking rows (1,6,7,8,…etc), I want it export rows (5,6,7,8,9,…etc). So from the last duplicated value. Hope that makes sense.Hi there,
Is there a way I can use the ‘unique’ function to return the last duplicate value instead of the first one? Refer to the images attached for clarity. Essentially I’m using
"[Sp_Au,ix] = unique(Data.SetPoint, ‘stable’);
DataE = Data(ix,:);"
to retireve the unique values as per the SetPoint column and save it into ‘DataE’, but instead of it starting at rows that correspond to the first "10", I want it to retrieve the last "10" row just before it starts increasing. Is there a way to do that?
So from table Data, instead of it taking rows (1,6,7,8,…etc), I want it export rows (5,6,7,8,9,…etc). So from the last duplicated value. Hope that makes sense. Hi there,
Is there a way I can use the ‘unique’ function to return the last duplicate value instead of the first one? Refer to the images attached for clarity. Essentially I’m using
"[Sp_Au,ix] = unique(Data.SetPoint, ‘stable’);
DataE = Data(ix,:);"
to retireve the unique values as per the SetPoint column and save it into ‘DataE’, but instead of it starting at rows that correspond to the first "10", I want it to retrieve the last "10" row just before it starts increasing. Is there a way to do that?
So from table Data, instead of it taking rows (1,6,7,8,…etc), I want it export rows (5,6,7,8,9,…etc). So from the last duplicated value. Hope that makes sense. unique, data MATLAB Answers — New Questions
How to identify duplicate rows between tables
I’m using R2020b, and I want to set up a master table for appending new data to – and as part of this I want to identify any duplicate rows in the new, incoming table to filter them out before appending. Ideally, the master table will live in a related directory in a .mat file, and the new data will be read in directly from a set-name, set-location .csv using e.g.
fullname = fullfile(‘relativepath’,’newdata.csv’);
% grab column headers from input sheet
opts = detectImportOptions(fullname);
% set all variable types to categorical
opts.VariableTypes(:) = {‘categorical’};
% read in new data
T = readtable(fullname,opts);
% make any modifications to new data headers to match old data
T = renamevars(T,"NewLabel","OldLabel");
% clean new table headers to match originally-wizard-imported headers (I’d ask why these exhibit different behaviour, but that’s a separate tragedy, and this current fix works – I think)
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘ ‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘(‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘)’, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘_’, ”);
I found the solution suggested here: https://au.mathworks.com/matlabcentral/answers/514921-finding-identical-rows-in-2-tables, but having done a quick test via:
foo = T(str2double(string(T.Year))<1943,:); % not my actual query, but structurally the same; this gave me ~40% of my original data
bar = T(str2double(string(T.Year))>1941,:); % similar, gave me ~70% of the original data
baz = ismember(foo,bar); % similar, gives the overlap for 1 particular year (should be about 14% of my original data)
blah = T(str2double(string(T.Year))==1942,:); % to directly extract the number of rows I am looking for
sum(baz) % What I expect here is the number of rows in the overlap
ans =
0
I found that ismember was not finding any duplicates (which were there by construction).
Note: due to categorical data I actually used T(str2double(string(T.Year))…)
Replacing
baz = ismember(foo,bar,’rows’);
sum(baz)
ans =
0
results in the same not finding any duplicates. Using double quotes "rows" does not change the behaviour.
On the other hand, using the function to assess single variables gives the expected behaviour (to some degree):
testest = ismember(foo.var1,bar.var1)
sum(testest)
The sum is now non-zero, and (because single variables are repeated more often than their combinations) gives more like 30% of the original data, which seems reasonable (the number of unique entries in the original set in that variable was about 40% of the total).
I guess I could create a logical index based on the product of multiple calls of this kind, but that seems rather… inefficient… and sensitive to the exact construction of the table/variables used in the filter. I’d rather have a generic solution for full table rows that will be robust if the overall table changes over the long term (or if/when I functionalise the code and use it for other work). Whilst most of the time, a couple of key variables can be used to identify unique rows, occasionally more information is required to distinguish pathological cases. I will probably use this approach if a more elegant solution doesn’t appear, though, and put some thought into which groups of variables are 100% correlated (and therefore useless for this distinction) to cut down the Boolean product.
I could also throw good coding practice to the winds and just write two nested loops (one for rows, one for variables) and exhaustively test every combination, but I suspect that would be even less efficient (although I wonder whether the scaling order would be the same given the nature of the comparisons required).
If it is pertinent, I imported all (>25) data columns from a .csv file as categorical variables. The original data before that were a mix of number and general columns from an Excel sheet; I could have used any or all of {double,string,categorical,datetime} to store the various variables, but there are some data which are best stored as categorical to avoid character trimming and consequent data cleaning / returning to original state steps.
Digging further, I also found this: https://au.mathworks.com/matlabcentral/answers/1775400-how-do-i-find-all-indexes-of-duplicate-names-in-a-table-column-then-compare-the-row-values-for-each which appears to imply that ismember should have the functionality I need here.
Similarly, methods using unique (see e.g. https://au.mathworks.com/matlabcentral/answers/1999193-find-duplicated-rows-in-matlab-without-for-loop or https://au.mathworks.com/matlabcentral/answers/1571588-table-find-duplicate-rows-double-char-datetime or https://au.mathworks.com/matlabcentral/answers/305987-identify-duplicate-rows-in-a-matrix) give:
size(unique([foo;bar],’rows’),1) == size(foo,1)+size(bar,1)
ans =
logical
1
instead of the expected 0 due to the lower amount of actual full-row matches. (Same for "rows" again.)
I’ve also looked into outerjoin/join/innerjoin, but those don’t seem to remove duplicates like I need.I’m using R2020b, and I want to set up a master table for appending new data to – and as part of this I want to identify any duplicate rows in the new, incoming table to filter them out before appending. Ideally, the master table will live in a related directory in a .mat file, and the new data will be read in directly from a set-name, set-location .csv using e.g.
fullname = fullfile(‘relativepath’,’newdata.csv’);
% grab column headers from input sheet
opts = detectImportOptions(fullname);
% set all variable types to categorical
opts.VariableTypes(:) = {‘categorical’};
% read in new data
T = readtable(fullname,opts);
% make any modifications to new data headers to match old data
T = renamevars(T,"NewLabel","OldLabel");
% clean new table headers to match originally-wizard-imported headers (I’d ask why these exhibit different behaviour, but that’s a separate tragedy, and this current fix works – I think)
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘ ‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘(‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘)’, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘_’, ”);
I found the solution suggested here: https://au.mathworks.com/matlabcentral/answers/514921-finding-identical-rows-in-2-tables, but having done a quick test via:
foo = T(str2double(string(T.Year))<1943,:); % not my actual query, but structurally the same; this gave me ~40% of my original data
bar = T(str2double(string(T.Year))>1941,:); % similar, gave me ~70% of the original data
baz = ismember(foo,bar); % similar, gives the overlap for 1 particular year (should be about 14% of my original data)
blah = T(str2double(string(T.Year))==1942,:); % to directly extract the number of rows I am looking for
sum(baz) % What I expect here is the number of rows in the overlap
ans =
0
I found that ismember was not finding any duplicates (which were there by construction).
Note: due to categorical data I actually used T(str2double(string(T.Year))…)
Replacing
baz = ismember(foo,bar,’rows’);
sum(baz)
ans =
0
results in the same not finding any duplicates. Using double quotes "rows" does not change the behaviour.
On the other hand, using the function to assess single variables gives the expected behaviour (to some degree):
testest = ismember(foo.var1,bar.var1)
sum(testest)
The sum is now non-zero, and (because single variables are repeated more often than their combinations) gives more like 30% of the original data, which seems reasonable (the number of unique entries in the original set in that variable was about 40% of the total).
I guess I could create a logical index based on the product of multiple calls of this kind, but that seems rather… inefficient… and sensitive to the exact construction of the table/variables used in the filter. I’d rather have a generic solution for full table rows that will be robust if the overall table changes over the long term (or if/when I functionalise the code and use it for other work). Whilst most of the time, a couple of key variables can be used to identify unique rows, occasionally more information is required to distinguish pathological cases. I will probably use this approach if a more elegant solution doesn’t appear, though, and put some thought into which groups of variables are 100% correlated (and therefore useless for this distinction) to cut down the Boolean product.
I could also throw good coding practice to the winds and just write two nested loops (one for rows, one for variables) and exhaustively test every combination, but I suspect that would be even less efficient (although I wonder whether the scaling order would be the same given the nature of the comparisons required).
If it is pertinent, I imported all (>25) data columns from a .csv file as categorical variables. The original data before that were a mix of number and general columns from an Excel sheet; I could have used any or all of {double,string,categorical,datetime} to store the various variables, but there are some data which are best stored as categorical to avoid character trimming and consequent data cleaning / returning to original state steps.
Digging further, I also found this: https://au.mathworks.com/matlabcentral/answers/1775400-how-do-i-find-all-indexes-of-duplicate-names-in-a-table-column-then-compare-the-row-values-for-each which appears to imply that ismember should have the functionality I need here.
Similarly, methods using unique (see e.g. https://au.mathworks.com/matlabcentral/answers/1999193-find-duplicated-rows-in-matlab-without-for-loop or https://au.mathworks.com/matlabcentral/answers/1571588-table-find-duplicate-rows-double-char-datetime or https://au.mathworks.com/matlabcentral/answers/305987-identify-duplicate-rows-in-a-matrix) give:
size(unique([foo;bar],’rows’),1) == size(foo,1)+size(bar,1)
ans =
logical
1
instead of the expected 0 due to the lower amount of actual full-row matches. (Same for "rows" again.)
I’ve also looked into outerjoin/join/innerjoin, but those don’t seem to remove duplicates like I need. I’m using R2020b, and I want to set up a master table for appending new data to – and as part of this I want to identify any duplicate rows in the new, incoming table to filter them out before appending. Ideally, the master table will live in a related directory in a .mat file, and the new data will be read in directly from a set-name, set-location .csv using e.g.
fullname = fullfile(‘relativepath’,’newdata.csv’);
% grab column headers from input sheet
opts = detectImportOptions(fullname);
% set all variable types to categorical
opts.VariableTypes(:) = {‘categorical’};
% read in new data
T = readtable(fullname,opts);
% make any modifications to new data headers to match old data
T = renamevars(T,"NewLabel","OldLabel");
% clean new table headers to match originally-wizard-imported headers (I’d ask why these exhibit different behaviour, but that’s a separate tragedy, and this current fix works – I think)
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘ ‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘(‘, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘)’, ”);
T.Properties.VariableNames = regexprep(T.Properties.VariableNames, ‘_’, ”);
I found the solution suggested here: https://au.mathworks.com/matlabcentral/answers/514921-finding-identical-rows-in-2-tables, but having done a quick test via:
foo = T(str2double(string(T.Year))<1943,:); % not my actual query, but structurally the same; this gave me ~40% of my original data
bar = T(str2double(string(T.Year))>1941,:); % similar, gave me ~70% of the original data
baz = ismember(foo,bar); % similar, gives the overlap for 1 particular year (should be about 14% of my original data)
blah = T(str2double(string(T.Year))==1942,:); % to directly extract the number of rows I am looking for
sum(baz) % What I expect here is the number of rows in the overlap
ans =
0
I found that ismember was not finding any duplicates (which were there by construction).
Note: due to categorical data I actually used T(str2double(string(T.Year))…)
Replacing
baz = ismember(foo,bar,’rows’);
sum(baz)
ans =
0
results in the same not finding any duplicates. Using double quotes "rows" does not change the behaviour.
On the other hand, using the function to assess single variables gives the expected behaviour (to some degree):
testest = ismember(foo.var1,bar.var1)
sum(testest)
The sum is now non-zero, and (because single variables are repeated more often than their combinations) gives more like 30% of the original data, which seems reasonable (the number of unique entries in the original set in that variable was about 40% of the total).
I guess I could create a logical index based on the product of multiple calls of this kind, but that seems rather… inefficient… and sensitive to the exact construction of the table/variables used in the filter. I’d rather have a generic solution for full table rows that will be robust if the overall table changes over the long term (or if/when I functionalise the code and use it for other work). Whilst most of the time, a couple of key variables can be used to identify unique rows, occasionally more information is required to distinguish pathological cases. I will probably use this approach if a more elegant solution doesn’t appear, though, and put some thought into which groups of variables are 100% correlated (and therefore useless for this distinction) to cut down the Boolean product.
I could also throw good coding practice to the winds and just write two nested loops (one for rows, one for variables) and exhaustively test every combination, but I suspect that would be even less efficient (although I wonder whether the scaling order would be the same given the nature of the comparisons required).
If it is pertinent, I imported all (>25) data columns from a .csv file as categorical variables. The original data before that were a mix of number and general columns from an Excel sheet; I could have used any or all of {double,string,categorical,datetime} to store the various variables, but there are some data which are best stored as categorical to avoid character trimming and consequent data cleaning / returning to original state steps.
Digging further, I also found this: https://au.mathworks.com/matlabcentral/answers/1775400-how-do-i-find-all-indexes-of-duplicate-names-in-a-table-column-then-compare-the-row-values-for-each which appears to imply that ismember should have the functionality I need here.
Similarly, methods using unique (see e.g. https://au.mathworks.com/matlabcentral/answers/1999193-find-duplicated-rows-in-matlab-without-for-loop or https://au.mathworks.com/matlabcentral/answers/1571588-table-find-duplicate-rows-double-char-datetime or https://au.mathworks.com/matlabcentral/answers/305987-identify-duplicate-rows-in-a-matrix) give:
size(unique([foo;bar],’rows’),1) == size(foo,1)+size(bar,1)
ans =
logical
1
instead of the expected 0 due to the lower amount of actual full-row matches. (Same for "rows" again.)
I’ve also looked into outerjoin/join/innerjoin, but those don’t seem to remove duplicates like I need. table, ismember, rows, duplicate MATLAB Answers — New Questions
