I have a large table of data which needs to be output as a csv and then read into a model. For some reason (which I cannot solve) the model will not take in values with scientific number formating (e.g. very small or large numbers, so 8.8817e-04, rather than 0.000888). To get around this I turn the table into text and use compose, but this is slow as the data are large. I have tried using round, which is fast, but Matlab still displays in scientific notation for some values and these remain in the csv, which does not solve the problem. Any faster ideas welcome!
Here’s an example. Note I know you don’t need all the DateTime info, but it gives the form of the data. In reality I have 30 years of data at least, which is why it is a problem. It also need to use a seperate workstation or HPC as the string/compose lines lead to out of memory problems on my own laptop, even if it can store this size of table ok.
Many thanks!
StartDate=datetime(’01/01/2019 00:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
EndDate=datetime(’31/10/2020 23:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
DateTime = StartDate:hours(1):EndDate;
DateTime = DateTime’;
datasize =size(DateTime,1);
No_St = 307;

Data = rand(datasize,No_St);

Test_table = array2table(Data);
Test_table.Time = DateTime;
Test_table = movevars(Test_table,"Time","Before",1);

Table_str = string(Test_table{:,2:end}); %convert to string
Table_str = compose(‘%.6f’,Table_str); %Reduce precisionI have a large table of data which needs to be output as a csv and then read into a model. For some reason (which I cannot solve) the model will not take in values with scientific number formating (e.g. very small or large numbers, so 8.8817e-04, rather than 0.000888). To get around this I turn the table into text and use compose, but this is slow as the data are large. I have tried using round, which is fast, but Matlab still displays in scientific notation for some values and these remain in the csv, which does not solve the problem. Any faster ideas welcome!
Here’s an example. Note I know you don’t need all the DateTime info, but it gives the form of the data. In reality I have 30 years of data at least, which is why it is a problem. It also need to use a seperate workstation or HPC as the string/compose lines lead to out of memory problems on my own laptop, even if it can store this size of table ok.
Many thanks!
StartDate=datetime(’01/01/2019 00:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
EndDate=datetime(’31/10/2020 23:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
DateTime = StartDate:hours(1):EndDate;
DateTime = DateTime’;
datasize =size(DateTime,1);
No_St = 307;

Data = rand(datasize,No_St);

Test_table = array2table(Data);
Test_table.Time = DateTime;
Test_table = movevars(Test_table,"Time","Before",1);

Table_str = string(Test_table{:,2:end}); %convert to string
Table_str = compose(‘%.6f’,Table_str); %Reduce precision I have a large table of data which needs to be output as a csv and then read into a model. For some reason (which I cannot solve) the model will not take in values with scientific number formating (e.g. very small or large numbers, so 8.8817e-04, rather than 0.000888). To get around this I turn the table into text and use compose, but this is slow as the data are large. I have tried using round, which is fast, but Matlab still displays in scientific notation for some values and these remain in the csv, which does not solve the problem. Any faster ideas welcome!
Here’s an example. Note I know you don’t need all the DateTime info, but it gives the form of the data. In reality I have 30 years of data at least, which is why it is a problem. It also need to use a seperate workstation or HPC as the string/compose lines lead to out of memory problems on my own laptop, even if it can store this size of table ok.
Many thanks!
StartDate=datetime(’01/01/2019 00:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
EndDate=datetime(’31/10/2020 23:00′,’InputFormat’,’dd/MM/yyyy HH:mm’);
DateTime = StartDate:hours(1):EndDate;
DateTime = DateTime’;
datasize =size(DateTime,1);
No_St = 307;

Data = rand(datasize,No_St);

Test_table = array2table(Data);
Test_table.Time = DateTime;
Test_table = movevars(Test_table,"Time","Before",1);

Table_str = string(Test_table{:,2:end}); %convert to string
Table_str = compose(‘%.6f’,Table_str); %Reduce precision compose, string, csv MATLAB Answers — New Questions

​