Removing outliers from the data creates gaps. Filling these gaps with missing values or the median of surrounding values does not address the issue.Why?
I am analyzing EMG data in windows. In each window, I apply z-score normalization to identify and remove outliers. To address the gaps created by removing these outliers, I attempt to fill the empty spaces with the median of the surrounding values. Additionally, I have experimented with MATLAB built-in functions such as ‘movmedian’ for this purpose.
here is my function:
function data_clean = remove_outliers_and_fill(data)
% Calculate z-scores for each column
z_scores = zscore(data);
% Define outlier threshold
threshold =3;
% Identify outliers
outliers = abs(z_scores) > threshold;
% Copy data to preserve original shape
data_clean = data;
% Loop through each column
[num_rows, num_cols] = size(data);
for col = 1:num_cols
for row = 1:num_rows
if outliers(row, col)
range_start = max(1, row-10);
range_end = min(num_rows, row+10);
neighbors = data(range_start:range_end, col);
% Exclude the outlier from median calculation
filtered_neighbors = neighbors(neighbors ~= data(row, col));
median_value = median(filtered_neighbors);
data_clean(row, col) = median_value;
end
end
end
end
here is the plot where it creates gaps after applying the above function.I am analyzing EMG data in windows. In each window, I apply z-score normalization to identify and remove outliers. To address the gaps created by removing these outliers, I attempt to fill the empty spaces with the median of the surrounding values. Additionally, I have experimented with MATLAB built-in functions such as ‘movmedian’ for this purpose.
here is my function:
function data_clean = remove_outliers_and_fill(data)
% Calculate z-scores for each column
z_scores = zscore(data);
% Define outlier threshold
threshold =3;
% Identify outliers
outliers = abs(z_scores) > threshold;
% Copy data to preserve original shape
data_clean = data;
% Loop through each column
[num_rows, num_cols] = size(data);
for col = 1:num_cols
for row = 1:num_rows
if outliers(row, col)
range_start = max(1, row-10);
range_end = min(num_rows, row+10);
neighbors = data(range_start:range_end, col);
% Exclude the outlier from median calculation
filtered_neighbors = neighbors(neighbors ~= data(row, col));
median_value = median(filtered_neighbors);
data_clean(row, col) = median_value;
end
end
end
end
here is the plot where it creates gaps after applying the above function. I am analyzing EMG data in windows. In each window, I apply z-score normalization to identify and remove outliers. To address the gaps created by removing these outliers, I attempt to fill the empty spaces with the median of the surrounding values. Additionally, I have experimented with MATLAB built-in functions such as ‘movmedian’ for this purpose.
here is my function:
function data_clean = remove_outliers_and_fill(data)
% Calculate z-scores for each column
z_scores = zscore(data);
% Define outlier threshold
threshold =3;
% Identify outliers
outliers = abs(z_scores) > threshold;
% Copy data to preserve original shape
data_clean = data;
% Loop through each column
[num_rows, num_cols] = size(data);
for col = 1:num_cols
for row = 1:num_rows
if outliers(row, col)
range_start = max(1, row-10);
range_end = min(num_rows, row+10);
neighbors = data(range_start:range_end, col);
% Exclude the outlier from median calculation
filtered_neighbors = neighbors(neighbors ~= data(row, col));
median_value = median(filtered_neighbors);
data_clean(row, col) = median_value;
end
end
end
end
here is the plot where it creates gaps after applying the above function. outliers, matlab function, emg signal MATLAB Answers — New Questions