Hello Everyone!
I am trying to develop a classifier for classification of different human activities i.e., standing, walking, sitting, laying in bed etc.
I have total of 6 classes in my dataset (concatenated all activities) and training multiple classifiers and also implementing K-fold cross validation during training.
I split my data into train and test set, train my classifier and then test my classifier and accuracies are pretty good (almost 95%). (I wont go into details of segmentation, feature extraction etc.)
The problem that I am facing right now is testing my classifier with completely new data. The following are some notable things about the new test data:
1) The new test data has 5 classes instead of 6.
2) The new test has continuous activities e.g., a person performing all 5 activities in quick succession without any break. Whereas, during training data, each activity was seprately recorded and then I concatenated all activities.
3) The accuracy that I am getting is around 70% (pretty bad) and also most of the misclassifications are being thrown into the class that is not in the new testing set e.g., class 6 is not present in new test set but when I check predictions, the classifier is saying (for major portion of activities) that it belongs to class 6.
I cannot reduce the number of classes in my training set because when I implement this in real-time, the subjects will perform all the 6 activities but ofcourse not all together and one by one e.g., in real-time, the subject will perform activity 1 (the classifier will classify it as 1) then activity 2 (the classifier will classify it as 2) and so on.

Can you people help me understand, which parameters I should observe to improve the accuracy of my classifier for the new test data? Because 70% is not acceptable to me. And any suggestions on the mistakes that you have seen here and can correct me and tell me the right path.

Thank you so much :)Hello Everyone!
I am trying to develop a classifier for classification of different human activities i.e., standing, walking, sitting, laying in bed etc.
I have total of 6 classes in my dataset (concatenated all activities) and training multiple classifiers and also implementing K-fold cross validation during training.
I split my data into train and test set, train my classifier and then test my classifier and accuracies are pretty good (almost 95%). (I wont go into details of segmentation, feature extraction etc.)
The problem that I am facing right now is testing my classifier with completely new data. The following are some notable things about the new test data:
1) The new test data has 5 classes instead of 6.
2) The new test has continuous activities e.g., a person performing all 5 activities in quick succession without any break. Whereas, during training data, each activity was seprately recorded and then I concatenated all activities.
3) The accuracy that I am getting is around 70% (pretty bad) and also most of the misclassifications are being thrown into the class that is not in the new testing set e.g., class 6 is not present in new test set but when I check predictions, the classifier is saying (for major portion of activities) that it belongs to class 6.
I cannot reduce the number of classes in my training set because when I implement this in real-time, the subjects will perform all the 6 activities but ofcourse not all together and one by one e.g., in real-time, the subject will perform activity 1 (the classifier will classify it as 1) then activity 2 (the classifier will classify it as 2) and so on.

Can you people help me understand, which parameters I should observe to improve the accuracy of my classifier for the new test data? Because 70% is not acceptable to me. And any suggestions on the mistakes that you have seen here and can correct me and tell me the right path.

Thank you so much 🙂 Hello Everyone!
I am trying to develop a classifier for classification of different human activities i.e., standing, walking, sitting, laying in bed etc.
I have total of 6 classes in my dataset (concatenated all activities) and training multiple classifiers and also implementing K-fold cross validation during training.
I split my data into train and test set, train my classifier and then test my classifier and accuracies are pretty good (almost 95%). (I wont go into details of segmentation, feature extraction etc.)
The problem that I am facing right now is testing my classifier with completely new data. The following are some notable things about the new test data:
1) The new test data has 5 classes instead of 6.
2) The new test has continuous activities e.g., a person performing all 5 activities in quick succession without any break. Whereas, during training data, each activity was seprately recorded and then I concatenated all activities.
3) The accuracy that I am getting is around 70% (pretty bad) and also most of the misclassifications are being thrown into the class that is not in the new testing set e.g., class 6 is not present in new test set but when I check predictions, the classifier is saying (for major portion of activities) that it belongs to class 6.
I cannot reduce the number of classes in my training set because when I implement this in real-time, the subjects will perform all the 6 activities but ofcourse not all together and one by one e.g., in real-time, the subject will perform activity 1 (the classifier will classify it as 1) then activity 2 (the classifier will classify it as 2) and so on.

Can you people help me understand, which parameters I should observe to improve the accuracy of my classifier for the new test data? Because 70% is not acceptable to me. And any suggestions on the mistakes that you have seen here and can correct me and tell me the right path.

Thank you so much 🙂 machine learning, activity recognition, training and testing classes, improve accuracy for real time MATLAB Answers — New Questions

​