That is most likely a bad idea. There are 2^20 possible subsets of 20 elements (1 048 576). To compare them, you would have to run cross validation on all of them, and then do another cross validation of the optimal feature classifier on a third independent validation set, not used to find the optimal parameters. My concern is that this feature selection process will be prone to over fitting. Also, svm classifiers weight the features internally, such that adding features to an otherwise optimal subset would not diminish accuracy, or maybe even slightly increase it due to over fitting.

I would try simpler approaches first, e.g. linear models, or linear or quadratic discriminant analysis. These can also give estimates for the relevance of features.