I am a newbie to data science. I have a 'short text' categorization problem where input variables are either unstructured texts (names, definition, description etc) or categorical. There is not much semantic to the fields as they are product names, territory name, sales order type etc. Issue is I do not have any sample data set from which I can derive training, test, validation set or divided it into k-fold for cross validation. So how should I generate sample data? I have about 20 target classes. I can classify some dataset using regex or lucene rule based matches and manually verify them and make sure each class have equal amount of samples. But I am open to other suggestion.
I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem.
Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone new to data science and machine learning.
Real data will give you experience with less contrived problems and you can often try out similar ML models on very different data sets in order to gain experience with the variations and pitfalls of data science. Be creative... you might not find exactly what you want, but you can subset the data by projecting out a few text fields and using them to classify another field.
With your recent edit, it is now apparent that you are seeking to turn your unsupervised training data into supervised training data in order to train a supervised learning classification model. The method that you suggested, "classify some dataset using regex or lucene rule based matches and manually verify them", is a deterministic unsupervised method without the human verification step.
Without the human verification of the target data that you have created, I would not consider feeding the derived targets back into the classification algortihm as your results will only be as strong as the derived target data and predictions will show similar errors. Instead, you should think about a semi-supervised learning method where you perhaps employ a clustering algorithm and then label the clusters with the target variable.
With human verification of the target data that you have created, this will work just fine as training data for a classification model. The only issue is that this can become tedious. There are mechanical turks (humans paid to perform repetitive tasks) that you can hire to perhaps perform the labeling for you, which may be a more scalable option.
Hope this helps!