Detect sensitive data from unstructured text documents

Question

I know this question is broad, but I need an advice to know if it's possible to achieve what I want to do.

The problem is that I have around 2500 documents with sensitive data being replaced by four dots. I do not have the original documents, so I wonder if there is a way to build a model that can detect sensitive data from any new documents (with sensitive data not being removed) using the previous documents? I want to apply machine learning or deep learning approaches. And what I know is that the original data set with annotated sensitive data should be used for training, which I can't obtain.

I am new at this field so any advice would be very appropriated

Answer 1

Welcome to the site! Assuming that I understand your problem correctly, I think you can achieve a working model.

If I was in your position I would:

1. Obtain the cleanest data possible from the documents. For example, you don't state if the docs are already in simple text or if you need to do something like OCR or whatnot. Having the cleanest set possible will be key for this.
2. Make sure you have a consistent marker for the sensitive data. You mention four dots - is that the case for ALL instances? If not, clean that data now
3. You're going to need to do standard NLP cleansing stuff like removing punctuation but you may or may not want to keep stop words (this will be part of your model testing). Also, this is key, be 100% certain that the four dots are viewed as a single work in your tokenization process - you should be able to verify this prior to committing to your tokenization file.
4. I would take all my documents and create 3 word ngrams. I would then separate out ngrams that contain sensitive data and not sensitive data. That, essentially, becomes your labeled dataset and you should label them accordingly.
5. My base model would use all entries that contain sensitive data in the second position of the ngram (the middle of the three words). I would train a neural network on that and see what kind of results I achieve with that. NOTE that your four dots will not be an input, only the word previous and after will be your inputs. You could almost treat this as a binary classification model - the middle word is either sensitive or it's not.
6. Future iterations of my model would maybe use a multi-classification approach with something like (1) No sensitive data (2) sensitive data in first position (3) sensitive data in second position and (3) sensitive data in third position and so on and so on.
7. From there, you can play with variations on the size of the ngram since the immediate words may or may not actually have an effect on the predictions. There's no limit to how crazy you can get with this - you won't know until you start modeling.

Finally, your entire project becomes even more interesting when you go to the prediction phase with new data. You will do the same and break down your document into ngrams and create a prediction for each one and output the result. In other words, you will need to break down your document only to turn around and build it up again - that should be a fun script to write! Good luck with this, let us know how it turns out.