0
$\begingroup$

My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting on X. How can it do this?

Improve this question
$\endgroup$
8
  • $\begingroup$ Welcome to DataScienceSE. This a complex question. My first idea would be to measure a distance between the word distributions of two individuals, and doing this for many pairs of indvidual A and B. Then a significance test be done between the case where A and B have the same gender and when they don't. $\endgroup$
    – Erwan
    Apr 30, 2022 at 10:36
  • $\begingroup$ Do you mind if I calculate the weights of the words in category X by gender with tf-idf? So, regardless of gender, I create a list where the words in category X are repeated only once and compare it with spss based on the last gender. But the semantic information will be lost :/ Any chance of doing it while preserving the semantics? I thought of LDA topic model: To train and compare 2 different topic models (Women and Man) with 2 (X,Y) topics but I don't know how to do it. $\endgroup$
    – nem0
    Apr 30, 2022 at 11:49
  • $\begingroup$ There are certainly many ways to represent the words and to design a distance/similarity measure which could work. However one important difficulty is to measure the difference between men and women as opposed to between two random persons, i.e. to know if the difference is just part of personal differences or due to gender. If you directly measure all women vs all men, I'm not sure how you would do that: maybe the difference would be the same between two random groups of people. $\endgroup$
    – Erwan
    Apr 30, 2022 at 14:53
  • $\begingroup$ For LDA it's an interesting idea but there's no guarantee that the topics will match gender. It's more likely that they will match the topics being discussed,e.g. sports, news, family,etc. But a similar idea would be to train a classification model to recognize gender: if it works significantly better than random, then it proves that there is a difference. Fyi there's research work in stylometry about this question, it's called gender profiling. $\endgroup$
    – Erwan
    Apr 30, 2022 at 14:59
  • $\begingroup$ Thank you :) Well, if I do not care about individual differences and make direct comparisons between genders, do you think it makes sense to get a word score with TF-IDF? do you have a better suggestion? $\endgroup$
    – nem0
    Apr 30, 2022 at 19:46

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.