0
$\begingroup$

Good evening, I wanted to ask this question about the accent in words or special characters can affect machine learning algorithms. I'm looking to do a job. I would like to know a recommendation for an article or book that I can use as a reference.

$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Sep 13, 2022 at 8:20

2 Answers 2

0
$\begingroup$

The short answer is yes.

The long answer is the following. Machine learning (ML) algorithms are designed to build models from data, but there is a general motto among ML practitioners:

"garbage in, garbage out."

That said, if you have trained your model in an intelligent way to distinguish between strange accents on words, then testing the resulting model on sentences with such accents is a good practice; in particular, I'm thinking of Spanish and Italian words, for instance, where a missing accent could give a completely different semantics to such sentences. Nevertheless, a more practical approach in the field of natural language processing (NLP) is to perform a text normalization (e.g., lemmatization) of your text before learning since specific words are less frequent than more general lemmas. For your specific task, there is a post that has Python code that could remove accents from words if you are willing to drop the take this journey.

I hope that this gives you more insight into the complexity of NLP tasks.

EDIT

There are papers on accent-related researches, such as 1, 2, and 3, among many others.

$\endgroup$
3
  • $\begingroup$ Thank you very much, it helped a lot, I've been trying to research this for a while, but would you know a reference to a scientific article or book that I can use to prove this type of claim that affects the algorithm? $\endgroup$
    – user139815
    Commented Sep 14, 2022 at 0:28
  • $\begingroup$ I don't want to get to withdraw, I want to compare two models one without performing this accent treatment in words and the other with treatment before passing it to a machine learning model. I wanted to see if there was something that could take this as an article and reference it $\endgroup$
    – user139815
    Commented Sep 14, 2022 at 0:31
  • $\begingroup$ thanks, @Oxedu . I will read the articles. From the return, also finding a fragment of the article. Add here in the comments for the next $\endgroup$
    – user139815
    Commented Sep 14, 2022 at 20:27
0
$\begingroup$

Yes, full stop. Characters with accents may look similar when displayed, but as data they're usually different characters, or additional characters combined with the original character. This is not to mention the other ways text can deviate from expectations, such as encoding errors or homoglyphs (can you tell the difference between E and E? because to the computer they're different characters), all of which should be identified and normalized (forced to a standard character/removed/what have you, whatever is most appropriate for your use case) before attempting to process any text.

$\endgroup$
3
  • $\begingroup$ Thank you very much, it helped a lot, I've been trying to research this for a while, but would you know a reference to a scientific article or book that I can use to prove this type of claim that affects the algorithm? $\endgroup$
    – user139815
    Commented Sep 14, 2022 at 0:28
  • $\begingroup$ You're looking for a scientific article that tells you if your data isn't what you think it is, that your algorithms won't work as well? $\endgroup$
    – Andy
    Commented Sep 14, 2022 at 0:49
  • $\begingroup$ yes, because this job and college. My guiding teacher asked for something that stated this, although I had to test it. I have to make reference on this subject. $\endgroup$
    – user139815
    Commented Sep 14, 2022 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.