I thought it was worth a shot, so I decided to embark on this challenge.
AbstractIn order to create a hate-speech-detecting algorithm, we are going to use Python-based NLP machine learning techniques.
Machine learning is basically teaching machines to accomplish various tasks by training them through data.
In this case, we are going to collect data from the Korean radical anti-male website, Womad, but you’re free to use different kinds of data as long as the data is labeled appropriately (more on that later).
Then, using a NLP (or Natural Language Processing) technique called Tf-Idf vectorization, we’ll extract keywords that convey importance within hate speech.
Finally, based on a machine learning technique called logistic regression, which is popular for probability calculations, we’ll train the computer to classify hate speech using the data extracted from Womad (or any kind of data you wish to utilize for training).
Data CollectionFor data collection, we are going to use web crawling, which is a technique for collecting data from web pages in an automated process.
Web crawling in Python could be accomplished by using the beautifulsoup library.
Using this technique, I have collected the comments of popular posts on the Womad website.
Using beautifulsoup, I collected all the texts within those tags and created a hate speech dataset.
In order to prepare the data for artificial intelligence training, I shuffled the dataset with normal sentences (texts that didn’t contain hate speech) and labeled the hate speech comments as 1, and the normal sentences as 0 so the computer could use the data for classification.
Tf-idf VectorizationSo now that we’ve prepared the data, it’s time to convert the texts into numbers so the computer could understand it.
How does a machine read a document?.The same way humans do; by focusing on the words that convey importance.
Similarly, computers extract keywords from documents in order to understand it.
Tf-idf vectorization is suitable for accomplishing such task.
This process is done by assigning Tf-idf scores (which is a numerical element that portrays the importance of a phrase) to each word in a document.
Tf-idf score is calculated as the following formula:The following image is a word cloud visualization of the keywords I’ve extracted from the hate speech data.
As displayed below, it seems that sexual and gender-related Korean insults are quite popular among Womad internet trolls.
Logistic RegressionFinally, it’s time to train our artificial intelligence.
We will use the logistic regression model in order to create a program that could classify hate speech.
Logistic regression model is a model for calculating probabilities between 0 and 1.
I labeled hate speech comments as 1 and normal sentences as 0, and determined the coefficients of the logistic function using the Tf-idf vectors.
As a result, I was able to successfully produce an algorithm that could detect hate speech with approximately 85% precision.
My ThoughtsSo what did I learn from conducting this project?.What are my thoughts?Well, first of all, I was amazed at how precise natural language processing can be.
I believe that this technology could utilized at all sorts of areas; from fake news detection to document topic classification.
On the other hand, this success also made me kind of afraid of artificial intelligence.
Now, I designed this algorithm in order to neutralize the cultures of hate based on online hate speech, but what if men of ill intent, such as dictators or fascists, gets a hold of this technology?.They’d probably use this tech to oppress the minorities and spread fear.
So I guess artificial intelligence, like nuclear energy, could be described as a two-edged sword.
I believe that it is our responsibility to think deeply about how we should develop and utilize this new technology, so that it could be used only for a good cause.
This is the link to my Github page of this project in case you want to try this yourself^^ Thank you for reading~https://github.
com/captainnemo9292/womad_hate_speech_detection_through_machine_learning.