Three language political leaning text classification using natural language processing methods

Abstract

In this article, the problem of political leaning classification of the text resource is solved. First, a detailed analysis of ten studies on the work’s topic was performed in the form of comparative characteristics of the used methodologies. Literary sources were
compared according to the problem-solving methods, the learning that was carried out, the evaluation metrics, and according to the
vectorizations. Thus, it was determined that machine learning algorithms and neural networks, as well as vectorization methods TFIDF and Word2Vec, were most often used to solve the problem. Next, various classification models of whether textual information is
pro-Ukrainian or pro-Russian were built based on a dataset containing messages from social media users about the events of the
large-scale Russian invasion of Ukraine from February 24, 2022. The problem was solved with the help of Support Vector Machines,
Decision Tree, Random Forest, Naïve Bayes classifier, eXtreme Gradient Boosting and Logistic Regression machine learning algorithms, Convolutional Neural Networks, Long short-term memory and BERT neural networks, techniques for working with unbalanced data Random Oversampling, Random Undersampling , SMOTE and SMOTETomek, as well as stacking ensembles of models.
Among the machine learning algorithms, LR performed best, showing a macro F1-score value of 0.7966 when features were transformed by TF-IDF vectorization and 0.7933 when BoW. Among neural networks, the best macro F1-score value of 0.76 was obtained using CNN and LSTM. Applying data balancing techniques failed to improve the results of machine learning algorithms.
Next, ensembles of models from machine learning algorithms were determined. Two of the constructed ensembles achieved the same
macro F1-score value of 0.7966 as with LR. Ensembles that was able to do so consisted of the TF-IDF vectorization, the B-NBC
meta-model, and the SVC, NuSVC LR, and SVC, LR base models, respectively. Thus, three classifiers, the LR machine learning
algorithm and two ensembles of models, which were defined as a combination of existing methods of solving the problem, demonstrated the largest macro F1-score value of 0.7966. The obtained models can be used for a detailed review of various news publications according to the political leaning characteristic, information about which can help people identify being isolated by a filter
bubble.