text2emotion

Text Preprocessing with preprocess_text

The preprocess_text function is designed to clean and preprocess raw text data, preparing it for further analysis or modeling. This function includes a variety of stages that handle common text preprocessing tasks such as basic cleaning, contraction expansion, slang handling, and punctuation standardization. Below is a detailed guide on how to use this function and what to consider when applying it to your data.

# Sample text with contractions, slang, and emoticons
text <- "I'm so excited!! 2nite we go 4 a gg :)"

# Preprocess the text
cleaned_text <- preprocess_text(text)

# View the processed text
cleaned_text
#> [1] "i am so excited ! tonight we go for a good game smiley"

# Define custom slang terms
custom_slang <- c(
  "bff" = "best friend forever",
  "omg" = "oh my god"
)

# Preprocess the text with custom slang
text_with_custom_slang <- preprocess_text("omg! My bff is here!", custom_slang = custom_slang)

# View the processed text
text_with_custom_slang
#> [1] "oh my god ! my best friends , forever is here !"

Predicting Emotion with Emoji Representation

The predict_emotion_with_emoji() function allows you to analyze input text and obtain a predicted emotion, its corresponding emoji, or both. It combines preprocessing, TF-IDF feature extraction, and a trained Random Forest classifier to produce interpretable results for sentiment or emotion-aware applications.

predict_emotion_with_emoji("I'm feeling great today!")
#> I'm feeling great today! 😊

predict_emotion_with_emoji("He's super angry!!", output_type = "emoji")
#> 😡

predict_emotion_with_emoji("I feel scared", output_type = "emotion")
#> fear

1. Text Preprocessing

Before modeling, we apply a series of preprocessing techniques to clean and standardize the input texts. This step is essential for improving downstream classification performance. Our preprocessing includes:

• Lowercasing and tokenization: All text is converted to lowercase and tokenized into words.
• Slang and abbreviation expansion: We replace informal terms (e.g., “idk” → “I don’t know”) using a custom slang dictionary.
• Negation handling: Special care is taken to preserve negation cues. If a token like "not" precedes a word, we combine them into a new token (e.g., "not happy" → "not_happy"), ensuring the model learns the distinct semantics of negated expressions.
• Stopword filtering: We use a curated stopword list, but explicitly retain sentiment-related words such as "not", "never", and "no".

The function handle_negation() is responsible for applying this rule-based transformation to token sequences.

2. Feature Engineering: TF-IDF Vectorization

For feature extraction, we utilize Term Frequency-Inverse Document Frequency (TF-IDF) to encode the preprocessed text. The function train_tfidf_model() performs this transformation with the following steps:

• Token filtering: Uninformative tokens are removed based on our custom stopword list.

• Support for unigrams and trigrams: We include both single words and short phrases to better capture emotional context (e.g., "so happy" or "not good").

• Vocabulary pruning: Low-frequency and overly frequent terms are filtered using min_df, max_df, and a maximum feature size.

• Sentiment-priority term boosting: Words known to carry strong emotional weight (e.g., “happy”, “furious”, “disappointed”) are manually prioritized to ensure they are retained in the final vocabulary.

The resulting sparse TF-IDF matrix is used as input for the classification model.

3. Model Selection and Evaluation

We experimented with several classification models for sentiment prediction, including:

• Decision Trees
  
• XGBoost
  
• Random Forests

After extensive evaluation, we found that Random Forest consistently outperformed both Decision Trees and XGBoost in terms of test accuracy and macro F1-score. Therefore, we adopted Random Forest as our final model of choice.

4. Hyperparameter Tuning

We perform a grid search over mtry and ntree to optimize the Random Forest model using the tune_rf_model() function. For each hyperparameter combination, the model is trained and evaluated on the training set, and accuracy is recorded:

• mtry (number of features to consider at each split)

• ntree (number of trees in the forest)

best_params <- tune_rf_model(
    train_matrix = tfidf_result$tfidf_matrix,
    train_labels = train_labels,
    mtry_grid = c(5, 10, 20),
    ntree_grid = c(100, 200, 300),
    seed = 123,
    verbose = TRUE
  )

The best-performing parameter set is chosen based on training accuracy. A caching mechanism is used to store the training dataset and accelerate re-runs.

rf_model <- train_rf_model(
    train_matrix = tfidf_result$tfidf_matrix,
    train_labels = train_labels,
    ntree = best_params$ntree,
    mtry = best_params$mtry,
    seed = 123,
    verbose = TRUE,
    train_df_cache_path = train_df_cache_path
  )

6. Model Evaluation on Test Data

To evaluate generalization, we use evaluate_rf_model() to assess model performance on held-out test data. The function includes:

• Preprocessing and TF-IDF encoding of test texts
• Ensuring vocabulary alignment with the training vectorizer
• Prediction and computation of standard metrics, including:
  • Accuracy
  • Confusion Matrix
  • Macro F1-score

eval_result <- evaluate_rf_model(
    rf_model = rf_model,
    test_texts = preprocessed_test_texts,
    test_labels = test_labels,
    tfidf_model = tfidf_result$tfidf_model,
    vectorizer = tfidf_result$vectorizer,
    stopwords = stopwords,
    verbose = TRUE
  )

eval_result$text_accuracy
eval_result$macro_f1
eval_result$confusion

The accuracy of our Random Forest model remains stable above 65% across different runs. This robust evaluation framework allows us to monitor both overall and per-class performance, which is particularly important in imbalanced sentiment datasets.