Statistics-based outlier detection and correction method for amazon customer reviews

Ishani Chatterjee, Mengchu Zhou, Abdullah Abusorrah, Khaled Sedraoui, Ahmed Alabdulwahab

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

People nowadays use the internet to project their assessments, impressions, ideas, and observations about various subjects or products on numerous social networking sites. These sites serve as a great source to gather data for data analytics, sentiment analysis, natural language processing, etc. Conventionally, the true sentiment of a customer review matches its corresponding star rating. There are exceptions when the star rating of a review is opposite to its true nature. These are labeled as the outliers in a dataset in this work. The state-of-the-art methods for anomaly detection involve manual searching, predefined rules, or traditional machine learning techniques to detect such instances. This paper conducts a sentiment analysis and outlier detection case study for Amazon customer reviews, and it proposes a statistics-based outlier detection and correction method (SODCM), which helps identify such reviews and rectify their star ratings to enhance the performance of a sentiment analysis algorithm without any data loss. This paper focuses on performing SODCM in datasets containing customer reviews of various products, which are (a) scraped from Amazon.com and (b) publicly available. The paper also studies the dataset and concludes the effect of SODCM on the performance of a sentiment analysis algorithm. The results exhibit that SODCM achieves higher accuracy and recall percentage than other state-of-the-art anomaly detection algorithms.

Original languageEnglish (US)
Article number1645
JournalEntropy
Volume23
Issue number12
DOIs
StatePublished - Dec 2021

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Mathematical Physics
  • Physics and Astronomy (miscellaneous)
  • Electrical and Electronic Engineering

Keywords

  • Big data analytics
  • Data scrapping
  • Imbalance dataset
  • Interquartile range
  • J-shaped distribution
  • Natural language processing
  • Outlier detection
  • Sentiment analysis
  • TextBlob

Fingerprint

Dive into the research topics of 'Statistics-based outlier detection and correction method for amazon customer reviews'. Together they form a unique fingerprint.

Cite this