Identify and Remove Duplicate Data in R - Datanovia (2024)

111181310101289

• Required packages
• Demo dataset
• Find and drop duplicate elements
• Extract unique elements
• Remove duplicate rows in a data frame
• Summary

Required packages

Load the tidyverse packages, which include dplyr:

library(tidyverse)

Demo dataset

We’ll use the R built-in iris data set, which we start by converting into a tibble data frame (tbl_df) for easier data analysis.

my_data <- as_tibble(iris)my_data

## # A tibble: 150 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 144 more rows

Find and drop duplicate elements

The R function duplicated() returns a logical vector where TRUE specifies which elements of a vector or data frame are duplicates.

Given the following vector:

x <- c(1, 1, 4, 5, 4, 6)

• To find the position of duplicate elements in x, use this:

duplicated(x)

## [1] FALSE TRUE FALSE FALSE TRUE FALSE

• Extract duplicate elements:

x[duplicated(x)]

## [1] 1 4

• If you want to remove duplicated elements, use !duplicated(), where ! is a logical negation:

x[!duplicated(x)]

## [1] 1 4 5 6

• Following this way, you can remove duplicate rows from a data frame based on a column values, as follow:

# Remove duplicates based on Sepal.Width columnsmy_data[!duplicated(my_data$Sepal.Width), ]

## # A tibble: 23 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 17 more rows

! is a logical negation. !duplicated() means that we don’t want duplicate rows.

Extract unique elements

Given the following vector:

x <- c(1, 1, 4, 5, 4, 6)

You can extract unique elements as follow:

unique(x)

## [1] 1 4 5 6

It’s also possible to apply unique() on a data frame, for removing duplicated rows as follow:

unique(my_data)

## # A tibble: 149 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 143 more rows

Remove duplicate rows in a data frame

The function distinct() [dplyr package] can be used to keep only unique/distinct rows from a data frame. If there are duplicate rows, only the first row is preserved. It’s an efficient version of the R base function unique().

Remove duplicate rows based on all columns:

my_data %>% distinct()

## # A tibble: 149 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 143 more rows

Remove duplicate rows based on certain columns (variables):

# Remove duplicated rows based on Sepal.Lengthmy_data %>% distinct(Sepal.Length, .keep_all = TRUE)

## # A tibble: 35 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 29 more rows

# Remove duplicated rows based on # Sepal.Length and Petal.Widthmy_data %>% distinct(Sepal.Length, Petal.Width, .keep_all = TRUE)

## # A tibble: 110 x 5## Sepal.Length Sepal.Width Petal.Length Petal.Width Species## <dbl> <dbl> <dbl> <dbl> <fct> ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa ## # ... with 104 more rows

The option .kep_all is used to keep all variables in the data.

Summary

In this chapter, we describe key functions for identifying and removing duplicate data:

• Remove duplicate rows based on one or more column values: my_data %>% dplyr::distinct(Sepal.Length)
• R base function to extract unique elements from vectors and data frames: unique(my_data)
• R base function to determine duplicate elements: duplicated(my_data)

Coursera - Online Courses and Specialization

Data science

• Course: Machine Learning: Master the Fundamentals by Stanford
• Specialization: Data Science by Johns Hopkins University
• Specialization: Python for Everybody by University of Michigan
• Courses: Build Skills for a Top Job in any Industry by Coursera
• Specialization: Master Machine Learning Fundamentals by University of Washington
• Specialization: Statistics with R by Duke University
• Specialization: Software Development in R by Johns Hopkins University
• Specialization: Genomic Data Science by Johns Hopkins University

Popular Courses Launched in 2020

• Google IT Automation with Python by Google
• AI for Medicine by deeplearning.ai
• Epidemiology in Public Health Practice by Johns Hopkins University
• AWS Fundamentals by Amazon Web Services

Trending Courses

• The Science of Well-Being by Yale University
• Google IT Support Professional by Google
• Python for Everybody by University of Michigan
• IBM Data Science Professional Certificate by IBM
• Business Foundations by University of Pennsylvania
• Introduction to Psychology by Yale University
• Excel Skills for Business by Macquarie University
• Psychological First Aid by Johns Hopkins University
• Graphic Design by Cal Arts

Amazon FBA

Amazing Selling Machine

• Free Training - How to Build a 7-Figure Amazon FBA Business You Can Run 100% From Home and Build Your Dream Life! by ASM

Books - Data Science

Our Books

• Practical Guide to Cluster Analysis in R by A. Kassambara (Datanovia)
• Practical Guide To Principal Component Methods in R by A. Kassambara (Datanovia)
• Machine Learning Essentials: Practical Guide in R by A. Kassambara (Datanovia)
• R Graphics Essentials for Great Data Visualization by A. Kassambara (Datanovia)
• GGPlot2 Essentials for Great Data Visualization in R by A. Kassambara (Datanovia)
• Network Analysis and Visualization in R by A. Kassambara (Datanovia)
• Practical Statistics in R for Comparing Groups: Numerical Variables by A. Kassambara (Datanovia)
• Inter-Rater Reliability Essentials: Practical Guide in R by A. Kassambara (Datanovia)

Others

• R for Data Science: Import, Tidy, Transform, Visualize, and Model Data by Hadley Wickham & Garrett Grolemund
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems by Aurelien Géron
• Practical Statistics for Data Scientists: 50 Essential Concepts by Peter Bruce & Andrew Bruce
• Hands-On Programming with R: Write Your Own Functions And Simulations by Garrett Grolemund & Hadley Wickham
• An Introduction to Statistical Learning: with Applications in R by Gareth James et al.
• Deep Learning with R by François Chollet & J.J. Allaire
• Deep Learning with Python by François Chollet