Chapter 4: Data Manipulation

Preamble

Learning Outcomes

professor darter iconThe objective of this chapter is to introduce you to the options available for manipulating your data after you have loaded them into dartRverse.

After completing this chapter, you should have a basic understanding of the fundamentals of the dartRverse functions for deleting populations, individuals and loci, recoding populations and individuals, assigning individuals to populations, merging populations, renaming populations, and other routine tasks.

Most importantly, you should be empowered to expand your knowledge beyond the basics presented here by exploring the help on these topics and associated scripts, and by applying the knowledge in other tutorials to come.

Prerequisites

r studio logo

RStudio is an integrated development environment (IDE) with a graphical user interface (GUI) used to manage R workflow and so to work through this chapter, you will need to have already completed Chapter 1. RStudio Refresher. For more advanced treatment refer to the official documentation.

a bird running

 

You should have completed Chapter 3 on data entry and data structures before attempting this chapter.

 

a bird listening

You may also find it helpful to listen to the podcast summary of the overview material in this chapter.

Mastering Data Input and Storage (mp3)

Workflow

a bird runningWe assume that you have established the RStudio project eBook1 and downloaded the datasets required for all chapters. Package eBook_Startup has been installed. All ready to go.

 

The workflow from this point again follows a formula.

  • Theory, unencumbered by competing options and technical asides;
  • Worked Examples where you are led through analyses with sample code;
  • Exercises where you are left on your own to integrate the knowledge you have gained in a problem-solving context.
  • Review the learning outcomes and the section on Where have we come? to confirm that you are moving to the next chapter with all that is required under your belt.

The only difference with Chapter 4 is that it cycles through these elements twice, once in a session on dartR reports and once in a session on manipulating data.

Accessing Worked Examples and Exercises

We covered how to access the worked examples and exercises in Chapter 1. Briefly, if you have not done so already:

 

source("http://georges.biomatix.org/storage/app/media/eBook1/eBook_data_download.txt")
install.packages("dartRverse")
library(dartRverse)
gl.set.verbosity(3)
install.packages(c("remotes", "learner")
remotes::install_github("Biomatix/eBook1")
library(eBook1)

Session 4-1: Interrogating a genlight object

Revision

professor darter icon

 

You can interrogate the genlight object using commands built into the {adegenet} package.

nLoc() number of loci
locNames() list of loci
nInd() number of individuals (specimens or samples)
indNames() list of individuals
nPop() number of populations
popNames() list of populations
pop() list of population assignments for each individual
table(pop()) generate a table of sample size by population
m <- as.matrix() generate a matrix of the SNP scores, with 0 as homozygous reference, 2 as homozygous alternate, and 1 as heterozygous.

These are all useful for interrogating your genlight object and, of course, can be used in your R script statements to subset and manipulate your data.

Report Functions

professor darter iconWe covered some of the above options for interrogating a SNP dataset in dartRverse in Chapter 3. In this Chapter 4, we will carry this beyond the basic interrogation with genlight accessors to introduce some of the report functions in dartRverse.

These report functions are particularly important because they inform analyses down the track, particularly the filtering functions described in Chapter 5. Indeed, they come in pairs, each report function having a corresponding function that applies a filter. The report function provides the information necessary for a judgement on the threshold to use for filtering.

The report functions typically apply to both SNP and SilicoDArT datasets. The common report functions that are matched with filtering functions are listed below.

gl.report.callrate() summarises CallRate values
gl.report.reproducibility() summarises repAvg (SNP) or reproducibility (SilicoDArT) values
gl.report.monomorphs() provides a count of polymorphic and monomorphic loci
gl.report.secondaries() provides a count of loci that are secondaries, that is, loci that reside on the one sequence tag.
gl.report.rdepth() reports the estimate of average read depth for each locus
gl.report.hamming() reports on Hamming distances between sequence tags
gl.report.overshoot() reports loci for which the SNP has been trimmed along with the adaptor sequence
gl.report.taglength() reports a frequency tabulation of sequence tag lengths
gl.report.sexlinked() reports the number and type of loci likely to be in sex chromosomes

A full list of reporting functions can be obtained by typing

??gl.report

and examining the pulldown menu.

Example: Reporting on Call Rate

As an example, let us consider reporting on call rate across loci. Recall that call rate is the rate at which individuals have been successfully called for a locus, or that loci have been called for an individual depending on the context.

 

gl.report.callrate(testset.gl)

Starting gl.report.callrate

Processing SNP data

Reporting Call Rate by Locus

No. of loci = 255

No. of individuals = 250

Minimum : 0.401932

1st quantile : 0.7879225

Median : 0.971014

Mean : 0.8634915

3rd quantile : 0.993237

Maximum : 1

Missing Rate Overall: 0.12

Quantile Threshold Retained Percent Filtered Percent
1 100%  1.0000000  7  2.7 248  97.3
2 95% 0.9990340 19 7.5 236  92.5
3 90% 0.9971010 34  13.3 221  86.7
4 85% 0.9961350 44 17.3 211  82.7
5 80% 0.9951690 53 20.8 202  79.2
6 75% 0.9932370 67 26.3 188 73.7
7 70% 0.9911108 77 30.2 178  69.8
8 65% 0.9884060 91 35.7 164 64.3
9 60% 0.9855070 103 40.4 152 59.6
10 55% 0.9803862 115  45.1  140  54.9
11 50% 0.9710140 128  50.2  127  49.8
12 45% 0.9577778  140  54.9  115 45.1
13 40% 0.9389374 153  60.0  102  40.0
14 35% 0.9021252 166 65.1 89 34.9
15 30% 0.8606764 178 69.8 77  30.2
16 25% 0.7879225  191  74.9  64  25.1
17 20% 0.6672464 204 80.0 51 20.0
18 15% 0.6039616 216 84.7 39 15.3
19 10% 0.5151690 229  89.8 26  10.2
20 5% 0.4506281 242  94.9 13 5.1
21 0% 0.4019320 255 100.0 0 0.0

Saving the ggplot to session tempfile

Saving tabulation to session tempfile

Completed: gl.report.callrate

 

The report is fairly self-evident. The table provides you with a basis for deciding an appropriate Call Rate to use as a threshold for filtering loci. For example, if you insist on retaining only those loci that have successfully been called across all individuals (setting threshold=1.0), you will retain only 7 loci with the loss of 97.3% of scored loci. That is way too stringent. However, a threshold of 0.95 might be appropriate, with 54.9% of loci retained.

The plot can make the decision simpler (Figure 4-1).

Figure 4-1. Standard output of the report callrate function showing the distribution of successful calls across loci. This enables an assessment of the quality of the run

Each report function prints out a table of thresholds with the accompanying losses should that threshold be used to filter and a frequency distribution plot.

The report functions provide quality estimates for your dataset and allow you to determine whether remedial action is required.

Smear plot

professor darter iconAn alternative way of visualizing call rate across individuals and loci is to examine a smear plot. In the plot generated by gl.smearplot(testset.gl), SNP loci are shown along the x-axis and individuals/specimens along the y-axis. Missing data are shown as white, so it can be seen that some loci (vertical white stripes) have exceptionally low call rates, and some individuals have a poor call rate for some loci (horizontal white bands). One would typically examine a smear plot before and after any manipulations to evaluate the impact of remedial action (Figure 4-2).

Figure 4-2. A smear plot showing the SNP state for loci (x axis) and individuals (y axis). A key consideration is the pattern of missing values (NA) which provides an indication of where remedial action might be required.

 

Worked Example: Basic Attributes and QC

a bird running

This worked example uses the earless dragon dataset we input and then saved in Chapter 3. In this example, we illustrate the application of functions to provide a basic summary of the contents of the dataset, application of the standard adegenet accessors, and application of the dartRverse report functions.

Open the RStudio project you set up in Chapter 3, namely eBook1.

Set the global verbosity to 3 so you get to see the full progress log when you submit commands.

You will be asked to load the earless dragon dataset that was created in Chapter 3, namely tympo_SNP_raw.RData.

To access the Worked Examples, navigate to the RStudio Tutorial Pane and then open the session eBook.

Undertake Worked Example 4-1.

Exercises

a bird exercising

Now is a good time to try some exercises and to try out what you have learned so far on your own data. These exercises ask you to analyse a small SNP dataset and a small SilicoDArT dataset to see what you have learned so far.

 

Navigate to the RStudio Tutorial Pane and then open the session eBook. Navigate to the Exercises for Chapter 4. Undertake the first two exercises.

Where have we come?

a bird celebrating

In this Session, you have learnt the various ways to interrogate a genlight object, namely by:

  • simply typing its name
  • examining the locus and individual metrics
  • using adegenet accessors
  • using the dartRverse report functions

These techniques are important for familiarising yourself with your dataset at the most fundamental level, for assessing the quality of the SNP and SilicoDArT calls and for detecting any errors or outliers likely to cause problems in later analyses.

Session 4-2. Data Manipulation

Overview

professor darter iconAlmost certainly you will need to undertake various manipulations of your data prior to analysis. This might include altering or correcting some individual or population labels, removing some individuals included in error or that are perhaps duplicated, and recalculating the locus metrics to include new metrics not provided by Diversity Arrays Technology.

During the course of analysis as well, you will need to amalgamate individuals from sampling sites, delete populations (say the outgroup taxa for an analysis directed at the ingroup taxa alone), or examine statistics from one population on its own.

An early cautionary note

While it is possible to manipulate the content of genlight objects directly, this can lead to serious miscalculations. For example, deleting loci directly with

index <- glNA(gl, alleleAsUnit=FALSE)<= 0.05*nInd(gl)

gl <- gl[, index]

is fine, provided you remember to apply the restriction to the locus metrics as well. Otherwise, the locus metrics will become out of sync with the loci, which is potentially disastrous.

gl@other$loc.metrics <- gl@other$loc.metrics[index,]

Even subsetting is potentially problematic. For example

gl <- gl[,1:100]

will generate a genlight object containing only the first 100 loci, but the locus metrics will retain data for all the original loci. This is a feature not included in adegenet that we have attempted to redress in the definition of a dartR object. A dartR object is a superset of the adegenet genlight object. Nevertheless, it is prudent not to bypass the dartRverse scripts and try to manipulate the genlight objects directly in R.

So even if you are well-versed in R programming, and tempted to take shortcuts, they are best avoided.

Instead, there are functions in dartRverse to assist you in manipulating a genlight object. It is safest to use them.

Removing, Renaming and Creating Populations

Dropping populations

professor darter icon

The following set of functions applies to populations.

gl <- gl.drop.pop() remove listed populations

gl <- gl.keep.pop() keep only the listed populations

Merging and renaming a population

You can merge two populations or rename a population, using

gl <- gl.merge.pop() merge two populations under a new name, or if applied to one population, to rename it.

gl <- gl.rename.pop() rename a population.

Reassigning a population using an individual metric

It may also be convenient to permanently assign one of your individual metrics as the population variable. Some analysts have a series of different population breakdowns in the individual metrics and move from one to the other during analysis. For example, the popNames(gl) might be based on sample sites, but another metric might be region. The region can be assigned as the pop variable using

gl <- gl.reassign.pop(gl, as.pop="region")

This will overwrite the existing population assignments with values of the individual metric region.

Some functions allow the temporary assignment of an individual metric as the population attribute. For example,

gl <- gl.drop.pop(gl, pop.list="NSW", as.pop="region")

will use region as the population for the purposes of deleting populations (e.g. NSW), then revert to the initial population assignments. If all individuals in an existing population are reassigned, the original population will be dropped.

Bulk population reassignment or deletion [Recode Tables]

An alternative way to manipulate population assignments is to use a recode table in the form of a comma-delimited .csv file. This is efficient if you wish to make lot of changes.

With a recode table you can rename, merge and delete populations. The recode table is a .csv file that can be set aside as part of your preliminary workflow to ensure reproducibility of your initial manipulations.

Note that populations or individuals assigned the label ‘Delete’ will be removed from the genlight object when the recode table is applied.

gl.make.recode.pop() make a recode table based on existing population labels. You will need to edit the second column of the recode table to specify the new labels to apply.
gl <- gl.recode.pop() apply the specified pop.recode table to the populations

 An interactive method of effecting population changes can be achieved with

gl <- gl.edit.recode.pop() edit population assignments, and apply the changes on closure

Subsampling populations

There may be occasions when you want to subsample populations in a genlight object containing SNP or SilicoDArT data. This can be done, with or without replacement, using

gl2 <- gl.subsample.pop (gl, n=200, replacement=FALSE)

This option is particularly useful when used in combination with population assignment, for example:

 

lat <- as.character(gl@other$ind.metrics$lat)
long <- as.character(longitude)
latlon <- paste0(lat,long)
gl2 <- gl.subsample.pop(gl, n=100, as.pop=latlon, replacement=FALSE)

 

Object gl2 will contain a subsample of 100 randomly-selected locations in the dataset.

Removing and Renaming Individuals

Keeping and dropping individuals

professor darter icon

The following set of functions applies to populations and individuals.

gl <- gl.drop.ind() remove listed individuals
gl <- gl.keep.ind() keep only the listed individuals

Bulk removal of individuals using an individual metric

You can select individuals to be kept or dropped based on one of the individual metrics using:

gl <- gl.keep.pop(gl,"Male", as.pop="sex")

where sex is one of the variables among the individual metrics in gl@other$ind.metrics, and where you want to retain only the males. This parameter works with gl.drop.pop also.

Reassigning individuals to a new population

You can define a new population for a set of listed individuals, merge two populations, using:

gl <- gl.define.pop() create a new population for listed individuals; their old population assignments will be discarded.

Bulk individual reassignment or deletion [Recode Tables]

An alternative way to manipulate individual labels is to use a recode table in the form of a comma-delimited .csv file. This is efficient if you have a lot of changes, as might occur when the genotype names submitted to the sequence provider are different or abbreviated labels for the individuals.

The recode table is a .csv file that can be set aside as part of your preliminary workflow to ensure reproducibility of your initial manipulations.

Note that individuals assigned the label ‘Delete’ will be removed from the genlight object when the recode table is applied.

gl.make.recode.ind() make a recode table based on existing individual labels. You will need to edit the second column of the recode table to specify the new labels to apply. Individuals assigned the new label ‘Delete’ will be removed from the genlight object.
gl <- gl.recode.ind() apply the specified ind.recode table to the individuals
gl <- gl.edit.recode.ind() edit individual assignments, and apply the changes on closure

Subsampling individuals

There may be occasions when you want to subsample individuals in a genlight object containing SNP or SilicoDArT data. This can be done, with or without replacement, using:

gl2 <- gl.subsample.ind(gl, n=200, replacement=FALSE)

Removing and Renaming Loci

Keeping and dropping loci

professor darter icon

The following set of functions applies to populations and loci.

gl <- gl.drop.loc() remove listed loci
gl <- gl.keep.loc() keep only the listed loci

Bulk locus reassignment or deletion [Recode Tables]

An alternative way to manipulate individual labels is to use a recode table in the form of a comma-delimited .csv file. This is efficient if you have a lot of changes, as might occur when the genotype names submitted to the sequence provider are different or abbreviated labels for the individuals.

The recode table is a .csv file that can be set aside as part of your preliminary workflow to ensure reproducibility of your initial manipulations.

Note that individuals assigned the label ‘Delete’ will be removed from the genlight object when the recode table is applied.

gl.make.recode.loc() make a recode table based on existing locus labels. You will need to edit the second column of the recode table to specify the new labels to apply. Loci assigned the new label ‘Delete’ will be removed from the genlight object.
gl <- gl.recode.loc() apply the specified loc.recode table to the loci.

Subsampling loci

There may be occasions when you want to subsample loci in a genlight object containing SNP or SilicoDArT data. This can be done, with or without replacement, using

gl2 <- gl.subsample.loci(gl, n=200, replacement=FALSE)

Recalculating locus metrics

professor darter iconSeveral scripts delete individuals, and so many of the metadata variables provided by Diversity Arrays Technology or your service provider no longer apply. For example, deleting individuals with a low call rate across loci will alter the call rate for each locus and so the CallRate for loci provided initially by DArT or your service provider will be incorrect in the new genlight object.

There are parameters to set in each of the relevant dartRverse scripts to recalculate the locus metadata. Flags are set as to whether the locus data are no longer correct, and action taken to update them when needed. So dartRverse looks after this behind the scenes.

However, if you are going to access the locus metadata directly in your workflows, be sure to do your manipulations and then run

gl <- gl.recalc.metrics(gl)

Some functions will generate monomorphic loci. These are not deleted automatically because the need to do so depends on subsequent analyses. Instead a warning is issued. You can then choose to delete these monomorphic loci using

gl <- gl.filter.monomorphs(gl)

Refer to Chapter 5 for further details.

Worked Example: Manipulating data

a bird running

This worked example will run through the application of functions used to manipulate data prior to more serious analyses. The genlight object contains data from a real-world example, the earless grassland dragon.

 

To access the Worked Examples, navigate to the RStudio Tutorial Pane and then open the session eBook.

You will be asked again to load the earless dragon dataset that was created in Chapter 3, namely tympo_SNP_raw.RData.

Undertake Worked Example 4-2.

Exercises

Now is a good time to try some more of what you have learned so far on some exercises and your own data. These exercises ask you to analyse a small SNP dataset and a small SilicoDArT dataset to see what you have learned so far.

The Exercises are also found by navigating to the RStudio Tutorial Pane and then opening the session eBook. Navigate to the Exercises for Chapter 4.

Undertake the remaining exercises under Chapter 4.

Winding up

Where have we come?

a bird celebrating

The above Session was designed to give you an overview of the scripts in dartRverse for undertaking basic manipulations of your data.

 

Having completed this Session, you should now be able to:

  • Visualise your data in a smear plot to assess structure and frequency of null alleles.
  • Delete individuals and populations, rename individuals and populations, merge populations, and assign individuals to new populations.
  • Recalculate locus metrics after deleting individuals or populations.

Ende

Licence

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Getting started with dartRverse Copyright © by University of Canberra is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.