First blat splits the reference sequence up into “tiles”. The manner in which it is split depends on two parameters, -tileSize and -stepSize, where -tileSize is the size of the tile and -stepSize specifies when to start the next tile. The default setting of both for DNA sequences is 11, which also means the tiles do not overlap.

For blat to report an alignment, your query sequence must match at least two tiles (set via -minMatch) with no mismatches (you can allow up to one mismatch in the tile by using -oneOff). So if you’re trying to align a 21 bp sequence to a reference using the default setting, blat will never report an alignment.

To illustrate, imagine this reference sequence (44bp):

>database
AAAAAAAAAAACCCCCCCCCCCGGGGGGGGGGGTTTTTTTTTTT

and this query sequence (12bp)

>test
GGGGGGGGGGGT:

The only way an alignment will be reported is if the tileSize is set to 1 and the minScore set to less than 12.

#returns no hit
blat -minScore=0 -stepSize=2 database.fa test.fa output.psl
#returns 2 hits
blat -minScore=0 -stepSize=1 database.fa test.fa output.psl

Here’s an old post showing how the blat parameters affect the output.

#!/bin/bash
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000571/SRR002321.fastq.bz2
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000571/SRR002323.fastq.bz2
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000604/SRR002322.fastq.bz2
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000605/SRR002320.fastq.bz2
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000605/SRR002325.fastq.bz2
wget ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA000/SRA000299/SRX000606/SRR002324.fastq.bz2

Sample summary

To setup TopHat

Download binaries for bowtie2 at http://sourceforge.net/projects/bowtie-bio/files/bowtie2/

Download binaries for tophat2 at http://tophat.cbcb.umd.edu/downloads/tophat-2.0.0.Linux_x86_64.tar.gz

Download test data at http://tophat.cbcb.umd.edu/downloads/test_data.tar.gz

Run a test job with the test_data

tophat -r 20 test_ref reads_1.fq reads_2.fq

The -r parameter

-r/–mate-inner-dist

This is the expected (mean) inner distance between mate pairs. For, example, for paired end runs with fragments selected at 300bp, where each end is 50bp, you should set -r to be 200. There is no default, and this parameter is required for paired end runs.

Interpreting the test results

The reference sequence where the string of A’s serve as introns.

cat test_ref.fa
>test_chromosome
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ACTACTATCTGACTAGACTGGAGGCGCTTGCGACTGAGCTAGGACGTGCC
ACTACGGGGATGACGACTAGGACTACGGACGGACTTAGAGCGTCAGATGC
AGCGACTGGACTATTTAGGACGATCGGACTGAGGAGGGCAGTAGGACGCT
ACGTATTTGGCGCGCGGCGCTACGGCTGAGCGTCGAGCTTGCGATACGCC
GTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAG
ACTATTACTTTATTATCTTACTCGGACGTAGACGGATCGGCAACGGGACT
GTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAG
TTTTCTACTTGAGACTGGGATCGAGGCGGACTTTTTAGGACGGGACTTGC
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

There are 100 paired end reads in reads_1.fq and reads_2.fq.

head -4 reads_1.fq
@test_mRNA_150_290_0/1
TCCTAAAAAGTCCGCCTCGGTCTCAGTCTCAAGTAGAAAAAGTCCCGTTGGCGATCCGTCTACGTCCGAGTAAGA
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

head -4 reads_2.fq
@test_mRNA_150_290_0/2
TACGTATTTGTCGCGCGGCCCTACGGCTGAGCGTCGAGCTTGCGATCCGCCACTATTACTTTATTATCTTACTCG
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

If TopHat ran properly, your output would look something like this using IGV and viewing the accepted_hits.bam file:

Continuing with the analysis

Build index or download from ftp://ftp.cbcb.umd.edu/pub/data/bowtie_indexes/hg19.ebwt.zip

bowtie2-build /path/to/hg19 hg19

Running TopHat

tophat /path/to/hg19 reads1.fq,reads2.fq,reads3.fq

To be updated; currently indexing hg19.

cat /etc/lsb-release
#DISTRIB_ID=Ubuntu
#DISTRIB_RELEASE=12.04
#DISTRIB_CODENAME=precise
#DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"
wget http://circos.ca/distribution/circos-0.60.tgz
sudo cpan App::cpanminus
#change directory into where you unzipped circos and then the bin directory
cd circos-0.60/bin
test.modules > toget
cat toget | grep -v "^ok"
#fail Config::General is not usable (it or a sub-module is missing)
#fail Font::TTF::Font is not usable (it or a sub-module is missing)
#fail GD is not usable (it or a sub-module is missing)
#fail GD::Image is not usable (it or a sub-module is missing)
#fail GD::Polyline is not usable (it or a sub-module is missing)
#fail Math::Bezier is not usable (it or a sub-module is missing)
#fail Math::VecStat is not usable (it or a sub-module is missing)
#fail Readonly is not usable (it or a sub-module is missing)
#fail Regexp::Common is not usable (it or a sub-module is missing)
#fail Set::IntSpan is not usable (it or a sub-module is missing)
#fail Text::Format is not usable (it or a sub-module is missing)

sudo cpanm Config::General
sudo cpanm Font::TTF::Font
sudo cpanm GD
sudo cpanm GD::Image
sudo cpanm GD::Polyline
sudo cpanm Math::Bezier
sudo cpanm Math::VecStat
sudo cpanm Readonly
sudo cpanm Regexp::Common
sudo cpanm Set::IntSpan
sudo cpanm Text::Format

test.modules  | grep ^fail
fail GD is not usable (it or a sub-module is missing)
fail GD::Image is not usable (it or a sub-module is missing)
fail GD::Polyline is not usable (it or a sub-module is missing)

sudo apt-get -y install libgd2-xpm-dev build-essential

sudo cpanm GD
sudo cpanm GD::Image
sudo cpanm GD::Polyline

#all modules should be installed now
test.modules | grep "^fail"

#test GD
gddiag
#see if gddiag.png looks the same as the image at http://www.circos.ca/tutorials/lessons/configuration/png_output/images

#But I was still getting an error about error/configuration.missing.txt
circos 

#debuggroup conf 0.09s welcome to circos v0.60 4 May 2012
#debuggroup conf 0.09s guessing configuration file
#
#  *** CIRCOS ERROR ***
#
#  CONFIGURATION FILE ERROR

#  ...error text from [error/configuration.missing.txt] could not be read...

#  If you are having trouble debugging this error, use this tutorial to learn how
#  to use the debugging facility

#      http://www.circos.ca/tutorials/lessons/configuration/debugging

#  If you're still stumped, get support in the Circos Google Group

#      http://groups.google.com/group/circos-data-visualization

#  Stack trace:
# at /home/tan118/src/circos-0.60/bin/../lib/Circos/Error.pm line 325
#	Circos::Error::fatal_error('configuration', 'missing') called at /home/tan118/src/circos-0.60/bin/../lib/#Circos.pm line 152
#	Circos::run('Circos') called at bin/circos line 232

#I wrote to the author of Circos and turns out that
#the configuration.missing.txt file is missing in the tarball
#download the file and put it inside the error directory
wget http://davetang.org/file/configuration.missing.txt
mv configuration.missing.txt error

#In retrospect, the missing file above wouldn't have been a problem
#i.e. you could still run Circos if you gave it the right parameters
#However I wasn't sure what the error message:
#...error text from [error/configuration.missing.txt] could not be read...
#meant

For loops

In R:

for (x in c(0:9))
   print(x)

In Python (indentation is required as part of the language):

#!/usr/bin/python
for num in range(0, 10):
   print num

In Perl:

#!/usr/bin/perl
for (my $i = 0; $i < 10; ++$i){
   print "$i\n";
}

While loops

In R:

n <- 10
i <- 1
while(i <= n) {
   print(i)
   i <- i + 1
}

In Python

#!/usr/bin/python

count = 0
while (count < 9):
   print count
   count = count + 1

In Perl:

#!/usr/bin/perl
my $i = 0;
while ($i < 10){
   print "$i\n";
   ++$i;
}

Arrays

In Python and more information here:

#!/usr/bin/python

from array import *
a=array('i',[1,2,3,4,5])
for i in a:
   print(i)

Will be continually updated

For example, I want to make a bed file of SNPs and load them as a custom track. When displayed on the Genome Browser, I want to directly go to the dbSNP website by clicking on the feature. All you need to do is add this line into your custom bed file:

track name=”dbSNPs” description=”SNPs from dbSNP” url=”http://www.ncbi.nlm.nih.gov/snp/?term=$$”

Then when you click on the SNP, you should see a page with an “Outside Link”.

library("biomaRt")
listMarts()
ensembl=useMart("ensembl")
listDatasets(ensembl)
ensembl = useMart("ensembl",dataset="hsapiens_gene_ensembl")
#building a query, requires filters, attributes and values
#listFilters shows all filters
filters = listFilters(ensembl)
#listAttributes shows all attributes
attributes = listAttributes(ensembl)
#the getBM function is the main query function in biomaRt
#as an example, let's convert affy ids into Entrez gene names
affyids=c("202763_at","209310_s_at","207500_at")
getBM(attributes=c('affy_hg_u133_plus_2', 'entrezgene'), filters = 'affy_hg_u133_plus_2', values = affyids, mart = ensembl)

The vignette contains other cool examples, which you should look at if you are interested in using biomaRt.

I will try to build a query that converts a human RefSeq id into the Entrez id

library("biomaRt")
ensembl = useMart("ensembl",dataset="hsapiens_gene_ensembl")
filters = listFilters(ensembl)
#look for filters with RefSeq
grep("refseq", filters$name, ignore.case=T, value=T)
# [1] "with_refseq_peptide_predicted"  "with_ox_refseq_genomic"         "with_ox_refseq_mrna"            #"with_ox_refseq_mrna_predicted"
# [5] "with_ox_refseq_ncrna"           "with_ox_refseq_ncrna_predicted" "refseq_mrna"                    #"refseq_mrna_predicted"
# [9] "refseq_ncrna"                   "refseq_ncrna_predicted"         "refseq_peptide"                 #"refseq_peptide_predicted"
#[13] "refseq_genomic"
attributes = listAttributes(ensembl)

#RefSeq for beta actin
my_refseq <- 'NM_001101'
getBM(attributes='ensembl_gene_id', filters = 'refseq_mrna', values = my_refseq , mart = ensembl)
#  ensembl_gene_id
#1 ENSG00000075624
getBM(attributes=c('ensembl_gene_id','description'), filters = 'refseq_mrna', values = my_refseq , mart = ensembl)
#  ensembl_gene_id                              description
#1 ENSG00000075624 actin, beta [Source:HGNC Symbol;Acc:132]

And lastly an example taken from http://www.stat.berkeley.edu/~sandrine/Teaching/PH292.S10/Durinck.pdf:

snp <- useMart("snp",dataset="hsapiens_snp")
out=getBM(attributes=c("refsnp_id","allele","chrom_start"), filters=c("chr_name","chrom_start","chrom_end"), values=list(8,148350, 158612), mart=snp)
nrow(out)
#[1] 465
head(out)
#   refsnp_id allele chrom_start
#1 rs78403279    C/A      148354
#2  rs4057463    T/G      148382
#3  rs3869584    T/C      148423
#4  rs4066939    T/C      148423
#5 rs71213765    -/T      148473
#6  rs3870584    G/A      148523

a       b       c       d       e
10      20      30      20      10

x <- read.table("some.file")
#the data needs to be in matrix format for barplot()
barplot(as.matrix(x[2,]))
#to label the x-axis
barplot(as.matrix(x[2,]),names.arg=as.matrix(x[1,]))
#to label the x-axis using horizontal labels
barplot(as.matrix(x[2,]),names.arg=as.matrix(x[1,]),las=2)
#export as postscript
postscript(file="some_file.ps")
barplot(as.matrix(x[2,]),names.arg=as.matrix(x[1,]),las=2)
dev.off()

x <- rpois(n=1000,lambda=100)
y <- rpois(n=1000,lambda=100)
ks.test(x,y)

#        Two-sample Kolmogorov-Smirnov test
#
#data:  x and y
#D = 0.028, p-value = 0.828
#alternative hypothesis: two-sided
#
#Warning message:
#In ks.test(x, y) : p-values will be approximate in the presence of ties

We cannot reject the null hypothesis that the two distributions are different, which in this case they aren’t i.e. they are both from a Poisson distribution with the same lambda.

x <- rpois(n=1000,lambda=100)
y <- rnorm(n=1000,mean=100)
ks.test(x,y)

#        Two-sample Kolmogorov-Smirnov test

#data:  x and y
#D = 1, p-value < 2.2e-16
#alternative hypothesis: two-sided

#Warning message:
#In ks.test(x, y) : p-values will be approximate in the presence of ties

In this case we get an extremely low p-value, and we can reject the null, which is that both the distributions are the same and they are not (one is a Normal distribution and the other a Poisson distribution).

The warning messages are due to the implementation of the KS test in R, which expects a continuous distribution and thus there should not be any identical values in the two datasets i.e. ties. I’ve read several sources and they all mention that the KS test can deal with both discrete and continuous data (I’m guessing because it mainly deals with cumulative quantiles) but I’m not sure about the implementation in R.

Related http://www.physics.csbsju.edu/stats/KS-test.html

You can follow me on Twitter for a list of facts on molecular biology and on bioinformatics that I didn’t know or have forgotten about over the years.

data <- read.table("pnas_expression_filtered.tsv",header=T,row.names=1)
mean <- ''
for (i in 1:nrow(data)){
   mean[i] <- log2(mean(c(data$lane1[i],data$lane2[i],data$lane3[i],data$lane4[i])))
}

var <- ''
for (i in 1:nrow(data)){
   var[i] <- log2(var(c(data$lane1[i],data$lane2[i],data$lane3[i],data$lane4[i])))
}

plot(var,mean)

library("edgeR")
d <- data[,1:4]
group <- c(rep("Control",4))
d <- DGEList(counts = d, group=group)
d <- calcNormFactors(d)
d <- estimateCommonDisp(d)
d$common.dispersion
#[1] 0.01897251
sqrt(d$common.dispersion)
#[1] 0.137740
getPriorN(d)
d <- estimateTagwiseDisp(d, prop.used=0.5, grid.length=500)
summary(d$tagwise.dispersion)
#    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
#0.008581 0.017790 0.034540 0.052080 0.070670 0.640000

Generate some random data from Poisson distribution

data <- matrix(rep("0",15000*6),ncol=6,nrow=15000)
for (i in c(1:15000)){
   mean <- sample(x=100000,size=1);
   blah <- c(rpois(n=6,lambda=mean));
   for (j in c(1:6)){
      data[i,j] <- blah[j]
   }
}
head(data)
#     [,1]    [,2]    [,3]    [,4]    [,5]    [,6]
#[1,] "90542" "90596" "90713" "90803" "90454" "90783"
#[2,] "78677" "78763" "78622" "78126" "78575" "78706"
#[3,] "67682" "67740" "67558" "67064" "67567" "67649"
#[4,] "84725" "85171" "84971" "84136" "84704" "84885"
#[5,] "22303" "22338" "22320" "22089" "22247" "22465"
#[6,] "58768" "58938" "59048" "58856" "58958" "58651"
data2 <- matrix(as.numeric(data),ncol=6,nrow=15000)
library("edgeR")
d <- data2[,1:6]
group <- c(rep("Control",6))
d <- DGEList(counts = d, group=group)
d <- calcNormFactors(d)
d <- estimateCommonDisp(d)
d$common.dispersion
#[1] 0.0001005378
sqrt(d$common.dispersion)
#[1] 0.01002685
getPriorN(d)
d <- estimateTagwiseDisp(d, prop.used=0.5, grid.length=500)
summary(d$tagwise.dispersion)
#     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
#1.571e-06 1.571e-06 1.571e-06 2.225e-06 1.571e-06 8.297e-05

mean2 <- ''
for (i in 1:nrow(data2)){
   mean2[i] <- log2(mean(data2[i,]))
}

var2 <- ''
for (i in 1:nrow(data2)){
   var2[i] <- log2(var(data2[i,]))
}

cor(as.numeric(var2),as.numeric(mean2))
#[1] 0.821295
cor(as.numeric(var2),as.numeric(mean2),method="spearman")
#[1] 0.7203007

plot(var2,mean2)