Skip to content
amcphersonPublic

About

Clone-specific genomic structure estimation in cancer

Resources

Stars

8 stars

Watchers

4 watching

Forks

Latest commit

 

History

861 Commits

Folders and files

Repository files navigation

ReMixT

ReMixT is a tool for joint inference of clone specific segment and breakpoint copy number in whole genome sequencing data. The input for the tool is a set of segments, a set of breakpoints predicted from the sequencing data, and normal and tumour bam files. Where multiple tumour samples are available, they can be analyzed jointly for additional benefit.

How to cite

If you find ReMixT useful, please consider citing our genome biology article.

Installation

Conda is a prerequisite, install anaconda python from the continuum website.

Installing from pip

The recommended method of installation for ReMixT is using pip.

pip install remixt

You will also need to shapeit and samtools on your path. They can be installed using conda:

conda install samtools
conda install -c dranew shapeit

Installing from conda

The conda distribution is now out of date. However, to use conda, add my channel, and the bioconda channel, and install ReMixT as follows.

conda config --add channels https://conda.anaconda.org/dranew
conda config --add channels 'bioconda'
conda install remixt

Installing from source

Clone Source Code

To install the code, first clone from bitbucket. A recursive clone is preferred to pull in all submodules.

git clone --recursive git@bitbucket.org:dranew/remixt.git

Dependencies

To install from source you will need several dependencies. A list of dependencies can be found in the conda yaml file in the repo at conda/remixt/meta.yaml.

Build executables and install

To build executables and install the ReMixT code as a python package run the following command in the ReMixT repo:

python setup.py install

Setup ReMixT

Reference genome

Download and setup of the reference genome is automated. The default is hg19. Select a directory on your system that will contain the reference data, herein referred to as $ref_data_dir. The $ref_data_dir directory will be used in many of the subsequent scripts when running destruct.

Download the reference data and build the required indexes:

remixt create_ref_data $ref_data_dir

Mappability file

Additionally, ReMixT requires a mappability file in $ref_data_dir, see the test dataset section below for an example of retrieving a precomputed mappability file.

Alternatively, calculate the mappability file by aligning every 100-mer of the genome back to the genome with bwa:

nextflow run nextflow/mappability.nf -profile singularity \
    --ref_data_dir $ref_data_dir \
    --ensembl_genome_version GRCh38 \
    --out_dir $ref_data_dir

The genome fasta and mappability filenames are listed in nextflow/conf/genomes.config. An existing bwa index next to the genome fasta is used, otherwise one is built. K-mers are aligned in parallel chunks of --mappability_chunk_size k-mers (default 10 million), so it is recommended you run this on a cluster.

Running ReMixT

Input Data

ReMixT takes multiple bam files as input. Bam files should be multiple samples from the same patient, with one bam sequenced from a normal sample from that patient.

Additionally, ReMixT takes a list of predicted breakpoints detected by paired end sequencing as an additional input.

Breakpoint Prediction Input Format

The predicted breakpoints should be provided in a tab separated file with the following columns:

  • prediction_id
  • chromosome_1
  • strand_1
  • position_1
  • chromosome_2
  • strand_2
  • position_2

The first line should be the column names, which should be identical to the above list. Each subsequent line is a breakpoint prediction. The prediction_id should be unique to each breakpoint prediction. The chromosome_, strand_ and position_ columns give the position and orientation of each end of the breakpoint. The values for strand_ should be either + or -. A value of + means that sequence to the right of chromosome_, position_ is preserved in the tumour chromosome containing the breakpoint. Conversely, a value of - means that sequence to the left of chromosome_, position_ is preserved in the tumour chromosome containing the breakpoint.

The following table may assist in understanding the strand of a break-end. Note that an inversion event produces two breakpoints, the strand configurations for both are shown. Additionally, for inter-chromosomal events, any strand configuration is possible.

Structural Variation Strand of Leftmost Break-End Strand of Rightmost Break-End
Deletion + -
Duplication - +
Inversion (Breakpoint A) + +
Inversion (Breakpoint B) - -

ReMixT Command Line

ReMixT is run using the Nextflow pipeline in the nextflow directory, see Running with Nextflow and Containers for container options. The result of ReMixT is an hdf5 file storing pandas tables.

Suppose we have the following list of inputs:

  • Normal sample with ID 123N and bam filename $normal_bam
  • Tumour sample with ID 123A and bam filename $tumour_a_bam
  • Tumour sample with ID 123B and bam filename $tumour_b_bam
  • Breakpoint table in TSV format with filename $breakpoints

Additionally, ReMixT will generate the following outputs:

  • Results as HDF5 files storing pandas tables, one per tumour sample, in directory $out_dir

Given the above inputs and outputs run ReMixT as follows:

nextflow run nextflow/main.nf -profile singularity \
    --ref_data_dir $ref_data_dir \
    --out_dir $out_dir \
    --breakpoint_file $breakpoints \
    --normal_sample_id 123N \
    --normal_bam_file $normal_bam \
    --tumour_sample_ids 123A,123B \
    --tumour_bam_files $tumour_a_bam,$tumour_b_bam

Many parts of ReMixT are massively parallelizable, thus it is recommended you run ReMixT on a cluster, for example with -profile slurm,singularity, adding site specific settings such as the queue and filesystem binds with -c site.config.

ReMixT is deterministic, rerunning with the same inputs and parameters produces the same results. Random sampling of GC positions, model initialization and model fitting are seeded using --random_seed (default 1234).

Output File Formats

The main output file is an HDF5 store containing pandas dataframes. These can be extracted in python or viewed using the ReMixT viewer. Important tables include:

  • stats: statistics for each restart
  • failed_fits: restarts for which model fitting failed to optimize, with the error message, these have no solution
  • solutions/solution_{idx}/cn: segment copy number table for solution idx
  • solutions/solution_{idx}/brk_cn: breakpoint copy number table for solution idx
  • solutions/solution_{idx}/h: haploid depths for solution idx

Statistics

ReMixT uses optimal restarts and model selection by BIC. The statistics table contains one row per restart, sorted by BIC. The table contains the following columns:

  • idx: the solution index, used to refer to solutions/solution_{idx}/* tables.
  • bic: the bic of this solution
  • log_posterior: log posterior of the HMM
  • log_posterior_graph: log posterior of the genome graph model
  • num_clones: number of clones including normal
  • num_segments: number of segments
  • h_converged: whether haploid depths estimation converged
  • h_em_iter: number of iterations for convergence of h
  • graph_opt_iter: number of iterations for convergence of genome graph copy number
  • decreased_log_posterior: whether the genome graph optimization stopped due to a move that decreased the log posterior

Segment Copy Number

The segment copy number table adds additional columns to the segment counts table described above, including but not limited to:

  • major_1
  • minor_1
  • major_2
  • minor_2

The columns refer to the major and minor copy number in tumour clone 1 and 2.

Breakpoints Copy Number

The breakpoint copy number table contains the following columns:

  • prediction_id
  • cn_1
  • cn_2

The prediction_id column matches the column of the same name in the input breakpoints file, and specifies for which breakpoint prediction the copy number is being provided. The cn_1 and cn_2 columns provide the clone specific copy number for tumour clone 1 and 2 respectively.

Haploid Depths

The haploid depths is a vector of M depths for each of the M clones including the normal. To recover cell mixture proportions, simply normalize h.

Extracting Tables as TSV files

If preferred, it is possible to extract copy number and metadata in TSV and YaML format. For results file $results_h5, extract segment copy number, breakpoint copy number and meta data to files $cn_table, $brk_cn_table, $meta_data respectively as follows:

remixt write_results \
    $results_h5 $cn_table $brk_cn_table $meta_data

The tables are those of the optimal solution selected by the pipeline. Any other solution can be written by its init id with --solution, the ids are listed in the report and in the stats table of the results file.

ReMixT Report

The Nextflow pipeline writes remixt_report.html to the output directory, a single standalone HTML report of all tumours. The results are embedded in the file, so it can be opened in a browser or shared without the results files or a network connection. The report has a cohort view summarizing the selected solution of each tumour, and a tumour view with genome wide raw, expected and clone specific copy number, breakpoints, read depth, and every solution, any of which can be selected for comparison with the optimal solution.

The report can also be created from results files with the remixt visualize_solutions sub-command, giving each results file as tumour_id:path, or as a path named <tumour_id>.h5:

remixt visualize_solutions report.html \
    tumour_a:tumour_a.h5 tumour_b:tumour_b.h5

Test Dataset for ReMixT

A test dataset is provided for providing the ability to run a quick analysis of a small dataset to ensure remixt is working correctly.

We will assume that the REMIXT_DIR environment variable points to a clone of the ReMixT source code. Additionally, create a directory, and set the environment variable WORK_DIR to the location of that directory.

First use the remixt create_ref_data sub-command to create a reference dataset. Specify a config, and use the example config that restricts to chromosome 15.

remixt create_ref_data $WORK_DIR/ref_data \
    --config $REMIXT_DIR/examples/chromosome_15_config.yaml

Use wget to retrieve a precomputed mappability file.

wget http://remixttestdata.s3.amazonaws.com/hg19.100.bwa.mappability.h5 --directory-prefix $WORK_DIR/ref_data/

Use wget to retrieve the example bam files and their indices for chromosome 15, and the breakpoints file with chromosome 15 breakpoints.

wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_chr15_grch38.bam --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_chr15_grch38.bam.bai --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395BL_chr15_grch38.bam --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395BL_chr15_grch38.bam.bai --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_breakpoints.tsv --directory-prefix $WORK_DIR/

Use the Nextflow pipeline to run a remixt analysis.

nextflow run $REMIXT_DIR/nextflow/main.nf -profile singularity \
    --ref_data_dir $WORK_DIR/ref_data \
    --out_dir $WORK_DIR \
    --breakpoint_file $WORK_DIR/HCC1395_breakpoints.tsv \
    --tumour_sample_ids HCC1395 \
    --tumour_bam_files $WORK_DIR/HCC1395_chr15_grch38.bam \
    --normal_sample_id HCC1395BL \
    --normal_bam_file $WORK_DIR/HCC1395BL_chr15_grch38.bam \
    --chromosomes chr15 \
    --chr_name_prefix chr

Use the remixt write_results sub-command to write out tables of results and a yaml file containing inferred parameters and other meta data.

remixt write_results $WORK_DIR/HCC1395.h5 \
    $WORK_DIR/HCC1395_cn.tsv \
    $WORK_DIR/HCC1395_brk_cn.tsv \
    $WORK_DIR/HCC1395_info.yaml

Finally, open $WORK_DIR/remixt_report.html in a browser to view the results. The report can also be recreated from the results using the remixt visualize_solutions sub-command.

remixt visualize_solutions $WORK_DIR/remixt_report.html \
    $WORK_DIR/HCC1395.h5

Build

Docker builds

Production build (from GitHub)

To build and push a release image from the GitHub repository:

docker build --platform linux/amd64 \
    --build-arg INSTALL_FROM=github \
    --build-arg REMIXT_VERSION=0.6.2 \
    -t quay.io/andrew_mcpherson/remixt:v0.6.2 .
docker push quay.io/andrew_mcpherson/remixt:v0.6.2

To also update the latest tag:

docker tag quay.io/andrew_mcpherson/remixt:v0.6.2 quay.io/andrew_mcpherson/remixt:latest
docker push quay.io/andrew_mcpherson/remixt:latest

Development build (from local source)

To build from your local working copy:

docker build --platform linux/amd64 \
    -t remixt:dev .

The container includes remixt, shapeit4, bingraphsample, and all required tools (bcftools, tabix, bgzip).

Running with Nextflow and Containers

The Nextflow pipeline can run entirely in containers using the docker profile:

nextflow run nextflow/main.nf -profile docker \
    --ref_data_dir $ref_data_dir \
    --out_dir $out_dir \
    --breakpoint_file $breakpoints \
    --tumour_sample_ids HCC1395 \
    --tumour_bam_files /path/to/tumour.bam \
    --normal_sample_id HCC1395BL \
    --normal_bam_file /path/to/normal.bam \
    --chromosomes 15 \
    --chr_name_prefix chr

For Singularity (common on HPC clusters):

nextflow run nextflow/main.nf -profile singularity \
    ...

Container versions

The container image defaults to the tag matching the pipeline version, remixt_version in nextflow/nextflow.config, quay.io/andrew_mcpherson/remixt:v0.6.2 for version 0.6.2. Running a tagged version of the pipeline therefore uses the container built for that version, either from a checkout of the tag:

git checkout v0.6.2
nextflow run nextflow/main.nf -profile singularity ...

or directly from GitHub, which checks out the tag into the Nextflow assets directory:

nextflow run amcpherson/remixt -r v0.6.2 -profile singularity ...

Each run logs the pipeline version and container at startup. To use another container, such as latest or a local development build:

nextflow run nextflow/main.nf -profile docker --remixt_container quay.io/andrew_mcpherson/remixt:latest
nextflow run nextflow/main.nf -profile docker --remixt_container remixt:dev

When releasing a version, set remixt_version in nextflow/nextflow.config in the commit that is tagged, then build and push the image with the same tag as described in Docker builds.

Pip build

To build with pip and distribute to pypi, use the following commands:

python setup.py build_ext --force sdist
twine upload --repository pypi dist/*

License

ReMixT is released under the MIT License.

About

Clone-specific genomic structure estimation in cancer

Resources

Stars

8 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages