ReMixT is a tool for joint inference of clone specific segment and breakpoint copy number in whole genome sequencing data. The input for the tool is a set of segments, a set of breakpoints predicted from the sequencing data, and normal and tumour bam files. Where multiple tumour samples are available, they can be analyzed jointly for additional benefit.
If you find ReMixT useful, please consider citing our genome biology article.
Conda is a prerequisite, install anaconda python from the continuum website.
The recommended method of installation for ReMixT is using pip.
pip install remixt
You will also need to shapeit and samtools on your path. They can be installed using conda:
conda install samtools
conda install -c dranew shapeit
The conda distribution is now out of date. However, to use conda, add my channel, and the bioconda channel, and install ReMixT as follows.
conda config --add channels https://conda.anaconda.org/dranew
conda config --add channels 'bioconda'
conda install remixt
To install the code, first clone from bitbucket. A recursive clone is preferred to pull in all submodules.
git clone --recursive git@bitbucket.org:dranew/remixt.git
To install from source you will need several dependencies. A list of dependencies can be found in the conda yaml file in the repo at conda/remixt/meta.yaml.
To build executables and install the ReMixT code as a python package run the following command in the ReMixT repo:
python setup.py install
Download and setup of the reference genome is automated. The default is hg19. Select a directory on your system that will contain the reference data, herein referred to as $ref_data_dir. The $ref_data_dir directory will be used in many of the subsequent scripts when running destruct.
Download the reference data and build the required indexes:
remixt create_ref_data $ref_data_dir
Additionally, ReMixT requires a mappability file in $ref_data_dir, see the test dataset section below for an example of retrieving a precomputed mappability file.
Alternatively, calculate the mappability file by aligning every 100-mer of the genome back to the genome with bwa:
nextflow run nextflow/mappability.nf -profile singularity \
--ref_data_dir $ref_data_dir \
--ensembl_genome_version GRCh38 \
--out_dir $ref_data_dir
The genome fasta and mappability filenames are listed in nextflow/conf/genomes.config. An existing bwa index next to the genome fasta is used, otherwise one is built. K-mers are aligned in parallel chunks of --mappability_chunk_size k-mers (default 10 million), so it is recommended you run this on a cluster.
ReMixT takes multiple bam files as input. Bam files should be multiple samples from the same patient, with one bam sequenced from a normal sample from that patient.
Additionally, ReMixT takes a list of predicted breakpoints detected by paired end sequencing as an additional input.
The predicted breakpoints should be provided in a tab separated file with the following columns:
prediction_idchromosome_1strand_1position_1chromosome_2strand_2position_2
The first line should be the column names, which should be identical to the above list. Each subsequent line is a breakpoint prediction. The prediction_id should be unique to each breakpoint prediction. The chromosome_, strand_ and position_ columns give the position and orientation of each end of the breakpoint. The values for strand_ should be either + or -. A value of + means that sequence to the right of chromosome_, position_ is preserved in the tumour chromosome containing the breakpoint. Conversely, a value of - means that sequence to the left of chromosome_, position_ is preserved in the tumour chromosome containing the breakpoint.
The following table may assist in understanding the strand of a break-end. Note that an inversion event produces two breakpoints, the strand configurations for both are shown. Additionally, for inter-chromosomal events, any strand configuration is possible.
| Structural Variation | Strand of Leftmost Break-End | Strand of Rightmost Break-End |
|---|---|---|
| Deletion | + | - |
| Duplication | - | + |
| Inversion (Breakpoint A) | + | + |
| Inversion (Breakpoint B) | - | - |
ReMixT is run using the Nextflow pipeline in the nextflow directory, see Running with Nextflow and Containers for container options. The result of ReMixT is an hdf5 file storing pandas tables.
Suppose we have the following list of inputs:
- Normal sample with ID
123Nand bam filename$normal_bam - Tumour sample with ID
123Aand bam filename$tumour_a_bam - Tumour sample with ID
123Band bam filename$tumour_b_bam - Breakpoint table in TSV format with filename
$breakpoints
Additionally, ReMixT will generate the following outputs:
- Results as HDF5 files storing pandas tables, one per tumour sample, in directory
$out_dir
Given the above inputs and outputs run ReMixT as follows:
nextflow run nextflow/main.nf -profile singularity \
--ref_data_dir $ref_data_dir \
--out_dir $out_dir \
--breakpoint_file $breakpoints \
--normal_sample_id 123N \
--normal_bam_file $normal_bam \
--tumour_sample_ids 123A,123B \
--tumour_bam_files $tumour_a_bam,$tumour_b_bam
Many parts of ReMixT are massively parallelizable, thus it is recommended you run ReMixT on a cluster, for example with -profile slurm,singularity, adding site specific settings such as the queue and filesystem binds with -c site.config.
ReMixT is deterministic, rerunning with the same inputs and parameters produces the same results. Random sampling of GC positions, model initialization and model fitting are seeded using --random_seed (default 1234).
The main output file is an HDF5 store containing pandas dataframes. These can be extracted in python or viewed using the ReMixT viewer. Important tables include:
stats: statistics for each restartfailed_fits: restarts for which model fitting failed to optimize, with the error message, these have no solutionsolutions/solution_{idx}/cn: segment copy number table for solutionidxsolutions/solution_{idx}/brk_cn: breakpoint copy number table for solutionidxsolutions/solution_{idx}/h: haploid depths for solutionidx
ReMixT uses optimal restarts and model selection by BIC. The statistics table contains one row per restart, sorted by BIC. The table contains the following columns:
idx: the solution index, used to refer tosolutions/solution_{idx}/*tables.bic: the bic of this solutionlog_posterior: log posterior of the HMMlog_posterior_graph: log posterior of the genome graph modelnum_clones: number of clones including normalnum_segments: number of segmentsh_converged: whether haploid depths estimation convergedh_em_iter: number of iterations for convergence of hgraph_opt_iter: number of iterations for convergence of genome graph copy numberdecreased_log_posterior: whether the genome graph optimization stopped due to a move that decreased the log posterior
The segment copy number table adds additional columns to the segment counts table described above, including but not limited to:
major_1minor_1major_2minor_2
The columns refer to the major and minor copy number in tumour clone 1 and 2.
The breakpoint copy number table contains the following columns:
prediction_idcn_1cn_2
The prediction_id column matches the column of the same name in the input breakpoints file, and specifies for which breakpoint prediction the copy number is being provided. The cn_1 and cn_2 columns provide the clone specific copy number for tumour clone 1 and 2 respectively.
The haploid depths is a vector of M depths for each of the M clones including the normal. To recover cell mixture proportions, simply normalize h.
If preferred, it is possible to extract copy number and metadata in TSV and YaML format. For results file $results_h5, extract segment copy number, breakpoint copy number and meta data to files $cn_table, $brk_cn_table, $meta_data respectively as follows:
remixt write_results \
$results_h5 $cn_table $brk_cn_table $meta_data
The tables are those of the optimal solution selected by the pipeline. Any other solution can be written by its init id with --solution, the ids are listed in the report and in the stats table of the results file.
The Nextflow pipeline writes remixt_report.html to the output directory, a single standalone HTML report of all tumours. The results are embedded in the file, so it can be opened in a browser or shared without the results files or a network connection. The report has a cohort view summarizing the selected solution of each tumour, and a tumour view with genome wide raw, expected and clone specific copy number, breakpoints, read depth, and every solution, any of which can be selected for comparison with the optimal solution.
The report can also be created from results files with the remixt visualize_solutions sub-command, giving each results file as tumour_id:path, or as a path named <tumour_id>.h5:
remixt visualize_solutions report.html \
tumour_a:tumour_a.h5 tumour_b:tumour_b.h5
A test dataset is provided for providing the ability to run a quick analysis of a small dataset to ensure remixt is working correctly.
We will assume that the REMIXT_DIR environment variable points to a clone of the ReMixT source code. Additionally, create a directory, and set the environment variable WORK_DIR to the location of that directory.
First use the remixt create_ref_data sub-command to create a reference dataset. Specify a config, and use the example config that restricts to chromosome 15.
remixt create_ref_data $WORK_DIR/ref_data \
--config $REMIXT_DIR/examples/chromosome_15_config.yaml
Use wget to retrieve a precomputed mappability file.
wget http://remixttestdata.s3.amazonaws.com/hg19.100.bwa.mappability.h5 --directory-prefix $WORK_DIR/ref_data/
Use wget to retrieve the example bam files and their indices for chromosome 15, and the breakpoints file with chromosome 15 breakpoints.
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_chr15_grch38.bam --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_chr15_grch38.bam.bai --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395BL_chr15_grch38.bam --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395BL_chr15_grch38.bam.bai --directory-prefix $WORK_DIR/
wget http://remixttestdata.s3.amazonaws.com/test_grch38_chrprefix/HCC1395_breakpoints.tsv --directory-prefix $WORK_DIR/
Use the Nextflow pipeline to run a remixt analysis.
nextflow run $REMIXT_DIR/nextflow/main.nf -profile singularity \
--ref_data_dir $WORK_DIR/ref_data \
--out_dir $WORK_DIR \
--breakpoint_file $WORK_DIR/HCC1395_breakpoints.tsv \
--tumour_sample_ids HCC1395 \
--tumour_bam_files $WORK_DIR/HCC1395_chr15_grch38.bam \
--normal_sample_id HCC1395BL \
--normal_bam_file $WORK_DIR/HCC1395BL_chr15_grch38.bam \
--chromosomes chr15 \
--chr_name_prefix chr
Use the remixt write_results sub-command to write out tables of results and a yaml file containing inferred parameters and other meta data.
remixt write_results $WORK_DIR/HCC1395.h5 \
$WORK_DIR/HCC1395_cn.tsv \
$WORK_DIR/HCC1395_brk_cn.tsv \
$WORK_DIR/HCC1395_info.yaml
Finally, open $WORK_DIR/remixt_report.html in a browser to view the results. The report can also be recreated from the results using the remixt visualize_solutions sub-command.
remixt visualize_solutions $WORK_DIR/remixt_report.html \
$WORK_DIR/HCC1395.h5
To build and push a release image from the GitHub repository:
docker build --platform linux/amd64 \
--build-arg INSTALL_FROM=github \
--build-arg REMIXT_VERSION=0.6.2 \
-t quay.io/andrew_mcpherson/remixt:v0.6.2 .
docker push quay.io/andrew_mcpherson/remixt:v0.6.2
To also update the latest tag:
docker tag quay.io/andrew_mcpherson/remixt:v0.6.2 quay.io/andrew_mcpherson/remixt:latest
docker push quay.io/andrew_mcpherson/remixt:latest
To build from your local working copy:
docker build --platform linux/amd64 \
-t remixt:dev .
The container includes remixt, shapeit4, bingraphsample, and all required tools (bcftools, tabix, bgzip).
The Nextflow pipeline can run entirely in containers using the docker profile:
nextflow run nextflow/main.nf -profile docker \
--ref_data_dir $ref_data_dir \
--out_dir $out_dir \
--breakpoint_file $breakpoints \
--tumour_sample_ids HCC1395 \
--tumour_bam_files /path/to/tumour.bam \
--normal_sample_id HCC1395BL \
--normal_bam_file /path/to/normal.bam \
--chromosomes 15 \
--chr_name_prefix chr
For Singularity (common on HPC clusters):
nextflow run nextflow/main.nf -profile singularity \
...
The container image defaults to the tag matching the pipeline version, remixt_version in nextflow/nextflow.config, quay.io/andrew_mcpherson/remixt:v0.6.2 for version 0.6.2. Running a tagged version of the pipeline therefore uses the container built for that version, either from a checkout of the tag:
git checkout v0.6.2
nextflow run nextflow/main.nf -profile singularity ...
or directly from GitHub, which checks out the tag into the Nextflow assets directory:
nextflow run amcpherson/remixt -r v0.6.2 -profile singularity ...
Each run logs the pipeline version and container at startup. To use another container, such as latest or a local development build:
nextflow run nextflow/main.nf -profile docker --remixt_container quay.io/andrew_mcpherson/remixt:latest
nextflow run nextflow/main.nf -profile docker --remixt_container remixt:dev
When releasing a version, set remixt_version in nextflow/nextflow.config in the commit that is tagged, then build and push the image with the same tag as described in Docker builds.
To build with pip and distribute to pypi, use the following commands:
python setup.py build_ext --force sdist
twine upload --repository pypi dist/*
ReMixT is released under the MIT License.