1000Bulls mapping pipeline at ABGC - Revision history

Megen002 at 13:17, 28 March 2014

2014-03-28T13:17:48Z

← Older revision		Revision as of 15:17, 28 March 2014
Line 45:		Line 45:
	python rename_files.py -t transtable.txt		python rename_files.py -t transtable.txt
	</source>		</source>
			The translation table should look like this; note that files are not ordered correctly lexicographically.

	bcded3e7eb29e423cfb7559a81b6f651 bov-wur-16 run0262 bov-wur-16_CGATGT_L004_R1_001.cleaned.fastq.gz		bcded3e7eb29e423cfb7559a81b6f651 bov-wur-16 run0262 bov-wur-16_CGATGT_L004_R1_001.cleaned.fastq.gz

Megen002 at 13:16, 28 March 2014

2014-03-28T13:16:27Z

← Older revision		Revision as of 15:16, 28 March 2014
Line 42:		Line 42:

	Conventions of the mapping pipeline dictate that for all fastq files in a folder, the file containing the second read should immediate follow the file containing the first read lexicographically. Therefore, if necessary, the files can be renamed with the following python script:		Conventions of the mapping pipeline dictate that for all fastq files in a folder, the file containing the second read should immediate follow the file containing the first read lexicographically. Therefore, if necessary, the files can be renamed with the following python script:
			<source lang='bash'>
			python rename_files.py -t transtable.txt
			</source>

			bcded3e7eb29e423cfb7559a81b6f651 bov-wur-16 run0262 bov-wur-16_CGATGT_L004_R1_001.cleaned.fastq.gz
			6f1e32498eb31d138bd4fdacf0cef5d0 bov-wur-16 run0262 bov-wur-16_CGATGT_L004_R2_001.cleaned.fastq.gz
			ec3f4ecd34402d6aaa33c897bf62430d bov-wur-16 run0264 bov-wur-16_CGATGT_L004_R1_001.cleaned.fastq.gz
			6cfbc04bf11fd5d71bb937a9d978056d bov-wur-16 run0264 bov-wur-16_CGATGT_L004_R2_001.cleaned.fastq.gz
			eafca33360fa81c68933363d393252db bov-wur-16 run0264 bov-wur-16_CGATGT_L005_R1_001.cleaned.fastq.gz
			22b24d014613d465b13c0d1a89281074 bov-wur-16 run0264 bov-wur-16_CGATGT_L005_R2_001.cleaned.fastq.gz
			e0e44096300bec49405c6c7591352ab3 bov-wur-16 run0269 bov-wur-16_CGATGT_L002_R1_001.cleaned.fastq.gz
			9e97fa54293f90b1f3d9738791e38603 bov-wur-16 run0269 bov-wur-16_CGATGT_L002_R1_002.cleaned.fastq.gz
			cec51a979f766b18e7a9cbec99b0a433 bov-wur-16 run0269 bov-wur-16_CGATGT_L002_R1_003.cleaned.fastq.gz
			7ac8f3eb1f0ff26503137b4428120e5b bov-wur-16 run0269 bov-wur-16_CGATGT_L002_R2_001.cleaned.fastq.gz

	<source lang='python'>		<source lang='python'>

Megen002 at 13:13, 28 March 2014

2014-03-28T13:13:48Z

← Older revision		Revision as of 15:13, 28 March 2014
Line 41:		Line 41:
	</source>		</source>

	If necessary, the files can be renamed with the following python script:		Conventions of the mapping pipeline dictate that for all fastq files in a folder, the file containing the second read should immediate follow the file containing the first read lexicographically. Therefore, if necessary, the files can be renamed with the following python script:

	<source lang='python'>		<source lang='python'>

Megen002 at 13:12, 28 March 2014

2014-03-28T13:12:10Z

@@ Line 39: / Line 39: @@
 <source lang='bash'>
 java -Xmx4G -jar GenomeAnalysisTK.jar -T DepthOfCoverage -R umd_3_1.fa -I BOV-WUR-2_rh.dedup_st.reA.bam --omitDepthOutputAtEachBase --logging_level ERROR --summaryCoverageThreshold 10 --summaryCoverageThreshold 20 --summaryCoverageThreshold 30 --summaryCoverageThreshold 40 --summaryCoverageThreshold 50 --summaryCoverageThreshold 80 --summaryCoverageThreshold 90 --summaryCoverageThreshold 100 --summaryCoverageThreshold 150 --minBaseQuality 15 --minMappingQuality 30 --start 1 --stop 1000 --nBins 999 -dt NONE -o BOV-WUR-2_rh.dedup_st.reA.coverage
 </source>

Hjmegens at 21:34, 26 November 2013

2013-11-26T21:34:54Z

← Older revision		Revision as of 23:34, 26 November 2013
Line 24:		Line 24:
	BOV-WUR-1 HOLNLDM000120873995		BOV-WUR-1 HOLNLDM000120873995
	BOV-WUR-2 HOLNLDM000811488961		BOV-WUR-2 HOLNLDM000811488961

			The code to generate the runfiles for the first 15 bulls to be analysed:

	<source lang='bash'>		<source lang='bash'>
Line 29:		Line 31:
	for i in `seq 1 15`; do echo $i; python3 ABGC_mapping_v2.py -i BOV-WUR-$i -a /srv/mds01/shared/Bulls1000/F12FPCEUHK0755_alq121122/cleandata/ -r /srv/mds01/shared/Bulls1000/UMD31/umd_3_1.fa -t 12 -s cow -m bwa-aln -b 5.9 -o; done		for i in `seq 1 15`; do echo $i; python3 ABGC_mapping_v2.py -i BOV-WUR-$i -a /srv/mds01/shared/Bulls1000/F12FPCEUHK0755_alq121122/cleandata/ -r /srv/mds01/shared/Bulls1000/UMD31/umd_3_1.fa -t 12 -s cow -m bwa-aln -b 5.9 -o; done
	</source>		</source>

	* [https://github.com/hjmegens/NGStools/blob/master/ABGC_mapping_v2.py The Python3-based masterscript]		* [https://github.com/hjmegens/NGStools/blob/master/ABGC_mapping_v2.py The Python3-based masterscript]
	* [https://github.com/hjmegens/NGStools/blob/master/runBOV-WUR-4.sh Example of a runfile created by the master script]		* [https://github.com/hjmegens/NGStools/blob/master/runBOV-WUR-4.sh Example of a runfile created by the master script]

Hjmegens at 21:34, 26 November 2013

2013-11-26T21:34:13Z

← Older revision		Revision as of 23:34, 26 November 2013
Line 19:		Line 19:
	</source>		</source>

	The table will hold one line per individual. Its purpose is to connect the trivial name used in the cow_schema_main table to the official 1000 Bulls Id.		The table will hold one line per individual. Its purpose is to connect the trivial name used in the cow_schema_main table to the official 1000 Bulls Id. In addition, it provides an overview of the animals represented in the sequence archives.
	TABLE bulls1K_id		TABLE bulls1K_id
	animal_id bull1K_id		animal_id bull1K_id

Hjmegens at 21:33, 26 November 2013

2013-11-26T21:33:28Z

← Older revision		Revision as of 23:33, 26 November 2013
Line 19:		Line 19:
	</source>		</source>

			The table will hold one line per individual. Its purpose is to connect the trivial name used in the cow_schema_main table to the official 1000 Bulls Id.
	TABLE bulls1K_id		TABLE bulls1K_id
	animal_id bull1K_id		animal_id bull1K_id

Hjmegens at 21:32, 26 November 2013

2013-11-26T21:32:23Z

← Older revision		Revision as of 23:32, 26 November 2013
Line 1:		Line 1:
	The 1000 Bulls mapping pipeline as currently implemented at ABGC is an extension of a generic pipeline developed for pig at ABGC. Modifications include use of bwa 5.9 in stead of later versions, as per the requirements of the 1000 Bulls consortium. Furthermore, the script will automate setting metadata in the BAM files such as including the official 1000 Bulls Ids of the individual cows. Another modification is the use of an sqlite-based database that holds some the metadata required for performing the mapping and setting the correct ids. The sqlite database should be called 'cow_schema.db' and should be in the same working directory as the Python3 master script. Currently, the following two tables should be present in the database: <code>cow_schema_main</code> and <code>bulls1K_id</code>		The 1000 Bulls mapping pipeline as currently implemented at ABGC is an extension of a generic pipeline developed for pig at ABGC. Modifications include use of bwa 5.9 in stead of later versions, as per the requirements of the 1000 Bulls consortium. Furthermore, the script will automate setting metadata in the BAM files such as including the official 1000 Bulls Ids of the individual cows. Another modification is the use of an sqlite-based database that holds some the metadata required for performing the mapping and setting the correct ids. The sqlite database should be called 'cow_schema.db' and should be in the same working directory as the Python3 master script. Currently, the following two tables should be present in the database: <code>cow_schema_main</code> and <code>bulls1K_id</code>.

			Code to create the <code>cow_schema_main</code> table:

	<source lang='sql'>		<source lang='sql'>
Line 5:		Line 7:
	</source>		</source>

			The table will hold one line for each fastq file (seq_file_name). Archive refers to the base directory that holds the fastq file. Animal ID can be a trivial name. Note that the pipeline will assume files to be gzipped plain-text fastq. The md5sums are from the gzipped files.
	Table: cow_schema_main:		Table: cow_schema_main:
	archive seq_file_name animal_id md5sum_zipped		archive seq_file_name animal_id md5sum_zipped
Line 10:		Line 13:
	SZAIPI019130-16 121202_I598_FCD1JRLACXX_L6_SZAIPI019130-16_2.fq.gz.clean.dup.clean.gz BOV-WUR-1 333aee62675082d42c2355cc7f21e89b		SZAIPI019130-16 121202_I598_FCD1JRLACXX_L6_SZAIPI019130-16_2.fq.gz.clean.dup.clean.gz BOV-WUR-1 333aee62675082d42c2355cc7f21e89b


			Code to create the <code>bulls1K_id</code> table:
	<source lang='sql'>		<source lang='sql'>
	CREATE TABLE bulls1K_id (animal_id text not null, bull1K_id text not null primary key, tmp_inserted datetime default current_timestamp);		CREATE TABLE bulls1K_id (animal_id text not null, bull1K_id text not null primary key, tmp_inserted datetime default current_timestamp);

Hjmegens at 21:27, 26 November 2013

2013-11-26T21:27:55Z

← Older revision		Revision as of 23:27, 26 November 2013
Line 1:		Line 1:
	The 1000 Bulls mapping pipeline as currently implemented at ABGC is an extension of a generic pipeline developed for pig at ABGC. Modifications include use of bwa 5.9 in stead of later versions, as per the requirements of the 1000 Bulls consortium. Furthermore, the script will automate setting metadata in the BAM files such as including the official 1000 Bulls Ids of the individual cows. Another modification is the use of an sqlite-based database that holds some the metadata required for performing the mapping and setting the correct ids. The sqlite database should be called 'cow_schema.db' and should be in the same working directory as the Python3 master script. Currently, the following two tables should be present in the database:		The 1000 Bulls mapping pipeline as currently implemented at ABGC is an extension of a generic pipeline developed for pig at ABGC. Modifications include use of bwa 5.9 in stead of later versions, as per the requirements of the 1000 Bulls consortium. Furthermore, the script will automate setting metadata in the BAM files such as including the official 1000 Bulls Ids of the individual cows. Another modification is the use of an sqlite-based database that holds some the metadata required for performing the mapping and setting the correct ids. The sqlite database should be called 'cow_schema.db' and should be in the same working directory as the Python3 master script. Currently, the following two tables should be present in the database: <code>cow_schema_main</code> and <code>bulls1K_id</code>

	<source lang='sql'>		<source lang='sql'>

Hjmegens at 21:26, 26 November 2013

2013-11-26T21:26:29Z

@@ Line 1: / Line 1: @@
 <source lang='bash'>
@@ Line 5: / Line 25: @@
 * [https://github.com/hjmegens/NGStools/blob/master/ABGC_mapping_v2.py The Python3-based masterscript]
 * [https://github.com/hjmegens/NGStools/blob/master/runBOV-WUR-4.sh Example of a runfile created by the master script]
 <source lang='bash'>
 java -Xmx4G -jar GenomeAnalysisTK.jar -T DepthOfCoverage -R umd_3_1.fa -I BOV-WUR-2_rh.dedup_st.reA.bam --omitDepthOutputAtEachBase --logging_level ERROR --summaryCoverageThreshold 10 --summaryCoverageThreshold 20 --summaryCoverageThreshold 30 --summaryCoverageThreshold 40 --summaryCoverageThreshold 50 --summaryCoverageThreshold 80 --summaryCoverageThreshold 90 --summaryCoverageThreshold 100 --summaryCoverageThreshold 150 --minBaseQuality 15 --minMappingQuality 30 --start 1 --stop 1000 --nBins 999 -dt NONE -o BOV-WUR-2_rh.dedup_st.reA.coverage
 </source>