samtools BAM to SAM tmp directory
Is there a reason that the samtools BAM to SAM does not respect the new_file_path set in the config file? The tmp directory handling by different tool wrappers seems to be an issue right now on systems with small system tmp directories. -- Matt Shirley Ph.D Candidate - BCMB Pevsner Lab <http://pevsnerlab.kennedykrieger.org/> Johns Hopkins Medicine
Hej Matt! This has been bugging me as well since we are running galaxy locally with a server that has a very restricted space for the /tmp directory. Here are the change I made to the sam-to-bam tools to have him writing to new_file_path rather than $TMP. First in the xml file (1.1.2 is the original version of the file in the June 2013 release) delhomme:samtools$ diff sam_to_bam.xml sam_to_bam.xml_1.1.2 1c1 < <tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.3"> ---
<tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.2"> 8d7 < --temp=$__app__.config.new_file_path
And in the python file: delhomme:samtools$ diff sam_to_bam.py sam_to_bam.py_1.1.2 5d4 < --temp: hidden option to set the tmp directory 39d37 < parser.add_option( '', '--temp', dest='temp', help='The temporary directory to use for writing tmp files' ) 61d58 < os.remove( tmp ) 63d59 < os.remove( tmp ) 74c70 < tmp_dir = tempfile.mkdtemp( dir=options.temp ) ---
tmp_dir = tempfile.mkdtemp()
For the galaxy developers - I don't know if there's a way in the galaxy API to do this more easily. If there's I'd love to hear about it. If not it would be a great addition. Cheers, Nico --------------------------------------------------------------- Nicolas Delhomme Nathaniel Street Lab Department of Plant Physiology Umeå Plant Science Center Tel: +46 90 786 7989 Email: nicolas.delhomme@plantphys.umu.se SLU - Umeå universitet Umeå S-901 87 Sweden --------------------------------------------------------------- On Apr 9, 2012, at 10:53 PM, Matt Shirley wrote:
Is there a reason that the samtools BAM to SAM does not respect the new_file_path set in the config file? The tmp directory handling by different tool wrappers seems to be an issue right now on systems with small system tmp directories.
-- Matt Shirley Ph.D Candidate - BCMB Pevsner Lab Johns Hopkins Medicine
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
In the spirit of this solution, I've implemented something similar in -central changeset: https://bitbucket.org/galaxy/galaxy-central/commits/c7a07f350887fe3b4e822ba6... The tool now creates temporary files in its working directory, which can be configured via the 'job_working_directory' setting; you might choose to set this to the same value as 'new_file_path' for simplicity. Best, J. On Jul 1, 2013, at 8:32 AM, Nicolas Delhomme wrote:
Hej Matt!
This has been bugging me as well since we are running galaxy locally with a server that has a very restricted space for the /tmp directory. Here are the change I made to the sam-to-bam tools to have him writing to new_file_path rather than $TMP.
First in the xml file (1.1.2 is the original version of the file in the June 2013 release)
delhomme:samtools$ diff sam_to_bam.xml sam_to_bam.xml_1.1.2
1c1 < <tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.3"> ---
<tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.2"> 8d7 < --temp=$__app__.config.new_file_path
And in the python file:
delhomme:samtools$ diff sam_to_bam.py sam_to_bam.py_1.1.2
5d4 < --temp: hidden option to set the tmp directory 39d37 < parser.add_option( '', '--temp', dest='temp', help='The temporary directory to use for writing tmp files' ) 61d58 < os.remove( tmp ) 63d59 < os.remove( tmp ) 74c70 < tmp_dir = tempfile.mkdtemp( dir=options.temp ) ---
tmp_dir = tempfile.mkdtemp()
For the galaxy developers - I don't know if there's a way in the galaxy API to do this more easily. If there's I'd love to hear about it. If not it would be a great addition.
Cheers,
Nico
--------------------------------------------------------------- Nicolas Delhomme
Nathaniel Street Lab Department of Plant Physiology Umeå Plant Science Center
Tel: +46 90 786 7989 Email: nicolas.delhomme@plantphys.umu.se SLU - Umeå universitet Umeå S-901 87 Sweden ---------------------------------------------------------------
On Apr 9, 2012, at 10:53 PM, Matt Shirley wrote:
Is there a reason that the samtools BAM to SAM does not respect the new_file_path set in the config file? The tmp directory handling by different tool wrappers seems to be an issue right now on systems with small system tmp directories.
-- Matt Shirley Ph.D Candidate - BCMB Pevsner Lab Johns Hopkins Medicine
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hej Jeremy! That's certainly neater. There are many more wrapper that do the same - using mkdtemp() without a dir argument. Is there any plan to perform the same changes on these (bam-to-sam, etc...)? Cheers, Nico --------------------------------------------------------------- Nicolas Delhomme Nathaniel Street Lab Department of Plant Physiology Umeå Plant Science Center Tel: +46 90 786 7989 Email: nicolas.delhomme@plantphys.umu.se SLU - Umeå universitet Umeå S-901 87 Sweden --------------------------------------------------------------- On Jul 2, 2013, at 3:27 PM, Jeremy Goecks wrote:
In the spirit of this solution, I've implemented something similar in -central changeset:
https://bitbucket.org/galaxy/galaxy-central/commits/c7a07f350887fe3b4e822ba6...
The tool now creates temporary files in its working directory, which can be configured via the 'job_working_directory' setting; you might choose to set this to the same value as 'new_file_path' for simplicity.
Best, J.
On Jul 1, 2013, at 8:32 AM, Nicolas Delhomme wrote:
Hej Matt!
This has been bugging me as well since we are running galaxy locally with a server that has a very restricted space for the /tmp directory. Here are the change I made to the sam-to-bam tools to have him writing to new_file_path rather than $TMP.
First in the xml file (1.1.2 is the original version of the file in the June 2013 release)
delhomme:samtools$ diff sam_to_bam.xml sam_to_bam.xml_1.1.2
1c1 < <tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.3"> ---
<tool id="sam_to_bam" name="SAM-to-BAM" version="1.1.2"> 8d7 < --temp=$__app__.config.new_file_path
And in the python file:
delhomme:samtools$ diff sam_to_bam.py sam_to_bam.py_1.1.2
5d4 < --temp: hidden option to set the tmp directory 39d37 < parser.add_option( '', '--temp', dest='temp', help='The temporary directory to use for writing tmp files' ) 61d58 < os.remove( tmp ) 63d59 < os.remove( tmp ) 74c70 < tmp_dir = tempfile.mkdtemp( dir=options.temp ) ---
tmp_dir = tempfile.mkdtemp()
For the galaxy developers - I don't know if there's a way in the galaxy API to do this more easily. If there's I'd love to hear about it. If not it would be a great addition.
Cheers,
Nico
--------------------------------------------------------------- Nicolas Delhomme
Nathaniel Street Lab Department of Plant Physiology Umeå Plant Science Center
Tel: +46 90 786 7989 Email: nicolas.delhomme@plantphys.umu.se SLU - Umeå universitet Umeå S-901 87 Sweden ---------------------------------------------------------------
On Apr 9, 2012, at 10:53 PM, Matt Shirley wrote:
Is there a reason that the samtools BAM to SAM does not respect the new_file_path set in the config file? The tmp directory handling by different tool wrappers seems to be an issue right now on systems with small system tmp directories.
-- Matt Shirley Ph.D Candidate - BCMB Pevsner Lab Johns Hopkins Medicine
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
That's certainly neater. There are many more wrapper that do the same - using mkdtemp() without a dir argument. Is there any plan to perform the same changes on these (bam-to-sam, etc...)?
I made the same change for the BAM to SAM wrapper so that it create temp files in the job working directory. While there is no plan to make these changes across all tools, we'd definitely like to see it happen. Making the changes yourself and creating pull requests would be a fast, efficient way to get these into the code base. Thanks, J.
participants (3)
-
Jeremy Goecks -
Matt Shirley -
Nicolas Delhomme