Troubleshooting BAM file upload issues
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration. Attempting to upload a BAM file returns the message "The uploaded binary file contains inappropriate content" and the upload is empty. The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly. If someone can suggest some troubleshooting steps, I would be most grateful. Thanks! Amelia Integrated Microbial Genomics (IMG) JGI / LBL
Hi Amelia Does this happen with "Auto-detect" or do you pre-set the data type to 'bam' ? Regards, Hans-Rudolf On 03/22/2016 07:28 PM, Amelia Ireland wrote:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Both, unfortunately. On 23 Mar 2016 00:41, "Hans-Rudolf Hotz" <hrh@fmi.ch> wrote:
Hi Amelia
Does this happen with "Auto-detect" or do you pre-set the data type to 'bam' ?
Regards, Hans-Rudolf
On 03/22/2016 07:28 PM, Amelia Ireland wrote:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Amelia, can you pleae check if you have samtools installed in your PATH? Ciao, Bjoern Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
The bam support is regrettably different to most of Galaxy. For some reason it can not make use of Wheels installed by Galaxy but has to use native samtools. Giving very confusing error messages such as file not found if samtools is not there. Also your bam files MUST be sorted by coordinates as BAM sorted by name is not a format supported by Galaxy. For those few tools that expect Bam sorted by name you have to sort inside that tool. Which breaks: https://toolshed.g2.bx.psu.edu/view/iuc/samtools_sort/38ea74bd4054 As it returns a bam file no longer(or never) supported by Galaxy I have wasted a number of days on these confusing behaviours. Christian University of Manchester ________________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Björn Grüning [bjoern.gruening@gmail.com] Sent: Wednesday, March 23, 2016 8:00 AM To: Amelia Ireland; galaxy-dev@u coordinateslists.galaxyproject.org Subject: Re: [galaxy-dev] Troubleshooting BAM file upload issues Hi Amelia, can you pleae check if you have samtools installed in your PATH? Ciao, Bjoern Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module. The tools dependencies file looks like this: <dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers> Debug messages when the tool runs: galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Amelia, How are you uploading the files? If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running. A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case). Thanks for using Galaxy, Dan On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov> wrote:
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote: Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module.
The tools dependencies file looks like this:
<dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers>
Debug messages when the tool runs:
galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I'm using the standard 'Upload file' tool, and choosing some local files. I can successfully upload these files to the galaxy installation on my laptop and to the usegalaxy.org server, but uploading them to the galaxy installation on the server results in their appearing to be empty and the error message about uploaded file containing inappropriate content. I'm not sure where to proceed from here. Some things I'd like to explore, but am not sure how to do so in Galaxy world, are: - check that the file has uploaded properly - manually run the tool script on the input file to ensure there's not some error that I'm missing On 23 March 2016 at 12:53, Daniel Blankenberg <dan@bx.psu.edu> wrote:
Hi Amelia,
How are you uploading the files?
If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running.
A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case).
Thanks for using Galaxy,
Dan
On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov> wrote:
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module.
The tools dependencies file looks like this:
<dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers>
Debug messages when the tool runs:
galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Try using samtools to sort the BAM before uploading. samtools sort original.bam sorted.bam This will help with two things. 1. Check the original file is not corrupt 2. Put the BAM in the only order galaxy will accept. Christian University of Manchester ________________________________ From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Amelia Ireland [aireland@lbl.gov] Sent: Wednesday, March 23, 2016 8:24 PM To: Daniel Blankenberg Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Troubleshooting BAM file upload issues I'm using the standard 'Upload file' tool, and choosing some local files. I can successfully upload these files to the galaxy installation on my laptop and to the usegalaxy.org<http://usegalaxy.org> server, but uploading them to the galaxy installation on the server results in their appearing to be empty and the error message about uploaded file containing inappropriate content. I'm not sure where to proceed from here. Some things I'd like to explore, but am not sure how to do so in Galaxy world, are: - check that the file has uploaded properly - manually run the tool script on the input file to ensure there's not some error that I'm missing On 23 March 2016 at 12:53, Daniel Blankenberg <dan@bx.psu.edu<mailto:dan@bx.psu.edu>> wrote: Hi Amelia, How are you uploading the files? If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running. A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case). Thanks for using Galaxy, Dan On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov<mailto:aireland@lbl.gov>> wrote: On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com<mailto:bjoern.gruening@gmail.com>> wrote: Hi Amelia, can you pleae check if you have samtools installed in your PATH? I've got it in my tool dependencies dir and it's also available as an Environment module. The tools dependencies file looks like this: <dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers> Debug messages when the tool runs: galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs<http://galaxy.jobs/> DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Christian, Thanks for the tip. I tried this, but to no avail. I think that if the problem was with the file ordering, I wouldn't be able to upload it successfully to other galaxy installations (e.g. the main Galaxy server, Galaxy install on my laptop). It seems as though the issue is either the file upload process somehow doing weird things to the file, or something going wrong in launching samtools and running the script to process the uploaded file. Thanks, Amelia. On 24 March 2016 at 02:03, Christian Brenninkmeijer < christian.brenninkmeijer@manchester.ac.uk> wrote:
Try using samtools to sort the BAM before uploading.
samtools sort original.bam sorted.bam
This will help with two things.
1. Check the original file is not corrupt
2. Put the BAM in the only order galaxy will accept.
Christian University of Manchester ------------------------------ *From:* galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Amelia Ireland [aireland@lbl.gov] *Sent:* Wednesday, March 23, 2016 8:24 PM *To:* Daniel Blankenberg *Cc:* galaxy-dev@lists.galaxyproject.org *Subject:* Re: [galaxy-dev] Troubleshooting BAM file upload issues
I'm using the standard 'Upload file' tool, and choosing some local files. I can successfully upload these files to the galaxy installation on my laptop and to the usegalaxy.org server, but uploading them to the galaxy installation on the server results in their appearing to be empty and the error message about uploaded file containing inappropriate content.
I'm not sure where to proceed from here. Some things I'd like to explore, but am not sure how to do so in Galaxy world, are:
- check that the file has uploaded properly - manually run the tool script on the input file to ensure there's not some error that I'm missing
On 23 March 2016 at 12:53, Daniel Blankenberg <dan@bx.psu.edu> wrote:
Hi Amelia,
How are you uploading the files?
If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running.
A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case).
Thanks for using Galaxy,
Dan
On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov> wrote:
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote:
Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module.
The tools dependencies file looks like this:
<dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers>
Debug messages when the tool runs:
galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Amelia, I’ve provided a patch below that will disable the html check, so you might be able to see the uploaded file content to help with debugging. Thanks, Dan $ git diff diff --git a/lib/galaxy/datatypes/sniff.py b/lib/galaxy/datatypes/sniff.py index ff9584d..5b608ee 100644 --- a/lib/galaxy/datatypes/sniff.py +++ b/lib/galaxy/datatypes/sniff.py @@ -440,8 +440,6 @@ def handle_uploaded_dataset_file( filename, datatypes_registry, ext='auto', is_m if check_binary( filename ): if not Binary.is_ext_unsniffable(ext) and not datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ): raise InappropriateDatasetContentError( 'The binary uploaded file contains inappropriate content.' ) - elif check_html( filename ): - raise InappropriateDatasetContentError( 'The uploaded file contains inappropriate HTML content.' ) return ext AUTO_DETECT_EXTENSIONS = [ 'auto' ] # should 'data' also cause auto detect? diff --git a/tools/data_source/upload.py b/tools/data_source/upload.py index 59fa187..0d659c8 100644 --- a/tools/data_source/upload.py +++ b/tools/data_source/upload.py @@ -280,11 +280,6 @@ def add_file( dataset, registry, json_file, output_path ): err_msg = "You must manually set the 'File Format' to '%s' when uploading %s files." % ( ext.capitalize(), ext ) file_err( err_msg, dataset, json_file ) return - if not data_type: - # We must have a text file - if check_html( dataset.path ): - file_err( 'The uploaded file contains inappropriate HTML content', dataset, json_file ) - return if data_type != 'binary': if link_data_only == 'copy_files': if dataset.type in ( 'server_dir', 'path_paste' ) and data_type not in [ 'gzip', 'bz2', 'zip' ]: On Mar 25, 2016, at 2:32 PM, Amelia Ireland <aireland@lbl.gov> wrote:
Hi Christian,
Thanks for the tip. I tried this, but to no avail. I think that if the problem was with the file ordering, I wouldn't be able to upload it successfully to other galaxy installations (e.g. the main Galaxy server, Galaxy install on my laptop).
It seems as though the issue is either the file upload process somehow doing weird things to the file, or something going wrong in launching samtools and running the script to process the uploaded file.
Thanks, Amelia.
On 24 March 2016 at 02:03, Christian Brenninkmeijer <christian.brenninkmeijer@manchester.ac.uk> wrote: Try using samtools to sort the BAM before uploading.
samtools sort original.bam sorted.bam
This will help with two things.
1. Check the original file is not corrupt
2. Put the BAM in the only order galaxy will accept.
Christian University of Manchester From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Amelia Ireland [aireland@lbl.gov] Sent: Wednesday, March 23, 2016 8:24 PM To: Daniel Blankenberg Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Troubleshooting BAM file upload issues
I'm using the standard 'Upload file' tool, and choosing some local files. I can successfully upload these files to the galaxy installation on my laptop and to the usegalaxy.org server, but uploading them to the galaxy installation on the server results in their appearing to be empty and the error message about uploaded file containing inappropriate content.
I'm not sure where to proceed from here. Some things I'd like to explore, but am not sure how to do so in Galaxy world, are:
- check that the file has uploaded properly - manually run the tool script on the input file to ensure there's not some error that I'm missing
On 23 March 2016 at 12:53, Daniel Blankenberg <dan@bx.psu.edu> wrote: Hi Amelia,
How are you uploading the files?
If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running.
A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case).
Thanks for using Galaxy,
Dan
On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov> wrote:
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote: Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module.
The tools dependencies file looks like this:
<dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers>
Debug messages when the tool runs:
galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (5)
-
Amelia Ireland -
Björn Grüning -
Christian Brenninkmeijer -
Daniel Blankenberg -
Hans-Rudolf Hotz