Hi Amelia, I’ve provided a patch below that will disable the html check, so you might be able to see the uploaded file content to help with debugging. Thanks, Dan $ git diff diff --git a/lib/galaxy/datatypes/sniff.py b/lib/galaxy/datatypes/sniff.py index ff9584d..5b608ee 100644 --- a/lib/galaxy/datatypes/sniff.py +++ b/lib/galaxy/datatypes/sniff.py @@ -440,8 +440,6 @@ def handle_uploaded_dataset_file( filename, datatypes_registry, ext='auto', is_m if check_binary( filename ): if not Binary.is_ext_unsniffable(ext) and not datatypes_registry.get_datatype_by_extension( ext ).sniff( filename ): raise InappropriateDatasetContentError( 'The binary uploaded file contains inappropriate content.' ) - elif check_html( filename ): - raise InappropriateDatasetContentError( 'The uploaded file contains inappropriate HTML content.' ) return ext AUTO_DETECT_EXTENSIONS = [ 'auto' ] # should 'data' also cause auto detect? diff --git a/tools/data_source/upload.py b/tools/data_source/upload.py index 59fa187..0d659c8 100644 --- a/tools/data_source/upload.py +++ b/tools/data_source/upload.py @@ -280,11 +280,6 @@ def add_file( dataset, registry, json_file, output_path ): err_msg = "You must manually set the 'File Format' to '%s' when uploading %s files." % ( ext.capitalize(), ext ) file_err( err_msg, dataset, json_file ) return - if not data_type: - # We must have a text file - if check_html( dataset.path ): - file_err( 'The uploaded file contains inappropriate HTML content', dataset, json_file ) - return if data_type != 'binary': if link_data_only == 'copy_files': if dataset.type in ( 'server_dir', 'path_paste' ) and data_type not in [ 'gzip', 'bz2', 'zip' ]: On Mar 25, 2016, at 2:32 PM, Amelia Ireland <aireland@lbl.gov> wrote:
Hi Christian,
Thanks for the tip. I tried this, but to no avail. I think that if the problem was with the file ordering, I wouldn't be able to upload it successfully to other galaxy installations (e.g. the main Galaxy server, Galaxy install on my laptop).
It seems as though the issue is either the file upload process somehow doing weird things to the file, or something going wrong in launching samtools and running the script to process the uploaded file.
Thanks, Amelia.
On 24 March 2016 at 02:03, Christian Brenninkmeijer <christian.brenninkmeijer@manchester.ac.uk> wrote: Try using samtools to sort the BAM before uploading.
samtools sort original.bam sorted.bam
This will help with two things.
1. Check the original file is not corrupt
2. Put the BAM in the only order galaxy will accept.
Christian University of Manchester From: galaxy-dev [galaxy-dev-bounces@lists.galaxyproject.org] on behalf of Amelia Ireland [aireland@lbl.gov] Sent: Wednesday, March 23, 2016 8:24 PM To: Daniel Blankenberg Cc: galaxy-dev@lists.galaxyproject.org Subject: Re: [galaxy-dev] Troubleshooting BAM file upload issues
I'm using the standard 'Upload file' tool, and choosing some local files. I can successfully upload these files to the galaxy installation on my laptop and to the usegalaxy.org server, but uploading them to the galaxy installation on the server results in their appearing to be empty and the error message about uploaded file containing inappropriate content.
I'm not sure where to proceed from here. Some things I'd like to explore, but am not sure how to do so in Galaxy world, are:
- check that the file has uploaded properly - manually run the tool script on the input file to ensure there's not some error that I'm missing
On 23 March 2016 at 12:53, Daniel Blankenberg <dan@bx.psu.edu> wrote: Hi Amelia,
How are you uploading the files?
If you are using a copy and pasted URL, can you make sure that the example file exists and is accessible to the machine where the Galaxy job is running.
A 404 error, for example, can often masquerade as a “The uploaded file contains inappropriate HTML content” error (although this shouldn’t be the outcome of this tool, it should be a red dataset stating a 404 error, in this case).
Thanks for using Galaxy,
Dan
On Mar 23, 2016, at 3:30 PM, Amelia Ireland <aireland@lbl.gov> wrote:
On 23 March 2016 at 01:00, Björn Grüning <bjoern.gruening@gmail.com> wrote: Hi Amelia,
can you pleae check if you have samtools installed in your PATH?
I've got it in my tool dependencies dir and it's also available as an Environment module.
The tools dependencies file looks like this:
<dependency_resolvers> <!-- the default configuration, first look for dependencies installed from the toolshed --> <tool_shed_packages /> <galaxy_packages /> <galaxy_packages versionless="true" /> <modules modulecmd="/usr/bin/modulecmd" /> <modules modulecmd="/usr/bin/modulecmd" find_by="avail" versionless="true" /> </dependency_resolvers>
Debug messages when the tool runs:
galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Building dependency shell command for dependency 'samtools' galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Find dependency samtools version None galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver tool_shed_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver galaxy_packages returned <galaxy.tools.deps.resolvers.NullDependency object at 0x33c71d0> (isnull? True) galaxy.tools.deps DEBUG 2016-03-23 12:23:05,556 Resolver modules returned <galaxy.tools.deps.resolvers.modules.ModuleDependency object at 0x9fc9590> (isnull? False) galaxy.jobs.runners DEBUG 2016-03-23 12:23:05,567 (217) command is: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/tool_script.sh; return_code=$?; MODULEPATH=/usr/share/Modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/common/usg/Modules/modulefiles:/usr/syscom/nsg/modulefiles:/usr/common/jgi/Modules/modulefiles; export MODULEPATH; eval `/usr/bin/modulecmd sh load samtools`; python "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/set_metadata_NFVQiC.py" "/global/homes/a/aireland/galaxy/database/tmp/tmp5vpBro" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy.json" "/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_in_HistoryDatasetAssociation_349_ZqiKgT,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_kwds_HistoryDatasetAssociation_349_CH5HZd,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_out_HistoryDatasetAssociation_349_Hb4hhP,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_results_HistoryDatasetAssociation_349_2e3dj6,/global/homes/a/aireland/galaxy/database/files/000/dataset_320.dat,/global/homes/a/aireland/galaxy/database/job_working_directory/000/217/metadata_override_HistoryDatasetAssociation_349_pEc2Gh" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,583 (217) executing job script: /global/homes/a/aireland/galaxy/database/job_working_directory/000/217/galaxy_217.sh galaxy.jobs DEBUG 2016-03-23 12:23:05,599 (217) Persisting job destination (destination id: local:///) galaxy.jobs.runners.local DEBUG 2016-03-23 12:23:05,799 execution finished: /global/homes/a/aireland/galaxy/database/job_working_directory/000/216/galaxy_216.sh
Am 22.03.2016 um 19:28 schrieb Amelia Ireland:
I'm setting up a Galaxy instance on a new server and am having some issues with BAM file uploads. I'm using the BAM files supplied with JBrowse for testing purposes. I have Galaxy installed on my laptop and file uploads are working fine, so it's evidently an issue with the server and/or OS configuration.
Attempting to upload a BAM file returns the message
"The uploaded binary file contains inappropriate content"
and the upload is empty.
The server is running Scientific Linux and uses the Environment Modules system for dependency management. I can successfully upload bed and bigwig files, which suggests that samtools is being loaded and functioning correctly.
If someone can suggest some troubleshooting steps, I would be most grateful.
Thanks! Amelia
Integrated Microbial Genomics (IMG) JGI / LBL
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/