- galaxy-dev - lists.galaxyproject.org

tool runtimes
by Matthias Bernt 13 Sep '17

13 Sep '17

Dear list, I have the impression that on our cluster the "auxiliary" computations (e.g. setting metadata) is very slow (minutes) compared to the run time of the actual tools (often just seconds). Do you have any suggestion how I can tweak this or how I can analyze this more detailed? Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

1 0

September Galaxy Newsletter
by Dave Clements 06 Sep '17

06 Sep '17

1 0

installation/conda problems
by Matthias Bernt 06 Sep '17

06 Sep '17

Dear list, I think I have messed up the environment for one tool (IntaRNA) -- can someone tell me what to do to get a clear environment? Here is what happened. I have started the installation (via ansible) and accidentally aborted it. Then the tool gave me the following error: ``` Failed to activate conda environment! Error was: CondaEnvironmentError: Environment error: Cannot activate environment bash. User does not have write access for conda symlinks. Fatal error: Exit code 1 (Error occurred. Please check Tool Standard Error) ``` Since uninstalling and reinstallation (via ansible) gave the same result I tried to reinstall from the Galaxy UI. Now I got (even if if the admin interface list the tool as installed): ``` Fatal error: Exit code 127 (Error occurred. Please check Tool Standard Error) /gpfs1/data/galaxy_server/galaxy-dev/jobs_dir/000/362/tool_script.sh: line 25: IntaRNA: command not found ``` Is there a list of directories and files that I could clear to get rid of any trace of the tool and start anew? Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

3 3

Issues with new galaxy and galaxy_lib
by Evan Clark 04 Sep '17

04 Sep '17

Is there a way to change the location of the galaxy lib directory? We migrated our galaxy and it is looking for the lib on the wrong location. Additionally this is causing errors like the one below the first error. galaxy.jobs.runners DEBUG 2017-09-01 09:35:13,673 executing external set_meta script for job 13: GALAXY_LIB="/var/web_services/galaxy/lib"; if [ "$GALAXY_LIB" != "None" ]; then if [ -n "$PYTHONPATH" ]; then PYTHONPATH="$GALAXY_LIB:$PYTHONPATH"; else PYTHONPATH="$GALAXY_LIB"; fi; export PYTHONPATH; fi; GALAXY_VIRTUAL_ENV="/var/web_services/galaxy/.venv"; if [ "$GALAXY_VIRTUAL_ENV" != "None" -a -z "$VIRTUAL_ENV" -a -f "$GALAXY_VIRTUAL_ENV/bin/activate" ]; then . "$GALAXY_VIRTUAL_ENV/bin/activate"; fi; python "/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/set_metadata_7mPxU0.py" "/mnt/ceph/home/galaxy/galaxy/scratch/tmpEn3vYB" "/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/working/galaxy.json" "/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/metadata_in_HistoryDatasetAssociation_13_WlHC7F,/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/metadata_kwds_HistoryDatasetAssociation_13_w_LWxj,/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/metadata_out_HistoryDatasetAssociation_13_Wjmehj,/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/metadata_results_HistoryDatasetAssociation_13_r2SVjv,/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/galaxy_dataset_13.dat,/mnt/ceph/home/galaxy/galaxy/scratch/jobs_directory/000/13/metadata_override_HistoryDatasetAssociation_13_OXLKhY" 5242880 edu/koko/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:48.0) Gecko/ Firefox/48.0" galaxy.jobs.runners DEBUG 2017-09-03 15:33:54,164 execution of external set_meta for job 19 finished galaxy.jobs.output_checker DEBUG 2017-09-03 15:33:54,241 Tool produced standard error failing job - [Traceback (most recent call last): File "/mnt/ceph/home/galaxy/galaxy/tools/data_source/upload.py", line 19, in <module> from galaxy import util ImportError: No module named galaxy ]

1 0

unipept visualization
by Matthias Bernt 01 Sep '17

01 Sep '17

Dear list, I can not get the wonderful unipept visualization to work on our Galaxy instance (a test instance on the dev branch). I get the message: "Unfortunately we could not identify a suitable plugin...." The file I try to visualize was generated by unipept (pept2lca) and is of format d3_hierarchy. Could someone tell me if I need to install / configure something to get this running? Thanks a lot. Best, Matthias P.S. This might be also of interest for others, since this visualization is used in the Metaproteomics tutorial of the GTN. -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

3 7

conda problem
by Matthias Bernt 01 Sep '17

01 Sep '17

Dear list, When running rmarkdown_wgcna I get the following error in the logs: PaddingError: Placeholder of length '80' too short in package bioconda::r-wgcna-1.51-r3.3.1_0. The package must be rebuilt with conda-build > 2.0. The command that seems to lead to this is: Executing command: /gpfs1/data/galaxy_server/galaxy-dev/database/dependencies/_conda/bin/conda create -y --name mulled-v1-3bfe8da4ed798836281cc963619917 317ae960c9a828f7ea05d1670c65b19619 pandoc=1.15.0.6-0 r-getopt=1.20.0 r-rmarkdown=1.2 r-plyr=1.8.4 r-highcharter=0.4.0 r-dt=0.2 r-htmltools=0.3.5 r-wgcna=1.51 Any ideas? Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 2

[hxr@hx42.org: [SECURITY] 2017-08 security patch. Update now!]
by E. Rasche 29 Aug '17

29 Aug '17

Two high priority security vulnerabilities were recently discovered by Eric Rasche and Manabu Ishii respectively. These vulnerabilities were to cross site scripting and session fixation attacks. Detailed descriptions of these categories of vulnerabilities can be found at: - https://www.owasp.org/index.php/Cross-site_Scripting_(XSS) - https://www.owasp.org/index.php/Session_fixation Per our new security policies (https://github.com/galaxyproject/galaxy/blob/dev/SECURITY_POLICY.md) we have created patches for versions of Galaxy including 16.10, 17.01, and 17.05. **With Git** If you have cloned Galaxy from the official GitHub repository and are on a supported release_branch, you can simply `git pull` to pull in the security patch which has been applied to those branches. **Manual Patching** The 16.10 patch can apply cleanly to 17.01 as well and can be fetched using: wget https://gist.githubusercontent.com/jmchilton/760bf8ba6055b9a47a48529fcc49a4… The 17.05 patch can be fetched using: wget https://gist.githubusercontent.com/jmchilton/2623e37a2ccb1820770278e348dfef… Once the patch has been downloaded and copied to the root of your Galaxy directory, it can be applied using the following patch command: % patch -p1 < 2017augsecurity_1610.patch -or- % patch -p1 < 2017augsecurity_1705.patch If you are having trouble applying the patch feel free to email galaxy-committers(a)lists.galaxyproject.org and we will try to help. -Eric (on behalf of the Galaxy Committers) Post script: this mail was intended to go out on Thursday the 24th of August, however I failed to send it then. My apologies to the community, it will be more timely in the future.

1 0

Error while building reference genome index
by laurent.fournier＠ulg.ac.be 29 Aug '17

29 Aug '17

Hi, As the topic says, i am trying to populate the "built in" list. Fetching with the tool "Create DBKey and Reference Genome" in the Local datas works just fine but when i try to complete the procedure with "BWA-MEM index" (as well as "SAM FASTA index"), the job stops quickly. There are some shell outputs : export PATH=$PATH:'/home/mass/ifilesets/GAL/APP/galaxy/database/dependencies/_conda/envs/__samtools(a)1.3.1/bin' ; python "/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/set_metadata_WKziP5.py" "/home/mass/GAL/DATATMP/tmpsX8JCw" "/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/working/galaxy.json" "/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/metadata_in_HistoryDatasetAssociation_81_RSHoLb,/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/metadata_kwds_HistoryDatasetAssociation_81_cGTIQh,/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/metadata_out_HistoryDatasetAssociation_81_6hf9L8,/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/metadata_results_HistoryDatasetAssociation_81_0CUrCX,/home/mass/GAL/DATA/032/dataset_32404.dat,/home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/metadata_override_HistoryDatasetAssociation_81_XBr_WZ" 5242880; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2017-08-29 12:06:50,938 (80) submitting file /home/mass/ifilesets/GAL/APP/galaxy/database/jobs_directory/000/80/galaxy_80.sh galaxy.jobs.runners.drmaa DEBUG 2017-08-29 12:06:50,938 (80) native specification is: -p all_5hrs galaxy.jobs.runners.drmaa INFO 2017-08-29 12:06:51,045 (80) queued as 2333510 galaxy.jobs DEBUG 2017-08-29 12:06:51,045 (80) Persisting job destination (destination id: slurm_default) galaxy.jobs.runners.drmaa DEBUG 2017-08-29 12:06:51,238 (80/2333510) state change: job is queued and active galaxy.jobs.runners.drmaa DEBUG 2017-08-29 12:06:52,247 (80/2333510) state change: job is running galaxy.jobs.runners.drmaa DEBUG 2017-08-29 12:06:57,373 (80/2333510) state change: job finished normally galaxy.jobs.output_checker INFO 2017-08-29 12:06:57,416 Job 80: Fatal error: Exit code 1 () galaxy.jobs.output_checker DEBUG 2017-08-29 12:06:57,417 Tool exit code indicates an error, failing job. galaxy.jobs DEBUG 2017-08-29 12:06:57,482 (80) setting dataset 32404 state to ERROR galaxy.jobs INFO 2017-08-29 12:06:57,578 Collecting metrics for Job 80 galaxy.jobs DEBUG 2017-08-29 12:06:57,584 job 80 ended (finish() executed in (170.899 ms)) galaxy.model.metadata DEBUG 2017-08-29 12:06:57,586 Cleaning up external metadata files and then, in galaxy's frontend : Traceback (most recent call last): File "/home/mass/ifilesets/GAL/APP/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_sam_fasta_index_builder/2a1ac1abc3f7/data_manager_sam_fasta_index_builder/data_manager/data_manager_sam_fasta_index_builder.py", line 87, in <module> if __name__ == "__main__": main() File "/home/mass/ifilesets/GAL/APP/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_sam_fasta_index_builder/2a1ac1abc3f7/data_manager_sam_fasta_index_builder/data_manager/data_manager_sam_fasta_index_builder.py", line 82, in main build_sam_index( data_manager_dict, options.fasta_filename, target_directory, options.fasta_dbkey, sequence_id, sequence_name, data_table_name=options.data_table_name or DEFAULT_DATA_TABLE_NAME ) File "/home/mass/ifilesets/GAL/APP/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_sam_fasta_index_builder/2a1ac1abc3f7/data_manager_sam_fasta_index_builder/data_manager/data_manager_sam_fasta_index_builder.py", line 30, in build_sam_index assert os.path.exists( fasta_filename ), 'FASTA file "%s" is missing, cannot build samtools index.' % fasta_filename AssertionError: FASTA file "/home/mass/ifilesets/GAL/APP/galaxy/tool-data/mm10/seq/mm10.fa,/home/mass/ifilesets/GAL/APP/galaxy/tool-data/mm10/seq/mm10.fa" is missing, cannot build samtools index. BUT if i check, the unexisting file is right where it is supposed to : [f043030@big-cloud19 tmp]$ ls -l /home/mass/ifilesets/GAL/APP/galaxy/tool-data/mm10/seq/ total 2720208 -rw-r--r--. 1 4294967294 4294967294 2785490220 Aug 28 09:47 mm10.fa Any ideas on what i do wrong ? Regards, Laurent Fournier. --- Laurent FOURNIER IT Specialist University of Liège, GIGA +1 laurent.fournier(a)ulg.ac.be +32 497 87 96 46 www.giga.ulg.ac.be

1 0

Re: [galaxy-dev] galaxy-dev Digest, Vol 134, Issue 19
by Matthias Bernt 28 Aug '17

28 Aug '17

Dear ..., what are the file permissions for the file and the folder? Is the drmaa_external_runner.py python script run with sudo? Which release/branch are you on? Best, Matthias > We are setting up run as real user but have run into a strange issue. > When the drmaa external script is called it could not load the job json > file. All permissions are set properly and the file exists. Upon > attempting to run the script by itself this error is created. When > loading a test script to open a file there is no issue. If the drmaa > external script is run as root the same issue occurs. > > /mnt/ceph/home/galaxy/galaxy/files_directory/database/slurm/11.jt_json > Traceback (most recent call last): > File "scripts/drmaa_external_runner.py", line 157, in <module> > main() > File "scripts/drmaa_external_runner.py", line 146, in main > load_job_template_from_file(jt, json_filename) > File "scripts/drmaa_external_runner.py", line 43, in > load_job_template_from_file > f = open(filename, 'r') > IOError: [Errno 13] Permission denied: > '/mnt/ceph/home/galaxy/galaxy/files_directory/database/slurm/11.jt_json'

1 0

Error with run as real user, unable to read file
by Evan Clark 25 Aug '17

25 Aug '17

We are setting up run as real user but have run into a strange issue. When the drmaa external script is called it could not load the job json file. All permissions are set properly and the file exists. Upon attempting to run the script by itself this error is created. When loading a test script to open a file there is no issue. If the drmaa external script is run as root the same issue occurs. /mnt/ceph/home/galaxy/galaxy/files_directory/database/slurm/11.jt_json Traceback (most recent call last): File "scripts/drmaa_external_runner.py", line 157, in <module> main() File "scripts/drmaa_external_runner.py", line 146, in main load_job_template_from_file(jt, json_filename) File "scripts/drmaa_external_runner.py", line 43, in load_job_template_from_file f = open(filename, 'r') IOError: [Errno 13] Permission denied: '/mnt/ceph/home/galaxy/galaxy/files_directory/database/slurm/11.jt_json'

1 0