- galaxy-dev - lists.galaxyproject.org

Running Galaxy on a cluster with SLURM ?
by Carlos Lijeron 23 Sep '15

23 Sep '15

Hello everyone, I¹m setting up Galaxy to run on our cluster which uses SLURM as the job manager. I am wondering if any of you have any expertise in this area, ideas to share or lessons learned. I carried out the following steps listed below, but I can¹t make the final connection just yet. Any pointers will be greatly appreciated. 1. Setup Galaxy to run on the head node 2. Setup MySQL and it¹s running well with Galaxy 3. Setup the drmaa-python library inside our Galaxy folder Main questions: What would be the next step to take? Where to setup Galaxy to use an exclusive pool of resources (called partition in SLURM). Thank you all ! Carlos.

2 1

Current directory in the galaxy job file script
by Ping Luo 22 Sep '15

22 Sep '15

I installed the latest Galaxy (v15.07) and tested with a job. I found my local environment setup file was not executed. The line of code that executes the local setup file is [ -f "config/local_env.sh" ] && . config/local_env.sh Further looking into the galaxy_xx.sh file shows the current directory of the script is $working_directory before "cd $working_directory" is executed. This is not the case in previous versions, where the job's current directory is the root of the Galaxy server in the beginning, until "cd $working_directory" is executed. I think making the current directory first at the root of the Galaxy server is better since I don't need to put a full path in galaxy.ini for the environment_setup_file. Will you change this back to it was? Thanks, Ping

1 0

uWSGI --daemonize option should be replaced with --daemonize2
by Federico Zambelli 22 Sep '15

22 Sep '15

Hello guys, I'm configuring a new Galaxy production instance and I did many tries with uWSGI. I have found that the -d (or --daemonize) option that the Galaxy wiki suggests should be replaced with the --daemonize2 option. In fact, when using the -d option for uWSGI it often happens that some (or sometimes all) of the uwsgi workers does not correctly bind to the Galaxy app when uWSGI is loaded just after Galaxy, as when you set up an init.d galaxy launch script. I don't know exactly why it happens but I think it has to do with the fact that uWSGI daemonize before the Galaxy server is fully loaded and this was giving me a lot of headaches. Example: This is a typical uWSGI log that I had at launch when using -d (or --daemonize) using six workers spawned uWSGI master process (pid: 32142) spawned uWSGI worker 1 (pid: 32183, cores: 4) spawned uWSGI worker 2 (pid: 32184, cores: 4) spawned uWSGI worker 3 (pid: 32185, cores: 4) galaxy.queue_worker INFO 2015-09-22 13:53:22,348 Binding and starting galaxy control worker for main.1 galaxy.queue_worker INFO 2015-09-22 13:53:22,351 Binding and starting galaxy control worker for main.2 spawned uWSGI worker 4 (pid: 32188, cores: 4) spawned uWSGI worker 5 (pid: 32189, cores: 4) galaxy.queue_worker INFO 2015-09-22 13:53:22,356 Binding and starting galaxy control worker for main.4 spawned uWSGI worker 6 (pid: 32191, cores: 4) *** Stats server enabled on 127.0.0.1:9191 fd: 20 *** galaxy.queue_worker INFO 2015-09-22 13:53:22,360 Binding and starting galaxy control worker for main.5 As you can see worker 3 and worker 6 are not bound to galaxy. And this behaviour is not deterministic: sometimes more workers are missing and in some cases I had no workers bound to Galaxy and so it was not accessible. Some other times instead all the workers were correctly bound to Galaxy. Using --daemonize2 ensures that uWSGI daemonize after the app (Galaxy) is fully loaded thus removing this strange behaviour. Best, Federico

1 0

parse history file/uploaded file with dynamic_options script
by Linda Bakker 22 Sep '15

22 Sep '15

Hi All, I am a bit lost here. In my xml I have the following sections: <inputs> <param type="data" format="tabular" name="amplitudes" label="amplitude file"/> <param name="samples" type="select" refresh_on_change="true" display="checkboxes" multiple="true" label="Mark quality control samples" dynamic_options='get_column_names(amplitudes)'/> </inputs> <code file="dynamic_options.py" /> In my dynamic_options.py script, I want to parse the headers from 'amplitudes' file and show them in the checkboxes section. My problem seems to be the following. In python, amplitudes is a galaxy object: <galaxy.model.HistoryDatasetAssociation object at 0x7f4fc020a210> How do I parse/access this file to get the headers? It seems to be partly the same problem as asked last week in this conversation: http://dev.list.galaxyproject.org/Question-about-using-dynamic-options-and-… Thanks! Linda -- Linda Bakker Bioinformatics Research Assistant Plant Research International, Wageningen UR Building 107 (Radix), room W1.Fa.054 Droevendaalsesteeg 1, 6708 PB, Wageningen, The Netherlands tel: +31-317-480756 email: linda.bakker(a)wur.nl

2 2

Need urgent help: ItemOwnershipException: History is not owned by user
by Zuzanna K. Filutowska 21 Sep '15

21 Sep '15

Dear All, I have recently started to receive errors like that while new LDAP users are trying to access the history or upload a file, while they can log into galaxy just fine. Everything under /home/galaxy is owned by galaxy:galaxy. I am using apache proxy. Details of error: galaxy.web.framework.decorators ERROR 2015-09-21 12:48:09,338 Uncaught exception in exposed API method: Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py", line 135, in decorator rval = func( self, trans, *args, **kwargs) File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/tools.py", line 234, in create target_history = self.history_manager.get_owned( decoded_id, trans.user, current_history=trans.history ) File "/home/galaxy/galaxy-dist/lib/galaxy/managers/secured.py", line 94, in get_owned return self.error_unless_owner( item, user, **kwargs ) File "/home/galaxy/galaxy-dist/lib/galaxy/managers/secured.py", line 104, in error_unless_owner raise exceptions.ItemOwnershipException( "%s is not owned by user" % ( self.model_class.__name__ ) ) ItemOwnershipException: History is not owned by user -- Pozdrawiam, -- Zuzanna K. Filutowska, HPC Systems Administrator Poznan Supercomputing and Networking Center Institute of Bioorganic Chemistry Polish Academy of Sciences Seize the day boys! Make your lifes extraordinary! --John Keating

4 11

Time Out
by Daniel Siqueira de Oliveira 21 Sep '15

21 Sep '15

Hello, i'm repeatexplorer user and i am with difficult for download of a archive. The archive weighs 13.5GB and always arriving at 600-700MB of download, the download fail. I need a solution. Thank you. -- Daniel Siqueira de Oliveira Acadêmico de Ciências Biológicas- UFSM Iniciação científica- LABDROS Fone: (55)- 9699-5834

1 0

Problem with load balancing.
by Zuzanna K. Filutowska 18 Sep '15

18 Sep '15

Dear All, I am trying to run galaxy in multiple processes. I followed the documentation: https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling And while the python processes seem to run fine, the balacing is broken - while accessing Galaxy server via www, I can only download a text file with "internal server error" inside. Here is my httpd.conf: http://www.wklej.org/id/1799330/ And here is my universe_wsgi.ini file: http://www.wklej.org/id/1799334/ Any help would be appreciated. -- Pozdrawiam, -- Zuzanna K. Filutowska, HPC Systems Administrator Poznan Supercomputing and Networking Center Institute of Bioorganic Chemistry Polish Academy of Sciences Seize the day boys! Make your lifes extraordinary! --John Keating

1 1

Error when Installing package_galaxy_utils_1_0
by Leslie Herrera 18 Sep '15

18 Sep '15

Hello: My lab is currently in the process of setting Galaxy up on our server. I have been installing the appropriate tools, but I am having difficulty installing the package_galaxy_utils_1_0. The error message is below. If this is something that is occuring on our end, would you please send instuctions on how to fix it? If I have sent this email to wrong address or you need more information, please let me know. Thank you very much for your help. Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz Extracting in /tmp/tmp3dIIS_ Now working in /tmp/tmp3dIIS_/distribute-0.6.38 Building a Distribute egg in /scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0 Traceback (most recent call last): File "setup.py", line 248, in <module> scripts = scripts, File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/tmp/tmp3dIIS_/distribute-0.6.38/setuptools/dist.py", line 225, in __init__ _Distribution.__init__(self,attrs) File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__ self.finalize_options() File "/tmp/tmp3dIIS_/distribute-0.6.38/setuptools/dist.py", line 257, in finalize_options ep.require(installer=self.fetch_build_egg) File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2027, in require working_set.resolve(self.dist.requires(self.extras),env,installer)) File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2237, in requires dm = self._dep_map File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2466, in _dep_map self.__dep_map = self._compute_dependencies() File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2499, in _compute_dependencies common = frozenset(reqs_for_extra(None)) File "/tmp/tmp3dIIS_/distribute-0.6.38/pkg_resources.py", line 2496, in reqs_for_extra if req.marker_fn(override={'extra':extra}): File "/tmp/tmp3dIIS_/distribute-0.6.38/_markerlib/markers.py", line 109, in marker_fn return eval(compiled_marker, environment) File "<environment marker>", line 1, in <module> NameError: name 'sys_platform' is not defined /scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0/distribute-0.6.38-py2.7.egg Traceback (most recent call last): File "setup.py", line 9, in <module> use_setuptools() File "/scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0/distribute_setup.py", line 152, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "/scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0/distribute_setup.py", line 132, in _do_download _build_egg(egg, tarball, to_dir) File "/scratch/tmp-toolshed-mtdKjroZR/galaxy_sequence_utils-1.0.0/distribute_setup.py", line 123, in _build_egg raise IOError('Could not build the egg.') IOError: Could not build the egg. -- Leslie A Herrera Starr Lab Junior Specialist MCB, 139 Briggs Hall University of California, Davis laherrera(a)ucdavis.edu

2 2

Re: [galaxy-dev] need help badly
by Fernandez Edgar 17 Sep '15

17 Sep '15

Hello gents, I figured out something: it has nothing to do with galaxy but maybe one of you could explain me why I'm having this behavior : I connect ssh on the torque master (where the galaxy web service is running) and: 1. su - galaxy 2. cd /home/galaxy/galaxy-prod/database/job_working_directory/000 I execute a job #1234, it fails with the error mentioned below. I execute the exact same job #1235 and while it's running I enter the directory 1235 --> job finishes successfully. My directory /home/galaxy is an nfs mount in /etc/fstab. Any ideas? Cordialement / Regards, Edgar Fernandez De : Fernandez Edgar Envoyé : September-17-15 9:44 AM À : galaxy-dev(a)bx.psu.edu Objet : [galaxy-dev] need help badly Hello gents, I hope everyone is doing ok... I'm in need of your help so badly it's a little bit urgent: So my galaxy wasn't used in a while but the last time everything was working well: 1. The web-site was functional 2. The local handler upload the files locally on the master 3. Torque was shooting all the jobs correctly and getting an answer back. But today, one of my students tried some typical trimmomatic and all his jobs came back with the following error: Job output not returned from cluster. However, when I checked his datasets with that error, the view data (icon that is an eye) had some results. Also, when I looking inside the logs, I got: Job output not returned from cluster: [Errno 2] No such file or directory: '/home/galaxy/galaxy-prod/database/job_working_directory/000/406/galaxy_406.o' But that file galaxy_406.o existed and had TrimmomaticPE: Completed successfully I tried changing the value from 0 to 5 of "retry_job_output_collection" in the config/galaxy.ini file without any success. I've look through the internet and I saw a lot of people had that error but no one found a solution. Can you guys please help? Cordialement / Regards, Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication * Bur. : 1-514-343-6111 poste 16568 Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218

3 5

need help badly
by Fernandez Edgar 17 Sep '15

17 Sep '15

Hello gents, I hope everyone is doing ok... I'm in need of your help so badly it's a little bit urgent: So my galaxy wasn't used in a while but the last time everything was working well: 1. The web-site was functional 2. The local handler upload the files locally on the master 3. Torque was shooting all the jobs correctly and getting an answer back. But today, one of my students tried some typical trimmomatic and all his jobs came back with the following error: Job output not returned from cluster. However, when I checked his datasets with that error, the view data (icon that is an eye) had some results. Also, when I looking inside the logs, I got: Job output not returned from cluster: [Errno 2] No such file or directory: '/home/galaxy/galaxy-prod/database/job_working_directory/000/406/galaxy_406.o' But that file galaxy_406.o existed and had TrimmomaticPE: Completed successfully I tried changing the value from 0 to 5 of "retry_job_output_collection" in the config/galaxy.ini file without any success. I've look through the internet and I saw a lot of people had that error but no one found a solution. Can you guys please help? Cordialement / Regards, Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication * Bur. : 1-514-343-6111 poste 16568 Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218

1 0