- galaxy-dev - lists.galaxyproject.org

jobs submitted to a cluster
by Shrum, Donald C 12 Jun '14

12 Jun '14

I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center

3 5

Toolshed upload error message
by Vipin TS 12 Jun '14

12 Jun '14

Hi Greg, When I trying to upload a next release version o my https://toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf converter program to Community Toolshed, I am getting an internal error message. I am uploading a tar.gz file to the page https://toolshed.g2.bx.psu.edu/ but this fails. Could you please pass the error message or let me how I can add new files to the repository and delete the old version. Thanks, Vipin | Rätsch Lab

3 3

Re: [galaxy-dev] Galaxy updated botched?
by Kandalaft, Iyad 12 Jun '14

12 Jun '14

This is a follow up for those that are interested with regards to my failed schema upgrade. I believe I have determined why all the tables are set to use the MyISAM engine. When I initialized galaxy on our enterprise servers, they were running a dated version of CentOS. Hence, the OS defaults to a dated MySQL version. MySQL only switched to using InnoDB (over MyISAM) as the default engine in version 5.5. If I'm not mistaken, I initialized galaxy in MySQL 5.1 without changing the default engine to InnoDB (big mistake). Due to my ignorance, I will now have to try to compare v118 of our database with a new install of galaxy running schema v118. Then, I will try "migrating" to the true schema state without destroying the data. I do feel that I should still modify the galaxy schema to set the MySQL engine to InnoDB to thwart problems like this for other unsuspecting users. If anyone can point me at some documentation about how Galaxy schema changes should occur in this case, that would be great. As per my previous comment, I'm not certain whether I would edit all schema versions to ensure table definitions include the mysql_engine=InnoDB attribute or whether using a DDL event in SQLAlchemy. I suspect that setting this option globally would mean that future developers don't need to remember to define mysql_engine on every new table. Iyad Kandalaft Microbial Biodiversity Bioinformatics Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada 960 Carling Ave.| 960 Ave. Carling Ottawa, ON| Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel Iyad.Kandalaft(a)agr.gc.ca Telephone | Téléphone 613-759-1228 Facsimile | Télécopieur 613-759-1701 Teletypewriter | Téléimprimeur 613-773-2600 Government of Canada | Gouvernement du Canada From: Kandalaft, Iyad Sent: Tuesday, June 10, 2014 1:39 PM To: 'galaxy-dev(a)bx.psu.edu' Subject: Re: Galaxy updated botched? Hi Everyone, This is follow-up information/questions to the issue I ran into with the galaxy June 2nd, 2014 update. I hope to receive feedback on how to proceed. Background: - Running Galaxy (DB Schema 118) with a MySQL 5.5 back-end - When updating galaxy to the june 2nd release, the v120 DB schema has referential integrity constraints, which produced errors during the upgrade. - Completed two galaxy updates in the past 4 months without encountering this before (schema changes included) Discussion: In the past, referential integrity in the DB schema was never an issue. I checked backups and the current database to find that the database tables are using the MyISAM engine. MyISAM = no referential integrity support, no transactions. I reviewed galaxy's SQLAlchemy templates and determined that mysql_engine='InnoDB' isn't set on tables. This explains why all tables were created with the MyISAM engine. If the mysql_engine is not innodb, SQL Alchemy is supposed to drop any referential integrity constraints defined in the schema. What I don't understand is why SQL Alchemy is no longer ignoring the referential integrity constraints. Going forward, can anyone propose how I can salvage the database or continue ignoring referential integrity for now? Assuming that my limited understanding of SQLAlchemy holds water, I was looking at fixing the galaxy code base but I need some clarification on the DB schema versioning. Do I edit schema v1 and add the appropriate table args to make every table an innodb engine table or do I add a new schema and modify all tables to use the innodb engine? Alternatively, I can use DDL events def after_create(target, connection, **kw): connection.execute("ALTER TABLE %s ENGINE=InnoDB; (target.name, target.name)) Thank you for your help. Regards, Iyad Kandalaft Bioinformatics Application Developer Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada KW Neatby Bldg | éd. KW Neatby 960 Carling Ave| 960, avenue Carling Ottawa, ON | Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel: Iyad.Kandalaft(a)agr.gc.ca<mailto:Iyad.Kandalaft@agr.gc.ca> Telephone | Téléphone 613- 759-1228 Facsimile | Télécopieur 613-759-1701 Government of Canada | Gouvernement du Canada

4 7

Cannot find 'Export this revision' option on main or test toolsheds
by Will Holtz 11 Jun '14

11 Jun '14

I am running local toolshed with a recent version (latest_2014.06.02 for changeset fb68af9a775a). When viewing one of my tools, I can click on the 'Repository Actions' button and see the option 'Export this revision.' When I am using the public main or test toolsheds, I don't see the 'Export this revision' option. In all cases I am a logged in user. On the local toolshed I own the tool I am looking at, while on the main and test toolsheds I am looking at tools owned by others (a specific example would be package_biopython_1_62). Is this expected behavior? How do I generate a capsule from the main or test toolsheds to import into my local toolsheld? thanks, -Will

2 1

sorting of files in data libraries
by Sarah Diehl 11 Jun '14

11 Jun '14

Hi everyone, I uploaded a bunch of files to the Galaxy data libraries with the option "Upload files from filesystem paths" and "Link to files without copying to Galaxy". Afterwards I edited the names that are displayed in Galaxy. However, now the files are not alphabetically sorted anymore. I tried moving them around, but the order stayed the same. Is this on purpose and is there a way to change it? Thanks, Sarah

2 1

Join Paired-End reads (version 1.0.3)
by Julien Daligault 11 Jun '14

11 Jun '14

Hello, I am interested by a wrapper in the galaxy-dev.jgi-psf.org : Join Paired-End reads (version 1.0.3) (https://galaxy-dev.jgi-psf.org/tool_runner?tool_id=jgi_rnd_merge_pe) I didn't find it in the tool shed, where is it possible to access to this wrapper ? Regards, Julien -- Julien Daligault Plateforme Génomique-Transcriptomique (583B) INSTITUT JACQUES MONOD CNRS-Université Paris Diderot Bât. Buffon - 15 rue Hélène Brion 75205 Paris Cedex 13 - France tel : +33 1 57 27 81 31 daligault(a)ijm.univ-paris-diderot.fr

2 1

SAM/BAM To Counts dataset generation errors
by Israel Bravo 11 Jun '14

11 Jun '14

When trying SAM/BAM to Counts I get the following error message: pysam not installed; please install it But a) pysam was not in this tool dependencies; b) pysam/0.7.7 is already installe -- [image: Israel Bravo on about.me] Israel Bravo about.me/bravobih <http://about.me/bravobih>

2 1

input file empty
by Sofie THIJS 11 Jun '14

11 Jun '14

Hi, I have a problem with uploading the picrust biom-file to start the Normalize By Copy Number, on the webpage: http://huttenhower.sph.harvard.edu/galaxy/root?tool_id=PICRUSt_normalize. In attachment is the template biom-file provided in the tutorial, so it should work. After I upload the biom in 'get data'; I can not choose the file in the window "Input file" on "the Normalize By Copy Number" page. Screenshot, notice the empty box in "Input file". Can you help me with this? Thanks, Sofie [image: Inline afbeelding 1] -- *Sofie Thijs* *PhD-student *|* FWO* Centrum voor Milieukunde (CMK) T +32(0)11 26 83 73 Universiteit Hasselt | Campus Diepenbeek Agoralaan Gebouw D | B-3590 Diepenbeek Kantoor G1.97

2 1

upload problems
by Shrum, Donald C 11 Jun '14

11 Jun '14

Hi all, I'm working with a problem with user uploaded files. After digging through the logs a bit and running the commands on at a time manually I think I've narrowed it to a permissions problem. This was confirmed by just running galaxy as root and the problem went away ;) -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/ -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf 6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat Traceback (most recent call last): File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module> __main__() File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf' -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30 /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf It does not look like galaxy is using sudo to run the script. Suggestions to work this out?

2 5

Setting multiple parameters for a workflow (API)
by Carlos del Ojo 11 Jun '14

11 Jun '14

Hi all, I am working in a project that requires running workflows with multiple parameter by using the Galaxy API. However I realised that /api/workflows webservice does not allow to set several parameters for one sigle step. You can check that ingalaxy-dist/lib/galaxy/workflow/run.py _update_step_parameters __doc__ string: PARAM_DICT = {'param': NAME, 'value': VALUE} Note that this format allows only one parameter to be set per step. I've done a change in the script so it allows setting multiple parameters per step. So the format of the PARAM_DICT would be: PARAM_DICT={'param': ['threshold','path'], 'value': [2.3,'/tmp/kYzoax02.tmp'] a list of parameters and a list of values, that will be combined and assigned to the step. I'm attaching a patch to the run.py file. Is there any thought to fix that? Could that be integrated in the main repository? We have been patching the same part of the code for the last 2 months and I think it would be interesting to add this feature permanently. Thanks, C

2 1