June 2014 - galaxy-dev - lists.galaxyproject.org

Cannot find 'Export this revision' option on main or test toolsheds
by Will Holtz 11 Jun '14

11 Jun '14

I am running local toolshed with a recent version (latest_2014.06.02 for changeset fb68af9a775a). When viewing one of my tools, I can click on the 'Repository Actions' button and see the option 'Export this revision.' When I am using the public main or test toolsheds, I don't see the 'Export this revision' option. In all cases I am a logged in user. On the local toolshed I own the tool I am looking at, while on the main and test toolsheds I am looking at tools owned by others (a specific example would be package_biopython_1_62). Is this expected behavior? How do I generate a capsule from the main or test toolsheds to import into my local toolsheld? thanks, -Will

2 1

sorting of files in data libraries
by Sarah Diehl 11 Jun '14

11 Jun '14

Hi everyone, I uploaded a bunch of files to the Galaxy data libraries with the option "Upload files from filesystem paths" and "Link to files without copying to Galaxy". Afterwards I edited the names that are displayed in Galaxy. However, now the files are not alphabetically sorted anymore. I tried moving them around, but the order stayed the same. Is this on purpose and is there a way to change it? Thanks, Sarah

2 1

Join Paired-End reads (version 1.0.3)
by Julien Daligault 11 Jun '14

11 Jun '14

Hello, I am interested by a wrapper in the galaxy-dev.jgi-psf.org : Join Paired-End reads (version 1.0.3) (https://galaxy-dev.jgi-psf.org/tool_runner?tool_id=jgi_rnd_merge_pe) I didn't find it in the tool shed, where is it possible to access to this wrapper ? Regards, Julien -- Julien Daligault Plateforme Génomique-Transcriptomique (583B) INSTITUT JACQUES MONOD CNRS-Université Paris Diderot Bât. Buffon - 15 rue Hélène Brion 75205 Paris Cedex 13 - France tel : +33 1 57 27 81 31 daligault(a)ijm.univ-paris-diderot.fr

2 1

SAM/BAM To Counts dataset generation errors
by Israel Bravo 11 Jun '14

11 Jun '14

When trying SAM/BAM to Counts I get the following error message: pysam not installed; please install it But a) pysam was not in this tool dependencies; b) pysam/0.7.7 is already installe -- [image: Israel Bravo on about.me] Israel Bravo about.me/bravobih <http://about.me/bravobih>

2 1

input file empty
by Sofie THIJS 11 Jun '14

11 Jun '14

Hi, I have a problem with uploading the picrust biom-file to start the Normalize By Copy Number, on the webpage: http://huttenhower.sph.harvard.edu/galaxy/root?tool_id=PICRUSt_normalize. In attachment is the template biom-file provided in the tutorial, so it should work. After I upload the biom in 'get data'; I can not choose the file in the window "Input file" on "the Normalize By Copy Number" page. Screenshot, notice the empty box in "Input file". Can you help me with this? Thanks, Sofie [image: Inline afbeelding 1] -- *Sofie Thijs* *PhD-student *|* FWO* Centrum voor Milieukunde (CMK) T +32(0)11 26 83 73 Universiteit Hasselt | Campus Diepenbeek Agoralaan Gebouw D | B-3590 Diepenbeek Kantoor G1.97

2 1

upload problems
by Shrum, Donald C 11 Jun '14

11 Jun '14

Hi all, I'm working with a problem with user uploaded files. After digging through the logs a bit and running the commands on at a time manually I think I've narrowed it to a permissions problem. This was confirmed by just running galaxy as root and the problem went away ;) -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/ -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf 6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat Traceback (most recent call last): File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module> __main__() File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf' -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30 /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf It does not look like galaxy is using sudo to run the script. Suggestions to work this out?

2 5

Setting multiple parameters for a workflow (API)
by Carlos del Ojo 11 Jun '14

11 Jun '14

Hi all, I am working in a project that requires running workflows with multiple parameter by using the Galaxy API. However I realised that /api/workflows webservice does not allow to set several parameters for one sigle step. You can check that ingalaxy-dist/lib/galaxy/workflow/run.py _update_step_parameters __doc__ string: PARAM_DICT = {'param': NAME, 'value': VALUE} Note that this format allows only one parameter to be set per step. I've done a change in the script so it allows setting multiple parameters per step. So the format of the PARAM_DICT would be: PARAM_DICT={'param': ['threshold','path'], 'value': [2.3,'/tmp/kYzoax02.tmp'] a list of parameters and a list of values, that will be combined and assigned to the step. I'm attaching a patch to the run.py file. Is there any thought to fix that? Could that be integrated in the main repository? We have been patching the same part of the code for the last 2 months and I think it would be interesting to add this feature permanently. Thanks, C

2 1

Non-admin tool install/import
by Karthik Gururaj 10 Jun '14

10 Jun '14

Hello, We have a centralized Galaxy installation for out cluster which can be used by many users (backend interfaces with HTCondor to run jobs). Is there any way for non-administrator users to import/install their own tools so that they may use it in their workflows (under a subsection called "unstable", for example? Ideally, without having to restart the Galaxy daemons whenever a change is made. It would be fine if these tools are not visible to other users till the administrator says so. Thanks, Karthik

3 5

inconsistent use of tempfile.mkstemp during upload causes problems
by John-Paul Robinson 10 Jun '14

10 Jun '14

There appears to be some inconsistent use of tempfile.mkstemp() within upload.py that causes problems when users import data files to galaxy from a cluster directory via the upload process and import/temp/dataset directories are on different file systems. The issue manifests when Galaxy's job directory, dataset directory and import directory are on different file systems (common for cluster environments) in conjunction with a configuration where users can copy their data files directly to the import directory from which Galaxy selects data sets to upload (as opposed to using an FTP gateway). While allowing users to copy files to an import directory rather than using the FTP gateway may not be that common, we use this configuration locally to help build a more seamless interface with our local collection of HPC resources. Users can be logged into their cluster account and move data into galaxy with a file copy command rather than having to use FTP. This configuration has worked well in our environment as long as the correct ownership configuration existed on the import directory and as long as the import directory, job temporary directory, and galaxy data set directory were all on the same file system. We now have our galaxy dataset directory on a different file system and are seeing inconsistent behavior during the upload.py runs depending on if the data is ordinary text, BAM files, or gzipped data. A subset of uploads will fail because of the way temporary files are created by Galaxy to facilitate the import and any associated conversion processes of different file types. During the import, 1) Galaxy will copy the original file to a temporary target file (converting as needed during the copy). 2) Once this first conversion step is complete, Galaxy then attempts to move the temporary file back to the original location, ie. the import directory. 3) If this move is succeeds, Galaxy completes the upload processing and the data becomes a registered data set in the user's dataset collection. Galaxy prefers the Python shutil.move method to move tempfile . This results in a simple os.rename if the files remain on the same file system. However, if os.rename raises OSError because a move was attempted across a file system boundary, shutil.move resorts to a copy2, which copies the data to the original import file and then tries to copy the file attributes (permissions and utimes) to the original import file from the source file (which will be the temporary file Galaxy created in step 1 to begin the conversion process). The os.rename and shutil.copy2 behave (and fail) differently depending on the file ownership of the original import file. The os.rename will succeed even if the Galaxy upload.py job process only maps to the group-owner of the original import file (which can be ensured with group sticky bit on the import dir or ACLs). The shutil.copy2 command, however, will fail if the Galaxy upload.py job process UID is not the user-owner of the original import file. We could ensure the os.rename succeeds by keeping the job temporary directory and the import directory on the same file system. However, it seems the temporary directories used by upload.py are inconsistent across data types which prevents this simple fix from working for all data types. When text files are imported, upload calls the sniff.* methods to perform conversion. These methods use a bare call to tempfile.mkstemp() which ensures the file is created in the directory specified by the env var $TMPDIR. For example in sniff.convert_newlines: fd, temp_name = tempfile.mkstemp() https://bitbucket.org/galaxy/galaxy-central/src/5884328e91724e9bdf4b43f012e… However, for compressed files, the upload.py script directly creates temp files but here it specifies the target directory as the same as the data set directory: fd, uncompressed = tempfile.mkstemp( prefix='data_id_%s_upload_gunzip_' % dataset.dataset_id, dir=os.path.dirname( output_path ), text=False ) https://bitbucket.org/galaxy/galaxy-central/src/5884328e91724e9bdf4b43f012e… It's not clear if there is any significance to using the data set directory as the tempdir for compressed files versus the job temporary directory for other data files. It seems like all temporary files created by upload.py should be consistently created in the same temporary location, and preferably in the job temp directory. Is there a reason that these file types use different temporary file locations? If they used the same tempfile location, we could use one consistent system configuration and ensure all our data files can be imported even when the import+tempdir are not on the same file system as the Galaxy dataset dir. It seems reasonable that all tempfile.mkstemp() calls should be unadorned and inherit the temp directory location from their environment. A more comprehensive solution that would correct the inconsistency in failures between os.rename and shutil.copy2 and also remove any constraint for Galaxy to have it's import, temp, and data set directories on the same file system, would be to simply delete the original import file before attempting the shutil.move. This would ensure the file that the upload.py job attempts to create in step 2 is new and created with full Galaxy process ownership. Finally, it seems odd that Galaxy attempts to reuse the users original import file in the first place. It seems that once galaxy begins processing the content of the to-be-imported file, it should not ever write back to that file. What's the motivation here? I'll be interested to learn more about the motivations of these different tempfile conventions and if this can be fixed in the upstream. Thanks, ~jpr

4 6

Re: [galaxy-dev] Galaxy updated botched?
by Kandalaft, Iyad 10 Jun '14

10 Jun '14

Hi Everyone, This is follow-up information/questions to the issue I ran into with the galaxy June 2nd, 2014 update. I hope to receive feedback on how to proceed. Background: - Running Galaxy (DB Schema 118) with a MySQL 5.5 back-end - When updating galaxy to the june 2nd release, the v120 DB schema has referential integrity constraints, which produced errors during the upgrade. - Completed two galaxy updates in the past 4 months without encountering this before (schema changes included) Discussion: In the past, referential integrity in the DB schema was never an issue. I checked backups and the current database to find that the database tables are using the MyISAM engine. MyISAM = no referential integrity support, no transactions. I reviewed galaxy's SQLAlchemy templates and determined that mysql_engine='InnoDB' isn't set on tables. This explains why all tables were created with the MyISAM engine. If the mysql_engine is not innodb, SQL Alchemy is supposed to drop any referential integrity constraints defined in the schema. What I don't understand is why SQL Alchemy is no longer ignoring the referential integrity constraints. Going forward, can anyone propose how I can salvage the database or continue ignoring referential integrity for now? Assuming that my limited understanding of SQLAlchemy holds water, I was looking at fixing the galaxy code base but I need some clarification on the DB schema versioning. Do I edit schema v1 and add the appropriate table args to make every table an innodb engine table or do I add a new schema and modify all tables to use the innodb engine? Alternatively, I can use DDL events def after_create(target, connection, **kw): connection.execute("ALTER TABLE %s ENGINE=InnoDB; (target.name, target.name)) Thank you for your help. Regards, Iyad Kandalaft Bioinformatics Application Developer Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada KW Neatby Bldg | éd. KW Neatby 960 Carling Ave| 960, avenue Carling Ottawa, ON | Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel: Iyad.Kandalaft(a)agr.gc.ca<mailto:Iyad.Kandalaft@agr.gc.ca> Telephone | Téléphone 613- 759-1228 Facsimile | Télécopieur 613-759-1701 Government of Canada | Gouvernement du Canada

1 0