- galaxy-dev - lists.galaxyproject.org

inconsistent use of tempfile.mkstemp during upload causes problems
by John-Paul Robinson 10 Jun '14

10 Jun '14

There appears to be some inconsistent use of tempfile.mkstemp() within upload.py that causes problems when users import data files to galaxy from a cluster directory via the upload process and import/temp/dataset directories are on different file systems. The issue manifests when Galaxy's job directory, dataset directory and import directory are on different file systems (common for cluster environments) in conjunction with a configuration where users can copy their data files directly to the import directory from which Galaxy selects data sets to upload (as opposed to using an FTP gateway). While allowing users to copy files to an import directory rather than using the FTP gateway may not be that common, we use this configuration locally to help build a more seamless interface with our local collection of HPC resources. Users can be logged into their cluster account and move data into galaxy with a file copy command rather than having to use FTP. This configuration has worked well in our environment as long as the correct ownership configuration existed on the import directory and as long as the import directory, job temporary directory, and galaxy data set directory were all on the same file system. We now have our galaxy dataset directory on a different file system and are seeing inconsistent behavior during the upload.py runs depending on if the data is ordinary text, BAM files, or gzipped data. A subset of uploads will fail because of the way temporary files are created by Galaxy to facilitate the import and any associated conversion processes of different file types. During the import, 1) Galaxy will copy the original file to a temporary target file (converting as needed during the copy). 2) Once this first conversion step is complete, Galaxy then attempts to move the temporary file back to the original location, ie. the import directory. 3) If this move is succeeds, Galaxy completes the upload processing and the data becomes a registered data set in the user's dataset collection. Galaxy prefers the Python shutil.move method to move tempfile . This results in a simple os.rename if the files remain on the same file system. However, if os.rename raises OSError because a move was attempted across a file system boundary, shutil.move resorts to a copy2, which copies the data to the original import file and then tries to copy the file attributes (permissions and utimes) to the original import file from the source file (which will be the temporary file Galaxy created in step 1 to begin the conversion process). The os.rename and shutil.copy2 behave (and fail) differently depending on the file ownership of the original import file. The os.rename will succeed even if the Galaxy upload.py job process only maps to the group-owner of the original import file (which can be ensured with group sticky bit on the import dir or ACLs). The shutil.copy2 command, however, will fail if the Galaxy upload.py job process UID is not the user-owner of the original import file. We could ensure the os.rename succeeds by keeping the job temporary directory and the import directory on the same file system. However, it seems the temporary directories used by upload.py are inconsistent across data types which prevents this simple fix from working for all data types. When text files are imported, upload calls the sniff.* methods to perform conversion. These methods use a bare call to tempfile.mkstemp() which ensures the file is created in the directory specified by the env var $TMPDIR. For example in sniff.convert_newlines: fd, temp_name = tempfile.mkstemp() https://bitbucket.org/galaxy/galaxy-central/src/5884328e91724e9bdf4b43f012e… However, for compressed files, the upload.py script directly creates temp files but here it specifies the target directory as the same as the data set directory: fd, uncompressed = tempfile.mkstemp( prefix='data_id_%s_upload_gunzip_' % dataset.dataset_id, dir=os.path.dirname( output_path ), text=False ) https://bitbucket.org/galaxy/galaxy-central/src/5884328e91724e9bdf4b43f012e… It's not clear if there is any significance to using the data set directory as the tempdir for compressed files versus the job temporary directory for other data files. It seems like all temporary files created by upload.py should be consistently created in the same temporary location, and preferably in the job temp directory. Is there a reason that these file types use different temporary file locations? If they used the same tempfile location, we could use one consistent system configuration and ensure all our data files can be imported even when the import+tempdir are not on the same file system as the Galaxy dataset dir. It seems reasonable that all tempfile.mkstemp() calls should be unadorned and inherit the temp directory location from their environment. A more comprehensive solution that would correct the inconsistency in failures between os.rename and shutil.copy2 and also remove any constraint for Galaxy to have it's import, temp, and data set directories on the same file system, would be to simply delete the original import file before attempting the shutil.move. This would ensure the file that the upload.py job attempts to create in step 2 is new and created with full Galaxy process ownership. Finally, it seems odd that Galaxy attempts to reuse the users original import file in the first place. It seems that once galaxy begins processing the content of the to-be-imported file, it should not ever write back to that file. What's the motivation here? I'll be interested to learn more about the motivations of these different tempfile conventions and if this can be fixed in the upstream. Thanks, ~jpr

4 6

Re: [galaxy-dev] Galaxy updated botched?
by Kandalaft, Iyad 10 Jun '14

10 Jun '14

Hi Everyone, This is follow-up information/questions to the issue I ran into with the galaxy June 2nd, 2014 update. I hope to receive feedback on how to proceed. Background: - Running Galaxy (DB Schema 118) with a MySQL 5.5 back-end - When updating galaxy to the june 2nd release, the v120 DB schema has referential integrity constraints, which produced errors during the upgrade. - Completed two galaxy updates in the past 4 months without encountering this before (schema changes included) Discussion: In the past, referential integrity in the DB schema was never an issue. I checked backups and the current database to find that the database tables are using the MyISAM engine. MyISAM = no referential integrity support, no transactions. I reviewed galaxy's SQLAlchemy templates and determined that mysql_engine='InnoDB' isn't set on tables. This explains why all tables were created with the MyISAM engine. If the mysql_engine is not innodb, SQL Alchemy is supposed to drop any referential integrity constraints defined in the schema. What I don't understand is why SQL Alchemy is no longer ignoring the referential integrity constraints. Going forward, can anyone propose how I can salvage the database or continue ignoring referential integrity for now? Assuming that my limited understanding of SQLAlchemy holds water, I was looking at fixing the galaxy code base but I need some clarification on the DB schema versioning. Do I edit schema v1 and add the appropriate table args to make every table an innodb engine table or do I add a new schema and modify all tables to use the innodb engine? Alternatively, I can use DDL events def after_create(target, connection, **kw): connection.execute("ALTER TABLE %s ENGINE=InnoDB; (target.name, target.name)) Thank you for your help. Regards, Iyad Kandalaft Bioinformatics Application Developer Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada KW Neatby Bldg | éd. KW Neatby 960 Carling Ave| 960, avenue Carling Ottawa, ON | Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel: Iyad.Kandalaft(a)agr.gc.ca<mailto:Iyad.Kandalaft@agr.gc.ca> Telephone | Téléphone 613- 759-1228 Facsimile | Télécopieur 613-759-1701 Government of Canada | Gouvernement du Canada

1 0

Toolshed update list?
by Eric Rasche 10 Jun '14

10 Jun '14

Is there any way to access a list of repository updates in a toolshed, perhaps as an RSS feed? Alternatively, is this data available internally and it just requires writing a route to handle that query? As of now when I log into my toolshed I'm faced with a page of meaningless hashes and version numbers which I cannot mentally track for updates across multiple days/projects. Cheers, Eric -- Eric Rasche Programmer II Center for Phage Technology Texas A&M University College Station, TX 77843 404-692-2048 <tel:4046922048> esr(a)tamu.edu <mailto:esr@tamu.edu>

2 1

Tracking tool execution time?
by Jim McCusker 10 Jun '14

10 Jun '14

Is there a way to look at how long a galaxy tool takes to run? Jim

2 1

ClustalW jobs aren't working
by Malcolm Tobias 10 Jun '14

10 Jun '14

All, When running clustalW jobs, the only output we're getting is an error message that appears to show the tool isn't receiving the proper run-time parameters: Unknown option: I Value "FILE=infile_copy" invalid for option N (number expected) Unknown option: F Unknown option: I Unknown option: L Unknown option: E ... This appears to come from the rgClustalw.py wrapper: cl = ['clustalw2 -INFILE=%s -OUTFILE=%s -OUTORDER=%s -TYPE=%s -OUTPUT=%s ' % (self.iname,self.opts.output,self.opts.out_order,self.opts.dnarna,self.opts. outform)] Strangely, the log files appear to show the correct paramters: galaxy.tools WARNING 2014-06-05 10:37:43,484 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.local DEBUG 2014-06-05 10:37:43,596 (779) executing: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/001/dataset_1243.dat" -o "/home/galaxy/galaxy-dist/database/files/001/dataset_1244.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/001/dataset_1245.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" I'll post the full output from the logs below, but I was hoping someone might be able to offer some suggestions on how to debug this further. Thanks! Malcolm 10.28.56.101 - - [05/Jun/2014:10:37:42 -0500] "POST /tool_runner/index HTTP/1.1" 200 - "http://galaxy.chpc.wustl.edu/tool_runner?tool_id=clustalw" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" galaxy.jobs DEBUG 2014-06-05 10:37:43,025 (779) Working directory for job is: /home/galaxy/galaxy-dist/database/job_working_directory/000/779 galaxy.jobs.handler DEBUG 2014-06-05 10:37:43,057 (779) Dispatching to local runner galaxy.jobs DEBUG 2014-06-05 10:37:43,205 (779) Persisting job destination (destination id: local) galaxy.jobs.handler INFO 2014-06-05 10:37:43,258 (779) Job dispatched 10.28.56.101 - - [05/Jun/2014:10:37:42 -0500] "GET /history HTTP/1.1" 200 - "http://galaxy.chpc.wustl.edu/tool_runner/index" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" galaxy.tools DEBUG 2014-06-05 10:37:43,466 Building dependency shell command for dependency 'clustalw2' galaxy.tools WARNING 2014-06-05 10:37:43,484 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.local DEBUG 2014-06-05 10:37:43,596 (779) executing: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/001/dataset_1243.dat" -o "/home/galaxy/galaxy-dist/database/files/001/dataset_1244.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/001/dataset_1245.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" 10.28.56.101 - - [05/Jun/2014:10:37:43 -0500] "GET /api/histories/1ac3b66e11b8fe50 HTTP/1.1" 200 - "http://galaxy.chpc.wustl.edu/history" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" galaxy.datatypes.interval ERROR 2014-06-05 10:37:43,732 Exception caught attempting to generate viewport for dataset '972' Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/interval.py", line 159, in get_estimated_display_viewport fh = open( dataset.file_name ) IOError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/files/000/dataset_769.dat' galaxy.jobs DEBUG 2014-06-05 10:37:43,736 (779) Persisting job destination (destination id: local) galaxy.datatypes.interval ERROR 2014-06-05 10:37:43,776 Exception caught attempting to generate viewport for dataset '985' Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/interval.py", line 159, in get_estimated_display_viewport fh = open( dataset.file_name ) IOError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/files/000/dataset_782.dat' galaxy.datatypes.interval ERROR 2014-06-05 10:37:43,830 Exception caught attempting to generate viewport for dataset '986' Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/interval.py", line 159, in get_estimated_display_viewport fh = open( dataset.file_name ) IOError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/files/000/dataset_783.dat' galaxy.datatypes.interval ERROR 2014-06-05 10:37:43,876 Exception caught attempting to generate viewport for dataset '987' Traceback (most recent call last): File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/interval.py", line 159, in get_estimated_display_viewport fh = open( dataset.file_name ) IOError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-dist/database/files/000/dataset_784.dat' 10.28.56.101 - - [05/Jun/2014:10:37:43 -0500] "GET /history/get_display_application_links HTTP/1.1" 200 - "http://galaxy.chpc.wustl.edu/history" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" galaxy.jobs.runners.local DEBUG 2014-06-05 10:37:45,983 execution finished: python /home/galaxy/galaxy-dist/tools/rgenetics/rgClustalw.py -i "/home/galaxy/galaxy-dist/database/files/001/dataset_1243.dat" -o "/home/galaxy/galaxy-dist/database/files/001/dataset_1244.dat" -s "ALIGNED" -l "/home/galaxy/galaxy-dist/database/files/001/dataset_1245.dat" -t "Clustal_run" -d "DNA" -f "CLUSTAL" galaxy.jobs.runners DEBUG 2014-06-05 10:37:46,609 executing external set_meta script for job 779: /home/galaxy/galaxy-dist/set_metadata.sh ./database/files /home/galaxy/galaxy-dist/database/job_working_directory/000/779 . /home/galaxy/galaxy-dist/universe_wsgi.ini /home/galaxy/galaxy-dist/database/tmp/tmpsnNodx /home/galaxy/galaxy-dist/database/job_working_directory/000/779/galaxy.json /home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_in_HistoryDatasetAssociation_1448_zaFZp3,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_kwds_HistoryDatasetAssociation_1448_Rltsef,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_out_HistoryDatasetAssociation_1448_12s67j,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_results_HistoryDatasetAssociation_1448_Qucqzx,,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_override_HistoryDatasetAssociation_1448_95Vr9q /home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_in_HistoryDatasetAssociation_1447_Zh2tCn,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_kwds_HistoryDatasetAssociation_1447_UeH8AF,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_out_HistoryDatasetAssociation_1447_cZxp2H,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_results_HistoryDatasetAssociation_1447_ck1M8o,,/home/galaxy/galaxy-dist/database/job_working_directory/000/779/metadata_override_HistoryDatasetAssociation_1447_pefoLl -- Malcolm Tobias 314.362.1594

4 8

installing htseq-count from tool-shed
by Karen Chait 10 Jun '14

10 Jun '14

Hi, I am trying to install the tool htseq-count from the tool-shed. I had a problem in the first installation so I uninstalled the tool and all its dependencies (numpy, samtools 0.1.19 and pysam) from the 'manage installed tool shed repositories) and installed it again but then I get installation status 'Installed, missing tool dependencies' for htseq-count. When running this tool I get the error Fatal error: The HTSeq python package is not properly installed, contact Galaxy administrators The path set in universe_wsgi.ini: tool_dependency_dir = /home/galaxy/galaxy-dist/tool_deps in the installation log htseq/0.6.1/lparsons/htseq_count/d5edaf8dc974/INSTALLATION.log I get the error: Setup script for HTSeq: Failed to import 'numpy'. Please install numpy and then try again to install HTSeq. I reinstalled numpy but that did not help. How should I continue? Thanks, Karen

3 2

Galaxy updated botched?
by Kandalaft, Iyad 08 Jun '14

08 Jun '14

Hi everyone I was looking to upgrade galaxy to the latest verison. I did the usual hg pull; hg update;. I start up galaxy, it fetches some eggs, and then it tells me that I need a DB schema update. Fine, I ran sh db_manage.sh upgrade. All I see is a list of stack traces similar to the following. I'm using mysql and all the tables that galaxy had created are running under the myisam engine.... Hence, no foreign key constraints allowed. Why did this happen and can I simply convert all my tables to InnoDB to complete the database upgrade? Should I restore the DB from a backup? IntegrityError: (IntegrityError) (1215, 'Cannot add foreign key constraint') '\nCREATE TABLE history_dataset_collection_rating_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\thistory_dataset_collection_id INTEGER, \n\tuser_id INTEGER, \n\trating INTEGER, \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(history_dataset_collection_id) REFERENCES history_dataset_collection_association (id), \n\tFOREIGN KEY(user_id) REFERENCES galaxy_user (id)\n)\n\n' () (IntegrityError) (1215, 'Cannot add foreign key constraint') '\nCREATE TABLE history_dataset_collection_tag_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\thistory_dataset_collection_id INTEGER, \n\ttag_id INTEGER, \n\tuser_id INTEGER, \n\tuser_tname VARCHAR(255), \n\tvalue VARCHAR(255), \n\tuser_value VARCHAR(255), \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(history_dataset_collection_id) REFERENCES history_dataset_collection_association (id), \n\tFOREIGN KEY(tag_id) REFERENCES tag (id), \n\tFOREIGN KEY(user_id) REFERENCES galaxy_user (id)\n)\n\n' () Creating history_dataset_collection_tag_association table failed: (IntegrityError) (1215, 'Cannot add foreign key constraint') '\nCREATE TABLE history_dataset_collection_tag_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\thistory_dataset_collection_id INTEGER, \n\ttag_id INTEGER, \n\tuser_id INTEGER, \n\tuser_tname VARCHAR(255), \n\tvalue VARCHAR(255), \n\tuser_value VARCHAR(255), \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(history_dataset_collection_id) REFERENCES history_dataset_collection_association (id), \n\tFOREIGN KEY(tag_id) REFERENCES tag (id), \n\tFOREIGN KEY(user_id) REFERENCES galaxy_user (id)\n)\n\n' () Traceback (most recent call last): File "lib/galaxy/model/migrate/versions/0120_dataset_collections.py", line 179, in __create table.create() File "build/bdist.linux-x86_64/egg/sqlalchemy/schema.py", line 593, in create checkfirst=checkfirst) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 2302, in _run_visitor conn._run_visitor(visitorcallable, element, **kwargs) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1972, in _run_visitor **kwargs).traverse_single(element) File "build/bdist.linux-x86_64/egg/sqlalchemy/sql/visitors.py", line 106, in traverse_single return meth(obj, **kw) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/ddl.py", line 86, in visit_table self.connection.execute(schema.CreateTable(table)) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1542, in _execute_ddl compiled File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) File "build/bdist.linux-x86_64-ucs4/egg/MySQLdb/cursors.py", line 173, in execute self.errorhandler(self, exc, value) File "build/bdist.linux-x86_64-ucs4/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue Regards, Iyad Kandalaft Bioinformatics Programmer Microbial Biodiversity Bioinformatics Science & Technology Branch Agriculture & Agri-Food Canada Iyad.Kandalaft(a)agr.gc.ca | (613) 759-1228

1 0

"Resume Paused Jobs"
by Björn Grüning 07 Jun '14

07 Jun '14

Hi, can anyone confirm that "Resume Paused Jobs" on crashed tools is broken in the latest Galaxy release? Thanks, Bjoern

2 2

Docker based tools
by Kyle Ellrott 07 Jun '14

07 Jun '14

I know a few people have talked about using docker to package and distribute Galaxy tools. The most recent version now runs on all major linux distributions, so it's a bit more inclusive now (but there probably won't ever be a windows or mac version). One strategy would be for tool configs to define an option Docker archive name and possibly an alternate command line to execute if the docker archive is being used. How far has that work gotten, and what are people thinking about it? Kyle

3 2

Are there examples of tools that run processes in the background?
by Melissa Cline 06 Jun '14

06 Jun '14

Hi folks, We're working on developing a VM which we'd like to be tightly coordinated with Galaxy, such that VM users can export data from the VM into Galaxy, perform further analyses on that data, and then import the analysis results into the VM for storage (from where it will serve additional applications). Right now, I'm picturing this as a set of Galaxy tools that launch the VM, manage the data import and export, and shut down the VM at the appropriate time. Or is there a better approach I didn't consider? Assuming the answer is "no", then are there any tools in the toolshed that launch background processes? If so, I'd like to look at them. And assuming that my VM will be running in the background, I'd like it to shut down when Galaxy shuts down. Is there a way to configure a process to be run when it's time to quit Galaxy? Thanks! Melissa

2 1