July 2012 - galaxy-dev - lists.galaxyproject.org

pipeline execution succeeds but galaxy shows failure
by Brian Haas 17 Aug '12

17 Aug '12

Hi, I wrote a pipeline (xml attached) that, from what I can gather, succeeds, but galaxy shows it as an error and doesn't make the output file accessible as a new data set. >From the server log, I can see that the command line is being constructed correctly, and it even indicates that it's captured the output, but in the display of the web browser, it just shows up in the error state. The script being run exits (0) on success. Any ideas? Here's what the output section of my xml file looks like: <outputs> <data format="bam" name="coordSortedBam" label="${tool.name} on ${on_string}: coord-sorted read alignments" from_work_dir="alignment/alignment.coordSorted.bam"/> </outputs> and here's what the server log states: galaxy.jobs.handler INFO 2012-07-20 09:52:05,240 (30) Job dispatched galaxy.jobs.runners.local DEBUG 2012-07-20 09:52:05,453 executing: alignReads.pl --target /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_26.dat -o alignment --aligner bowtie --single /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_23.dat --seqType fq galaxy.jobs DEBUG 2012-07-20 09:52:16,673 finish(): Moved /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/job_working_directory/000/30/alignment/alignment.coordSorted.bam to /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_50.dat as directed by from_work_dir Again, as far as I can tell, everything worked - but the browser doesn't think so. I've run the exact command above on the command-line, and it exits(0) indicating success. Also, I've verified that when run through my galaxy instance, the galaxy-relocated output file is as expected. Many thanks for your help. I'm still getting my feet wet with galaxy, reading through all the documentation and searching the mailing list for additional help. best regards, -brian -- -- Brian J. Haas Manager, Genome Annotation and Analysis, Research and Development The Broad Institute http://broad.mit.edu/~bhaas

7 10

Set metadata from within the tool
by roberto.dilauro＠uniparthenope.it 13 Aug '12

13 Aug '12

Hi everyone, How can I set dataset's metadata from within the tool? For example: From the command tag (in the xml tool specification) i call myscript.sh: <command> myscript.sh -input=$input -output=$output </command> My script.sh produce a value that i want to store as metadata of $input. I've checked the tool config syntax http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax and seems that i can read metadata (${input.metadata.somemetadata}) but not write. Maybe <code> tag can help me but it is deprecated. Thanks for the help, Roberto

2 1

Old tool not showing up properly in Tool Shed
by Peter Cock 13 Aug '12

13 Aug '12

Hi all, I've just uploaded an update to my (previously protein only) sequence analysis bundle on the ToolShed, which is stuck with the original name: tmhmm_and_signalp This now offers Galaxy wrappers for several standalone command line sequence analysis tools: * SignalP 3.0 predicts signal peptides, see: http://www.cbs.dtu.dk/services/SignalP/ * TMHMM 2.0 predicts transmembrane helices, see: http://www.cbs.dtu.dk/services/TMHMM/ * Promoter 2.0 predicts eukaryotic PolII promoters [NEW WRAPPER], see: http://www.cbs.dtu.dk/services/Promoter/ * WoLF PSORT predicts Eukaryote protein sub-cellular localization, see: http://wolfpsort.org/ * Oomycete RXLR motif prediction - a Python script implementing three published methods, which calls SignalP 3.0 and HMMER 2.3.2 internally. In this update I added the new Promoter 2.0 wrapper, and made some small changes to the SignalP and TMHMM wrappers. The other tools are unchanged. Looking at the tool bundle via the Tool Shed, for revision 7:9b45a8743100 it only shows four "valid" tools: * Promoter 2.0 - Find eukaryotic PolII promoters in DNA sequences * RXLR Motifs - Find RXLR Effectors of Plant Pathogenic Oomycetes * SignalP 3.0 - Find signal peptides in protein sequences * TMHMM 2.0 - Find transmembrane domains in protein sequences Why is this missing?: * WoLF PSORT - Eukaryote protein subcellular localization prediction The missing tool is NOT showing up under "Browse my invalid tools", and the wrapper XML seems to still be there if I browse the tip files. Thanks, Peter

2 4

Problem with get data/uploading file
by David Roquis 09 Aug '12

09 Aug '12

Dear Galaxy community, I have a problem with uploading data on Galaxy, using the get data/upload file tool, and I can't figure out where the trouble comes from. Basically, when I try to upload files (size ranging from less than 1 Mo up to 600 Mo), sometimes it works, sometimes it doesn't (keep uploading forever), and I don't get any error message when it doesn't work. I can't find a pattern on when it works or not, it seems pretty random. It don't think it is related to the files I use, because I tried with different files, of different types and format, and the exact same file can sometimes be uploaded, sometimes not. One thing I have noticed is that when the upload is successful, a file is created in /tmp/, and when the upload runs forever, no new file appears in /tmp/. I have tried a couple of basic things to solve the problem: emptying the /tmp/, restarting galaxy, deleting permanently my history, etc. Sometimes I can upload 4-5 files in a row, then it stops working, and some other time I can't even upload one. When one upload keep running forever, even if I cancel it, no further uploads will work. I have to stop the uploads and restart galaxy, but it doesn't work 100% of the time (sometimes I have to try 2-3 times before I can upload again). I am facing this problem on two different instances of galaxy. A development one that I simply installed on my /home/ using mercurial and didn't modified at all, and a production one that I just set up following the Galaxy Team recommendations (using PostgreSQL and nginx with the upload module, and working on a dedicated Galaxy user account). I am a beginner in informatics, and since I get no error message, I don't know where to look for a way to solve this annoying problem. Uploading files using data libraries under the admin menu seems to be working perfectly all the time. Thank you very much for your help and advice. Sincerely David Roquis Ph.D. student University of Perpignan additional info: - I am using the latest version of galaxy, and my OS is Fedora 16 - I tried working with and without virtualenv (made no difference). - I don't know if this is related or not, but it seems quite strange to me. Whenever I try to access "manage jobs" in the admin section (on both instances), I get the following error message: "This Galaxy instance is not the job manager. If using multiple servers, please directly access the job manager instance to manage jobs." How is that possible since I am not using multiple servers (especially for the development instance, which I haven't modified at all)? - debug = False is causing me a server error in the history frame (so I use True instead).

2 3

Odd (possible latency/race?) problems post-upgrade
by Fields, Christopher J 08 Aug '12

08 Aug '12

I'm seeing a possible latency issue or race condition when starting Galaxy after the latest hg upgrade (July 20) from galaxy-dist; the prior upgrade doesn't have this problem. We have small setup with one job manager/runner and two web front-ends for testing load balancing: …from universe_wsgi.ini: ------------------------------ [server:web0] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:web1] use = egg:Paste#http port = 8081 host = 127.0.0.1 use_threadpool = true threadpool_workers = 7 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 ------------------------------ If I run: GALAXY_RUN_ALL=1 sh run.sh --daemon I will intermittently see the following in the paster log for any of the above services (example below is web1, but I have seen this for manager and web0 as well). The traceback and error is the same in all cases ('File exists: /home/a-m/galaxy/dist-database/tmp/work_tmp'): ------------------------------ galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loading references to tool sheds from tool_sheds_conf.xml galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loaded reference to tool shed: Galaxy main tool shed galaxy.tool_shed.tool_shed_registry DEBUG 2012-07-30 11:40:10,194 Loaded reference to tool shed: Galaxy test tool shed galaxy.model.migrate.check DEBUG 2012-07-30 11:40:10,650 psycopg2 egg successfully loaded for postgres dialect galaxy.model.migrate.check INFO 2012-07-30 11:40:10,845 At database version 103 galaxy.tool_shed.migrate.check DEBUG 2012-07-30 11:40:10,940 psycopg2 egg successfully loaded for postgres dialect galaxy.tool_shed.migrate.check INFO 2012-07-30 11:40:10,986 At migrate_tools version 3 galaxy.model.custom_types DEBUG 2012-07-30 11:40:10,994 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/app.py", line 66, in __init__ self.installed_repository_manager.load_proprietary_datatypes() File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/tool_shed/__init__.py", line 47, in load_proprietary_datatypes installed_repository_dict = galaxy.util.shed_util.load_installed_datatypes( self.app, tool_shed_repository, relative_install_dir ) File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py", line 1269, in load_installed_datatypes work_dir = make_tmp_directory() File "/home/a-m/galaxy/galaxy-dist/lib/galaxy/util/shed_util.py", line 1305, in make_tmp_directory os.makedirs( work_dir ) File "/usr/lib64/python2.6/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 17] File exists: '/home/a-m/galaxy/dist-database/tmp/work_tmp' Removing PID file web1.pid ------------------------------ I was also seeing this using separate runner/webapp ini files and 'run_multiple_processes.sh --daemon', but we decided to go ahead and migrate over to a unified universe_wsgi.ini file. Anyway, we found a workaround by rerunning 'GALAXY_RUN_ALL=1 sh run.sh --daemon' which skips any running services, but I'm curious whether anyone else has seen this and whether there is a fix (or maybe added config setting we are missing?) chris

3 3

Galaxy and Toolshed issues
by SHAUN WEBB 08 Aug '12

08 Aug '12

Hi, I have been trying out the latest version of Galaxy and a newly installed local toolshed and I have a few questions/suggestions/bug reports: 1. Tool Versioning: a) As I understand from the wiki, when I create a new version of a tool in a tool shed I must then install the latest version in my Galaxy instance. This then shows up as two separate tools in my tool menu. Presumably it is best to change the xml description tag to reflect the version number and make tool versions distinguishable in the Galaxy tool menu? b) Would it make more sense to have a single entry for each tool in the tool menu and a drop down to choose which version to run in the tool gui? Similar to when I try to rerun a tool that is no longer installed I get a drop down option to use a derivative of that tool if it exists. d) Say I create a workflow then install a new version of a tool within the workflow. I would like to be able to run that workflow using either version of the tool without creating a new workflow. Is this possible? To me it would make sense to again provide a drop down to select tool versions on the workflow ui just before execution. c) The toolshed interface allows me to change the name of my tool. If I do this, then go to my Galaxy admin ui and try to get updates for my tool then it fails. Does it make sense to change a tools name after it is created? Is it possible to track previous tool names to avoid this happening? e) Email alerts: I could not find an option to receive email alerts about tool updates when installing via the Galaxy admin ui. Perhaps this should be an option during installation. Running toolshed tools: I have no problems installing tools from toolsheds via Galaxy admin interface, however I can't get these tools to run. They stay Grey forever and look like this in the manage jobs table: 65 xxxx(a)xxxx.xx.ac.uk 0 minutes ago toolshed.g2.bx.psu.edu/repos/aaronquinlan/bedtools/bedtools_coveragebed_cou… new None None None Do I have to set a tool runner or some other option for shed_tools? Migrating tools: I upgraded to the latest dist release of galaxy and when I restarted the server I expected to get a message about migrating freebayes tools. I didn't see this message. Will this be because I restart using --daemon? Is it advisable to therefore restart without --daemon after an upgrade to see if there are any tools to be migrated? Genome indexing: 1. I realise this is just a beta feature and it will be great when fully integrated. I tried to generate bowtie2 indexes for an installed build. I can see the job appear below the table in yellow and it contains the message "...added to the job queue to be indexed with bowtie2". However it seems to just sit there, even after restarting. Should I be able to see the job under 'manage jobs'? 2. Where are the generated indexes stored and is the .loc file automatically updated? Workflow menus: Would it be possible to categorise workflows the way we do with tools. I have a long list of workflows and it is sometimes difficult to find the one I need. I realise that workflows are user specific so this is not as easy to implement as with tools. Perhaps it would be possible to have category menus for published workflows at least? Trackster: I have noticed a few bugs with trackster in the latest release (tested on Chrome and Firefox): 1. Overview button does not work 2. If I add tracks to a group and make a composite track, then showindividual tracks again I get new menu options (filters, tool, tool parameter space visualisation). None of these seem to work and trackster often stops working after doing this. Thanks for all the effort put in to these great new features. I hope this lengthy email is helpful! Shaun Webb Edinburgh University -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

4 9

PBS Server Throwing Errors
by Daniel Patrick Sullivan 01 Aug '12

01 Aug '12

Hi, Galaxy Community, Greetings from The University of Chicago; I hope all who attended the Galaxy conference enjoyed it as much as I did. I have searched the mailing list archives as well as Google to a resolve problem I am seeing, however I am somewhat at a loss as to the next course of action I should be taking to bring this issue to a close. I am hoping that one of the bright minds on this mailing list could help me shed some light on the solution to my problem, or at least help me identify a root cause. I have configured Galaxy to integrate with TORQUE (version 4.0.2) server, and successfully built the PBS python egg as specified in the Galaxy documentation. I am using Python version 2.6 and the latest build of Galaxy. Whenever I launch a job from the Galaxy UI, I get the following error message(s) on the PBS server: 07/30/2012 15:42:48;0080;PBS_Server;Req;dis_request_read;conflicting version numbers, 1 detected, 2 expected 07/30/2012 15:42:48;0080;PBS_Server;Req;req_reject;Reject reply code=15058(Bad DIS based Request Protocol MSG=cannot decode message), aux=0, type=AlternateUserAuthentication, from galaxy@ 07/30/2012 15:42:48;0080;PBS_Server;Req;dis_request_read;conflicting version numbers, 1 detected, 2 expected 07/30/2012 15:42:48;0080;PBS_Server;Req;req_reject;Reject reply code=15058(Bad DIS based Request Protocol MSG=cannot decode message), aux=0, type=QueueJob, from galaxy@ 07/30/2012 15:42:48;0080;PBS_Server;Req;dis_request_read;conflicting version numbers, 1 detected, 2 expected 07/30/2012 15:42:48;0080;PBS_Server;Req;req_reject;Reject reply code=15058(Bad DIS based Request Protocol MSG=cannot decode message), aux=0, type=Disconnect, from galaxy@ 07/30/2012 15:43:01;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 4.0.2, loglevel = 1 One thing I did notice, that suggests there might be a problem, is that there is no hostname after the galaxy@; most of the other messages in this log file have a host name appended to log entry, i.e.: 07/30/2012 15:43:38;0100;PBS_Server;Req;;Type StatusJob request received from root@sc01, sock=10 I have completed a tcpdump on the schedule node, and I can definitely see bi-directional traffic between the Galaxy server and the scheduler node on TCP port 15001. In addition to this, I have installed the TORQUE client tools on on the Galaxy server, and can spawn an interactive job with qsub -I, as well as check the status of queued jobs using qstat (from the Galaxy server). This suggests to me that there a potential problem with the PBS Egg, although I am not certain. Has anybody seen something like this before, or could somebody point me in the right direction? We do have a support contract with Adaptive Computing, and I am opening a ticket with them as well, however I wanted to reach out to the Galaxy community to cover all of my bases. Thank-you so much for taking the time to read my email. Dan Sullivan

2 2

Local instance and storage space to provide
by Sarah Maman 01 Aug '12

01 Aug '12

Dear all, To purchase equipment related to a local instance of Galaxy, I would like to have your feedback. Did you notice load problems when 10 to 20 users launch jobs at the same time and on a single instance of Galaxy, with mi-RNA or RNA seq data ? What do you recommend in terms of storage space, number of nodes, number of instances? Thank you in advance, Sarah Maman

2 1

Update Galaxy instance with Mercurial error
by Makis Ladoukakis 01 Aug '12

01 Aug '12

Dear Galaxy users, I am running a Galaxy instance in my institution's server and I want to keep it up to date with the latest release. Nevertheless when i try hg incoming i get the following error: abort: There is no Mercurial repository here (.hg not found)! Most probably the administrator of the server has downloaded the Galaxy-dist directory without the use of Mercurial (I was suspicious of that all along as the directory of the Galaxy installation is named galaxy_dist and not galaxy-dist). Is there another way to update my instance? Thank you, Makis Ladoukakis

2 1

Error in loading tools.
by Mohammad Heydarian 01 Aug '12

01 Aug '12

Hi Galaxy Community! Hope all is well. I've been trying to install some tools in my Galaxy Cloudman instance (using image: ami-da58aab3 ) from the main toolshed. When I try to "install to local Galaxy" I get the error: Not Found The resource could not be found. No action for /admin_toolshed/prepare_for_install I have tried to load tools on a couple of different Cloudman instances and tried to load a variety of tools and end up with the same error. Any help would be fantastic. Cheers, Mo Heydarian

2 2