June 2012 - galaxy-dev - lists.galaxyproject.org

Hiding a tool via tool_conf.xml
by Peter Cock 30 Oct '12

30 Oct '12

Hi all, I'm aware that in an individual tool's XML wrapper you can add hidden="true" to the <tool> element to hide the tool from the listing shown in Galaxy's left hand panel (but the tool is still loaded and can be called by old workflows etc). Can this be done via the tool_conf.xml file as well? This would seem useful from a system configuration point of view where it seems wrong to edit the tool wrappers themselves. In fact, it might also be nice to add hidden="true" support to <section> as well as <tool> in tool_conf.xml - although I'm not so sure how useful that would be. Peter

1 2

Re: [galaxy-dev] Using $NSLOTS in tools to control thread number
by Peter Cock 19 Oct '12

19 Oct '12

On Fri, Jun 15, 2012 at 4:38 PM, James Taylor <james(a)jamestaylor.org> wrote: > This is exactly what I think we should do (and have for a long time), but I > think the variable should be something like: > > GALAXY_CPUS > > (threads is not accurate, a multithread or multiprocess job might want to use > this info, something even more abstract than CPUS might make sense, but > SLOTS has never made sense to me). I agree that a Galaxy specific name makes a lot of sense, and that the SGE term "slots" is a bit odd. Using CPUS however is potentially ambiguous with CPUs vs cores - my desktop has two quad core CPUs, i.e. 2 CPUs but 8 cores. Where do you think this number should come from? A new entry in the runner URL is simple albeit potentially redundant with cluster-specific entries in the runner URL. As to the alternative (doing it automatically), for PBS and SGE determining the number of cores from the cluster configuration and/or parsing the cluster runner URL sounds doable - what about the other backends? Peter

2 2

Re: [galaxy-dev] Sample tracking data transfer hangs "in queue" forever
by TerAvest, Emily 27 Jul '12

27 Jul '12

Hi Leandro, I am also experiencing the same problem with the latest version of galaxy. I just attempted to connect our sequencer for the first time yesterday. I do not have an older version of galaxy to test and compare to see if it works in earlier versions. I am able to transfer data from the sequence to the import directory, however it is not moved to the data library. My data_transfer.log also has the same error. "The server could not comply with the request since it is either malformed or otherwise incorrect." Does anyone have a solution for this? Thanks Emily ------------------------------ Message: 17 Date: Wed, 28 Mar 2012 13:15:47 +0200 From: Leandro Hermida <softdev(a)leandrohermida.com> To: Luobin Yang <yangluob(a)isu.edu> Cc: Galaxy Dev <galaxy-dev(a)bx.psu.edu> Subject: Re: [galaxy-dev] Sample tracking data transfer hangs "in queue" forever Message-ID: <CAOhzmPJ67udRQBhfChgyxNrXsm3QcY1c7jna1kR0aJBNKEKJxA(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Dear Galaxy Dev, Ok I have gotten further found out what was going on, in my data_transfer.log I was getting HTTP 404 Not Authorized when the data transfer was trying to access Galaxy API URLs http://galaxyserver/api/... This is because we are using external user authentication as documented in http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy and this puts all of Galaxy behind this authentication. Since the Galaxy API uses API keys to essentially authenticate you have to change you <Location /> ... </Location> container to not match URLs starting with /api. To do this you have to change it to <LocationMatch "^/(?!api)"> .... </LocationMatch>. best, leandro On Mon, Mar 26, 2012 at 6:10 PM, Luobin Yang <yangluob(a)isu.edu> wrote: > I've got the same problem and when I looked at the data_transfer.log, I saw > the following message: > > 2012-03-01 15:12:27,338 - datatx_13870 - > (u'9c17d84742cd2acb63d88b5bd41d968f', > u'http://xxx.xxx.xxx.xxx/api/samples/2d9035b3fc152403', > {'sample_dataset_ids': ['a799d38679e985db', '33b43b4e7093c91f'], > 'error_msg': '', 'update_type': 'sample_dataset_transfer_status', > 'new_status': 'Adding to data library'}) > 2012-03-01 15:12:27,342 - datatx_13870 - Error. <!DOCTYPE HTML PUBLIC > "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>405 Method Not Allowed</title> > </head><body> > <h1>Method Not Allowed</h1> > <p>The requested method PUT is not allowed for the URL > /api/samples/2d9035b3fc152403.</p> > <hr> > > Using Galaxy instead of Apache as the web server changed the sample status > from "in queue" to complete but didn't add the downloaded files to the data > library. > > Luobin > > On Mon, Mar 26, 2012 at 9:54 AM, Leandro Hermida > <softdev(a)leandrohermida.com> wrote: >> >> Dear Galaxy Dev, >> >> I've set up the Galaxy sample tracking system data transfer >> functionality exactly as specified here >> https://main.g2.bx.psu.edu/u/rkchak/p/data-transfer, but when I >> attempt to transfer datasets it puts them into the queue and then >> never seems to transfer anything. >> >> The galaxy_listener.log shows: >> >> 2012-03-26 17:16:24,515 - GalaxyAMQP - GALAXY LISTENER PID: 8738 - >> {'config_file': 'universe_wsgi.ini', 'http_server_section': >> 'server:main'} >> 2012-03-26 17:16:24,518 - GalaxyAMQP - {'exchange': 'galaxy_exchange', >> 'rabbitmqctl_path': >> >> '/gpfshpc/software/pmrdcdb/galaxy/rabbitmq/rabbitmq_server-2.8.1/sbin/rabbitmqctl', >> 'userid': 'galaxy_dev', 'routing_key': 'bar_code_scanner', 'queue': >> 'galaxy_queue', 'host': '127.0.0.1', 'virtual_host': >> 'galaxy_dev_messaging_engine', 'password': 'mypassword', 'port': >> '5672'} >> 2012-03-26 17:16:24,565 - GalaxyAMQP - Connected to rabbitmq server - >> 127.0.0.1:5672 >> 2012-03-26 17:31:23,393 - GalaxyAMQP - MESSAGE RECVD: data_transfer >> 2012-03-26 17:31:23,393 - GalaxyAMQP - DATA TRANSFER >> 2012-03-26 17:31:23,401 - GalaxyAMQP - Started process (25406): python >> >> "/gpfshpc/software/pmrdcdb/galaxy/galaxy_dist_dev/scripts/galaxy_messaging/server/data_transfer.py" >> " <data_transfer> >> <galaxy_host>rd-galaxydev.app.pmi</galaxy_host> >> <api_key>69bbd1a9b18eda2a888e9963820d8ef0</api_key> >> <data_host>chp723</data_host> >> <data_user>galaxy</data_user> >> <data_password>mypassword</data_password> >> <request_id>12</request_id> >> <sample_id>12</sample_id> >> <library_id>15</library_id> >> <folder_id>52</folder_id> ? ? ? ? ? ? ? ? ? ?<dataset> >> ?<dataset_id>10</dataset_id> >> <name>ttt_2.fastq.gz</name> >> <file>/gpfshpc/data/pmrdcdb/RNA-Seq/datasets/ttt-2.fastq.gz</file> >> ? ? ? ? ?</dataset><dataset> >> <dataset_id>9</dataset_id> >> <name>ttt_1.fastq.gz</name> >> <file>/gpfshpc/data/pmrdcdb/RNA-Seq/datasets/ttt-1.fastq.gz</file> >> ? ? ? ? ?</dataset> ? ? ? ? ? ? ? ?</data_transfer>" >> "universe_wsgi.ini" >> >> The FASTQ files are for testing and aren't very big (7.6 MB), what >> could be going wrong? >> >> regards, >> Leandro >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. ?To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> ?http://lists.bx.psu.edu/ > > ************ This email message and any electronic files transmitted with it relating to the official business of Codexis, Inc. and its affiliates are proprietary, strictly confidential, and may be legally privileged. This email message is intended solely for the use of the individual(s) to whom it is addressed. If you believe you have received this email in error, please delete this email from your system and notify the sender immediately. If you are not the intended recipient, you should not disclose, distribute, use or copy this email or take any action in reliance on its contents.

4 9

HMMER wrappers
by Peter Cock 26 Jul '12

26 Jul '12

Hello all, Has anyone done a wrapper for HMMER2 or HMMER3? http://hmmer.janelia.org/software I can't find anything in the Tool Shed http://toolshed.g2.bx.psu.edu/ so we are likely to look into doing this ourselves - starting with hmmerscan. Thanks, Peter

4 16

output file
by 張詩婷 19 Jul '12

19 Jul '12

Hi, After execution, my tool created many files in a directory , how can I to find this directory? command: bash mytool.sh $input

2 1

How to make trackster jobs run locally instead of on a cluster?
by Dorset, Daniel C 13 Jul '12

13 Jul '12

We have an instance of Galaxy deployed on a local institutional cluster. It's working well, but when we import BED files for visualization, Galaxy runs the processing for these on the cluster. I would like to run these tasks locally instead. I understand how to make individual tools run locally, but I'm hoping there's a better way to accomplish what I would like to do without having to specify each of the many conversion tools individually in the universe_wsgi.ini config file. Thanks! Dan

2 1

Galaxy not killing split cluster jobs
by Peter Cock 10 Jul '12

10 Jul '12

Hi all, We're running our Galaxy with an SGE cluster, using the DRMAA support in Galaxy, and job splitting. I've noticed if the user cancels a job (that was running or queued on the cluster) while the job is shows as deleted in Galaxy, looking at the queue on the cluster with qstat shows it persists. I've not seen anything similar reported except for this PBS issue: http://lists.bx.psu.edu/pipermail/galaxy-dev/2010-October/003633.html When I don't use job splitting, cancelling jobs seems to work: galaxy.jobs.handler DEBUG 2012-05-01 14:46:47,755 stopping job 57 in drmaa runner galaxy.jobs.runners.drmaa DEBUG 2012-05-01 14:46:47,756 (57/26504) Being killed... galaxy.jobs.runners.drmaa DEBUG 2012-05-01 14:46:47,757 (57/26504) Removed from DRM queue at user's request galaxy.jobs.runners.drmaa DEBUG 2012-05-01 14:46:48,441 (57/26504) state change: job finished, but failed galaxy.jobs.runners.drmaa DEBUG 2012-05-01 14:46:48,441 Job output not returned from cluster When I am using job splitting, cancelling jobs fails: galaxy.jobs.handler DEBUG 2012-05-01 14:28:30,364 stopping job 56 in tasks runner galaxy.jobs.runners.tasks WARNING 2012-05-01 14:28:30,386 stop_job(): 56: no PID in database for job, unable to stop That warning comes from lib/galaxy/jobs/runners/tasks.py which starts: def stop_job( self, job ): # DBTODO Call stop on all of the tasks. #if our local job has JobExternalOutputMetadata associated, then our primary job has to have already finished if job.external_output_metadata: pid = job.external_output_metadata[0].job_runner_external_pid #every JobExternalOutputMetadata has a pid set, we just need to take from one of them else: pid = job.job_runner_external_id if pid in [ None, '' ]: log.warning( "stop_job(): %s: no PID in database for job, unable to stop" % job.id ) return pid = int( pid ) ... I'm a little confused about tasks.py vs drmaa.py but that TODO comment looks pertinent. Is that the problem here? Regards, Peter

3 9

Galaxy Czars: Doodle pool for first Teleconference
by Ann Black-Ziegelbein 09 Jul '12

09 Jul '12

Hi everyone - We are targeting a web conference call for Galaxy Deployers/Operators for July 9th or 10th. This teleconference will be held using Elluminate Live/Blackboard Collborate web conference - so all data (including voice) will be over the internet. We have good luck at the University of Iowa with this technology for world wide remote conferences (our Ponseti International group uses it for their large distributed collaboration and others) and we can record our session and post it for playback off of the wiki. Finding a time that will suit everyone will be a challenge. If you can take a moment to respond to the doodle poll with when you can join a web conference, it will help us find the best time for the majority: http://www.doodle.com/f4f2f37smgcv3z6n By the way, if you have not found it yet, here is a link the Galaxy Czar wiki page: http://wiki.g2.bx.psu.edu/Community/GalaxyCzars Here is a draft agenda for our first call - comments are welcome : *For our first Mtg.* * Logistics: Address how we want to tackle these calls. Go over generic agenda. Frequency of calls * Presentation: Galaxy at Iowa: Discuss our issues with big data, how NGS tools take in/output data and finding the right storage server solution. * Group Goals: what do we want to accomplish with this group beyond discussions/sharing? * Open Mic & recruit new volunteers for the next call. *My shot at a generic agenda outline - to be customized each User Group Mtg ? * 20 min: Galaxy in Our Town. - presentation from a local galaxy institution on what they are doing or a problem they are troubleshooting - or have someone walk through their use cases and pain points. 20 min: Galaxy Today/Tomorrow. - presentation on a galaxy coding item. Either from penn state team, or from someone working on a new feature or customization. 20 mins: Open Mic Discussion & make point to point connections. IE - organize smaller breakouts if someone wants to host a call specific to an issue. Thanks everyone! Ann

2 4

Interested in speaking with other institutions deploying Galaxy locally?
by Ann Black-Ziegelbein 05 Jul '12

05 Jul '12

Hi everyone - Here at the University of Iowa we are working on deploying Galaxy locally for campus wide access. I am interested in forming a community of other institutions trying to deploy Galaxy locally and mange/operate it on a broad level. Is anyone else? If there is enough interest, possibly we could have a community conference call every other month to have an open discussion on how we are all deploying galaxy, customizations we are making, problems we are encountering, bugs, and any add-on operations management for galaxy being developed, etc. Would love to hear from others operating Galaxy or in process of standing up a local deployment. Thanks! Ann Black-Ziegelbein

22 32

Looking for recommendations: How to run galaxy workflows in batch
by Dave Lin 05 Jul '12

05 Jul '12

Hi All, I'm looking to batch process 40 large data sets with the same galaxy workflow. This obviously can be done in a brute-force manual manner. However, is there a better way to schedule/invoke these jobs in batch 1) from the UI with a plugin 2) command-line 3) web-service Thanks in advance for any pointers. Dave

5 8