August 2012 - galaxy-dev - lists.galaxyproject.org

generating composite files
by Jorrit Boekel 31 Aug '12

31 Aug '12

Dear list, I'm on galaxy-dist. I have a tool that outputs an unknown amount of files in a directory, all of which I need for input to the next tool. I used to tar them, but after a tip from John Chilton, I tried to fit this into Galaxy's composite datatypes. The tool now works, but I wonder if there is a better way to do it than how I have implemented it. I ran into some trouble when my new datatype's generate_primary_file was never called. (Is this only used after an upload?) My tool now outputs to the output.extra_files_path given to it on the xml command line, and writes an HTML file to the output file it is given. The next tool reads input.extra_files_path, and uses the files there. In other words, no use has been made of add_composite_file, generate_primary_file, etc. I can even set composite_file=None in the datatype, and it will still work. The whole thing feels a bit like a hack, so I wanted to ask if there is a way for tools to generate composite files the same way uploads can. cheers, jorrit boekel

1 0

missing slash when installing from toolshed?
by Dan Sullivan 30 Aug '12

30 Aug '12

Hi, Galaxy Developers, I have a question that is very similar to the following thread; http://dev.list.galaxyproject.org/Problem-fetching-updates-to-toolshed-tool… Basically whenever I try to install a tool from the toolshed, it appears that a trailing slash is not appended to the Galaxy URL, which is causing a DNS lookup failure. For example, the URL specified in the attached screenshot should be https://crigalaxy.uchicago.edu/admin_toolshed, however it is appearing as https://crigalaxy.uchicago.eduadmin_toolshed. Does anybody know if there are any known bugs that would cause this behavior? I might venture down the road of modifying lib/galaxy/webapps/community/controllers/repository.py as suggested by the thread above, although I would like to avoid doing this if possible. I am seeing experiencing this behavior on the following version of Galaxy: > hg parent changeset: 7400:ec29ce8e27a1 In addition to this, I feel that its worthwhile to mention that I am using the apache proxy balancer with web scaling, however based on what I have seen and tested, I do not believe this (apache) to be the root cause of my problem. The URL below is a screenshot of the network debug I am seeing from the firebug extension in firefox. https://webshare.uchicago.edu/users/dansully/Public/Screen%20Shot%202012-08… If somebody has seen this, or could offer a suggestion to bring this issue to resolve, I would definitely be interested in your strategy. Thank-you so much for your help and for your time. Dan Sullivan 312-607-3702

3 4

Setup questions: data locations, MySQL, and cores
by Steven Peckins 30 Aug '12

30 Aug '12

I've been asked to set up a local Galaxy installation specifically for large datasets (tens of terabytes). Is there a list of default locations where Galaxy puts data? As an admin, that would be my first question, but it's not obvious from the documentation, and for large datasets, it's important to know. (During testing, with Galaxy in the default location, /home/galaxy, an attempt to upload and decompress a 50GB file wreaked havoc on the server due to heavy I/O. Changing the tmp directory so it wasn't reading and writing massive amounts of data to the same filesystem at the same time helped, but that would have been nice to know ahead of time.) Is the database used for datasets at all or just user account data? In other words, if users are crunching terabytes of data, do I need to worry about the amount of space on the filesystem that hosts the database? What exactly are the disadvantages of using MySQL over PostgreSQL? Several places in the docs state that it is preferred but not why. Is it a bigger problem than installing another database that I have no experience with over one that is already installed and with which I am already familiar? Is an MPI configuration necessary for getting full use out of a multicore system? The docs seem to indicate that Galaxy will use multiple cores ("Without a cluster, you'll be limited to the number of cores in your server...") but take pains to say that GIL won't allow more than a single thread ("This means that regardless of the number of cores in your server, Galaxy can only use one" and "having a multi-core system will not improve the Galaxy framework's performance out of the box since Galaxy can use (at most) one core at a time"). We have a 48-core machine, and there will only be two or three users. Thanks, --steve

2 1

Feature Request: Nested sections in tool_conf.xml
by Ted Goldstein 30 Aug '12

30 Aug '12

Dear Galaxy Developers, First, thanks for the great work. Galaxy is a amazingly well architected and extraordinary useful system. Of course, I have a feature request to make it even more perfect ;-) The list of tool sections is already very long. It would be nice to organize the sections in nested sections. For example: <toolbox> <section name="Percussive Hammer" id="hammer"> <section name="hand hammers" id="hammer"> <tool file="Air" id="Air hammer"> <tool file="Blacksmith" id="Blacksmith hammer"> <tool file="Body" id="Body hammer"> <tool file="Bossing mallet" id="Bossing mallet"> <tool file="Brass" id="Brass hammer"> <tool file="Brass" id="Brass hammer"> <tool file="Bricklayer" id="Bricklayer hammer"> <tool file="Bumping" id="Bumping hammer"> <tool file="Chipping" id="Chipping hammer"> <tool file="Claw" id="Claw hammer"> <tool file="Copper-faced" id="Copper-faced hammer"> <tool file="Finish" id="Finish hammer"> <tool file="Nail" id="Nail hammer"> <tool file="Nylon" id="Nylon hammer"> <tool file="Panel" id="Panel hammer"> <tool file="Pick" id="Pick hammer"> <tool file="Planishing" id="Planishing hammer"> <tool file="Ripping" id="Ripping hammer"> <tool file="Riveting" id="Riveting hammer"> <tool file="Rotary" id="Rotary hammer"> <tool file="Rubber mallet" id="Rubber mallet"> <tool file="Shingler's" id="Shingler's hammer"> <tool file="Shrinking" id="Shrinking hammer"> <tool file="Sledge" id="Sledge hammer"> <tool file="Slide" id="Slide hammer"> <tool file="Tack" id="Tack hammer"> <tool file="Tinners" id="Tinner's hammer"> <tool file="Wooden_mallet" id="Wooden mallet"> </section> <section name="Power hammer" id="power_hammer"> <tool file="Jack Hammer " id="jackhammer"> <tool file="Electromechanical Hammer " id="e_jackhammer"> <tool file="Pneumatic Hammer " id="p_jackhammer"> <tool file="Hydraulic Hammer " id="h_jackhammer"> </section> .. etc .. </section> <section name="Wrenches" id="wrenches"> <section name="Power wrench" id="power_wrench"> .. etc .. </section> <section name="hand wrench" id="hand_wrench"> .. etc .. </section> </section> </toolbox> BTW, Its great that tools can conveniently be in more than one section! This makes it every useful to group common tasks together. Best regards, Ted

1 0

Re: [galaxy-dev] Blastxml error on Galaxy Cloudman
by Peter Cock 30 Aug '12

30 Aug '12

Hi Mohammad, Don't forget to CC the mailing list ;) It sounds like something has gone wrong with the update... Peter On Wednesday, August 29, 2012, Mohammad Heydarian wrote: > Hi Peter, > Thanks for your response. > > I am not using any of the Blast tools, I have only tried using the "filter > and sort" and "text manipulation" tools and gotten this error. > > Could this be fixed by re-updating my instance of Galaxy Cloudman? Or some > similar fix? I am not able to install tools to my instance of Galaxy > Cloudman, the error I get is: > The *tool_config_file* setting in *universe_wsgi.ini* must include at > least one shed tool configuration file name with a *<toolbox>* tag that > includes a *tool_path* attribute value which is a directory relative to > the Galaxy installation directory in order to automatically install tools > from a Galaxy tool shed (e.g., the file name *shed_tool_conf.xml* whose * > <toolbox>* tag is *<toolbox tool_path="../shed_tools">*). > > > Cheers, > Mo Heydarian > > PhD candidate > The Johns Hopkins School of Medicine > Department of Biological Chemistry > 725 Wolfe Street > 414 Hunterian > Baltimore, MD 21205 > > > > On Wed, Aug 29, 2012 at 3:24 PM, Peter Cock <p.j.a.cock(a)googlemail.com<javascript:_e({}, 'cvml', 'p.j.a.cock(a)googlemail.com');> > > wrote: > >> Hi >> >> The BLAST+ tools and the definition of the blastxml format were moved to >> the tool shed. You should have seen a migration message during the update... >> >> If you are not using these tools, perhaps you have a legacy reference to >> the blastxml file which should be removed? >> >> If you are using BLAST+ they and the blastxml format need to be installed >> from the ToolShed, >> >> Peter >> >> On Wednesday, August 29, 2012, Mohammad Heydarian wrote: >> >>> Hello, >>> I recently updated my instance of Galaxy Cloudman (using Amazon AWS) and >>> I have since not been able to run any jobs. I get the following error: >>> >>> WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' >>> WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' >>> >>> >>> Any help would greatly be appreciated. >>> >>> >>> Cheers, >>> Mo Heydarian >>> >>> >

2 1

work flow tool saving problem
by YuFei Luo 30 Aug '12

30 Aug '12

Hi, I just added a new tool to my existed work flow. And I would like to set the parameter of this tool to 'set at runtime', but I got an error message as following: Server error URL: http://127.0.0.1:8085/workflow/editor_form_post Module weberror.evalexception.middleware:*364* in respond >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>app_iter *=* self*.*application*(*environ*,* detect_start_response*)* Module paste.debug.prints:*98* in __call__ >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>environ*,* self*.*app*)* Module paste.wsgilib:*539* in intercept_output >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>app_iter *=* application*(*environ*,* replacement_start_response*)* Module paste.recursive:*80* in __call__ >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>*return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*632* in __call__ >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>*return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*160* in __call__ >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>body *=* method*(* trans*,* ****kwargs *)* Module galaxy.web.framework:*67* in decorator >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>*return* simplejson*.*dumps*(* func*(* self*,* trans*,* ***args*,* ****kwargs *)* *)* Module galaxy.web.controllers.workflow:*706* in editor_form_post >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>'tool_state' *:* incoming*.*pop*(*"tool_state"*)* Module galaxy.workflow.modules:*363* in from_dict >> <http://127.0.0.1:8085/workflow/editor?id=d413a19dec13d11e#>*assert* type *in* self*.*module_types *AssertionError: * [] And when I added an annotation/notes, and I wanted to save this work flow, I got the same error. Any idea? Thanks! Yufei

1 0

LSF DRMAA/FedStage config-make errors
by Douglas W Cloud 30 Aug '12

30 Aug '12

Hello all, I am trying to get a Galaxy server configured to submit jobs to an LSF cluster running version 7 update 2, generally CentOS/Redhat 5 systems. I have the Galaxy host as a client in LSF, but when I tried to configure/make/install the FedStage drmaa code, it errors out in the make stage (see stdout below). Don't know what to fix, if it's possible. If anyone has experience configuring/troubleshooting this code to get Galaxy conversant with LSF, I'd appreciate any feedback. Thanks! Regards, Doug _________ util.c: In function 'lsfdrmaa_map_lsberrno': util.c:730: error: 'LSBE_ADRSV_ID_UNIQUE' undeclared (first use in this function) util.c:730: error: (Each undeclared identifier is reported only once util.c:730: error: for each function it appears in.) util.c:731: error: 'LSBE_BAD_RSVNAME' undeclared (first use in this function) util.c:732: error: 'LSBE_BAD_CHKPNTDIR' undeclared (first use in this function) util.c:734: error: 'LSBE_JOB_REQUEUE_BADEXCLUDE' undeclared (first use in this function) util.c:737: error: 'LSBE_ADVRSV_ACTIVESTART' undeclared (first use in this function) util.c:739: error: 'LSBE_ADRSV_ID_USED' undeclared (first use in this function) util.c:740: error: 'LSBE_ADRSV_PREVDISABLED' undeclared (first use in this function) util.c:742: error: 'LSBE_ADRSV_DISABLECURR' undeclared (first use in this function) util.c:744: error: 'LSBE_ADRSV_NOT_RSV_HOST' undeclared (first use in this function) util.c:749: error: 'LSBE_ADRSV_MOD_REMOTE' undeclared (first use in this function) util.c:751: error: 'LSBE_JOB_RUSAGE_EXCEED_LIMIT' undeclared (first use in this function) util.c:752: error: 'LSBE_APP_RUSAGE_EXCEED_LIMIT' undeclared (first use in this function) util.c:755: error: 'LSBE_ADRSV_DISABLE_DATE' undeclared (first use in this function) util.c:759: error: 'LSBE_ADRSV_DETACH_MIX' undeclared (first use in this function) util.c:761: error: 'LSBE_ADRSV_DETACH_ACTIVE' undeclared (first use in this function) util.c:765: error: 'LSBE_MISSING_START_END_TIME' undeclared (first use in this function) make[3]: *** [libdrmaa_la-util.lo] Error 1 make[3]: Leaving directory `/root/installers/lsf_drmaa-1.0.4/lsf_drmaa' make[2]: *** [all] Error 2 make[2]: Leaving directory `/root/installers/lsf_drmaa-1.0.4/lsf_drmaa' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/root/installers/lsf_drmaa-1.0.4' (Same output with version 1.0.3.) This communication is for use by the intended recipient and contains information that may be Privileged, confidential or copyrighted under applicable law. If you are not the intended recipient, you are hereby formally notified that any use, copying or distribution of this e-mail, in whole or in part, is strictly prohibited. Please notify the sender by return e-mail and delete this e-mail from your system. Unless explicitly and conspicuously designated as "E-Contract Intended", this e-mail does not constitute a contract offer, a contract amendment, or an acceptance of a contract offer. This e-mail does not constitute a consent to the use of sender's contact information for direct marketing purposes or for transfers of data to third parties. Francais Deutsch Italiano Espanol Portugues Japanese Chinese Korean http://www.DuPont.com/corp/email_disclaimer.html

2 1

run_reports.sh not starting
by Joachim Jacob 30 Aug '12

30 Aug '12

Hi all, When running galaxy-dist/run_reports.sh I get errors. It has been awhile since I run it, and it worked perfect then however. python path is: /home/galaxy/galaxy-dist/eggs/pycrypto-2.5-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/bx_python-0.7.1_7b95ff194725-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/amqplib-0.6.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/pexpect-2.4-py2.6.egg, /home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg, /home/galaxy/galaxy-dist/eggs/MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/WebHelpers-0.2-py2.6.egg, /home/galaxy/galaxy-dist/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/wchartype-0.1-py2.6.egg, /home/galaxy/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg, /home/galaxy/galaxy-dist/eggs/docutils-0.7-py2.6.egg, /home/galaxy/galaxy-dist/eggs/WebOb-0.8.5-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Routes-1.12.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg, /home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg, /home/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg, /home/galaxy/galaxy-dist/lib, /usr/local/lib64/python2.6/site-packages, /opt/MACS/1.4.2/lib/python2.6/site-packages, /opt/MACS/1.3.7/lib/python2.6/site-packages, /usr/lib64/python26.zip, /usr/lib64/python2.6, /usr/lib64/python2.6/plat-linux2, /usr/lib64/python2.6/lib-tk, /usr/lib64/python2.6/lib-old, /usr/lib64/python2.6/lib-dynload, /usr/lib64/python2.6/site-packages/PIL, /usr/lib64/python2.6/site-packages/gtk-2.0, /usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info, /usr/lib/python2.6/site-packages, /usr/lib64/python2.6/site-packages galaxy.model.custom_types DEBUG 2012-08-30 09:06:58,051 psycopg2 egg successfully loaded for postgres dialect Traceback (most recent call last): File "./scripts/paster.py", line 34, in <module> command.run() File "/home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run invoke(command, command_name, options, args[1:]) File "/home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke exit_code = runner.run(args) File "/home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run result = self.command() File "/home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command relative_to=base, global_conf=vars) File "/home/galaxy/galaxy-dist/eggs/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp **kw) File "/home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 225, in loadobj return context.create() File "/home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 625, in create return self.object_type.invoke(self) File "/home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 110, in invoke return fix_call(context.object, context.global_conf, **context.local_conf) File "/home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call val = callable(*args, **kw) File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/reports/buildapp.py", line 52, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/reports/app.py", line 22, in __init__ create_tables = True ) File "/home/galaxy/galaxy-dist/lib/galaxy/model/mapping.py", line 1892, in init metadata.create_all() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 1796, in create_all bind.create(self, checkfirst=checkfirst, tables=tables) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1129, in create self._run_visitor(self.dialect.schemagenerator, entity, connection=connection, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1154, in _run_visitor conn = self.contextual_connect(close_with_result=False) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1229, in contextual_connect return self.Connection(self, self.pool.connect(), close_with_result=close_with_result, **kwargs) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 142, in connect return _ConnectionFairy(self).checkout() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 304, in __init__ rec = self._connection_record = pool.get() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 161, in get return self.do_get() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 639, in do_get con = self.create_connection() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 122, in create_connection return _ConnectionRecord(self) File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 198, in __init__ self.connection = self.__connect() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/pool.py", line 261, in __connect connection = self.__pool._creator() File "/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/strategies.py", line 80, in connect raise exc.DBAPIError.instance(None, None, e) sqlalchemy.exc.OperationalError: (OperationalError) could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? None None Removing PID file reports_webapp.pid The postgres database is running, user credentials are correct. I tried also running as root, as opposed to user galaxy, but no avail. Thanks for your consideration, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 2

September 2012 Galaxy Update
by Dave Clements 29 Aug '12

29 Aug '12

1 0

Modules
by Fabien Mareuil 29 Aug '12

29 Aug '12

Dear Colleagues, Many teams use module, a user interface for the Modules package. The Modules package enable to manage easily several versions of a program with a dynamic modification of the user's environment via modulefiles. I would like to know if it is planned to integrate the ability to load modules via galaxy? For example, we could imagine a configuration file that would link the name and version of a tool with the module name and allowing to load the module before running the tool locally or on a cluster. Thank you for your answer, Best regards, Fabien Mareuil

2 1