- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] (no subject)
by Dave Bouvier 16 Apr '13

16 Apr '13

David, The reason shed_tools should be outside the Galaxy path is that tool shed mercurial repositories can interfere with the Galaxy mercurial repository, since mercurial will find more than one .hg directory. As long as the data you want to store is not a mercurial repository, you should be safe, but my advice would still be to locate it outside the Galaxy path to avoid any possibility of conflicts. --Dave B. On 4/16/13 16:28:01.000, David Bonsall wrote: > Thanks Dave > > A quick update from /galaxy-dist .... > > % hg update stable > > > .....did the trick! > > > Btw, I have another question I have since added sub-directories to /galaxy-dist to store blast databases etc. Is this bad? Will mercurial be able to update. I remember a warning to not install shed-tools directory in the main galaxy stem for this reason. Does this rule apply to all directories? > > > Best Wishes > > David > > > > On Tue, Apr 16, 2013 at 9:16 PM, Dave Bouvier <dave(a)bx.psu.edu > <mailto:dave@bx.psu.edu>> wrote: > > David, > > Thank you for reporting this error, which was due to a new tool shed > feature being introduced in -central, which had not yet been wrapped > in a backwards compatibility layer for -dist. As of > 9440:bbb229927c68, the issue should be resolved. > > --Dave B. > > On 4/11/13 12:51:03.000, David Bonsall wrote: > > Dear Galaxy team, > > I'm trying to install NCBI + tools on a local instance of galaxy > (rep: > 8.1546099212f), but I get the following error, which I do not > get when > trying to install other tools. > > Thank you for all your hard work. It's all much appreciated > > David > > > > Error Traceback: > View as: Interactive | Text | XML (full) > ⇝ HTTPError: HTTP Error 500: Internal Server Error > URL: > http://127.0.0.1:8080/admin___toolshed/prepare_for_install?__tool_shed_url=… > <http://127.0.0.1:8080/admin_toolshed/prepare_for_install?tool_shed_url=http…> > Module weberror.evalexception.__middleware:364 in respond > view > >> app_iter = self.application(environ, detect_start_response) > Module paste.recursive:84 in __call__ view > >> return self.application(environ, start_response) > Module paste.httpexceptions:633 in __call__ view > >> return self.application(environ, start_response) > Module galaxy.web.framework.base:125 in __call__ view > >> return self.handle_request( environ, start_response ) > Module galaxy.web.framework.base:182 in handle_request view > >> body = method( trans, **kwargs ) > Module galaxy.web.framework:228 in decorator view > >> return func( self, trans, *args, **kwargs ) > Module galaxy.webapps.galaxy.__controllers.admin_toolshed:894 in > prepare_for_install view > >> common_install_util.get___dependencies_for_repository( trans, > tool_shed_url, repo_info_dict, includes_tool_dependencies ) > Module tool_shed.util.common_install___util:90 in > get_dependencies_for___repository view > >> required_repo_info_dicts = get_required_repo_info_dicts( > tool_shed_url, util.listify( repo_info_dict ) ) > Module tool_shed.util.common_install___util:283 in > get_required_repo_info_dicts view > >> response = urllib2.urlopen( url ) > Module urllib2:126 in urlopen view > >> return _opener.open(url, data, timeout) > Module urllib2:400 in open view > >> response = meth(req, response) > Module urllib2:513 in http_response view > >> 'http', request, response, code, msg, hdrs) > Module urllib2:438 in error view > >> return self._call_chain(*args) > Module urllib2:372 in _call_chain view > >> result = func(*args) > Module urllib2:521 in http_error_default view > >> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > HTTPError: HTTP Error 500: Internal Server Error > > > _____________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/__search/mailinglists/ > <http://galaxyproject.org/search/mailinglists/> > >

1 0

Re: [galaxy-dev] Toolshed errors
by Greg Von Kuster 16 Apr '13

16 Apr '13

Hello Björn, On Apr 6, 2013, at 9:06 AM, Björn Grüning wrote: > Hi Greg and Dave, > > good to know that it is at least installing :) > I also tried without apache, fresh checkout and get the same timeouts. > Maybe my computer is just so slow that its triggers the timeout. Also my > Memory is very limited and nearly swapped during compilation. > > But with the new API I'm able to install all of them just fine. Is that > explainable? > During my experiments I found that im not able to install the tools if a > new-panel-section is supplied: > > galaxy.web.framework ERROR 2013-04-06 14:29:05,411 Uncaught exception in > exposed API method: > > Traceback (most recent call last): > File > "/home/bag/projects/galaxy/galaxy-central/lib/galaxy/web/framework/__init__.py", line 196, in decorator > rval = func( self, trans, *args, **kwargs) > File > "/home/bag/projects/galaxy/galaxy-central/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py", line 254, in install_repository_revision > reinstalling=False ) > File > "/home/bag/projects/galaxy/galaxy-central/lib/tool_shed/galaxy_install/repository_util.py", line 358, in install_tool_shed_repository > if tool_panel_section_key: > File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__ > raise KeyError(key) > KeyError: 'section_chemicaltoolbox' This issue should be fixed in revision 3a07307 - thanks for reporting this! > > Without that option it works. I tried to understand the code-path but > found no section where a new entry is created under the toolbox. What I > found is that: tool_panel_section_keys in repository_dependecy_util.py > line 235 is updated every time and you will end up with several > identical instances of 'section_chemicaltoolbox'. Hope that helps a > little bit. I'm not able to reproduce this (assuming it is not related to the issue I fixed above). Is this issue now resolved for your? > > Also please see a pull-request for a small typo: > https://bitbucket.org/galaxy/galaxy-central/pull-request/150/change-error-m… This pull request has been accepted - thanks for the fixes! > > Thanks for your work! > Bjoern > > > > >> Hello Björn, >> >> Dave Bouvier (cc'd on this) spent some time yesterday inspecting the logs for out test tool shed and found no timeouts in either our paster log or our nginx log. Dave was also successful in installing your repository and dependencies. If I remember correctly, this timeout issue may be intermittent, but again, Dave found no logged timeout events at our end. If you uncover any additional information, please let us know, and we'll continue to keep an eye on this. >> >> Thanks very much, >> >> Greg Von Kuster >> >> >> On Apr 4, 2013, at 5:32 AM, Björn Grüning wrote: >> >>> Hi Greg, >>> >>> sorry forgot to attach the new traceback. Its now attached. >>> If the grepping with the IP address did not work, maybe you can grep for >>> the chemicaltoolbox/chemfp/openbabel/osra repositories. Im probably the >>> only one working with it. >>> >>> Have a nice day! >>> Bjoern >>> >>>> Hello Björn, >>>> >>>> Thanks very much for your message! Please see my inline comments. >>>> >>>> >>>> On Apr 2, 2013, at 4:33 PM, Björn Grüning wrote: >>>> >>>>> I have tested a few things regarding to the toolshed and prepared a >>>>> complex module-dependency-beast, with meta-package and >>>>> orhphan-tool-dependencies and so on. A few problems I encountered are >>>>> summarized below. You can try it by yourself, if you install the >>>>> chemicaltoolbox package from the testtoolshed. >>>>> >>>>> - three crashes with traceback attached, during tool-installation >>>>> >>>>> - one 'name' not defined error, with attached patch >>>> >>>> >>>> Thanks very much for finding this issue and for the patch. We've applied it in changeset 9252:da2d6ee29f30 in our stable branch, so it is running on both public Galaxy tool sheds. >>>> >>>> >>>>> >>>>> - For me the the installation of tools with really complex and long >>>>> compiling steps are stalled in 'Installing' forever. Try to install the >>>>> osra package (it often hangs, but not every time). >>>>> Its not possible to uninstall/reinstall these dependencies >>>>> without touching the database. And the error says, tool is already >>>>> installed. Would be nice if the admin is able to reset that repository >>>>> easily, even if it is limbo ... (is there a timeout somewhere?) >>>>> Error: paste.httpserver.ThreadPool WARNING 2013-04-02 22:02:41,256 Thread 140246687659776 hung (working on task for 1964 seconds) >>>>> (The SystemExit: traceback attached) >>>> >>>> I've create the following Trello card for this issue: >>>> >>>> https://trello.com/card/toolshed-enable-galaxy-admin-to-reset-tool-dependen… >>>> >>>> There are 4 places where this timeout may be occurring: Galaxy tool shed server, Galaxy tool shed nginx, a proxy server at your end, or your local Galaxy server. There is not enough information in your attached traceback log to determine where the timeout is occurring. Can you provide me your Galaxy server's IP address so I can look in logs at our end? >>>> >>>> >>>>> >>>>> - It would be great if the tool-developer or the admin has more control >>>>> over the location of the tools in the toolbar. Installing meta package >>>>> leaves you with only one choice, regardless how many tools are bundled. >>>> >>>> >>>> Support for this will be available in the next Galaxy release. There are Trello cards for this here: >>>> >>>> https://trello.com/card/toolshed-enhance-the-current-galaxy-api-for-install… >>>> https://trello.com/card/toolshed-add-features-that-leverage-the-galaxy-api-… >>>> >>>> >>>>> >>>>> - Installing dependencies will leave me with empty folders under >>>>> galaxy-root with the name of the dependency. My guess is, >>>>> that if I manually download a file/folder with wget, the folder is not purged, but the content is. >>>> >>>> >>>> Our intent is that installing repositories from the tool shed is completely automated with no manual intervention required. Tool dependency definitions should not use utilities like wget to download dependencies because these utilities may not be available in the environments (e.g. vanilla mac os) into which the repositories are being installed. I'll enhance the tool dependency tag sets as necessary to accommodate additional download and install requirements. Can you let me know if the current download_by_url needs enhancements? >>>> >>>> >>>>> >>>>> - Often after upgrading the tool_dependencies.xml file, I was not able >>>>> to 'get updates' from the toolshed. The version was up-to-data, but it >>>>> wasn't. Deinstalling and Installing fixed it. But I think its ab bug, >>>>> isn't it? >>>> >>>> This very well may be a bug. Can you please provide additional details so that I can reproduce this behavior? I'll get a fix asap. >>>> >>>> >>>>> >>>>> - Is there a way, to just download a python file, not zipped and add it to PYTHONPATH? >>>>> If I download it with <action type="download_by_url"> I have problems to copy that file with move. >>>>> I would like to download it with wget, but one download by URL is required. My current solution is, >>>>> to include a dummy download_by_url and than fetch with wget, or download a archive version. >>>> >>>> I've created the following Trello card for this issue: >>>> >>>> https://trello.com/card/toolshed-enhance-the-download-by-url-process-for-in… >>>> >>>> >>>>> >>>>> - Is it correct that a repository referenced only in >>>>> tool_dependencies.xml is not enough to install? numpy was only >>>>> referenced as tool_dependency and I got the error that the tool (with >>>>> owner and revision) was not found. After adding to repository_dependency >>>>> it was ok. >>>> >>>> Can you provide more details about your question above (e.g., the tool_dependencies.xml fie would help). I want to make sure i understand the issue. >>>> >>>> Thanks very much Björn, >>>> >>>> Greg Von Kuster >>>> >>>> >>>> >>> >>> <trackback.txt> >> > >

1 0

(no subject)
by David Bonsall 16 Apr '13

16 Apr '13

Dear Galaxy team, I'm trying to install NCBI + tools on a local instance of galaxy (rep: 8.1546099212f), but I get the following error, which I do not get when trying to install other tools. Thank you for all your hard work. It's all much appreciated David Error Traceback: View as: Interactive | Text | XML (full) ⇝ HTTPError: HTTP Error 500: Internal Server Error URL: http://127.0.0.1:8080/admin_toolshed/prepare_for_install?tool_shed_url=http… Module weberror.evalexception.middleware:364 in respond view >> app_iter = self.application(environ, detect_start_response) Module paste.recursive:84 in __call__ view >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ view >> return self.application(environ, start_response) Module galaxy.web.framework.base:125 in __call__ view >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:182 in handle_request view >> body = method( trans, **kwargs ) Module galaxy.web.framework:228 in decorator view >> return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:894 in prepare_for_install view >> common_install_util.get_dependencies_for_repository( trans, tool_shed_url, repo_info_dict, includes_tool_dependencies ) Module tool_shed.util.common_install_util:90 in get_dependencies_for_repository view >> required_repo_info_dicts = get_required_repo_info_dicts( tool_shed_url, util.listify( repo_info_dict ) ) Module tool_shed.util.common_install_util:283 in get_required_repo_info_dicts view >> response = urllib2.urlopen( url ) Module urllib2:126 in urlopen view >> return _opener.open(url, data, timeout) Module urllib2:400 in open view >> response = meth(req, response) Module urllib2:513 in http_response view >> 'http', request, response, code, msg, hdrs) Module urllib2:438 in error view >> return self._call_chain(*args) Module urllib2:372 in _call_chain view >> result = func(*args) Module urllib2:521 in http_error_default view >> raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error

2 2

the release_2013.04.01 update crashes our galaxy server
by Christophe Antoniewski 16 Apr '13

16 Apr '13

Hi, After a crash recovery and a second trial I must admit that I cannot upgrade our galaxy instance. I need some help... the facts: initial instance: $ hg head changeset: 8796:961ae35ba612 branch: stable tag: tip parent: 8794:1c7174911392 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Feb 08 11:23:04 2013 -0500 summary: Added tag release_2013.02.08 for changeset 1c7174911392 changeset: 8795:9fd7fe0c5712 parent: 8785:25bd9d387135 parent: 8794:1c7174911392 user: Nate Coraor <nate(a)bx.psu.edu> date: Fri Feb 08 11:22:54 2013 -0500 summary: merge from stable Then I pull the upgrade using mercurial and update the postgresql database as follows: $ hg pull pulling from https://bitbucket.org/galaxy/galaxy-dist/ searching for changes adding changesets adding manifests adding file changes added 516 changesets with 1568 changes to 710 files (-1 heads) (run 'hg update' to get a working copy) galaxy@GED-Server:~/galaxy-dist$ hg update release_2013.04.01 merging lib/galaxy/web/form_builder.py merging tools/ngs_rna/tophat_wrapper.xml 750 files updated, 2 files merged, 433 files removed, 0 files unresolved galaxy@GED-Server:~/galaxy-dist$ hg branch stable galaxy@GED-Server:~/galaxy-dist$ sh manage_db.sh upgrade 110 -> 111... Add support for job destinations to the job table done 111 -> 112... Migration script to add the data_manager_history_association table and data_manager_job_association. 0112_add_data_manager_history_association_and_data_manager_job_association_tables DEBUG 2013-04-16 00:59:22,412 Created data_manager_history_association table 0112_add_data_manager_history_association_and_data_manager_job_association_tables DEBUG 2013-04-16 00:59:22,412 Created data_manager_history_association table 0112_add_data_manager_history_association_and_data_manager_job_association_tables DEBUG 2013-04-16 00:59:22,428 Created data_manager_job_association table 0112_add_data_manager_history_association_and_data_manager_job_association_tables DEBUG 2013-04-16 00:59:22,428 Created data_manager_job_association table done 112 -> 113... Migration script to update the migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate. done 113 -> 114... Migration script to update the migrate_tools.repository_path column to point to the new location lib/tool_shed/galaxy_install/migrate. done When I restarts the server, I have $ sh run.sh --reload Initializing data_manager_conf.xml from data_manager_conf.xml.sample Initializing shed_data_manager_conf.xml from shed_data_manager_conf.xml.sample Some eggs are out of date, attempting to fetch... Fetched http://eggs.galaxyproject.org/PasteDeploy/PasteDeploy-1.5.0-py2.7.egg Removed conflicting egg: /home/galaxy/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg Fetched http://eggs.galaxyproject.org/raven/raven-3.1.8-py2.7.egg Fetch successful. > I suspect that some new python eggs may cause my troubles... and then, ... some usual bla bla... and finishes by .... galaxy.webapps.galaxy.buildapp DEBUG 2013-04-16 01:00:30,077 Enabling 'Request ID' middleware Traceback (most recent call last): File "./scripts/paster.py", line 33, in <module> serve.run() File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1056, in run invoke(command, command_name, options, args[1:]) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 1062, in invoke exit_code = runner.run(args) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 227, in run result = self.command() File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py", line 650, in command app = loadapp( app_spec, name=app_name, relative_to=base, global_conf=vars) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 350, in loadapp return loadobj(APP, uri, name=name, **kw) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 375, in loadobj return context.create() File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 813, in create return self.object_type.invoke(self) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 332, in invoke filtered = context.next_context.create() File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 813, in create return self.object_type.invoke(self) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 249, in invoke return fix_call(context.object, context.global_conf, **context.local_conf) File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line 97, in fix_call val = callable(*args, **kw) File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 172, in app_factory webapp = wrap_in_static( webapp, global_conf, **kwargs ) File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 327, in wrap_in_static urlmap["/static"] = Static( conf.get( "static_dir" ), cache_time ) File "/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/static.py", line 16, in __init__ StaticURLParser.__init__( self, directory ) File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/urlparser.py", line 430, in __init__ self.directory = self.normpath(directory) File "/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/urlparser.py", line 435, in normpath return os.path.normcase(os.path.abspath(path)) File "/usr/lib/python2.7/posixpath.py", line 343, in abspath if not isabs(path): File "/usr/lib/python2.7/posixpath.py", line 53, in isabs return s.startswith('/') *AttributeError: 'NoneType' object has no attribute 'startswith'* *galaxy.jobs.handler INFO 2013-04-16 01:00:30,112 sending stop signal to worker thread* *galaxy.jobs.handler INFO 2013-04-16 01:00:30,112 job handler queue stopped* *galaxy.jobs.runners INFO 2013-04-16 01:00:30,112 LWRRunner: Sending stop signal to monitor thread* *galaxy.jobs.runners INFO 2013-04-16 01:00:30,112 LWRRunner: Sending stop signal to 3 worker threads* *galaxy.jobs.runners INFO 2013-04-16 01:00:30,113 LocalRunner: Sending stop signal to 5 worker threads* *galaxy.jobs.handler INFO 2013-04-16 01:00:30,115 sending stop signal to worker thread* *galaxy.jobs.handler INFO 2013-04-16 01:00:30,115 job handler stop queue stopped* does anyone has a clue about what is going wrong (eggs or something...) Thanks thanks thanks in advance.... -- Christophe Antoniewski Drosophila Genetics and Epigenetics Laboratoire de Biolologie du Développement 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 Tel +33 1 44 27 34 39 Fax +33 1 44 27 34 45 Mobile +33 6 68 60 51 50 http://drosophile.org

2 3

where the log in information saved?
by shenwiyn 16 Apr '13

16 Apr '13

Hi everyone, I installed my local galaxy successfully.But I have a problem about how to find out the log in information of each user ,such as the register mail and the password .And which files do this informations saved? Thank you! shenwiyn

4 3

Re: [galaxy-dev] Tophat didn't run
by Jennifer Jackson 16 Apr '13

16 Apr '13

Hi Sarah, On 4/11/13 8:02 AM, Sarah Maman wrote: > Thnaks Jennifer, > > Excuse me, my previous mail contains an error : In fact, the reference > genome from my history was a fasta format (the name was GTF file but > the format was fasta...). > So, when I run tophat with a reference genome from your history, here > is the error message (my reference genome is a FASTA file) : OK, now this looks like a tool/index mismatch problem. Most likely rooted in a binary path issue. > Error in tophat: > > [2013-04-11 14:57:12] Beginning TopHat run (v2.0.5) > ----------------------------------------------- > [2013-04-11 14:57:12] Checking for Bowtie > Bowtie version: 2.0.0.7 > [2013-04-11 14:57:12] Checking for Samtools > Samtools version: 0.1.19.0 > [2013-04-11 14:57:13] Checking for Bowtie index files > Error: Could not find Bowtie 2 index files (/tmp/1078173.1.workq/tmpzxEFNK/dataset_6485.*.bt2) > > Settings: > blablabla OK..... > Total time for backward call to driver() for mirror index: 00:00:57 > TopHat v2.0.5 > tophat -p 4 /tmp/1078173.1.workq/tmpzxEFNK/dataset_6485 /work/galaxy/database/files/006/dataset_6528.dat > [bam_header_read] EOF marker is absent. The input is probably truncated. > [bam_header_read] invalid BAM binary header (this is not a BAM file). > [bam_index_core] Invalid BAM header.[bam_index_build2] fail to index the BAM file. > Epilog : job finished at jeu. avril 11 14:57:18 CEST 2013 > > And here is my bowtie and tophat versions : > > $ which bowtie > bowtie -v > 0.12.8 This is good > $ which tophat > tophat -v > 2.0.5 This is most likely the problem. There is probably a symbolic link pointing from tophat -> tophat2. You will want to remove that. The tool wrappers will be looking for the correct binary/indexes for the version they are each dependent on. This means that if you are running Tophat2 for Illumina, you want both tophat2 and bowtie2 to be used, along with the bowtie2 indexes. This is detailed in the wikis I sent in the Tophat/Bowtie sections, for both dependencies and the index set up. My guess at this point, without seeing your exact files, is that you need to add the index path to the bowtie2 loc file, and remove/adjust the symbolic link as I stated above, then restart, and test again to see if that fixes the problem. > But we have also, available on our cluster : > bowtie2 --version is 2.0.0-beta7 This is also good, if "which bowtie" == v0.12.8 and "which bowtie2" == v2. If "bowtie" is pointing to v2 on your cluster nodes, then remove that symbolic link, so that this is instead pointing to the correct binary (v0.12.8). same for bowtie2, should point to the v2 binary. bowtie/tophat and bowtie2/tophat2 are not the same executable and use different indexes - this is most likely why you had to use the bowtie v0.12.8 loc to get tophat2 going to begin with. Hope it works this time! Please keep replies on the list to help us with tracking, Jen Galaxy team > > Could you please tell me how to point to the v2 binaries (how to > change symbolic links) ? > > Thanks in advance, > Sarah > > > Jennifer Jackson a écrit : >> Hi Sarah, >> >> It still sounds like there is a path problem - this is why the tools >> are looking in the wrong loc file. When bowtie2/tophat2 installs, it >> will create a symbolic link that names itself as just "bowtie" or >> "tophat", pointing to the v2 binaries. >> >> When you run these, what do you get? >> >> $ which bowtie >> >> $ which tophat >> >> My guess is that these are symbolic links pointing to the v2 >> binaries. You will want to remove these. This is noted in the NGS >> set-up wiki, but easy to miss. >> >> For the custom _reference genome _portion below, there is a mix-up >> here. A custom _reference genome_ is in fasta format, not GTF format. >> I think what you are doing is using a _reference annotation_ file >> with the process. Both can be used with RNA-seq tools, but the >> _reference genome_ is the one with the indexes. The link I sent about >> _custom reference genomes_ explains how to use one of these, if you >> still what want to try. >> >> I think it is worth reviewing the path and loc info, plus the binary >> commands above. Unless this helps you to solve the problem on your >> own now. >> >> Thanks! >> >> Jen >> Galaxy team >> >> On 4/11/13 6:16 AM, Sarah Maman wrote: >>> Thanks a lot Jennifer, >>> >>> Restart, full paths were OK. >>> >>> I don't know why but the 2nd version of Tophat (so the tophat tool >>> available from Galaxy) search indexs in bowtie-index.loc file and >>> not in bowtie2-index.loc >>> So, I've added my bowtie2 index paths in bowtie-index. loc file and >>> tophat run... >>> >>> >>> But when I want to run tophat with a reference genome from your >>> history, here is the error message (my reference genome is a GFT file) : >>> Error in tophat: >>> >>> [2013-04-11 14:57:12] Beginning TopHat run (v2.0.5) >>> ----------------------------------------------- >>> [2013-04-11 14:57:12] Checking for Bowtie >>> Bowtie version: 2.0.0.7 >>> [2013-04-11 14:57:12] Checking for Samtools >>> Samtools version: 0.1.19.0 >>> [2013-04-11 14:57:13] Checking for Bowtie index files >>> Error: Could not find Bowtie 2 index files (/tmp/1078173.1.workq/tmpzxEFNK/dataset_6485.*.bt2) >>> >>> >>> Settings: >>> blablabla OK..... >>> Total time for backward call to driver() for mirror index: 00:00:57 >>> TopHat v2.0.5 >>> tophat -p 4 /tmp/1078173.1.workq/tmpzxEFNK/dataset_6485 /work/galaxy/database/files/006/dataset_6528.dat >>> [bam_header_read] EOF marker is absent. The input is probably truncated. >>> [bam_header_read] invalid BAM binary header (this is not a BAM file). >>> [bam_index_core] Invalid BAM header.[bam_index_build2] fail to index the BAM file. >>> Epilog : job finished at jeu. avril 11 14:57:18 CEST 2013 >>> Thanks in advance, >>> Sarah >>> >>> >>> Jennifer Jackson a écrit : >>>> Hi Sarah, >>>> >>>> Let's try to sort this out. Your problem does not seem to be the >>>> same as in the question referenced, but we can see. First - just to >>>> double check - since setting up the genome, you have restarted the >>>> server? If not, do that first and check to see if that fixes the >>>> problem. Basically, you want to follow this checklist and >>>> restarting is the final step: >>>> http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup >>>> >>>> If the problem persists, then would you please send a few more details: >>>> >>>> 1 - full paths* on you system where you keep the .bt2 indexes, sam >>>> index, and .fa file. Maybe do an "ls -l" on these dirs so we can >>>> check the symbolic links are in place and named correctly. >>>> >>>> * as a note, these should be "hard paths" and not symbolic (except >>>> for the .fa links), and must have permissions set to be accessible >>>> to the "galaxy user" >>>> >>>> 2 - lines from your bowtie2_indices.loc and sam_fa_indices.loc file >>>> for this genome. I may have you double check your builds.txt file >>>> later. if this doesn't sounds familiar, it could be the problem, >>>> the genome must be in there, too. - see this wiki: >>>> http://wiki.galaxyproject.org/Admin/Data%20Integration >>>> >>>> 3 - full error message you get when you try to run this using a >>>> genome in fasta format from your history. It really shouldn't be >>>> the same error - something is not right with the settings and a >>>> custom genome is not actually being used if that is the case. Give >>>> it another try and see what happens, then send that info. This is a >>>> bit of a side case, we should get your basic install correct, but >>>> knowing how to do this is a good thing and easy to learn. >>>> http://wiki.galaxyproject.org/Support#Custom_reference_genome >>>> >>>> It is OK to masked out anything like user names/groups you don't >>>> want to share. Please keep on the list in case we need other feedback. >>>> >>>> Thanks! >>>> >>>> Jen >>>> Galaxy team >>>> >>>> On 4/10/13 3:15 AM, Sarah Maman wrote: >>>>> Hello, >>>>> >>>>> >>>>> When I run tophat ("Tophat for Illumina Find splice junctions >>>>> using RNA-seq data ), the job failed with truncated files. >>>>> However, index files are available and I get exactly the same >>>>> error message using built-in index or one of my history. >>>>> >>>>> / >>>>> Tool execution generated the following error message: >>>>> >>>>> Error in tophat: >>>>> >>>>> [2013-04-10 09:17:07] Beginning TopHat run (v2.0.5) >>>>> ----------------------------------------------- >>>>> [2013-04-10 09:17:07] Checking for Bowtie >>>>> Bowtie version: 2.0.0.7 >>>>> [2013-04-10 09:17:07] Checking for Samtools >>>>> Samtools version: 0.1.19.0 >>>>> [2013-04-10 09:17:07] Checking for Bowtie index files >>>>> Error: Could not find Bowtie 2 index files >>>>> (/work/galaxy/Danio_rerio.Zv9.62.dna.chromosome.22.fa.*.bt2) >>>>> >>>>> >>>>> The tool produced the following additional output: >>>>> >>>>> TopHat v2.0.5 >>>>> tophat -p 4 /work/galaxy/Danio_rerio.Zv9.62.dna.chromosome.22.fa >>>>> /work/galaxy/database/files/006/dataset_6528.dat >>>>> [bam_header_read] EOF marker is absent. The input is probably >>>>> truncated. >>>>> [bam_header_read] invalid BAM binary header (this is not a BAM file). >>>>> [bam_index_core] Invalid BAM header.[bam_index_build2] fail to >>>>> index the BAM file. >>>>> Epilog : job finished at mer. avril 10 09:17:12 CEST 2013 / >>>>> >>>>> In this post >>>>> (http://dev.list.galaxyproject.org/tophat-for-illumina-looking-in-wrong-dire…) >>>>> the solution isn't found. >>>>> >>>>> Do you have any idea, >>>>> Sarah Maman >>>>> -- >>>>> --*-- >>>>> Sarah Maman >>>>> INRA - LGC - SIGENAE >>>>> http://www.sigenae.org/ >>>>> Chemin de Borde-Rouge - Auzeville - BP 52627 >>>>> 31326 Castanet-Tolosan cedex - FRANCE >>>>> Tel: +33(0)5.61.28.57.08 >>>>> Fax: +33(0)5.61.28.57.53 >>>>> --*-- >>>>> >>>>> >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>> -- >>>> Jennifer Hillman-Jackson >>>> Galaxy Support and Training >>>> http://galaxyproject.org >>> >>> >>> -- >>> --*-- >>> Sarah Maman >>> INRA - LGC - SIGENAE >>> http://www.sigenae.org/ >>> Chemin de Borde-Rouge - Auzeville - BP 52627 >>> 31326 Castanet-Tolosan cedex - FRANCE >>> Tel: +33(0)5.61.28.57.08 >>> Fax: +33(0)5.61.28.57.53 >>> --*-- >> >> -- >> Jennifer Hillman-Jackson >> Galaxy Support and Training >> http://galaxyproject.org > > > -- > --*-- > Sarah Maman > INRA - LGC - SIGENAE > http://www.sigenae.org/ > Chemin de Borde-Rouge - Auzeville - BP 52627 > 31326 Castanet-Tolosan cedex - FRANCE > Tel: +33(0)5.61.28.57.08 > Fax: +33(0)5.61.28.57.53 > --*-- -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org

2 1

Broken tools ids from ./run_functional_tests.sh -list
by Peter Cock 15 Apr '13

15 Apr '13

Hi all, Does anyone else see this problem with broken tool ids in the function test script? I found this because one of my own tools wasn't being recognised because for some reason the id was treated as _name= [sic]. This happens on the default branch: $ ./run_functional_tests.sh -list | grep "_name=" _name= RNA/DNA _name= Convert _name= Compute quality statistics _name= Draw quality score boxplot _name= DAVID $ hg branch default hg log -b default | head changeset: 9349:199b62339f26 user: jeremy goecks <jeremy.goecks(a)emory.edu> date: Thu Apr 11 08:44:42 2013 -0400 summary: Whitespace fixes for Vcf datatype. changeset: 9348:dbfc964167ae user: guerler date: Wed Apr 10 17:30:27 2013 -0400 summary: Enhance button for trackster visualization in data display viewer Have I found a recent regression, or is my system messed up somehow? Thanks, Peter

2 4

drmaa and JSV
by jean-François Taly 15 Apr '13

15 Apr '13

Hi, Our galaxy instance runs jobs in a SGE cluster using 2 job-handlers. The SGE cluster uses a Job Submission Verifier (JSV) that rejects any job submission that specify core binding strategies. When Galaxy starts, the first jobs we submit works perfectly: First job submission: galaxy.jobs.manager DEBUG 2013-04-15 14:29:59,285 (194) Job assigned to handler 'handler0' galaxy.jobs DEBUG 2013-04-15 14:29:59,934 (194) Working directory for job is: /scratch/nfs/galaxy.crg.es/job_working_directory/000/194 galaxy.jobs.handler DEBUG 2013-04-15 14:29:59,942 dispatching job 194 to drmaa runner galaxy.jobs.handler INFO 2013-04-15 14:30:00,166 (194) Job dispatched galaxy.jobs.runners.drmaa DEBUG 2013-04-15 14:30:00,468 (194) submitting file /scratch/nfs/galaxy.crg.es/ogs/galaxy_194.sh galaxy.jobs.runners.drmaa DEBUG 2013-04-15 14:30:00,468 (194) command is: python /data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/tools/fastq/fastq_stats.py '/data/www-bi/galaxy.crg.es/files/000/dataset_4.dat' '/data/www-bi/galaxy.crg.es/files/000/dataset_238.dat' 'sanger' galaxy.jobs.runners.drmaa INFO 2013-04-15 14:30:01,538 (194) queued as 458816 galaxy.jobs.runners.drmaa DEBUG 2013-04-15 14:30:02,115 (194/458816) state change: job is queued and active # qstat -cb -j 458816 ============================================================== job_number: 458816 exec_file: job_scripts/458816 submission_time: Mon Apr 15 14:30:01 2013 owner: www-bi uid: 66401 group: www-bi gid: 501 sge_o_home: /data/www-bi sge_o_log_name: www-bi sge_o_path: /data/galaxy/apache/galaxy.crg.es/htdocs/scripts/galaxy-env/bin:/software/galaxy/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/data/www-bi/bin sge_o_shell: /bin/bash sge_o_workdir: /data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist sge_o_host: galaxy account: sge stderr_path_list: NONE:galaxy:/scratch/nfs/galaxy.crg.es/job_working_directory/000/194/194.drmerr reserve: y hard resource_list: virtual_free=12G,h_rt=21600 mail_list: www-bi(a)galaxy.crg.es notify: FALSE job_name: g194_fastq_stats_jtaly_crg_es stdout_path_list: NONE:galaxy:/scratch/nfs/galaxy.crg.es/job_working_directory/000/194/194.drmout jobshare: 0 hard_queue_list: www-el6 env_list: script_file: /scratch/nfs/galaxy.crg.es/ogs/galaxy_194.sh parallel environment: smp range: 2 verify_suitable_queues: 2 binding: set linear:2:0,0 scheduling info: queue instance "pr-el6(a)fenn.linux.crg.es" dropped because it is overloaded: np_load_avg=1.703333 (= 1.703333 + 0.50 * 0.000000 with nproc=12) >= 1.7 queue instance "short(a)node-ib0209bi.linux.crg.es" dropped because it is overloaded: np_load_avg=2.837500 (= 2.837500 + 0.50 * 0.000000 with nproc=8) >= 1.3 queue instance "long(a)node-ib0209bi.linux.crg.es" dropped because it is overloaded: np_load_avg=2.837500 (= 2.837500 + 0.50 * 0.000000 with nproc=8) >= 1.3 The core binding has been added by our jsv script. This is correct. But our second submission fails: galaxy.jobs.runners.drmaa ERROR 2013-04-15 14:30:56,263 Uncaught exception queueing job Traceback (most recent call last): File "/data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 144, in run_next self.queue_job( obj ) File "/data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 232, in queue_job job_id = self.ds.runJob(jt) File "/data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "/data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer)))) File "/data/www-bi/apache/galaxy.crg.es/htdocs/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) DeniedByDrmException: code 17: contact us: XXX(a)XXX.es if we look at the submited params: # cat /tmp/qsub_err.txt $VAR1 = { 'w' => 'e', 'N' => 'g195_fastq_stats_jtaly_crg_es', 'binding_amount' => '2', 'CMDNAME' => '/scratch/nfs/galaxy.crg.es/ogs/galaxy_195.sh', 'binding_type' => 'set', 'M' => { 'www-bi(a)galaxy.crg.es' => undef }, 'binding_strategy' => 'linear', 'l_hard' => { 'virtual_free' => '12G', 'h_rt' => '6:00:00' }, 'shell' => 'n', 'pe_min' => '2', 'USER' => 'www-bi', 'binding_socket' => '0', 'e' => { '/scratch/nfs/galaxy.crg.es/job_working_directory/000/195/195.drmerr' => undef }, 'GROUP' => 'www-bi', 'binding_core' => '0', 'pe_max' => '2', 'CMDARGS' => '0', 'q_hard' => { 'www-el6' => undef }, 'pe_name' => 'smp', 'CLIENT' => 'drmaa', 'b' => 'y', 'R' => 'y', 'VERSION' => '1.0', 'CONTEXT' => 'client', 'o' => { '/scratch/nfs/galaxy.crg.es/job_working_directory/000/195/195.drmout' => undef } }; There's a core binding strategy. The problem is that second job submission is inheriting submission parameters from the first job, and, as the JSV script does not allow to specify core binding strategy by the user, the job is rejected. If you wait some time (600 seconds), the new submit works again... We are wondering if anyone can help us to understand why the submission parameters been inherit by each job? Maybe the DRMAA session is not properly closed? or the environment not cleaned? Thank you for your help Best Jean-François $hg summary parent: 8795:9fd7fe0c5712 merge from stable branch: default commit: 1 modified, 59 unknown update: (current) -- ##################################### Jean-François Taly Bioinformatician Bioinformatics Core Facility http://biocore.crg.cat CRG - Centre de Regulació Genòmica (Room 439) Parc de Recerca Biomèdica de Barcelona (PRBB) Doctor Aiguader, 88 08003 Barcelona Spain email: jean-francois.taly(a)crg.eu phone: +34 93 316 0202 fax: +34 93 316 0099 #####################################

1 0

Research in Genomics
by Twaha Mlwilo 15 Apr '13

15 Apr '13

Hello, all First , I apologise if am posting in wrong forum!, I would like to learn how to conduct research on population genomic and am looking for recommendations for good material such as tutorials,books and course outline. thank you huu

1 0

tasks (parralelism) and from_work_dir bug
by Hagai Cohen 14 Apr '13

14 Apr '13

Hi, I have tried to use tasks with tophat2, but I got the following bug. > multi.py +153 msg = 'nothing to merge for %s (expected %i files)' \ % (output_file_name, len(task_dirs)) This occurs because tophat2 tool uses from_work_dir feature to get the output files. But on the line: output_files = [ f for f in output_files if os.path.exists(f) ] It returnes an empty list because no real output files exists. Did anyone else have tried this before? Hagai

2 1