December 2012 - galaxy-dev - lists.galaxyproject.org

Nominate topics for the 2013 Galaxy Community Conference (GCC2013) Training Day
by Dave Clements 07 Jan '13

07 Jan '13

Hello all, Nominations are now open <http://bit.ly/gcc2013nom> for Training Day topics<http://wiki.galaxyproject.org/Events/GCC2013/TrainingDay>at the 2013 Galaxy Community Conference (GCC2013)<http://wiki.galaxyproject.org/Events/GCC2013>. GCC2013 <http://wiki.galaxyproject.org/Events/GCC2013> will start on 30 June with a Training Day featuring 4 parallel tracks, each with three, two hour workshops, for a total of twelve sessions. Any topic of interest to the Galaxy Community can be nominated<http://bit.ly/gcc2013nom>(see what was offered at GCC2012<http://wiki.galaxyproject.org/Events/GCC2012/TrainingDay>). Topics can be repeats from last year or brand new, and you are encouraged to nominate as many topics as you like. The 2013 Galaxy Community Conference (GCC2013) <http://wiki.galaxyproject.org/Events/GCC2013> will start on 30 June with a Training Day featuring 4 parallel tracks, each with three, two hour workshops, for a total of twelve sessions. Nominations will be compiled into a uniform list, and then topics will be voted on by the Galaxy Community starting in late January. Topic nomination closes 11 January. Thanks, Dave Clements, on behalf of the GCC2013 Organizing Committee<http://wiki.galaxyproject.org/Events/GCC2013/Organizers#Organizing_Committee> Links: http://bit.ly/gcc2013nom - nominate a topic http://bit.ly/gcc2013td - Training Day page http://galaxyproject.org/GCC2013 - Conference home page http://bit.ly/gcc2013prom - help get the word out -- <http://galaxyproject.org/wiki/GCC2012>http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

1 1

Uploading large files to history through API (some random thoughts)
by John Chilton 07 Jan '13

07 Jan '13

I went down something of a rabbit hole last night, I thought that uploading files through the API using multipart/form-data worked. It is only at the very end of my adventure that I realized it only works for libraries uploads. I think it would be great to get that working to histories as well. I cannot spend much more time on this right now (I thought I only needed to spend an hour to implement the client side stuff), but I thought I would post the progress I made here in case someone wants to take up the fight someday: The first thing is that when issuing multipart/form-data requests, inputs are not currently being deserialized from JSON (so for instance in the tools api code the "inputs" variable would be the string containing the json, not the python dictionary). To bring multipart/form-data requests in line with other content requests I modified this in ilb/galaxy/web/framework/__init__.py payload = kwargs.copy() named_args, _, _, _ = inspect.getargspec(func) for arg in named_args: payload.pop(arg, None) with this: payload = kwargs.copy() named_args, _, _, _ = inspect.getargspec(func) for arg in named_args: payload.pop(arg, None) for k, v in payload.iteritems(): if isinstance(v, (str, unicode)): try: payload[k] = simplejson.loads(v) except: # may not actually be json, just continue pass payload = util.recursively_stringify_dictionary_keys( payload ) One could also imagine doing this replacement in the tools api controller directly on payload['input'] instead. After that change, files still weren't being matched up with inputs properly: This debug statement I added to upload_common.py, demonstrates that file_data is None. galaxy.tools.actions.upload_common INFO 2012-11-27 00:22:17,585 Uploaded datasets is {'NAME': u'galxtest694734465387762969.txt', 'file_data': None, 'space_to_tab': None, 'url_paste': None, '__index__': 0, 'ftp_files': None} To address this, the files_*|file_data parameters need to be moved inside of the inputs dict. That means in lib/galaxy/webapps/galaxy/api/tools.py, changing this: inputs = payload[ 'inputs' ] To this: inputs = payload[ 'inputs' ] for k, v in payload.iteritems(): if k.startswith("files_"): inputs[k] = v Then the debug line becomes this: galaxy.tools.actions.upload_common INFO 2012-11-27 00:33:15,168 Uploaded datasets is {'NAME': u'galxtest2484272839208214846.txt', 'file_data': FieldStorage('files_0|file_data', u'galxtest2484272839208214846.txt'), 'space_to_tab': None, 'url_paste': None, '__index__': 0, 'ftp_files': None} Which matches pretty well with the same line coming from a web browser request: galaxy.tools.actions.upload_common INFO 2012-11-27 00:23:57,590 Uploaded datasets is {'NAME': u'', 'file_data': FieldStorage('files_0|file_data', u'second_step.png'), 'space_to_tab': None, 'url_paste': u'', '__index__': 0, 'ftp_files': None} These changes aside I still wasn't it sill didn't work, that is why this is a rambling e-mail and not a pull request. I got this exception http://pastebin.com/hhs1pjtP. I guess something to do with the session handling stuff being different between API calls and normal web calls. Anyway, I've inspected the actual requests I was generating and I think they are reasonable, Galaxy just needs to be augmented to accept them :). If someone does land up taking a look at this, I have committed my test case to blend4j so it can be used to really quickly test such a client request (requires git, Java, and maven): % git checkout git@github.com:jmchilton/blend4j.git % cd blend4j % mvn test -Dtest=ToolsTest -Dtest.galaxy.instance=http://localhost:8080/ -Dtest.galaxy.key=<testapikey> Thanks, -John

2 1

hg pull aborted
by Sarah Maman 04 Jan '13

04 Jan '13

Hello, When updating local instance of our Galaxy, the hg aborted: pulling from https://bitbucket.org/galaxy/galaxy-dist/ searching for changes adding changesets adding manifests adding file changes added 1482 changesets with 3989 changes to 1340 files abort: untracked file in working directory differs from file in requested revision: 'doc/Makefile' and : -bash-3.2$ hg update abort: untracked file in working directory differs from file in requested revision: 'doc/source/conf.py' Could you please tell me what this may be the problem? Thanks

2 1

Re: [galaxy-dev] Where is the output of tools stored in Galaxy?
by Hans-Rudolf Hotz 03 Jan '13

03 Jan '13

Please keep all replies on the list by using "reply all"! You can write a wrapper around tophat2 and redirect the output to any directory you want - as long as galaxy has write access to the directory. However, the output will no longer be accessible as a history item. Regards, Hans On 12/27/2012 11:19 AM, Sachit Adhikari wrote: > Hi Hans, > > Oh yes it is stored in database/files. What I wanted to know was if I > can change the output path of one or two tools. Let's say if I want to > change the path of tophat2, what should I do? > > Thanks, > > Sachit > > On Thu, Dec 27, 2012 at 9:38 AM, Hans-Rudolf Hotz <hrh(a)fmi.ch > <mailto:hrh@fmi.ch>> wrote: > > Hi Sachit > > > They are stored in ~/database/files/ > > and you can change it by changing the value for "file_path" in the > 'universe_wsgi.ini' file. > > Regardd, Hans > > > > > On 12/26/2012 11:40 AM, Sachit Adhikari wrote: > > Hello Everyone, > > *Merry Christmas and Happy New Year* > * > > * > I was wondering where are the output of the tools stored in the > machine > where the galaxy is hosted? And, can I change the path of the > output of > the tools? > > Thanks, > > Sachit > > > > > _____________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > >

4 3

MGTAXA taxonomic prediction does not work...
by Edlund, Anna 02 Jan '13

02 Jan '13

Hello. My name is Anna Edlund and I have loaded several files containing contigs of assembled metgenomic data to generate taxonomic predictions. However, no one of the jobs I submitted yesterday afternoon are finished. Do you know what the problem is? How can I make MGTAXA work faster? Any help with this is very appreciated. Thank you! Anna

2 2

Tool error occurred when adding my tool to galaxy
by Tan Jun 31 Dec '12

31 Dec '12

Hi, all I'm new here. These days I'm trying to add a tool in Galaxy, as showed in screencast. I did the same thing with that in screencast. But an error occurred. The error says: tool error An error occurred running this job:failure preparing job The message in the terminal says: galaxy.jobs.handler DEBUG 2012-12-29 15:54:25,504 dispatching job 7 to local runner galaxy.jobs.handler INFO 2012-12-29 15:54:25,733 (7) Job dispatched galaxy.jobs.runners.local DEBUG 2012-12-29 15:54:25,818 Local runner: starting job 7 galaxy.jobs.runners.local ERROR 2012-12-29 15:54:26,061 failure running job 7 Traceback (most recent call last): File "/home/tanjun/galaxy/galaxy-galaxy-dist-36ad8aa7b922/lib/galaxy/jobs/runners/local.py", line 72, in run_job job_wrapper.prepare() File "/home/tanjun/galaxy/galaxy-galaxy-dist-36ad8aa7b922/lib/galaxy/jobs/__init__.py", line 180, in prepare config_filenames = self.tool.build_config_files( param_dict, self.working_directory ) File "/home/tanjun/galaxy/galaxy-galaxy-dist-36ad8aa7b922/lib/galaxy/tools/__init__.py", line 2430, in build_config_files f.write( fill_template( template_text, context=param_dict ) ) File "/home/tanjun/galaxy/galaxy-galaxy-dist-36ad8aa7b922/lib/galaxy/util/template.py", line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File "/home/tanjun/galaxy/galaxy-galaxy-dist-36ad8aa7b922/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ return getattr(self, mainMethName)() File "cheetah_DynamicallyCompiledCheetahTemplate_1356767666_06_36079.py", line 149, in respond KeyError: 'thype' 127.0.0.1 - - [29/Dec/2012:15:54:27 +0800] "GET /api/histories/f597429621d6eb2b HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/11.10 Chromium/18.0.1025.168 Chrome/18.0.1025.168 Safari/535.19" 127.0.0.1 - - [29/Dec/2012:15:54:27 +0800] "GET /api/histories/f597429621d6eb2b/contents?ids=ebfb8f50c6abde6d HTTP/1.1" 200 - "http://localhost:8080/history" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/11.10 Chromium/18.0.1025.168 Chrome/18.0.1025.168 Safari/535.19" Could anybody help me? Thanks! =================================================== Regards! School of Computer Engineering and Science Shanghai University, Shanghai China Tan Jun 谭军

3 3

MySQL Returning Empty Set Result in Galaxy
by Sachit Adhikari 30 Dec '12

30 Dec '12

Hello all, *Happy New Year* * * I have configured MySQL in Galaxy. I have edited universe_wsgi.ini file: *database_connection = mysql://username:password@ /database_name?unix_socket=/var/run/mysqld/mysqld.sock * * * I restarted the galaxy and all the galaxy_tables has been created but all of those table returns empty sets. There are no data's on the table. I then created a new workflow and again saw the database. The workflow table was empty. The tables has been created but all the tables are empty. Am I missing something? Regards, Sachit

1 1

Re: [galaxy-dev] expand disk size limit
by Dannon Baker 28 Dec '12

28 Dec '12

The maximum size of any EBS volume is 1TB (this is currently an Amazon limit, not Galaxy or Cloudman). We're hopeful that they'll eventually raise this, but for now we're planning on using S3 as a way around this for Galaxy. If the resize UI didn't indicate that your other entries were invalid, then that's a bug and I'll look into it. Lastly, please remember to email/cc the list so that others can benefit from questions like this. -Dannon On Dec 28, 2012, at 12:32 PM, "Sun, Wenping [USA]" <Sun_Wenping(a)bah.com> wrote: > Hi Donnon, > > I am working to try to expand the disk size on cloudman. I used the upper limit 1tb and there was no response from cloudman interface. However, when I changed to 800gb, it worked. Do you have any suggestion if I need the disk size more than 1tb? I may think to have at least 10tb as the start point if possible. > > Thank you in advance, > Kathryn

1 0

Re: [galaxy-dev] Where is the workflow stored in Galaxy?
by Dannon Baker 28 Dec '12

28 Dec '12

Yes, there is a relationship - in the database there's also a WorkflowStepConnection table that relates inputs and outputs (WorkflowStep + i/o name) between steps. On Dec 28, 2012, at 8:33 AM, Sachit Adhikari <sachit.technerves(a)gmail.com> wrote: > Hi Dannon, > > Thank you for your answer. What I meant by the last question is that: > > When I create a workflow (tophat-bowtie-bowtie2). The input is first processed by tophat and the output of tophat is processed by bowtie and then the output of bowtie and processed by bowtie2 and that's the final output. There's some sort of relationship, right? Is this relationship stored in database? > > On Fri, Dec 28, 2012 at 7:34 AM, Dannon Baker <dannonbaker(a)me.com> wrote: > A workflow is just the definition of a particular process or set of steps. This information is stored in the database (see Workflow, StoredWorkflow, WorkflowStep tables). Running that workflow creates datasets in a history (datasets which are stored in database/files). > > And, I'm not sure exactly what you're asking in your last question, can you clarify? > > -Dannon > > > On Dec 28, 2012, at 2:16 AM, Sachit Adhikari <sachit.technerves(a)gmail.com> wrote: > > > I have searched the mailing list and found that the output, histories and workflows are stored in ~/database/files. To do a little experiment I deleted all the inside that directory. I then open my Galaxy and browse the workflow. But the workflow is still there. I was wondering where is the workflow stored in Galaxy? Also, if possible can I know how is the relationship specified by user while creating workflow is stored? > > > > Happy New year > > > > Regards, > > > > Sachit > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > >

2 1

Where is the workflow stored in Galaxy?
by Sachit Adhikari 28 Dec '12

28 Dec '12

I have searched the mailing list and found that the output, histories and workflows are stored in ~/database/files. To do a little experiment I deleted all the inside that directory. I then open my Galaxy and browse the workflow. But the workflow is still there. I was wondering where is the workflow stored in Galaxy? Also, if possible can I know how is the relationship specified by user while creating workflow is stored? *Happy New year* Regards, Sachit

2 1