- galaxy-dev - lists.galaxyproject.org

Galaxy instance file upload problem
by Bongsoo Park 20 Oct '14

20 Oct '14

Hi folks, I encountered the problem in file upload. The error message is like below 'Failed: Not found (404)'. Attached is the screen shot for this error message. Any idea? Thanks! Best, Bongsoo

4 4

question about GALAXY_SLOTS
by Wolfgang Maier 17 Oct '14

17 Oct '14

Hi, this is just to make sure: the GALAXY_SLOTS environmental variable set by Galaxy when running tools will always be a number >= 1 with 1 being the default if nothing else is configured in the job runner settings ? Correct ? Thanks, Wolfgang

5 6

Using LEfSe for paired/matched data analysis
by Greathouse, Leigh (NIH/NCI) [E] 17 Oct '14

17 Oct '14

Hi, I have heard Curtis mention that LEfSe is not capable of doing paired Wilcoxan rank sum tests on matched samples. Is there a way to do this by command line in the LEfSe program? Thanks, Leigh Research Fellow Laboratory of Human Carcinogenesis National Cancer Institute 37 Convent Dr., Rm 3068A, MSC 4258 Bethesda, MD. 20892-4258 Phone: 301-496-5835 Fax: 301-496-0497 Email: greathousekl(a)mail.nih.gov

1 0

galaxy.eggs.EggNotFetchable: ['pexpect']
by Shrum, Donald C 17 Oct '14

17 Oct '14

I'm getting the error below... the dataset appears fine. The egg appears to be in the right place... [root@admin galaxy-dist]# ls -l ./eggs | grep pexpect -rw-rw-r-- 1 galaxy galaxy 78539 Jul 30 08:11 pexpect-2.4-py2.6.egg [root@admin galaxy-dist]# python --version Python 2.6.6 Traceback (most recent call last): File "./scripts/set_metadata.py", line 31, in <module> import galaxy.model.mapping # need to load this before we unpickle, in order to setup properties assigned by the mappers File "/panfs/storage.local/opt/galaxy-dist/lib/galaxy/model/__init__.py", line 9, in <module> eggs.require("pexpect") File "/panfs/storage.local/opt/galaxy-dist/lib/galaxy/eggs/__init__.py", line 414, in require raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) ) galaxy.eggs.EggNotFetchable: ['pexpect'] I found a mention of this problem in a few places but no solution other than to use python 2.6.6.

1 0

Re: [galaxy-dev] galaxy-dev Digest, Vol 100, Issue 15
by Iry Witham 17 Oct '14

17 Oct '14

So Ihave finally determined the cause and found a resolution for the snpEff/java problem. The issue was the fact that the vrsion of Java on the Ubuntu linux VM from AWS is 1.6.x snpEff 4.0e requires version 1.7.0 or higher. This is fine, but you willneed to manually upgrade the java version. To do so you will need to complete the following: SSH to the VM sdo su - root CD to /mnt/galaxy/ Complete the following steps: 1. sudo apt-get purge openjdk* (This removes java 1.6 completely) 2. Modify this file: vi /etc/apt/sources.list.d/cloudbiolinu.list (Remove the leading section of line 11) 3. sudo add-apt-repository ppa:webupd8team/java 4. sudo apt-get update 5. sudo apt-get installoracle-java7-installer 6. java -version Now if you plan to add nodes via the Cloudman Console you will need to perform these tasks for each node you install. I worked with AWS Support to setup an "Auto Scaling Group" to accommodate this process. This required getting my Master instance upgraded and creating an AMI from it. >From that point you can build the group based on the following: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/creating-your- auto-scaling-groups.html This was, in theory, a great idea. However, it did not work for me. Each of the node that were generated by this tool had Java version 1.6.x and this caused snpEff t fail. My recommendation is that if you have time you should paly around with this more, but I did not have that luxury for this project. Iry On 10/16/14 12:00 PM, "galaxy-dev-request(a)lists.bx.psu.edu" <galaxy-dev-request(a)lists.bx.psu.edu> wrote: >Send galaxy-dev mailing list submissions to > galaxy-dev(a)lists.bx.psu.edu > >To subscribe or unsubscribe via the Worl Wide Web, visit > http://lists.bx.psu.edu/listinfo/galaxy-dev >or, via email, send a message with subject or body 'help' to > galaxy-ev-request(a)lists.bx.psu.edu > >You can reach the person managing the list at > galaxy-dev-owner(a)lists.bx.psu.edu > >When replying, please edit your Subjct line so it is more specific >than "Re: Contents of galaxy-dev digest..." > > >HEY! This is important! If you reply to a thread in a igest, please >1. Change the subject of your response from "Galaxy-dev Digest Vol ..." >to the original subject for the thread. >2. Strip out everything else in the dgest that is not part of the thread >you are responding to. > >Why? >1. This will keep the subject meaningful. People will have some idea>from the subject line if they should read it or not. >2. Not doing this greatly increases the number of emails that match >search ueries, but that aren't actually informative. > >Today's Topics: > > 1. Re: Set output dbkey from parameter value (Daniel Blankenberg)> 2. Re: Set output dbkey from parameter value (Nikos Sidiropoulos) > 3. Re: snpeff tool for Galay extra_files_path (John Chilton) > 4. Re: snpeff tool for Galaxy extra_files_path (Bj?rn Gr?nng) > 5. Re: snpeff tool for Galaxy extra_files_path (John Chilton) > 6. Re: snpeff tool for Galaxy extra_files_path (Jim Johnson) > 7. snpEff and java issue(Iry Witham) > 8. Re: HOWTO share tool parameter settings? (Lukasse, Pieter) > 9.Help with Galaxy server migration (Sarah Diehl) > 10. Re: Help with Galaxy server migrtion (John Chilton) > 11. Login issue with a nginx proxy (Alexandre Loywick) > 12. Re: Help with Galaxy server migration (Sarah Diehl) > > >--------------------------------------------------------------------- > >Message: 1 >Date: Wed, 15 Oct 2014 12:14:28 -0400 >From: Daniel Blankenberg <dan(a)bx.psu.edu> >To: Nikos Sidiropoulos <nikos.sidiro(a)gmail.com> >Cc: "<galaxy-dev(a)bx.psu.edu>" <galaxy-dev(a)bx.psu.edu> >Subject: Re: [galaxy-dev] Set output dbkey from parameter value >Message-ID: <E77C8A92-617A-455E-B5EB-209F4505BECA(a)bx.psu.edu> >Content-Type: text/plain; charset=windows-1252 > >Does removing the ?param_attribute=?value"' attribute help? > > >On Oct 15, 2014, at 11:23 AM, Nikos Sidiropoulos <nikos.sidiro(a)gmail.com> >wrote: > >> Hi Daniel, >> >> Thanks for the response. >> >> I've edited the output to: >> >> <data format="bedgraph" name="bedgraph_slograt" >> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >> from_work_dir="output_dir/slograt.bedgraph"> >> <filter> bedgraph['check'] == 'yes' and slograt['check'] >> == 'yes' </filter> >> <actions> >> <conditional name="bedgraph.check"> >> <when value="yes"> >> <action type="metadata" name="dbkey"> >> <option type="from_param" >> name="bedgraph.genome" param_attribute="value" /> >> </action> >> </when> >> </conditional> >> </actions> >> </data> >> >> Now I'm getting a tool execution error. >> >> Error executing tool: 'unicode' object has no attribute 'value' >> >> I've tried to change the param_attribute to "ext", "dbkey" (ones that >> I know that exist) and got a similar error. >> >> Bests, >> Nikos >> >> On 15 October 2014 16:58, Daniel Blankenberg <dan(a)bx.psu.edu> wrote: >>> Hi Nikos, >>> >>> In the very least, you?ll want to make sure that you have a bounding >>><actions></actions> tag set around your actions. It is probably also >>>advisable to add a set of conditional/whens around the action, since >>>you?re only setting the dbkey under certain circumstances. >>> >>> >>> Thanks for using Galaxy, >>> >>> Dan >>> >>> >>> On Oct 15, 2014, at 6:37 AM, Nikos Sidiropoulos >>><nikos.sidiro(a)gmail.com> wote: >>> >>>> Hi all, >>>> >>>> I'm trying to set the dbkey of an output file from the value (text) of >>>> a parameter. >>>> >>>> The parameter I want to use is genome. >>>> >>>> <conditional name="bedgraph"> >>>> <param name="check" type="select" label="Produce BedGraph >>>> output" help="Can be displayed directly on UCSC browser. One file per >>>> normalisation method." > >>>> <option value="no" selected="True">No</option> >>>> <option value="yes">Yes</option> >>>> </param> >>>> <when value="yes"> >>>> <param name="bed_file" type="data" format="bed" >>>> label="Transcripts ins BED format" help="12 column BED file containing >>>> trancript definitions." /> >>>> <param name="genome" type="text" label="Genome Build" >>>> help="E.g. hg19" /> >>>> <param name="track_name" type="text" label="Track >>>> Name" size="20" value="Track Name" /> >>>> </when> >>>> <when value="no" /> >>>> </conditional> >>>> >>>> and this is how I've set the output: >>>> >>>> <data format="bedgraph" name="bedgraph_slograt" >>>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>>> from_work_dir="output_dir/slograt.bedgraph"> >>>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>>> == 'yes' </filter> >>>> <action type="metadata" name="dbkey"> >>>> <option type="from_param" name="bedgraph.genome" >>>> param_attribute="value" /> >>>> </action> >>>> </data> >>>> >>>> >>>> When I run the tool the dbkey isn't set to the output file. Does >>>> anyone know a workaround? >>>> >>>> Bests, >>>> Nikos >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> http://lists.bx.psu.edu/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>> >> > > > > >------------------------------ > >Message: 2 >Date: Wed, 15 Oct 2014 18:43:17 +0200 >From: Nikos Sidiropoulos <nikos.sidiro(a)gmail.com> >To: Daniel Blankenberg <dan(a)bx.psu.edu> >Cc: "<galaxy-dev(a)bx.psu.edu>" <galaxy-dev(a)bx.psu.edu> >Subject: Re: [galaxy-dev] Set output dbkey from parameter value >Message-ID: > <CAMjao2O1iw6ZHBrmuJ3VMH8kYe+7dVp2wk7sdOMb32OodFNgiw(a)mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >Yes! > >Thank you very much Daniel. > >On 15 October 2014 18:14, Daniel Blankenberg <dan(a)bx.psu.edu> wrote: >> Does removing the ?param_attribute=?value"' attribute help? >> >> >> On Oct 15, 2014, at 11:23 AM, Nikos Sidiropoulos >><nikos.sidiro(a)gmail.com> wrote: >> >>> Hi Daniel, >>> >>> Thanks for the response. >>> >>> I've edited the output to: >>> >>> <data format="bedgraph" name="bedgraph_slograt" >>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>> from_work_dir="output_dir/slograt.bedgraph"> >>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>> == 'yes' </filter> >>> <actions> >>> <conditional name="bedgraph.check"> >>> <when value="yes"> >>> <action type="metadata" name="dbkey"> >>> <option type="from_param" >>> name="bedgraph.genome" param_attribute="value" /> >>> </action> >>> </when> >>> </conditional> >>> </actions> >>> </data> >>> >>> Now I'm getting a tool execution error. >>> >>> Error executing tool: 'unicode' object has no attribute 'value' >>> >>> I've tried to change the param_attribute to "ext", "dbkey" (ones that >>> I know that exist) and got a similar error. >>> >>> Bests, >>> Nikos >>> >>> On 15 October 2014 16:58, Daniel Blankenberg <dan(a)bx.psu.edu> wrote: >>>> Hi Nikos, >>>> >>>> In the very least, you?ll want to make sure that you have a bounding >>>><actions></actions> tag set around your actions. It is probably also >>>>advisable to add a set of conditional/whens around the action, since >>>>you?re only setting the dbkey under certain circumstances. >>>> >>>> >>>> Thanks for using Galaxy, >>>> >>>> Dan >>>> >>>> >>>> On Oct 15, 2014, at 6:37 AM, Nikos Sidiropoulos >>>><nikos.sidiro(a)gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm tryingto set the dbkey of an output file from the value (text) >>>>>of >>>>>a parameter. >>>>> >>>>> The parameter I want to use is genome. >>>>> >>>> <conditional name="bedgraph"> >>>>> <param name="ceck" type="select" label="Produce BedGraph >>>>> output" help="Can be displayed directly on UCSC browser. One file per >>>>> normalisation method." > >>>>> <option value="no" selected="True">No</option> >>>>> <option value="yes">Yes</option> >>>>> </param> >>>>> <when value="yes"> >>>>> <param name="bed_file" type="data" format="bed" >>>>> label="Transcripts ins BED format" help="12 column BED file >>>>>containing >>>>> trancript definitions." /> >>>>> <param name="genome" type="text" label="Genome Build" >>>>> help="E.g. hg19" /> >>>>> <param name="track_name" type="text" label="Track >>>>> Name" size="20" value="Track Name" /> >>>>> </when> >>>>> <when value="no" /> >>>>> </conditional> >>>>> >>>>> and this is how I've set the output: >>>>> >>>>> <data format="bedgraph" name="bedgraph_slograt" >>>>> label="${tool.name} on ${on_string}: Smoot Log2ratio (bedGraph)" >>>>> from_work_dir="output_dir/slograt.bedgraph"> >>>>> <filter> bedgraph['check'] == 'yes' and slograt['check'] >>>>> == 'yes' </filter> >>>>> <action type="metadata" name="dbkey"> >>>>> <option type="from_param" name="bedgraph.genome" >>>>> param_attribute="value" /> >>>>> </action> >>>>> </data> >>>>> >>>>> >>>>> When I run the tool the dbkey isn't set to the output file. Does >>>>> anyone know a workaround? >>>>> >>>>> Bests, >>>>> Nikos >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>> >> > > > >------------------------------ > >Message: 3 >Date: Wed, 15 Oct 2014 13:05:55 -0400 >From: John Chilton <jmchilton(a)gmail.com> >To: Jim Johnson <jj(a)umn.edu> >Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu>, Nuwan > Goonasekera <nuwan.goonasekera(a)unimelb.edu.au>, Andrew Lonie > <alonie(a)unimelb.edu.au> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: > <CANwbokevqaN=G_cDTet7E-s25cRZ2No1At-xaiAJmi_t_tSUTg(a)mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >Hey JJ, > >Opened a pull request to stable with my best guess at the right to >proceed and hopefully a best practice recommendation we can all get >behind. Do you want to try it out and let me know if it fixes snpeff? >(It does fix the velvet datatypes you contributed to Galaxy.) > >https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-datat >ypes-consuming-output-extra/diff > >Dan Bjoern - does this make sense - can we move forward with this >approach ($input.extra_files_path or inputs and $output.files_path >for outputs) as the best practices forhow to reference these >directories. > >-John > >On Wed, Oct 15, 2014 at 1144 AM, Jim Johnson <johns198(a)umn.edu> wrote: >> I agree with you about theinadvisable use of: input.dataset.*. >> >> I'm looking at: >> >> lib/galay/model/__init__.py >> class Dataset( object ): >> ... >> def __iit__( self, id=None, state=None, external_filename=None, >> extra_files_path=None, file_size=None, purgable=True, uid=None ): >> ... >> self._extra_files_path = extra_fils_path >> ... >> @property >> def extra_files_path( self ):>> return self.object_store.get_filename( self, dir_only=True, >> extra_dir=self._extrafiles_path or "dataset_%d_files" % self.id ) >> >> I'm trying to see whenself._extra_files_path gets set. Otherwise, would >> this return the path rlative to the current file location of dataset? >> >> >> >> >> On 0/15/14, 9:36 AM, John Chilton wrote: >>> >>> Okay - so this is what boke things: >>> >>> >>> >>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201b >>>73a4dd99b56282169d86 >> >>> My feeling with the commit was that wrappers and tools should nevr be >>> explicitly accessing paths explicitly through input.dataset.*. I tink >>> this would circumvent options like outputs_to_working_directory and>>> break remote job execution through Pulsar. It also breaks the object >>> store abstraction I think - which is why I made the change for Bjoer >>> I guess. >>> >>> I did not (and this was stupid on my part) realize hat datatype code >>> would be running on the remote host and accessng these model >>> properties directly outside the abstractions setup by the wrappers >>> supplied to cheetah code and s they have become out of sync as of >>> that commit. >> >>> I am thinking somehow changing what the datatype code gets is the >>>right approach and not fixing things by circumvent the wrapper and >>> accessing properties directly on the datset. Since you will find that >>> doing this breaks things for Bjoern object store and could probably >>> never run on usegalaxy.org say for the same reason. >>> >>> Too many different competng deployment options all being incompatible >>> with each other :(. >>> >>> Will keep thinking about this and respond again. >>> >>> -John >> >>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchilton(a)gmail.com> >>>wrote: >>>> >>>> JJ, >>>> >>>> Arg this is a mess. I am very sorry about this - Istill don't >>>> understand extra_files_path versus files_path myself. There are open >>>> questions on Peter's blast repo and no one ever followed up on my >>>> object store questions about this with Bjoern's issues a couple >>>> release cycles ago. We need to get these to work - write documetation >>>> explicitly declaring best practices we can all agree on and then write >>>> some tests to ensure things don't break in the future. >>>> >>>> When you say your tools broke recently - can you say fo certain which >>>> release broke these - the August14, October14, something older? >>>> >>>> I'll try to do some more research and get back to you. >>>> >>>> -John >>>> >>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns198(a)umn.edu> wrote: >>>>> >>>>> Andrew, >>>>> >>>>> Thanks for investigating this. I changed the subject and sent to the >>>>> galaxy >>>>> dev lst. >>>>> >>>>> I've had a number of tools quit working recently. Particularly >>>>>tools >>>>> that >>>>> inspect the extra_files_pth when setting metadata, Defuse, Rsem, >>>>> SnpEff. >>>>> >>>>> I think there was a change in the galaxy framework: >>>>> The extra_files_path when reerenced from an input or output in the >>>>> cheetah >>>>> template sections of the tool config xml will be relative to the job >>>>> woring >>>>> directly rather than the files location. >>>>> I've just changed a few of my tools on my server yesterday >>>>> from: <param_name>.extra_files_path >>>>> o: <param_name>.dataset.extra_files_path >>>>> and they now work again. >>>>> >>>>> Dan or John, is that the right way to handle this? >>>> Thanks, >>>>> >>>>> JJ >>>>> >>>>> >>>>> >>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>> >>>>>> Hi Jim. I am probably oing about this the wrong way, but I am not >>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>error!) >>>>>> >>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>> getting a consistent eror with the SnpEff Download and SnpEff sub >>>>>> tools (the SnpSift dbNSFP works fine). The prolem seems to be with >>>>>>an >>>>>> attribute declaration and manifests during database download as: >>>>>> >>>>>> Traceback (most recent call last): >>>>>> Fle >>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>> line 54, in finish_job >>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code )>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", line >>>>>> 1107, in finish >>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # call >>>>> datatype.set_meta directly for the initial set_meta call during >>>>>> dataset creation >>>>>> File >>>>>> >>>>>> >>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/1 >>>>>>938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>> line 21, in set_meta >>>>>> data_dir = dataset.files_path >>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>> 'files_path' >>>>>> >>>>>> >>>>>> We fiddled around with the wrapper, eventually replacing >>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>> which fixed the download bug, but then SnpEff subtool itself threw a >>>>>> similar error when I tried to use that database from the history. >>>>>> >>>>>> I chased up a bit more but cannot understand the various posts on >>>>>> files_path vs extra_files_path >>>>>> >>>>>> I've shared a history with both of these errors here: >>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>> >>>>>> Maybe this is a problem with our Galaxy image? >>>>>> >>>>>> Any help appreciated! >>>>>> >>>>>> Andrew >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> A/Prof Andrew Lonie >>>>>> University of Melbourne >>>>> >>>>> >>>>> >>>>> -- >>>>> James E. Jhnson Minnesota Supercomputing Institute University of >>>>> Minnesota >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >> >> >> >> -- >> James E. Johnson Minnesota Supercomputing Institute University of >>Minnesota > > >------------------------------ > >Message: 4 >Date: Wed, 15 Oct 2014 19:47:40 +0200 >From: Bj?rn Gr?ning <bjoern.gruening(a)gmail.com> >To: galaxy-dev(a)lists.bx.psu.edu, John Chilton <jmchilton(a)gmail.com> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: <543EB33C.3070607(a)gmail.com> >Content-Type: text/plain; charset=windows-1252 > >Hi John, > >glad to see this gets some attention! > >Am 15.10.2014 um 19:05 schrieb John Chilton: >> Hey JJ, >> >> Opened a pull request to stable with my best guess at the right to >> proceed and hopefully a best practice recommendation we can all get >> behind. Do you want to try it out and let me know if it fixes snpeff? >> (It does fix the velvet datatypes you contributed to Galaxy.) >> >> >>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-data >>types-consuming-output-extra/diff >> >> Dan, Bjoern - does this make sense - can we move forward with this >> approach ($input.extra_files_path for inputs and $output.files_path >> for outputs) as the best practices for how to reference these >> directories. > >I'm not sure why we need this distinction? Can we not simply choose one >for both, inputs and outputs? >Otherwise we need to explain it very well, why this is needed and I >would vote to rename it to reflect that files_path can be only used by >$outputs ... > >Salve, >Bjoern > >> -John >> >> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns198(a)umn.edu> wrote: >>> I agree with you about the inadvisable use of: input.dataset.*. >>> >>> I'm looking at: >>> >>> lib/galaxy/model/__init__.py >>> class Dataset( object ): >>> ... >>> def __init__( self, id=None, state=None, external_filename=None, >>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>> ... >>> self._extra_files_path = extra_files_path >>> ... >>> @property >>> def extra_files_path( self ): >>> return self.object_store.get_filename( self, dir_only=True, >>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>> >>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>would >>> this return the path relative to the current file location of dataset? >>> >>> >>> >>> >>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>> >>>> Okay - so this is what broke things: >>>> >>>> >>>> >>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201 >>>>b73a4dd99b56282169d86 >>>> >>>> My feeling with the commit was that wrappers and tools should never be >>>> explicitly accessing paths explicitly through input.dataset.*. I think >>>> this would circumvent options like outputs_to_working_directory and >>>> break remote job execution through Pulsar. It also breaks the object >>>> store abstraction I think - which is why I made the change for Bjoern >>>> I guess. >>>> >>>> I did not (and this was stupid on my part) realize that datatype code >>>> would be running on the remote host and accessing these model >>>> properties directly outside the abstractions setup by the wrappers >>>> supplied to cheetah code and so they have become out of sync as of >>>> that commit. >>>> >>>> I am thinking somehow changing what the datatype code gets is the >>>> right approach and not fixing things by circumvent the wrapper and >>>> accessing properties directly on the dataset. Since you will find that >>>> doing this breaks things for Bjoern object store and could probably >>>> never run on usegalaxy.org say for the same reason. >>>> >>>> Too many different competing deployment options all being incompatible >>>> with each other :(. >>>> >>>> Will keep thinking about this and respond again. >>>> >>>> -John >>>> >>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchilton(a)gmail.com> >>>>wrote: >>>>> >>>>> JJ, >>>>> >>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>> understand extra_files_path versus files_path myself. There are open >>>>> questions on Peter's blast repo and no one ever followed up on my >>>>> object store questions about this with Bjoern's issues a couple >>>>> release cycles ago. We need to get these to work - write documetation >>>>> explicitly declaring best practices we can all agree on and then >>>>>write >>>>> some tests to ensure things don't break in the future. >>>>> >>>>> When you say your tools broke recently - can you say for certain >>>>>which >>>>> release broke these - the August14, October14, something older? >>>>> >>>>> I'll try to do some more research and get back to you. >>>>> >>>>> -John >>>>> >>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns198(a)umn.edu> >>>>>wrote: >>>>>> >>>>>> Andrew, >>>>>> >>>>>> Thanks for investigating this. I changed the subject and sent to the >>>>>> galaxy >>>>>> dev list. >>>>>> >>>>>> I've had a number of tools quit working recently. Particularly >>>>>>tools >>>>>> that >>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>> SnpEff. >>>>>> >>>>>> I think there was a change in the galaxy framework: >>>>>> The extra_files_path when referenced from an input or output in the >>>>>> cheetah >>>>>> template sections of the tool config xml will be relative to the job >>>>>> working >>>>>> directly rather than the files location. >>>>>> I've just changed a few of my tools on my server yesterday >>>>>> from: <param_name>.extra_files_path >>>>>> to: <param_name>.dataset.extra_files_path >>>>>> and they now work again. >>>>>> >>>>>> Dan or John, is that the right way to handle this? >>>>>> Thanks, >>>>>> >>>>>> JJ >>>>>> >>>>>> >>>>>> >>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>> >>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>error!) >>>>>>> >>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>with an >>>>>>> attribute declaration and manifests during database download as: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>> line 564, in finish_job >>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", line >>>>>>> 1107, in finish >>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # call >>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>> dataset creation >>>>>>> File >>>>>>> >>>>>>> >>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/ >>>>>>>1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>> line 21, in set_meta >>>>>>> data_dir = dataset.files_path >>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>>> 'files_path' >>>>>>> >>>>>>> >>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>> which fixed the download bug, but then SnpEff subtool itself threw >>>>>>>a >>>>>>> similar error when I tried to use that database from the history. >>>>>>> >>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>> files_path vs extra_files_path >>>>>>> >>>>>>> I've shared a history with both of these errors here: >>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>> >>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>> >>>>>>> Any help appreciated! >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> A/Prof Andrew Lonie >>>>>>> University of Melbourne >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>> Minnesota >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>> >>> >>> >>> -- >>> James E. Johnson Minnesota Supercomputing Institute University of >>>Minnesota >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > >------------------------------ > >Message: 5 >Date: Wed, 15 Oct 2014 14:06:14 -0400 >From: John Chilton <jmchilton(a)gmail.com> >To: Bj?rn Gr?ning <bjoern.gruening(a)gmail.com> >Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: > <CANwbokfuD83XLiBR9EWsKLAPD+KEcq6SsAVM6yYeDnqJqqmeUQ(a)mail.gmail.com> >Content-Type: text/plain; charset=UTF-8 > >On Wed, Oct 15, 2014 at 1:47 PM, Bj?rn Gr?ning ><bjoern.gruening(a)gmail.com> wrote: >> Hi John, >> >> glad to see this gets some attention! >> >> Am 15.10.2014 um 19:05 schrieb John Chilton: >>> Hey JJ, >>> >>> Opened a pull request to stable with my best guess at the right to >>> proceed and hopefully a best practice recommendation we can all get >>> behind. Do you want to try it out and let me know if it fixes snpeff? >>> (It does fix the velvet datatypes you contributed to Galaxy.) >>> >>> >>>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-dat >>>atypes-consuming-output-extra/diff >>> >>> Dan, Bjoern - does this make sense - can we move forward with this >>> approach ($input.extra_files_path for inputs and $output.files_path >>> for outputs) as the best practices for how to reference these >>> directories. >> >> I'm not sure why we need this distinction? Can we not simply choose one >> for both, inputs and outputs? >> Otherwise we need to explain it very well, why this is needed and I >> would vote to rename it to reflect that files_path can be only used by >> $outputs ... > >I sympathize with you that this adds complexity - I really do. But if >we do anything else we restrict the range of Galaxy versions these >tools can target even further - and we still have to maintain backward >compatibility on all of this junk anyway which is really weighing down >the wrapper and now metadata code as well. > >If you want input.files_path to work - that is fine - I wouldn't be >eager for the change given the complexity it would add to the >implementation but I would probably accept a pull request for that. If >you want $input.input_files_path and $output.output_files_path to work >- I would probably accept pull requests for those as well but I would >not be excited. Finally, I don't personally really want to put the >time in given my reservations and the benefits would not be so great I >don't think because I would think it would be awhile before we could >really recommend those as best practices anyway - given the range of >Galaxy versions people run. > >How about we reach an agreement that with a fictitious Tool 2.0 spec >(https://trello.com/c/AWVobyv1) where we fix all the problems we will >not grant access to $input.dataset directly and we will uniformly only >allow $input.files_path and $output.files_path. > >-John > >> >> Salve, >> Bjoern >> >>> -John >>> >>> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns198(a)umn.edu> wrote: >>>> I agree with you about the inadvisable use of: input.dataset.*. >>>> >>>> I'm looking at: >>>> >>>> lib/galaxy/model/__init__.py >>>> class Dataset( object ): >>>> ... >>>> def __init__( self, id=None, state=None, external_filename=None, >>>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>>> ... >>>> self._extra_files_path = extra_files_path >>>> ... >>>> @property >>>> def extra_files_path( self ): >>>> return self.object_store.get_filename( self, dir_only=True, >>>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>>> >>>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>>would >>>> this return the path relative to the current file location of dataset? >>>> >>>> >>>> >>>> >>>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>>> >>>>> Okay - so this is what broke things: >>>>> >>>>> >>>>> >>>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e20 >>>>>1b73a4dd99b56282169d86 >>>>> >>>>> My feeling with the commit was that wrappers and tools should never >>>>>be >>>>> explicitly accessing paths explicitly through input.dataset.*. I >>>>>think >>>>> this would circumvent options like outputs_to_working_directory and >>>>> break remote job execution through Pulsar. It also breaks the object >>>>> store abstraction I think - which is why I made the change for Bjoern >>>>> I guess. >>>>> >>>>> I did not (and this was stupid on my part) realize that datatype code >>>>> would be running on the remote host and accessing these model >>>>> properties directly outside the abstractions setup by the wrappers >>>>> supplied to cheetah code and so they have become out of sync as of >>>>> that commit. >>>>> >>>>> I am thinking somehow changing what the datatype code gets is the >>>>> right approach and not fixing things by circumvent the wrapper and >>>>> accessing properties directly on the dataset. Since you will find >>>>>that >>>>> doing this breaks things for Bjoern object store and could probably >>>>> never run on usegalaxy.org say for the same reason. >>>>> >>>>> Too many different competing deployment options all being >>>>>incompatible >>>>> with each other :(. >>>>> >>>>> Will keep thinking about this and respond again. >>>>> >>>>> -John >>>>> >>>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchilton(a)gmail.com> >>>>>wrote: >>>>>> >>>>>> JJ, >>>>>> >>>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>>> understand extra_files_path versus files_path myself. There are open >>>>>> questions on Peter's blast repo and no one ever followed up on my >>>>>> object store questions about this with Bjoern's issues a couple >>>>>> release cycles ago. We need to get these to work - write >>>>>>documetation >>>>>> explicitly declaring best practices we can all agree on and then >>>>>>write >>>>>> some tests to ensure things don't break in the future. >>>>>> >>>>>> When you say your tools broke recently - can you say for certain >>>>>>which >>>>>> release broke these - the August14, October14, something older? >>>>>> >>>>>> I'll try to do some more research and get back to you. >>>>>> >>>>>> -John >>>>>> >>>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns198(a)umn.edu> >>>>>>wrote: >>>>>>> >>>>>>> Andrew, >>>>>>> >>>>>>> Thanks for investigating this. I changed the subject and sent to >>>>>>>the >>>>>>> galaxy >>>>>>> dev list. >>>>>>> >>>>>>> I've had a number of tools quit working recently. Particularly >>>>>>>tools >>>>>>> that >>>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>>> SnpEff. >>>>>>> >>>>>>> I think there was a change in the galaxy framework: >>>>>>> The extra_files_path when referenced from an input or output in the >>>>>>> cheetah >>>>>>> template sections of the tool config xml will be relative to the >>>>>>>job >>>>>>> working >>>>>>> directly rather than the files location. >>>>>>> I've just changed a few of my tools on my server yesterday >>>>>>> from: <param_name>.extra_files_path >>>>>>> to: <param_name>.dataset.extra_files_path >>>>>>> and they now work again. >>>>>>> >>>>>>> Dan or John, is that the right way to handle this? >>>>>>> Thanks, >>>>>>> >>>>>>> JJ >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>>> >>>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>>error!) >>>>>>>> >>>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>>with an >>>>>>>> attribute declaration and manifests during database download as: >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File >>>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>>> line 564, in finish_job >>>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", >>>>>>>>line >>>>>>>> 1107, in finish >>>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # >>>>>>>>call >>>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>>> dataset creation >>>>>>>> File >>>>>>>> >>>>>>>> >>>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff >>>>>>>>/1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>>> line 21, in set_meta >>>>>>>> data_dir = dataset.files_path >>>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no >>>>>>>>attribute >>>>>>>> 'files_path' >>>>>>>> >>>>>>>> >>>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>>> which fixed the download bug, but then SnpEff subtool itself >>>>>>>>threw a >>>>>>>> similar error when I tried to use that database from the history. >>>>>>>> >>>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>>> files_path vs extra_files_path >>>>>>>> >>>>>>>> I've shared a history with both of these errors here: >>>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>>> >>>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>>> >>>>>>>> Any help appreciated! >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> A/Prof Andrew Lonie >>>>>>>> University of Melbourne >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>>> Minnesota >>>>>>> ___________________________________________________________ >>>>>>> Please keep all replies on the list by using "reply all" >>>>>>> in your mail client. To manage your subscriptions to this >>>>>>> and other Galaxy lists, please use the interface at: >>>>>>> http://lists.bx.psu.edu/ >>>>>>> >>>>>>> To search Galaxy mailing lists use the unified search at: >>>>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>> >>>> >>>> -- >>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>Minnesota >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> http://lists.bx.psu.edu/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >>> > > > >------------------------------ > >Message: 6 >Date: Wed, 15 Oct 2014 13:07:40 -0500 >From: Jim Johnson <johns198(a)umn.edu> >To: John Chilton <jmchilton(a)gmail.com> >Cc: Jim Johnson <jj(a)umn.edu>, "galaxy-dev(a)lists.bx.psu.edu" > <galaxy-dev(a)lists.bx.psu.edu>, Nuwan Goonasekera > <nuwan.goonasekera(a)unimelb.edu.au>, Andrew Lonie > <alonie(a)unimelb.edu.au> >Subject: Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path >Message-ID: <543EB7EC.2070606(a)umn.edu> >Content-Type: text/plain; charset=UTF-8; format=flowed > >Looks good, John. > >I tested with: >https://testtoolshed.g2.bx.psu.edu/view/jjohnson/snpsift_dbnsfp_datatypes > >lib/galaxy/datatypes/converters/tabular_to_dbnsfp.xml > >reverting from hack: > <command interpreter="python">tabular_to_dbnsfp.py $input >$dbnsfp.dataset.extra_files_path/dbNSFP.gz</command> >back to: > <command interpreter="python">tabular_to_dbnsfp.py $input >$dbnsfp.files_path/dbNSFP.gz</command> > >On 10/15/14, 12:05 PM, John Chilton wrote: >> Hey JJ, >> >> Opened a pull request to stable with my best guess at the right to >> proceed and hopefully a best practice recommendation we can all get >> behind. Do you want to try it out and let me know if it fixes snpeff? >> (It does fix the velvet datatypes you contributed to Galaxy.) >> >> >>https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-data >>types-consuming-output-extra/diff >> >> Dan, Bjoern - does this make sense - can we move forward with this >> approach ($input.extra_files_path for inputs and $output.files_path >> for outputs) as the best practices for how to reference these >> directories. >> >> -John >> >> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson <johns198(a)umn.edu> wrote: >>> I agree with you about the inadvisable use of: input.dataset.*. >>> >>> I'm looking at: >>> >>> lib/galaxy/model/__init__.py >>> class Dataset( object ): >>> ... >>> def __init__( self, id=None, state=None, external_filename=None, >>> extra_files_path=None, file_size=None, purgable=True, uuid=None ): >>> ... >>> self._extra_files_path = extra_files_path >>> ... >>> @property >>> def extra_files_path( self ): >>> return self.object_store.get_filename( self, dir_only=True, >>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id ) >>> >>> I'm trying to see when self._extra_files_path gets set. Otherwise, >>>would >>> this return the path relative to the current file location of dataset? >>> >>> >>> >>> >>> On 10/15/14, 9:36 AM, John Chilton wrote: >>>> Okay - so this is what broke things: >>>> >>>> >>>> >>>>https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201 >>>>b73a4dd99b56282169d86 >>>> >>>> My feeling with the commit was that wrappers and tools should never be >>>> explicitly accessing paths explicitly through input.dataset.*. I think >>>> this would circumvent options like outputs_to_working_directory and >>>> break remote job execution through Pulsar. It also breaks the object >>>> store abstraction I think - which is why I made the change for Bjoern >>>> I guess. >>>> >>>> I did not (and this was stupid on my part) realize that datatype code >>>> would be running on the remote host and accessing these model >>>> properties directly outside the abstractions setup by the wrappers >>>> supplied to cheetah code and so they have become out of sync as of >>>> that commit. >>>> >>>> I am thinking somehow changing what the datatype code gets is the >>>> right approach and not fixing things by circumvent the wrapper and >>>> accessing properties directly on the dataset. Since you will find that >>>> doing this breaks things for Bjoern object store and could probably >>>> never run on usegalaxy.org say for the same reason. >>>> >>>> Too many different competing deployment options all being incompatible >>>> with each other :(. >>>> >>>> Will keep thinking about this and respond again. >>>> >>>> -John >>>> >>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton <jmchilton(a)gmail.com> >>>>wrote: >>>>> JJ, >>>>> >>>>> Arg this is a mess. I am very sorry about this - I still don't >>>>> understand extra_files_path versus files_path myself. There are open >>>>> questions on Peter's blast repo and no one ever followed up on my >>>>> object store questions about this with Bjoern's issues a couple >>>>> release cycles ago. We need to get these to work - write documetation >>>>> explicitly declaring best practices we can all agree on and then >>>>>write >>>>> some tests to ensure things don't break in the future. >>>>> >>>>> When you say your tools broke recently - can you say for certain >>>>>which >>>>> release broke these - the August14, October14, something older? >>>>> >>>>> I'll try to do some more research and get back to you. >>>>> >>>>> -John >>>>> >>>>> On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson <johns198(a)umn.edu> >>>>>wrote: >>>>>> Andrew, >>>>>> >>>>>> Thanks for investigating this. I changed the subject and sent to the >>>>>> galaxy >>>>>> dev list. >>>>>> >>>>>> I've had a number of tools quit working recently. Particularly >>>>>>tools >>>>>> that >>>>>> inspect the extra_files_path when setting metadata, Defuse, Rsem, >>>>>> SnpEff. >>>>>> >>>>>> I think there was a change in the galaxy framework: >>>>>> The extra_files_path when referenced from an input or output in the >>>>>> cheetah >>>>>> template sections of the tool config xml will be relative to the job >>>>>> working >>>>>> directly rather than the files location. >>>>>> I've just changed a few of my tools on my server yesterday >>>>>> from: <param_name>.extra_files_path >>>>>> to: <param_name>.dataset.extra_files_path >>>>>> and they now work again. >>>>>> >>>>>> Dan or John, is that the right way to handle this? >>>>>> Thanks, >>>>>> >>>>>> JJ >>>>>> >>>>>> >>>>>> >>>>>> On 10/13/14, 9:29 PM, Andrew Lonie wrote: >>>>>>> Hi Jim. I am probably going about this the wrong way, but I am not >>>>>>> clear on how to report tool errors (if in fact this is a tool >>>>>>>error!) >>>>>>> >>>>>>> I've been trialling your snpeff wrapper from the test toolshed and >>>>>>> getting a consistent error with the SnpEff Download and SnpEff sub >>>>>>> tools (the SnpSift dbNSFP works fine). The problem seems to be >>>>>>>with an >>>>>>> attribute declaration and manifests during database download as: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>>"/mnt/galaxy/galaxy-app/lib/galaxy/jobs/runners/__init__.py", >>>>>>> line 564, in finish_job >>>>>>> job_state.job_wrapper.finish( stdout, stderr, exit_code ) >>>>>>> File "/mnt/galaxy/galaxy-app/lib/galaxy/jobs/__init__.py", >>>>>>>line >>>>>>> 1107, in finish >>>>>>> dataset.datatype.set_meta( dataset, overwrite=False ) # >>>>>>>call >>>>>>> datatype.set_meta directly for the initial set_meta call during >>>>>>> dataset creation >>>>>>> File >>>>>>> >>>>>>> >>>>>>>"/mnt/galaxy/shed_tools/testtoolshed.g2.bx.psu.edu/repos/iuc/snpeff/ >>>>>>>1938721334b3/snpeff/lib/galaxy/datatypes/snpeff.py", >>>>>>> line 21, in set_meta >>>>>>> data_dir = dataset.files_path >>>>>>> AttributeError: 'HistoryDatasetAssociation' object has no attribute >>>>>>> 'files_path' >>>>>>> >>>>>>> >>>>>>> We fiddled around with the wrapper, eventually replacing >>>>>>> 'dataset.files_path' with 'dataset.extra_files_path' in snpeff.py, >>>>>>> which fixed the download bug, but then SnpEff subtool itself threw >>>>>>>a >>>>>>> similar error when I tried to use that database from the history. >>>>>>> >>>>>>> I chased up a bit more but cannot understand the various posts on >>>>>>> files_path vs extra_files_path >>>>>>> >>>>>>> I've shared a history with both of these errors here: >>>>>>> http://130.56.251.62/galaxy/u/alonie/h/unnamed-history >>>>>>> >>>>>>> Maybe this is a problem with our Galaxy image? >>>>>>> >>>>>>> Any help appreciated! >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> A/Prof Andrew Lonie >>>>>>> University of Melbourne >>>>>> >>>>>> >>>>>> -- >>>>>> James E. Johnson Minnesota Supercomputing Institute University of >>>>>> Minnesota >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>> >>> >>> -- >>> James E. Johnson Minnesota Supercomputing Institute University of >>>Minnesota > > >-- >James E. Johnson Minnesota Supercomputing Institute University of >Minnesota > > >------------------------------ > >Message: 7 >Date: Wed, 15 Oct 2014 20:50:56 +0000 >From: Iry Witham <Iry.Witham(a)jax.org> >To: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> >Subject: [galaxy-dev] snpEff and java issue >Message-ID: <D064566E.39080%iry.witham(a)jax.org> >Content-Type: text/plain; charset="us-ascii" > >Hi Team, > >I had manually installed the latest version of snpEff based on the >recommendation of Pablo and after modifying the XML files I had a working >version of snpEff that produced a vcf. However, this morning I reran my >workflow and it is failing again, but this time I am getting the >following error: > >Fatal error: Exit code 1 (Error) >Exception in thread "main" java.lang.UnsupportedClassVersionError: >ca/mcgill/mcb/pcingola/snpEffect/commandLine/SnpEff : Unsupported >major.minor version 51.0 >at java.lang.ClassLoader.defineClass1(Native Method) >at java.lang.ClassLoader.defineClass(ClassLoader.java:643) >at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >at java.net.URLClassLoader.access$000(URLClassLoader.java:73) >at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >at java.lang.ClassLoader.loadClass(ClassLoader.java:323) >at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >at java.lang.ClassLoader.loadClass(ClassLoader.java:268) >Could not find the main class: >ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff. Program will exit. > >Nothing has been changed since I had success. > >Regards, >Iry > > >The information in this email, including attachments, may be confidential >and is intended solely for the addressee(s). If you believe you received >this email by mistake, please notify the sender by return email as soon >as possible. >

1 0

Re: [galaxy-dev] galaxy-dev Digest, Vol 100, Issue 10
by Iry Witham 17 Oct '14

17 Oct '14

So I have finally determined the cause and found a resolution. The issue was the fact that the version of Java on the Ubuntu linux VM from AWS is 1.6.x. GATK2 requires version 1.7.0 or higher. This is fine, but you will need to manually upgrade the java version. To do so you will need to complete the following: SSH to the VM sudo su - root CD to /mnt/galaxy/ Complete the following steps: 1. sudo apt-get purge openjdk* (This removes java 1.6 completely) 2. Modify this file: vi /etc/apt/sources.list.d/cloudbiolinux.list (Remove the leading section of line 11) 3. sudo add-apt-repository ppa:webupd8team/java 4. sudo apt-get update 5. sudo apt-get install oracle-java7-installer 6. java -version Now if you plan to add nodes via the Cloudman Console you will need to perform these tasks for each node you install. I worked with AWS Support to setup an "Auto Scaling Group" to accommodate this process. This required getting my Master instance upgraded and creating an AMI from it. >From that point you can build the group based on the following: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/creating-your- auto-scaling-groups.html This was, in theory, a great idea. However, it did not work for me. Each of the node that were generated by this tool had Java version 1.6.x and this caused GATK2 to fail. My recommendation is that if you have time you should paly around with this more, but I did not have that luxury for this project. Iry On 10/11/14 12:00 PM, "galaxy-dev-request(a)lists.bx.psu.edu" <galaxy-dev-request(a)lists.bx.psu.edu> wrote: >Send galaxy-dev mailing list submissions to > galaxy-dev(a)lists.bx.psu.edu > >To subscribe or unsubscribe via the World Wide Web, visit > http://lists.bx.psu.edu/listinfo/galaxy-dev >or, via email, send a message with subject or body 'help' to > galaxy-dev-request(a)lists.bx.psu.edu > >You can reach the person managing the list at > galaxy-dev-owner(a)lists.bx.psu.edu > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of galaxy-dev digest..." > > >HEY! This is important! If you reply to a thread in a digest, please >1. Change the subject of your response from "Galaxy-dev Digest Vol ..." >to the original subject for the thread. >2. Strip out everything else in the digest that is not part of the thread >you are responding to. > >Why? >1. This will keep the subject meaningful. People will have some idea >from the subject line if they should read it or not. >2. Not doing this greatly increases the number of emails that match >search queries, but that aren't actually informative. > >Today's Topics: > > 1. Re: Tool development: Is it possible to pass a user's id in > dynamic_options() parameter? (Dooley, Damion) > 2. Re: Tool development: Is it possible to pass a user's id in > dynamic_options() parameter? (Daniel Blankenberg) > 3. Anyone else having issues with cloudman cloudlaunch? > (Chris Dagdigian) > 4. Re: Anyone else having issues with cloudman cloudlaunch? > (Dannon Baker) > 5. Re: Anyone else having issues with cloudman cloudlaunch? > (Dannon Baker) > 6. Re: Anyone else having issues with cloudman cloudlaunch? > (Chris Dagdigian) > 7. Re: Anyone else having issues with cloudman cloudlaunch? > (Dannon Baker) > 8. Re: Anyone else having issues with cloudman cloudlaunch? > (Chris Dagdigian) > 9. GATK2 java version related issues (Iry Witham) > 10. Why does Galaxy CloudMan 2.3 (ami-a7dbf6ce) run the boinc > client? (Chris Dagdigian) > 11. Re: GATK2 java version related issues (Daniel Blankenberg) > 12. Re: Why does Galaxy CloudMan 2.3 (ami-a7dbf6ce) run the boinc > client? (Dannon Baker) > 13. Re: GATK2 java version related issues (Iry Witham) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Fri, 10 Oct 2014 09:02:50 -0700 >From: "Dooley, Damion" <Damion.Dooley(a)bccdc.ca> >To: Daniel Blankenberg <dan(a)bx.psu.edu> >Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> >Subject: Re: [galaxy-dev] Tool development: Is it possible to pass a > user's id in dynamic_options() parameter? >Message-ID: > <7891813F3C8F424B97D8BF2E5600E51903302322C5B5(a)VEXCCR02.phsabc.ehcnet.ca> > >Content-Type: text/plain; charset="Windows-1252" > >Hi Dan, > >I'd like to get the user's galaxy user name directly - but the <code >file="..."> that drives my dynamic_options select list has no access to >any information in a galaxy's user session, right? (Ultimately I'm >trying to get the dynamic_options code to make galaxy api calls using the >user's api key.) I saw the GenomeSpace tool earlier - but got the >impression that a user was entering their genome login in the tool form. >I was hoping just to get a user's galaxy id without any re-entry of it. > >p.s. Its been really great getting advice from all of you; I hope to >contribute likewise. > >d. > >Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC >Centre for Disease Control >655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada >________________________________________ >From: Daniel Blankenberg [dan(a)bx.psu.edu] >Sent: Friday, October 10, 2014 8:08 AM >To: Dooley, Damion >Cc: galaxy-dev(a)lists.bx.psu.edu >Subject: Re: [galaxy-dev] Tool development: Is it possible to pass a >user's id in dynamic_options() parameter? > >Hi Damion, > >Text fields are not currently able to be populated by dynamic_options, >but in a dynamic_options select list you could access the user?s username >directly. If you want the user to be able to override it, you could have >a conditionally used empty text box that your tool defaults to using when >it is populated. > >An example that you might want to look at could be the GenomeSpace Export >tool (tools/genomespace/genomespace_exporter.*), which queries the >GenomeSpace API to get a list of available folders for export for a >specific user using attributes of the current user?s preferences >(GenomeSpace username and token). It generates a drill_down type select >parameter, but the same can be done for a standard select list. > > >Thanks for using Galaxy, > >Dan > > >------------------------------ > >Message: 2 >Date: Fri, 10 Oct 2014 12:59:06 -0400 >From: Daniel Blankenberg <dan(a)bx.psu.edu> >To: "Dooley, Damion" <Damion.Dooley(a)bccdc.ca> >Cc: "galaxy-dev(a)lists.bx.psu.edu" <galaxy-dev(a)lists.bx.psu.edu> >Subject: Re: [galaxy-dev] Tool development: Is it possible to pass a > user's id in dynamic_options() parameter? >Message-ID: <1A8CFB92-0432-4FAC-B139-84E05F0EDE57(a)bx.psu.edu> >Content-Type: text/plain; charset="windows-1252" > >Hi Damion, > >In the genomespace exporter case, the call to >?dynamic_options="galaxy_code_get_genomespace_folders( genomespace_site = >'prod', trans=__trans__, value=__value__, input_dataset=input1, >base_url=base_url )?? is passing in trans, which it is using to >determine user and an attribute of the user. You could then check to see >if trans.user is not None, and then get trans.user.username. > >You may be able to access user or username here by the more recently >added ?__user__? or even ?__user_name__? / ?__user_id__? / >?__user_email__" variables directly, but I didn?t test that they were >available in this dynamic_options context example (if they are available, >they would be preferred over accessing trans). > >For accessing a user?s API key, this trello card is of interest: >https://trello.com/c/hL4NYsSp/2108-request-from-jxtx-provide-consistent-in >terface-to-tools-for-communicating-with-the-api. Of course the Key can be >obtained directly from the user object as well. > > >Hope this helps, > >Dan > > >On Oct 10, 2014, at 12:02 PM, Dooley, Damion <Damion.Dooley(a)bccdc.ca> >wrote: > >> Hi Dan, >> >> I'd like to get the user's galaxy user name directly - but the <code >>file="..."> that drives my dynamic_options select list has no access to >>any information in a galaxy's user session, right? (Ultimately I'm >>trying to get the dynamic_options code to make galaxy api calls using >>the user's api key.) I saw the GenomeSpace tool earlier - but got the >>impression that a user was entering their genome login in the tool form. >> I was hoping just to get a user's galaxy id without any re-entry of it. >> >> p.s. Its been really great getting advice from all of you; I hope to >>contribute likewise. >> >> d. >> >> Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC >>Centre for Disease Control >> 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada >> ________________________________________ >> From: Daniel Blankenberg [dan(a)bx.psu.edu] >> Sent: Friday, October 10, 2014 8:08 AM >> To: Dooley, Damion >> Cc: galaxy-dev(a)lists.bx.psu.edu >> Subject: Re: [galaxy-dev] Tool development: Is it possible to pass a >>user's id in dynamic_options() parameter? >> >> Hi Damion, >> >> Text fields are not currently able to be populated by dynamic_options, >>but in a dynamic_options select list you could access the user?s >>username directly. If you want the user to be able to override it, you >>could have a conditionally used empty text box that your tool defaults >>to using when it is populated. >> >> An example that you might want to look at could be the GenomeSpace >>Export tool (tools/genomespace/genomespace_exporter.*), which queries >>the GenomeSpace API to get a list of available folders for export for a >>specific user using attributes of the current user?s preferences >>(GenomeSpace username and token). It generates a drill_down type select >>parameter, but the same can be done for a standard select list. >> >> >> Thanks for using Galaxy, >> >> Dan > >

1 0

CloudMan auto-scaling - Use spot instance for worker node?
by Petit III, Robert A. 16 Oct '14

16 Oct '14

Hi all, I realize when you manually add a node you can request that it be a spot instance. Is it also possible to use spot instances for auto-scaling? Thank you very much, Robert ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

2 1

Galaxy Training Network
by Dave Clements 16 Oct '14

16 Oct '14

Hello all, We are pleased to announce the *Galaxy Training Network (GTN) <https://wiki.galaxyproject.org/Teach/GTN>*, a network of trainers who teach bioinformatics using Galaxy, or teach about Galaxy itself. The GTN aims to make it easy to find Galaxy trainers <https://wiki.galaxyproject.org/Teach/Trainers>, and to share and discover the wealth of training resources available for Galaxy. This includes training materials <https://wiki.galaxyproject.org/Teach/Resources>, a trainer directory <https://wiki.galaxyproject.org/Teach/Trainers>,best practices <https://wiki.galaxyproject.org/Teach/BestPractices>, and guidance on computing platforms <https://wiki.galaxyproject.org/Teach/ComputingPlatforms> for teaching with Galaxy. The Galaxy Training Network is accessible to the entire community. If you teach with Galaxy, then please consider adding your organization <https://wiki.galaxyproject.org/Teach/Trainers#Add_a_Trainer>, materials <https://wiki.galaxyproject.org/Teach/Resources#Add_a_Training_Resource>, and best practices <https://wiki.galaxyproject.org/Teach/BestPractices>. The Trainer Directory <https://wiki.galaxyproject.org/Teach/Trainers> already includes 16 organizations and there may be one (or 6!) near you <http://bit.ly/gxytrnmap>. Thanks, The Galaxy Training Network (GTN) <https://wiki.galaxyproject.org/Teach/Trainers> -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

Create select list parameter from input
by Nikos Sidiropoulos 16 Oct '14

16 Oct '14

Hi all, Is it possible to read id's from a column of an input dataset and create a checkbox parameter with the unique id's of that column within the same tool? The only remotely similar action that I found is on the sorting tool where it reads the number of the columns of that dataset and creates a dropdown select list. Bests, Nikos

1 0

Help with Galaxy server migration
by Sarah Diehl 16 Oct '14

16 Oct '14

Dear all, I'm trying to migrate our local Galaxy server from a standalone server running on CentOS 5 to a setup that runs Galaxy in a Docker container (bgruening/galaxy-stable). Due to Docker and the setup of the container, almost all data, tools, configs etc. have to move to a different place. Since our old server is pretty cluttered up anyway, I would like to move just user data (logins, histories, datasets ...), but none of the tool installations, tool data or configs. My plan was to start with a fresh and clean new Galaxy installation, transfer the datasets and the database and then reinstall all tools through the toolshed (of course there are many more small odds and ends that need fixing). The problem is that after transfering the database, I always get an error when going to "Manage installed tool shed repositories" (see end of the mail). I tried just overwriting the pg_data directory (PostgreSQL versions are identical) and also exporting to sql (pg_dump) and importing in the new DB. Both times I made sure to run manage_db.sh upgrade afterwards. The error stays the same. I can't make much sense of the error message, my only guess is that it looks for something that's not there. However, I really don't want to transfer the tool directories and tool configurations, because that will get messy and difficult, since many paths need to be adjusted. There's a high chance for error and this way I also carry over the clutter. So is there a way to remove all tool installation information from the database, before I transfer it? Can this even be disentangled from the histories and datasets? Any help or other ideas on how to do this migration are highly appreciated! Thanks, Sarah 10.1.5.190 - - [16/Oct/2014:09:54:35 +0000] "GET /admin_toolshed/browse_repositories HTTP/1.1" 500 - "http://deep2:8080/admin/index" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:32.0) Gecko/20100101 Firefox/32.0" Error - <type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xe2 in position 393: ordinal not in range(128) URL: http://deep2:8080/admin_toolshed/browse_repositories File 'lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/galaxy-central/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File 'lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File 'lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File 'lib/galaxy/web/framework/decorators.py', line 87 in decorator return func( self, trans, *args, **kwargs ) File 'lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 167 in browse_repositories return self.installed_repository_grid( trans, **kwd ) File 'lib/galaxy/web/framework/helpers/grids.py', line 306 in __call__ kwargs=kwargs ) File 'lib/galaxy/web/framework/webapp.py', line 771 in fill_template return self.fill_template_mako( filename, **kwargs ) File 'lib/galaxy/web/framework/webapp.py', line 786 in fill_template_mako return template.render( **data ) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render return runtime._render(self, self.callable_, args, data) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render **_kwargs_for_callable(callable_, data)) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/galaxy-central/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/galaxy-central/database/compiled_templates/base.mako.py', line 66 in render_body __M_writer(unicode(next.body())) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 91 in render_body __M_writer(unicode(self.load())) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 120 in render_load __M_writer(unicode( h.dumps( self.get_grid_config( embedded=embedded, insert=insert ) ) )) File '/galaxy-central/database/compiled_templates/grid_base.mako.py', line 303 in render_get_grid_config value = column.get_value( trans, grid, item ) File 'lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py', line 106 in get_value return suc.get_tool_shed_repository_status_label( trans.app, tool_shed_repository ) File 'lib/tool_shed/util/shed_util_common.py', line 829 in get_tool_shed_repository_status_label elif tool_shed_repository.tool_dependencies_being_installed: File 'lib/galaxy/model/tool_shed_install/__init__.py', line 408 in tool_dependencies_being_installed for tool_dependency in self.tool_dependencies: File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 168 in __get__ File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/attributes.py', line 453 in get File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 508 in _load_for_state File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/strategies.py', line 574 in _emit_lazyload File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2115 in all File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/query.py', line 2341 in instances File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', line 3204 in fetchall File '/galaxy-central/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py', line 3171 in _fetchall_impl UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 393: ordinal not in range(128)

2 2