January 2016 - galaxy-dev - lists.galaxyproject.org

access system variables in tool xml (Matthias De Smet)
by Dooley, Damion 28 Jan '16

28 Jan '16

Others might have responded to this, and I can't remember the very latest approach, but I used the approach below... Its not easy though since api is read via <code> script [BTW, topically, I still need <code> on a regular basis!], but is passed over to command line app securely via a <configfile>. For error situation messaging back to user I was using a dynamic box radio drill_down input. All this code is in toolshed under "versioned_data". Teasing out what you need might require some work though - the code also has a chunk of stuff for maintaining use of an administrator's api key as well as the user's since the app needed to do admin level api work on behalf of a user. >------------------------------ > >Message: 11 >Date: Wed, 27 Jan 2016 12:42:17 +0000 >From: Matthias De Smet <Matthias.DeSmet(a)UGent.be> >... >Hi again > >I’m trying to create a tool that interacts with galaxy through the API. >For this, I need to get the users API key (or equivalent) en the domain >name of the Galaxy host. > >The purpose of this tool is to get data and put it in a data library, so >I suppose I need to use the API. Or is there another way to do this with >a tool? > >Thanks! >Matthias <command interpreter="python"> #assert $__user__, Exception( 'You must be logged in to use this tool.' ) versioned_data.py ... -O "$__app__.security.encode_id($log.id)" --api_info_path "$api_info_path" ##Actually a file path to configfile that holds api key </command> <inputs> ... <param name="api_info" display="radio" type="drill_down" label="For user with Galaxy API Key" dynamic_options="vdb_init_tool_user(__trans__)" /> </inputs> <configfiles> <configfile name="api_info_path">${__user__.api_keys[0].key} $api_info </configfile> </configfiles> <code file="versioned_data_form.py" /> And over in versioned_data_form.py ( there's a reference to "set_trans()" which feeds through to set_user_api() below... import vdb_retrieval def vdb_init_tool_user(trans): """ Retrieves a user's api key if they have one, otherwise lets them know they need one This function is automatically called from versioned_data.xml form on presentation to user Note that this is how self.api_url gets back into form, for passage back to 2nd call via versioned_data.py self.api_key is passed via secure <configfile> construct. ALSO: squeezing history_id in this way since no other way to pass it. "trans" is provided only by tool form presentation via <code file="..."> BUT NOW SEE John Chilton's: https://gist.github.com/jmchilton/27c5bb05e155a611294d See galaxy source code at https://galaxy-dist.readthedocs.org/en/latest/_modules/galaxy/web/framework .html, See http://dev.list.galaxyproject.org/error-using-get-user-id-in-xml-file-in-ne w-Galaxy-td4665274.html See http://dev.list.galaxyproject.org/hg-galaxy-2780-Real-Job-tm-support-for-th e-library-upload-to-td4133384.html master api key, set in galaxy config: #self.master_api_key = trans.app.config.master_api_key """ global retrieval_obj api_url = trans.request.application_url + '/api' history_id = str(trans.security.encode_id(trans.history.id)) user_api_key = None if trans.user: user_name = trans.user.username if trans.user.api_keys and len(trans.user.api_keys) > 0: user_api_key = trans.user.api_keys[0].key #First key is always the active one? items = [ { 'name': user_name, 'value': api_url + '-' + history_id, 'options':[], 'selected': True } ] else: items = [ { 'name': user_name + ' - Note: you need a key (see "User" menu)!', 'value': '0', 'options':[], 'selected': False } ] else: items = [ { 'name': 'You need to be logged in to use this tool!', 'value': '1', 'options':[], 'selected': False } ] retrieval_obj = vdb_retrieval.VDBRetrieval() retrieval_obj.set_trans(api_url, history_id, user_api_key) return items And over in tool xml's target code (versioned_data.py) it includes vdb_retrieval.py BUT THIS HAS A LOT OF STUFF YOU PROBABLY DON'T NEED! Sorry!: from bioblend.galaxy import GalaxyInstance class VDBRetrieval(object): def __init__(self, api_key=None, api_url=None): """ This gets either trans.x.y from <code file="..."> call in versioned_data.xml, or it gets a call with api_key and api_url from versioned_data.py @param api_key_path string File path to temporary file containing user's galaxy api_key @param api_url string contains http://[ip]:[port] for handling galaxy api calls. """ # Initialized constants during the life of a request: self.global_retrieval_date = None self.library_id = None self.history_id = None self.data_stores = [] # Entire json library structure. item.url, type=file|folder, id, name (library path) # Note: changes to library during a request aren't reflected here. self.library = None self.user_api_key = None self.user_api = None self.admin_api_key = None self.admin_api = None self.api_url = None def set_trans(self, api_url, history_id, user_api_key=None): #master_api_key=None, """ Used only on initial presentation of versioned_data.xml form. Doesn't need admin_api """ self.history_id = history_id self.api_url = api_url self.user_api_key = user_api_key self.set_user_api() self.set_admin_api() self.set_datastores() ... def set_user_api(self): """ Note: error message tacked on to self.data_stores for display back to user. """ self.user_api = GalaxyInstance(url=self.api_url, key=self.user_api_key) if not self.user_api: self.data_stores.append({'name':'Error: user Galaxy API connection was not set up correctly. Try getting another user API key.', 'id':'none'}) return

1 0

Re: [galaxy-dev] Setting up visualization on Galaxy on CentOS
by Oksana Korol 27 Jan '16

27 Jan '16

It's un-commented in both local and production. Still there is no icon in production. Cheers, Oksana On Mon, Jan 25, 2016 at 9:52 AM, Björn Grüning <bjoern.gruening(a)gmail.com> wrote: > Hi, > > can you please check this setting? > > > https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample#L… > > Thanks, > Bjoern > > Am 25.01.2016 um 15:46 schrieb Oksana Korol: > > Thanks, Bjoern! > > > > For my local install of Galaxy, logging in as a user worked and the icon > > is showing up (although I still had to manually go and change > > permissions on all the mako templates in ./database/compiled_templates > > directory). But our production instance of Galaxy (also v15.07 on CentOS > > 6.7) does not show the chart-symbol for Phyloviz, even when I'm logged > > in as a user. In production, it is not a clean install, but an upgrade > > from previous versions, so it is harder to figure out what's going on. > > There are no errors in the logs, so I'm still looking for any kind of > > clues of what could have gone wrong with the visualization. > > > > Cheers, > > Oksana > > > > Message: 1 > > Date: Sat, 23 Jan 2016 19:55:47 +0100 > > From: Björn Grüning <bjoern.gruening(a)gmail.com > > <mailto:bjoern.gruening@gmail.com>> > > To: Oksana Korol <okorol(a)gmail.com <mailto:okorol@gmail.com>>, > > galaxy-dev(a)lists.galaxyproject.org > > <mailto:galaxy-dev@lists.galaxyproject.org> > > Subject: Re: [galaxy-dev] Setting up visualization on Galaxy on > CentOS > > Message-ID: <56A3CCB3.9020503(a)gmail.com > > <mailto:56A3CCB3.9020503@gmail.com>> > > Content-Type: text/plain; charset=utf-8 > > > > Hi Oksana, > > > > this work for me in 16.01 (upcoming). Please make sure you are logged > > into your Galaxy as some user, not anonymous and than the Phyloviz > > Visualisation will be available from the small chart-symbol next to > the > > (i) symbol. > > > > Hope this helps, > > Bjoern > > > > Am 22.01.2016 um 23:53 schrieb Oksana Korol: > > > Hi, > > > > > > I'm having trouble getting visualization icons, like Phyloviz, to > show > > > up on my Galaxy instance running in CentOS 6.7. Here are the steps > I > > > took, expecting it to work: > > > > > > 1. cloned Galaxy tag v15.07 (that's the version we need to run in > > > production) > > > > > > 2. Followed the steps in > > https://wiki.galaxyproject.org/VisualizationSetup > > > - Renamed datatypes_conf.xml.sample to > > > datatypes_conf.xml > > > - Downloaded wigToBigWig, bedGraphToBigWig, and faToTwoBit > > > - Installed BED Tools > > > - Added above tools to my user path (galaxy_utils dir is where > > > wigToBigWig and others are): > > > > > > /usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/galaxyuser/bin:/home/galaxyuser/bin:/home/galaxyuser/galaxy_utils:/home/galaxyuser/bin:/*home/galaxyuser/galaxy_utils:/home/galaxyuser/galaxy_utils/BEDTools/bedtools2/bin > > > * > > > 3. Started Galaxy. > > > > > > 4. Uploaded a newick tree file (nhx), which is known to be > displayable > > > in phyloviz on a Ubuntu instance of Galaxy > > > > > > 5. Made sure the datatype is nhx > > > > > > 6. Expecting to see a Phyloviz "graph" icon on the left, when I > expand > > > the file in history, but there is nothing there. > > > > > > At this point I don't know where to dig to find how I can fix it. > Any > > > feedback, including your own stories for visualization setup or > how I > > > can debug what's not being picked up where, would be greatly > > appreciated. > > > > > > Cheers, > > > Oksana > > > -- > >

3 4

Filtering dynamic options in tool with <filter> tagset
by Matthias De Smet 27 Jan '16

27 Jan '16

Hi All, I’m trying to implement a tool that get a list of options through the “dynamic_options” tag. I try to filter the options that get returned with another param, but it doensn’t seem to work. my tool xml looks like this <tool id="archivetolibrary" name="Copy files" version="0.1.0"> <description>from the archive to a data library</description> <code file="list_files.py"/> <command interpreter="bash"> test.sh > $log </command> <inputs> <param name="archive_source" type="select" display="radio" label="Choose source archive"> <options from_file="archives.loc"> <column name="name" index="0"/> <column name="value" index="1"/> </options> </param> <param name="query" type="text" label="Search for files"/> <param name="selected_file" type="select" label="Choose your file" dynamic_options="list_files(archive_source)"> <filter type="param_value" ref="query" column="0" /> </param> </inputs> <outputs> <data name="log" format="txt" label="${tool.name} on ${on_string}"></data> </outputs> <tests> <test> </test> </tests> <help> </help> </tool> Anyone with experience implementing filters? (Gx 15.10 on Ubuntu 14.04.3) Thanks! Matthias

3 5

Retain the dbkey specified for an input dataset through a Galaxy workflow?
by Greg Von Kuster 27 Jan '16

27 Jan '16

Hello Galaxians, I’m running Galaxy 15.10 and running workflows that include tools that require reference genomes (e.g. Extract Genomic DNA). I set the dbkey for the input dataset and it is retained for some tools, but not others. Running the workflow multiple times, it looks like the dbekey is lost at different tool points in the workflow. Is this a known issue or is there some setting I’ve missed. I’ve seen where the output dattype can be set for each tool, but not the dbkey. This is a problem because any tools that require a dbkey downstream result in errors. I was running the dev branch for a while, but workflow bugs in that branch forced me to revert back to 15.10. I’ve searched biostar and the mail lists, but haven’t seen an answer for this specific issue, although there are several related threads from the past. Sorry if it’s been answered and I missed it. Thanks very much for any help you can provide, Greg Von Kuster

1 3

access system variables in tool xml
by Matthias De Smet 27 Jan '16

27 Jan '16

Hi again I’m trying to create a tool that interacts with galaxy through the API. For this, I need to get the users API key (or equivalent) en the domain name of the Galaxy host. The purpose of this tool is to get data and put it in a data library, so I suppose I need to use the API. Or is there another way to do this with a tool? Thanks! Matthias

1 0

Importing workflows via command-line/Docker script
by Cameron Jack 27 Jan '16

27 Jan '16

Hi all, I've been developing a Dockerfile and repositories to enable users to set up an NGS forensics pipeline. So far so good, I can build the image (based on Bjoern Gruening's excellent galaxy-stable Docker image) and incorporate all the various tools I need, including custom ones with custom interfaces. I've also built some workflows within a running image and exported them to .ga files, but now I'm wondering if there is a way to import those back in to a new image via the Dockerfile? I'd rather have everything set up for the users upon building the image, rather than logging in as admin and then importing the workflow. Is this possible? If so, how? Thanks for any and all answers. Best regards, Cam Cameron Jack Bioinformatician ANU Bioinformatics Consulting Unit The Australian National University The John Curtin School of Medical Research Building 131 Garran Road Acton 0200, ACT Ph (office): +61 2 612 51128 Ph (mobile): +61 4 2368 0745 Email: cameron.jack(a)anu.edu.au Group email: abc(a)anu.edu.au

3 4

galaxy job manager with celery and rabbitmq
by 杜云飞 26 Jan '16

26 Jan '16

Hi, We are reading the source code for galaxy job manager. We found that the job manager works like amqp. We want to know why galaxy implements the job manager using rabbitmq and celery. What is the difference between them? Thanks! Best regards, forest

2 1

Python on cluster for setting metadata after jobs complete
by Peter Cock 25 Jan '16

25 Jan '16

On Mon, Jan 25, 2016 at 11:33 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hello all, > > We're currently looking at changing our Galaxy setup to link user accounts > with Linux user accounts for better cluster integration (running jobs as the > actual user on SGE). As part of this, we've tried setting up a fresh > installation on a new VM which has thrown up some issues. > > [snip] The next problem on my list was jobs successfully submitting to SGE and running, then failing with a Python exception: galaxy.eggs.EggNotFetchable These emails from Donald Shrum and Jingchao Zhang (BCC'd) look very similar, although I don't see they ever had a reply/resolution: https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-October/020719.ht… https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-January/018034.ht… We eventually realised that the new Galaxy VM server was running Python 2.7 (default under CentOS 7), but the cluster nodes were running Python 2.6 (default under CentOS 6). (Our current Galaxy server is also still running CentOS 6, which is likely why I never hit this before.) Replacing the VM with CentOS 6 we were able to get this to work - but was that necessary? How exactly do the cluster jobs invoke Python (especially when run as the associated Linux user account rather than under the Galaxy Linux account)? Sadly the cluster documentation does not mention Python at all: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster/ This seems to be an oversight, Regards, Peter

2 4

Want to Contribute in galaxy project
by Manish Bisht 25 Jan '16

25 Jan '16

Hello Developers, I am Manish Bisht. I am in second year of my engineering in information technology. I want to provide the code for Galaxy Project. I have created many projects outside of my college projects. You can see all at https:// <https://github.com/manishbisht>github.com <https://github.com/manishbisht>/ <https://github.com/manishbisht>manishbisht <https://github.com/manishbisht>. I am familiar with many languages some of them are HTML, CSS, JavaScript, PHP, C/C++, Java, and I am currently learning Python also. Also apart from my college hours i have also started my own startup with Run4Offers (http://www.run4offers.com/) I have also interested in Graphics Designing also. I have the knowledge of AdobePhotoshop, Adobe Illustrator and Adobe In Design. I have created many images of various festivals for my College Entrepreneurship Cell(TOPAZ). This is all about about me. I am new to open source communities. I want to work on the projects that can make world wide impact. So I have selected Galaxy Project and want to start contributing to it. Can anyone please help me in contributing to Galaxy Project. From where I can start contributing to Galaxy Project. Waiting for all the useful suggestions. Thanks :)

2 1

Creating cluster_files_directory (default database/pbs)
by Peter Cock 25 Jan '16

25 Jan '16

On Mon, Jan 25, 2016 at 11:33 AM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > Hello all, > > We're currently looking at changing our Galaxy setup to link user accounts > with Linux user accounts for better cluster integration (running jobs as the > actual user on SGE). As part of this, we've tried setting up a fresh > installation on a new VM which has thrown up some issues. > > [snip] We eventually got Galaxy to talk to SGE and submit jobs successfully as the individual user's Linux account, with external_chown_script.py being called to handle ownership of the files. One small stumbling block was missing the cluster_files_directory directory. https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample says: # If using a cluster, Galaxy will write job scripts and stdout/stderr to this # directory. #cluster_files_directory = database/pbs According the the current code, this is still the default value being used: https://github.com/galaxyproject/galaxy/blob/274af50792aaaf367520c22d7c13e2… However, Galaxy itself did not seem to create the cluster_files_directory directory - and there doesn't seem to be a ready-made pbs folder in the git repository. Should that happen automatically? At one stage, run.sh would create database/files and database/tmp - I'm note sure what that happens now: https://github.com/galaxyproject/galaxy/commit/4d720d63c631862778e54bd68c4e… https://github.com/galaxyproject/galaxy/commit/7560751dc87492471880c82b0085… Peter

2 2