- galaxy-dev - lists.galaxyproject.org

Can't stop uwsgi processes with supervisod
by Marco Tangaro 08 Jul '16

08 Jul '16

Dear experts, I've a strange behaviour with uWSGI and supervisor. This is my settings in galaxy.ini file: [uwsgi] processes = 3 socket = 127.0.0.1:4001 stats = 127.0.0.1:9191 #socket = /var/log/galaxy/uwgi.sock pythonpath = /home/galaxy/galaxy/lib pythonhome = /home/galaxy/galaxy/.venv threads = 4 logto = /var/log/galaxy/uwsgi.log And this is my supervisord settings: [program:galaxy_web] command = /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini directory = /home/galaxy/galaxy umask = 022 autostart = true autorestart = true startsecs = 20 user = galaxy environment = VIRTUAL_ENV="/home/galaxy/galaxy/.venv",PATH="/home/galaxy/galaxy/.venv/bin:%(ENV_PATH)s",PYTHONHOME="/home/galaxy/galaxy/.venv" numprocs = 1 stopsignal = INT startretries = 15 The problem is that when I run galaxy with "supervisorctl start galaxy:" I've 4 galaxy process (ps -aux | grep uwsgi): galaxy 28795 13.0 1.0 1465320 177332 ? Sl 16:42 0:12 /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini galaxy 28950 1.2 1.0 1465320 170108 ? Sl 16:42 0:01 /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini galaxy 28951 1.1 1.0 1465320 170108 ? Sl 16:42 0:01 /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini galaxy 28953 0.0 1.0 1465320 166060 ? S 16:42 0:00 /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini Once I did "supervisorctl stop galaxy:" I've: galaxy 28953 0.0 1.0 1465320 166060 ? S 16:42 0:00 /home/galaxy/galaxy/.venv/bin/uwsgi --ini-paste /home/galaxy/galaxy/config/galaxy.ini there is still a uwsgi process (sometimes more, it depends on the uwsgi process number). I'm able to stop only those processes which have a "Sl" in the STAT column of "ps -aux | grep uwsgi": S Interruptible sleep (waiting for an event to complete) l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do) (ps wiki) The problem is that I can't stop all the uwsgi processes, therefore I'm unable to restart galaxy, without killing all uwsgi processes. Thanks a lot, Marco.

1 0

Metadata access using tool config
by Katherine Beaulieu 08 Jul '16

08 Jul '16

Hi everyone, I am working on a tool which attempts to create a file which stores all the metadata associated with a job execution. The things I know how to access right now are the file extension, name history id, dataset id. I'd like to know how to access other things like the job id, uuid, file size, job command-line, job end time, job start time, core allocated, job runtime and any other important information associated with tool execution. I would prefer to get this information from the tool config file or my tool script and not have the user create an api key which they have to submit as a parameter to my tool. Let me know if you guys have any ideas! Katherine

2 4

Python tool wrapper with multiple input and output files
by Marco Tangaro 08 Jul '16

08 Jul '16

Dear experts, my name is Marco and I'm working to port our python tool to the Galaxy framework. The main script needs a rna.bam file as input, a reference fasta file, both mandatory. Finally, you can add a dna.bam file, but this is optional. Therefore an example command is: script.py -i rna.bam -f reference.fa -j dna.bam The outout is a tabular. Again the -j dna.bam option is completely optional. So quite soon it turned out that I had to use a python wrapper to parse our script. Now the wrapper works fine. The next step is to run the tool over multiple input file and we would like to avoid to use a workflow. The idea is that to each input file corresponds an output file. The reference is still the same. For instance, we have: rna_1.bam + dna_1.bam -> output_1.txt rna_2.bam + dna_2.bam -> output_2.txt rna_3.bam + dna_3.bam -> output_3.txt ... and so on. But I don't know the best strategy to give to my wrapper multiple input files. Moreover I have to be sure, when the dna_xyz.bam files are uploaded, that they correspond to the right rna_xyz.bam file. I would like to have as output a page which is showing as results the link to the single output files as suggested here. https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files planning to integrate a javascript interface. I've browsed a lot, but on multiple input file the posts are old. I'm using the last galaxy release (16_04). I'm quite new to the galaxy world... Thanks a lot for your suggestions, Marco

2 3

Auto detection of output format Galaxy custom tool
by Katherine Beaulieu 07 Jul '16

07 Jul '16

Hi everyone, I am looking to auto-detect the file format of a file outputted by a python tool I have written. Does anyone have any idea how to do this? I have tried setting the format attribute to "auto" with no luck. Let me know if you would like to see my tool config file. Cheers, Katherine

4 11

Mapping requirements to conda recipe names (granularity?)
by Peter van Heusden 07 Jul '16

07 Jul '16

Hi there I'm trying to satisfy requirements on our RNA-seq oriented Galaxy server at SANBI using conda/bioconda. There's a disconnect between the names used in tool dependency tags and the names of conda recipes though, for example: 1) the BAM to BigWig converter (tool ID CONVERTER_bam_to_bigwig_0) has a requirement for ucsc_tools. The tool it needs, bedGraphToBigWig, is provided by the ucsc-bedgraphtobigwig package, so in this case ucsc_tools is a broader requirement than what bioconda provides. 2) the featurecounts tool requires featurecounts, which is part of the subreads recipe, so in this case featurecounts is a narrow requirement than what is provided in bioconda. In the second case, one could say subreads provides the featurecounts requirement. In the first case one could say the tool requires a narrower requirement than what it specifies, although the bioconda recipes split a single upstream package (ucsc_tools) into multiple recipes that invariable will be updated together, a somewhat strange choice. Right now I'm resolving these problems by manually creating environments (e.g. __package__featurecounts@__unversioned__) that will get picked up by the conda dependency resolver, but I'd like to hear from the list what the best way to support this is? Probably the easiest is keeping the tool requirements definition in line with what the conda recipe is called, but in some cases a mapping table between requirements and conda recipe names could be useful. Peter

2 1

Using operations on multiple datasets
by D K 06 Jul '16

06 Jul '16

Sorry for the crosspost to Biostars, but I didn't get a response, so I thought I'd post here: I just noticed that Galaxy allows creation of lists of datasets by clicking on the "Operations on multiple datasets" in the history panel. How does one use this with a custom tool (e.g. what does one put between the <input> </input> tags). Is the list treated as a text file with a list of each file? I couldn't find any examples or documentation online. I'd appreciate it if someone could point me in the right direction. Thanks!

3 2

Re: [galaxy-dev] Building tool dependencies
by Matthias De Smet 06 Jul '16

06 Jul '16

Hi All! Thanks for chiming in! I’ll give the conda recipe approach a go, this seems to be the easiest way to go, and the most futureproof! Thanks! M > Message: 1 > Date: Tue, 5 Jul 2016 06:56:00 +0000 > From: Matthias De Smet <Matthias.DeSmet(a)UGent.be> > To: "galaxy-dev(a)lists.galaxyproject.org" > <galaxy-dev(a)lists.galaxyproject.org> > Subject: Re: [galaxy-dev] Building tool dependencies > Message-ID: <D70EED58-F05E-4084-9323-1B78C707B95F(a)ugent.be> > Content-Type: text/plain; charset="utf-8" > > >> >> Hi >> >> I’m trying to put together a tool + dependency package to use umi-tools (https://github.com/CGATOxford/UMI-tools) from Galaxy. >> I’m at a loss on how I should organise this. >> >> The package is available from either github or pip. Which one is preferable to use? How do I make sure the package dependencies are installed? I’ve found the <action type="download_by_url”> and <action type=“create_virtualenv”> actions, but I’m not really sure that’s the way to go. I havent found any real resources on how to build package dependencies either. >> >> Any help? Now, approaching it by downloading the package from github with <action type="download_by_url”> and installing it with <action type="shell_command">python setup.py install</action>. This, however, tries to install the package to /usr/bin/ which isn’t desirable (obviously). >> >> Thanks a lot! >> M >> > > > ------------------------------ > > Message: 2 > Date: Tue, 5 Jul 2016 09:33:09 +0200 > From: Peter van Heusden <pvh(a)sanbi.ac.za> > To: galaxy-dev(a)lists.bx.psu.edu > Subject: Re: [galaxy-dev] Building tool dependencies > Message-ID: > <CAK1reXiH9JXQ19gysXdkdr6ynC9aZgAbKfzXAKtFM0M+Tc2-4w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Actually, for recent Galaxy, I'd recommend making a conda recipe for > UMI-tools and contributing it to bioconda: > > https://bioconda.github.io/ > > Then your tool can just depend on this and the conda dependency resolver > (default in Galaxy since 16.01) can find it. Others can comment on the > recent work on conda/toolshed integration. > > Peter > > On 5 July 2016 at 08:56, Matthias De Smet <Matthias.DeSmet(a)ugent.be> wrote: > >> >>> >>> Hi >>> >>> I’m trying to put together a tool + dependency package to use umi-tools ( >> https://github.com/CGATOxford/UMI-tools) from Galaxy. >>> I’m at a loss on how I should organise this. >>> >>> The package is available from either github or pip. Which one is >> preferable to use? How do I make sure the package dependencies are >> installed? I’ve found the <action type="download_by_url”> and <action >> type=“create_virtualenv”> actions, but I’m not really sure that’s the way >> to go. I havent found any real resources on how to build package >> dependencies either. >>> >>> Any help? Now, approaching it by downloading the package from github >> with <action type="download_by_url”> and installing it with <action >> type="shell_command">python setup.py install</action>. This, however, tries >> to install the package to /usr/bin/ which isn’t desirable (obviously). >>> >>> Thanks a lot! >>> M >>> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20160705/0…> > > ------------------------------ > > Message: 3 > Date: Tue, 5 Jul 2016 08:49:54 +0100 > From: Peter Briggs <peter.briggs(a)manchester.ac.uk> > To: galaxy-dev(a)lists.galaxyproject.org > Subject: Re: [galaxy-dev] Building tool dependencies > Message-ID: <577B66A2.60007(a)manchester.ac.uk> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hello Matthias > > From what I've seen there are a number of different recipes used for > installing Python packages via the toolshed; the approach you're using > seems to be pretty common. In these cases the usual 'shell_command' > seems to be: > > python setup.py install --install-lib $INSTALL_DIR/lib/python > --install-scripts $INSTALL_DIR/bin > > which should avoid installing to /usr/bin. > > You'll then need to do something like: > > <action type="set_environment"> > <environment_variable action="prepend_to" > name="PYTHONPATH">$INSTALL_DIR/lib/python</environment_variable> > <environment_variable action="prepend_to" > name="PATH">$INSTALL_DIR/bin</environment_variable> > ... > </action> > > to ensure that the correct environment is set up when the tool is used. > > Aside from this: elsewhere I've seen other approaches including a > "magic" "package" command where the whole download and install is taken > care of in a single tag. For example: > > ... > <action type="setup_python_environment"> > <repository name="package_python_2_7_10" owner="iuc"> > <package name="python" version="2.7.10" /> > </repository> > <package > sha256sum="e01853dfe111f3aea005315573400b7216ddbabbf1f28d482a71217d67ae4f81">https://pypi.python.org/packages/source/p/pandas/pandas-0.16.2.tar.gz</package> > ... > > (see > https://github.com/galaxyproject/tools-iuc/blob/master/packages/package_pyt… > for the full recipe that I've pulled this fragment from.) > > HTH > > Best wishes > > Peter > > On 05/07/16 07:56, Matthias De Smet wrote: >> >>> >>> Hi >>> >>> I’m trying to put together a tool + dependency package to use umi-tools (https://github.com/CGATOxford/UMI-tools) from Galaxy. >>> I’m at a loss on how I should organise this. >>> >>> The package is available from either github or pip. Which one is preferable to use? How do I make sure the package dependencies are installed? I’ve found the <action type="download_by_url”> and <action type=“create_virtualenv”> actions, but I’m not really sure that’s the way to go. I havent found any real resources on how to build package dependencies either. >>> >>> Any help? Now, approaching it by downloading the package from github with <action type="download_by_url”> and installing it with <action type="shell_command">python setup.py install</action>. This, however, tries to install the package to /usr/bin/ which isn’t desirable (obviously). >>> >>> Thanks a lot! >>> M >>> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > -- > Peter Briggs peter.briggs(a)manchester.ac.uk > Bioinformatics Core Facility University of Manchester > B.1083 Michael Smith Bldg Tel: (0161) 2751482

1 0

July Galactic News: GCC2016, GCC2017(!), Admin/Dev Trainings, GAMe 2017!
by Dave Clements 05 Jul '16

05 Jul '16

1 0

Error while running Cuffdiff
by Patel, Ravi Kanu 05 Jul '16

05 Jul '16

Dear Galaxy development team, I am trying to analyze RNAseq data using the following pipeline: FASTQ Trimmer | HISAT | Cufflinks | Cuffmerge | Cuffdiff | CummeRbund I am able to run everything without a problem up to the CuffDiff step. I always get an error message. I sent a bug report and posted on Biostars-Galaxy. Jen replied to my comment on Biostars and recommended I email galaxy-dev. Please see the details of the error message here: https://biostar.usegalaxy.org/p/18331/ I think it may have a problem loading the SQlite database? Even though CuffDiff gives an error message, I was still able to download the data from CuffDiff in tabular format and perform some data analysis using R. However, it would be useful to be able to use cummeRbund to analyze the cuffdiff data. Thank you in advance for all your help. Let me know if there is any other information I can provide to clarify the error, etc. Best Regards, Ravi

1 0

Re: [galaxy-dev] Building tool dependencies
by Matthias De Smet 05 Jul '16

05 Jul '16

> > Hi > > I’m trying to put together a tool + dependency package to use umi-tools (https://github.com/CGATOxford/UMI-tools) from Galaxy. > I’m at a loss on how I should organise this. > > The package is available from either github or pip. Which one is preferable to use? How do I make sure the package dependencies are installed? I’ve found the <action type="download_by_url”> and <action type=“create_virtualenv”> actions, but I’m not really sure that’s the way to go. I havent found any real resources on how to build package dependencies either. > > Any help? Now, approaching it by downloading the package from github with <action type="download_by_url”> and installing it with <action type="shell_command">python setup.py install</action>. This, however, tries to install the package to /usr/bin/ which isn’t desirable (obviously). > > Thanks a lot! > M >

3 2