data sharing without duplication?
by Fernandez Edgar
Good morning Gents,
I hope everyone is well.
I was wondering if it's possible to share data on Galaxy without all the users downloading the data to their history.
Let me explain, teachers will be sharing data with their students.
And the students will need to work with that specific data.
However, I would like them to be able to read that data without replicating it on their history.
Is that possible?
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
* Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
5 years, 7 months
Nothing being tested on Test and main Tool Shed?
by Peter Cock
Hello all,
I am currently hoping to review the automated test results for
some repositories which I have recently updated, in one case
for dependency handling, the other functional changes:
https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer
https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus
These have not yet been tested. On further investigation
of a sample of my other tools, it appears none of them have
been tested on the Test Tool Shed since 2014-09-15, e.g.
https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_rename
https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats
https://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3
https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod
Similarly, some of my tools on the Main Tool Shed appear
not to have been tested since 2014-09-21, e.g.
https://toolshed.g2.bx.psu.edu/view/peterjc/seq_rename
https://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3
https://toolshed.g2.bx.psu.edu/view/peterjc/clinod
or 2014-10-27,
https://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_idxstats
Is there a known problem with the automated tool testing
(previously every second night) on the Tool Sheds?
Or have you had to further reduce the testing cycle?
Testing less frequently seems fine, say fortnightly, if this can
be supplemented by testing updated tools everynight. That
would give Tool Authors prompt feedback on their updates,
but also catch regressions where changes in Galaxy break
a previously working tool.
Regards,
Peter
5 years, 7 months
SGE integration
by Ryan G
I'm trying to get my instance of Galaxy working with Sun Grid Engine. The
page https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster is not
100% clear on how to do this but I here's what my job_conf.xml looks like:
<?xml version="1.0"?>
<job_conf>
<plugins>
<plugin id="local" type="runner"
load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
<plugin id="sge" type="runner"
load="galaxy.jobs.runners.drmaa:DRMAAJobRunner">
<!-- Override the $DRMAA_LIBRARY_PATH environment
variable -->
<param
id="drmaa_library_path">/apps/sys/sge/ge2011.11/lib/linux-x64/libdrmaa.so</param>
</plugin>
</plugins>
<handlers>
<handler id="sge"/>
</handlers>
<destinations default="sge">
<destination id="local" runner="local" />
<destination id="sge" runner="sge">
</destination>
</destinations>
</job_conf>
I tried to submit a job via Galaxy and at first the log files indicated the
shell script got created in the job_working_dir and I was a job ID, but
when the job completed, Galaxy didn't know about it. Now, I have no jobs
runnings and I can't submit anything via Galaxy. It doesn't appear to
submit them to SGE or even create the directory for the job. Pastor.log
doesn't show anything either. So, I guess I could use a little help with
getting Galaxy running with SGE. I have a simple set up so nothing fancy.
Any ideas where to get started?
5 years, 7 months
Re: [galaxy-dev] NFS configuration questions + NFS /etc/fstab configuration issue with AMI issue
by Dannon Baker
Hey Scott,
Glad you figured out the other two issues. Which file exactly are you
editing that is getting overwritten? And when does it get overwritten? (on
reboot, randomly, cluster terminate/start, etc?)
-Dannon
On Sun Jan 25 2015 at 12:20:12 PM Scott Jeschonek <scottj(a)averesystems.com>
wrote:
> Revising all of the below…
>
> I figured out the configuration location and file system issues.
>
> The only issue I have now is that my configured universe_wsgi.ini file is
> being overwritten with defaults. I modified it in the s3 bucket and
> locally but to no avail. Any ideas?
>
>
>
> On Jan 24, 2015, at 12:17 PM, Scott Jeschonek <scottj(a)averesystems.com>
> wrote:
>
> Hi,
>
> I’m setting up a ‘local’ Galaxy server in AWS and had a few questions that
> I can’t seem to answer.
>
> 1. *Directory configuration in universe_wsgi.ini* — I want to use an NFS
> from another server. The purpose would be to deposit results there. I may
> also want to point to another NFS server as a centralized reference / index
> file system. I tried changing the path for the Indices setting and
> restarted Galaxy, but CloudMan is showing the file system as “error”.
> Basically I am trying to wrap my head around the steps to re-pointing to
> NFS mounts (is it just change the paths in universe_wsgi.ini then restart
> everything?)
>
> 2. *CloudMan AMI and fstab* — I’m using the latest AMI Cloudman instance
> and it seems to have an issue with NFS configurations in the fstab. I am
> not quite sure what is going on, the entry is correct. I’m able to
> manually mount the filesystems as well so the paths are correct. Is there
> something specific I need to change on that image?
>
> Thanks in advance for any assistance!
>
> Scott Jeschonek
> Avere Systems
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
5 years, 7 months
Data manager .loc file not installing in correct place
by Jeremy Liu
Galaxy Devs,
I have a (semi) working data manager and I'm stuck on what exactly is
going wrong. Most of it appears to be working properly, the intended
file (pouya_test_motifs.bed.bgz) is being downloaded to the right
directory (/galaxy-dist/tool-data/motifs). However, the .loc file that
describes the data table isn't being set to the right place. It should
be going to (/galaxy-dist/tool-data/motif_databases.loc) but it ends
up in (./database/tmp/tmp-toolshed-gmfcrTJ8T4R/motif_databases.loc). I
think it's because /galaxy-dist/tool_data_conf.xml isn't being
updated.
In addition, I have been working off of the example data manager
data_manager_fetch_genome_all_fasta. What is the usage of
tool_data_table_conf.xml.sample? Does it get automatically merged with
tool_data_table_conf.xml in the galaxy root directory?
For reference, the tools is "region_motif_data_manager" in the
testtoolshed. Has anyone else run into this problem? Thanks!
Jeremy Liu
jeremy.liu(a)yale.edu
PO Box 207077
New Haven, CT 06520-7077
352.871.1258
5 years, 7 months
Galaxy XML from python's argparse
by Eric Rasche
Howdy devs,
I put together a library recently, and since it seems to be functional I
thought I'd share with the rest of -dev in case it's of interest to anyone.
If anyone has feedback, bugs/issues, or PRs, I'd be happy to receive them!
https://github.com/erasche/gxargparse
So, what is it?
gxargparse is a drop in replacement for argparse which can generate Galaxy
Tool XML on demand.
When I say "drop in" replacement, I mean it. Through some python magic, as
soon as you `pip install gxargparse`, your argparse will (maybe*) be
transparently wrapped by gxargparse, and you'll have the
--generate_galaxy_xml flag available, which will generate Galaxy Tool XML.
This means *no code changes required*, and free tool XML generation.
However, beware that it is "free tool XML"; you will likely need to make
some manual corrections to it before publishing tools (repeat labels for
instance). However, if you're converting an argparse tool with hundreds of
arguments for use in Galaxy, this could save you a lot of initial manual
work.
* I say *maybe* because it depends a bit on your python module load order,
which is something completely outside of my control. The package comes with
a command line tool <https://github.com/erasche/gxargparse#it-doesnt-work>
which spits out a path you can stick in PYTHONPATH to fix this issue.
Where to get it?
Now available on pypi (gxargparse
<https://mail.yandex.ru/re.jsx?h=a,u3kuzrWUkilmdvbgYLueaQ&l=aHR0cHM6Ly9weX...>)
and github (erasche/gxargparse
<https://mail.yandex.ru/re.jsx?h=a,JjiBfNrf6Qgagy0RTH93mw&l=aHR0cHM6Ly9naX...>).
I *strongly* recommend against installing it system wide, as any bugs in it
could render all argparse based python tools broken on your system. It's
much more reasonable to use it in a virtualenv.
(gx
Known Problems
- argument_groups are not dealt with specially
- prefix_chars and other lesser used features are not (yet) supported
- anything with a repeat is a bit of a hack
- no translation from argparse to conditionals/which yet figured out.
Bugs reports/suggestions are welcome
<https://github.com/erasche/gxargparse/issues/>!
Cheers,
Eric
--
Eric Rasche
Programmer II
Center for Phage Technology
Rm 312A, BioBio
Texas A&M University
College Station, TX 77843
404-692-2048
esr(a)tamu.edu
rasche.eric(a)yandex.ru
5 years, 7 months
Re: [galaxy-dev] CloudMan + Ansible + AWS
by Brad Chapman
Enis, John and all;
I spotted ansible-cloudman on GitHub today which reminded me I've been
meaning to write about the approach we setup late last year to run bcbio
on AWS. It uses elasticluster (https://github.com/gc3-uzh-ch/elasticluster)
which has the advantage of being all Ansible scripts and bootstrapping
from standard images -- so no more making AMIs. It also uses SLURM
instead of SGE, which is a nice change.
We wrote an interface that automates all of the stuff you need to setup
on AWS: IAM users, VPCs and what not. It is a pretty streamlined process
from the command line, including specifying the cluster size and
stopping/starting it:
https://bcbio-nextgen.readthedocs.org/en/latest/contents/cloud.html#aws-s...
I also wrote up some benchmarking work using to give an idea of using
it in practice:
http://bcb.io/2014/12/19/awsbench/
All of the code is here:
https://github.com/chapmanb/bcbio-nextgen-vm
As always, happy to overlap/share with whatever y'all decide to
do. We could make bcbio specific stuff optional as needed, although it
is pretty lightweight -- just the driver scripts and a Docker image of
bcbio. It's basically a ready to use cluster with this little extra
added so hopefully could be useful for future plans with CloudMan.
Hope this is useful,
Brad
5 years, 7 months
data collections - workflow - bug?
by Torsten Houwaart
Hello Galaxy Devs,
I was using data collections (for the first time) for a new workflow of ours and I ran into this problem. There was no complaint by the workflow-editor and I could start the workflow but then <see below> happened.
If you need more information about the workflow or otherwise let me know.
Best,
Torsten H.
job traceback:
Traceback (most recent call last):
File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 565, in finish_job
job_state.job_wrapper.finish( stdout, stderr, exit_code )
File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1250, in finish
self.sa_session.flush()
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py", line 114, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", line 1718, in flush
self._flush(objects)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py", line 1789, in _flush
flush_context.execute()
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute
rec.execute(self)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute
uow
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj
mapper, table, update)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements
execute(statement, params)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 1449, in execute
params)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
context)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
context)
File "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py", line 331, in do_execute
cursor.execute(statement, parameters)
DBAPIError: (TransactionRollbackError) deadlock detected
DETAIL: Process 3144 waits for ShareLock on transaction 2517124; blocked by process 3143.
Process 3143 waits for ShareLock on transaction 2517123; blocked by process 3144.
HINT: See server log for query details.
'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE workflow_invocation.id = %(workflow_invocation_id)s' {'update_time': datetime.datetime(2015, 1, 26, 14, 20, 4, 155440), 'workflow_invocation_id': 5454}
5 years, 7 months
Tool development - Selecting a single item from input dataset.
by Vimalkumar Velayudhan
Hi all,
I am trying to create a select box with the possibility of selecting only a
single item from the input dataset (figure 1). This works fine but the
option for selecting multiple files is still visible (figure 2). The
multiple="false" attribute has no effect.
Figure: http://i.imgur.com/oJVFCoF.png
I have the following in my XML.
<param format="tabular" name="ribo_files" type="data"
label="Select Ribo-Seq alignment file" multiple="false" >
</param>
Any suggestions?
galaxy-dist revision 5f4c13d622b8
Regards,
Vimalkumar Velayudhan
5 years, 7 months
