On Jul 17, 2015 11:00 AM, <galaxy-dev-request@lists.galaxyproject.org> wrote:
Send galaxy-dev mailing list submissions to
galaxy-dev@lists.galaxyproject.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.galaxyproject.org/listinfo/galaxy-dev
or, via email, send a message with subject or body 'help' to
galaxy-dev-request@lists.galaxyproject.org
You can reach the person managing the list at
galaxy-dev-owner@lists.galaxyproject.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of galaxy-dev digest..."
HEY! This is important! If you reply to a thread in a digest, please
1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to the original subject for the thread.
2. Strip out everything else in the digest that is not part of the thread you are responding to.
Why?
1. This will keep the subject meaningful. People will have some idea from the subject line if they should read it or not.
2. Not doing this greatly increases the number of emails that match search queries, but that aren't actually informative.
Today's Topics:
1. Re: yaml import problem on startup (Marius van den Beek)
2. Re: yaml import problem on startup (Marius van den Beek)
3. Re: yaml import problem on startup (Poole, Richard)
4. Re: Data Libraries: Unable to Link to Files When Paths
Contain Spaces (John Chilton)
5. Re: Error sorting or uploading Bam File (John Chilton)
6. Re: CloudMan auto-scaling - Use spot instance for worker
node? (Alexander Vowinkel)
7. Re: Download data by data manager with non generic ftp-proxy
(John Chilton)
8. Re: CloudMan auto-scaling - Use spot instance for worker
node? (Enis Afgan)
----------------------------------------------------------------------
Message: 1
Date: Fri, 17 Jul 2015 12:23:06 +0200
From: Marius van den Beek <m.vandenbeek@gmail.com>
To: "Poole, Richard" <r.poole@ucl.ac.uk>
Cc: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] yaml import problem on startup
Message-ID:
<CACJqqiQo3uti7-doVVg_1L7-rzkwSxh9ccz2NSfJFUrg+kGsOQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Richard,
Well, I do have the same problem now (running bamtools), after applying the
patch that should fix
metadata issue we discussed yesterday.
I'll dig into this.
Marius
On 17 July 2015 at 11:37, Poole, Richard <r.poole@ucl.ac.uk> wrote:
> Hi all,
>
> With the most recent update of Galaxy and also OS X 10.10.4 I am seeing
> this error message when first starting up the server:
>
> /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning:
> Module yaml was already imported from
> /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc,
> but
> /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg
> is being added to sys.path
> self.check_version_conflict()
>
> This then is preventing certain tools from running correctly (such as
> samtools_filter (jjohnson) and they give the same error message when they
> fail:
>
> /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path
> self.check_version_conflict()
>
> Any thoughts?
>
> Thanks,
> Richard
>
> *Richard J Poole PhD*
> Wellcome Trust Fellow
> Department of Cell and Developmental Biology
> University College London
> 21 University Street, London WC1E 6DE
> Office (518 Rockefeller): +44 20 7679 6577 (int. 46577)
> Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150717/37d41e9a/attachment-0001.html>
------------------------------
Message: 2
Date: Fri, 17 Jul 2015 13:00:24 +0200
From: Marius van den Beek <m.vandenbeek@gmail.com>
To: "Poole, Richard" <r.poole@ucl.ac.uk>
Cc: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] yaml import problem on startup
Message-ID:
<CACJqqiQh9F2410hc6H-r+5RCNHSDViwzkg1M+UngYK-mddaWCw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Richard,
Reverting the patch allows normal finishing of jobs, so unfortunately it
seems
we get a double import somewhere in the patch ... i'm suspecting it's
*from* galaxy *import* app
in lib/galaxy/model/custom_types.py, but I don't know how to circumvent
this.
Marius
On 17 July 2015 at 12:23, Marius van den Beek <m.vandenbeek@gmail.com>
wrote:
> Hi Richard,
>
> Well, I do have the same problem now (running bamtools), after applying
> the patch that should fix
> metadata issue we discussed yesterday.
> I'll dig into this.
>
> Marius
>
> On 17 July 2015 at 11:37, Poole, Richard <r.poole@ucl.ac.uk> wrote:
>
>> Hi all,
>>
>> With the most recent update of Galaxy and also OS X 10.10.4 I am seeing
>> this error message when first starting up the server:
>>
>> /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning:
>> Module yaml was already imported from
>> /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc,
>> but
>> /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg
>> is being added to sys.path
>> self.check_version_conflict()
>>
>> This then is preventing certain tools from running correctly (such as
>> samtools_filter (jjohnson) and they give the same error message when they
>> fail:
>>
>> /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path
>> self.check_version_conflict()
>>
>> Any thoughts?
>>
>> Thanks,
>> Richard
>>
>> *Richard J Poole PhD*
>> Wellcome Trust Fellow
>> Department of Cell and Developmental Biology
>> University College London
>> 21 University Street, London WC1E 6DE
>> Office (518 Rockefeller): +44 20 7679 6577 (int. 46577)
>> Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150717/f97ee45b/attachment-0001.html>
------------------------------
Message: 3
Date: Fri, 17 Jul 2015 12:05:20 +0000
From: "Poole, Richard" <r.poole@ucl.ac.uk>
To: Marius van den Beek <m.vandenbeek@gmail.com>
Cc: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] yaml import problem on startup
Message-ID: <D6388E22-7088-4E9B-85A5-6EC26350D596@live.ucl.ac.uk>
Content-Type: text/plain; charset="utf-8"
Hi Marius,
I can also confirm that switching branches to revert the patch allows normal finishing of jobs. So its clearly related to the patches we applied yesterday to fix the metadata issue (see [galaxy-dev] Slow responses viewing histories).
However, I have noticed that even on an older non-patched branch of 15.05 I am still getting the same UserWarning about double yaml import upon Galaxy startup. So perhaps something about the metadata patches is somehow turning this warning into a tool fail?
Rich
On 17 Jul 2015, at 12:00, Marius van den Beek <m.vandenbeek@gmail.com<mailto:m.vandenbeek@gmail.com>> wrote:
Hi Richard,
Reverting the patch allows normal finishing of jobs, so unfortunately it seems
we get a double import somewhere in the patch ... i'm suspecting it's
from galaxy import app
in lib/galaxy/model/custom_types.py, but I don't know how to circumvent this.
Marius
On 17 July 2015 at 12:23, Marius van den Beek <m.vandenbeek@gmail.com<mailto:m.vandenbeek@gmail.com>> wrote:
Hi Richard,
Well, I do have the same problem now (running bamtools), after applying the patch that should fix
metadata issue we discussed yesterday.
I'll dig into this.
Marius
On 17 July 2015 at 11:37, Poole, Richard <r.poole@ucl.ac.uk<mailto:r.poole@ucl.ac.uk>> wrote:
Hi all,
With the most recent update of Galaxy and also OS X 10.10.4 I am seeing this error message when first starting up the server:
/Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path
self.check_version_conflict()
This then is preventing certain tools from running correctly (such as samtools_filter (jjohnson) and they give the same error message when they fail:
/Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path
self.check_version_conflict()
Any thoughts?
Thanks,
Richard
Richard J Poole PhD
Wellcome Trust Fellow
Department of Cell and Developmental Biology
University College London
21 University Street, London WC1E 6DE
Office (518 Rockefeller): +44 20 7679 6577<tel:%2B44%C2%A020%207679%206577> (int. 46577)
Lab (529 Rockefeller): +44 20 7679 6133<tel:%2B44%2020%C2%A07679%206133> (int. 46133)
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150717/2f5c646d/attachment-0001.html>
------------------------------
Message: 4
Date: Fri, 17 Jul 2015 15:52:22 +0100
From: John Chilton <jmchilton@gmail.com>
To: Joseph Brent Greer <joseph.greer@northwestern.edu>
Cc: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] Data Libraries: Unable to Link to Files When
Paths Contain Spaces
Message-ID:
<CANwbokd0JcB_NcVZPm9tbXpJhCi+8x8daM8Hk5wddJZe2WZedw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
This is a good news, bad news sort of thing. I have opened a PR to fix
the actual uploads here -
https://github.com/galaxyproject/galaxy/pull/474. But I think many
tools don't properly quote file paths with names in them - so I
suspect you will find a lot of tool bugs using these files.
Thanks for the bug report.
-John
On Mon, Jul 6, 2015 at 10:11 PM, Joseph Brent Greer
<joseph.greer@northwestern.edu> wrote:
>
>
> Hi All,
>
>
>
> We’re having problems adding files to data libraries.
>
>
>
> We’d like to create links to files instead of copying them into Galaxy. We
> can create links if there aren’t spaces in the paths. Also, we can copy
> files into libraries even if there are spaces in the paths.
>
>
>
> Unfortunately, we cannot create links to files when their paths contain
> spaces.
>
>
>
> The job standard error and miscellaneous information are:
>
>
>
> Miscellaneous information:uploaded RAW file Traceback (most recent call
> last): File "/share/galaxy/working/002/2615/set_metadata_hnsR90.py", line 1,
> in <module> from galaxy_ext.metadata.set_metadata import set_metadata;
> set_metadata() File "/home/galaxy/Galaxy/galaxy-dist
>
> Job Standard Error
>
> Traceback (most recent call last):
>
> File "/share/galaxy/working/002/2615/set_metadata_hnsR90.py", line 1, in
> <module>
>
> from galaxy_ext.metadata.set_metadata import set_metadata;
> set_metadata()
>
> File
> "/home/galaxy/Galaxy/galaxy-dist/lib/galaxy_ext/metadata/set_metadata.py",
> line 80, in set_metadata
>
> filename_out = fields.pop( 0 )
>
> IndexError: pop from empty list
>
> error
>
>
>
> We’ve set allow_library_path_paste = True in galaxy.ini.
>
>
>
> We’ve also tried adding “” or ‘’ around the path and get the following error
> message:
>
>
>
> Invalid paths: “home/galaxy/Galaxy/galaxy-dist/”/share/projects/2014 Test””
>
>
>
> Any help or suggestions would be great.
>
>
>
> Thanks,
>
>
>
> Joe Greer
>
> Proteomics Center of Excellence
>
> Northwestern University
>
>
>
>
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
------------------------------
Message: 5
Date: Fri, 17 Jul 2015 16:08:19 +0100
From: John Chilton <jmchilton@gmail.com>
To: julie dubois <dubjulie@gmail.com>
Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu>
Subject: Re: [galaxy-dev] Error sorting or uploading Bam File
Message-ID:
<CANwbokeLSgB0uQpxn3kXWpvXLzKijiNBwmNvaCbJwsKoHr8Q-Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
I'm not the right person to respond to this but since no one else has
I will explain my limited (probably incorrect) understanding of the
problem and what needs to be done. I believe Galaxy assumes all bam
files are coordinated sorted by default - so Galaxy's bam datatype
would be better thought of as a coordinate-sorted bam. Certain people
do not want to allow non-coordinate sorted bam files because these
would be allowed by Galaxy as valid as inputs to tools that expect
sorted bams - and things would break. For the same reason I imagine
these people would argue the bai file needs to be generated because
certain tools will depend on its existence.
I believe the proposed solution is to add an unsorted bam datatype to
Galaxy. I don't know if there is any work in progress on this but it
seems to be needed by many researchers.
Hope this clarifies things somewhat,
-John
On Mon, Jun 29, 2015 at 11:06 AM, julie dubois <dubjulie@gmail.com> wrote:
> Hi all,
>
> My galaxy is the last update and I work with samtools 1.2.
> I have a bug with the sorting and the upload of bam file.
> This bug appears only when I try to upload bam sorted by name or when
> I try to sort by name.
> When I work with bam sorted by coordinate, there is no problem.
>
> So, when I try to sort a bam file by name, this error appears under
> the "bug icon" :
>
> Tool execution generated the following error message:
> [bam_sort_core] merging from 20 files...
> *** Error in `python': double free or corruption (!prev): 0x0000000002897990 ***
> Aborted (core dumped)
>
> The tool produced the following additional output:
>
> [E::hts_idx_push] chromosome blocks not continuous
> Galaxy attempted to build the BAM index with samtools 1.0+ but failed:
> [Errno 2] No such file or directory:
> '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai'
>
> And the same type of error appears when I try to upload bam sorted by name.
>
>
> What I understand is that the bam can be sorted properly (The command
> line to sort is ok in a terminal with no bug and I can follow this
> step in the job_working_directory during the run) but Galaxy seems to
> fail to create a bai.
>
> For our analysis pipeline, I think the bai is not crucial, is it
> possible to avoid this creation of bai during uploading and sorting of
> bam file by name ?
>
> Is there another problem than this creation of bai ?
>
> Thanks.
>
> Julie
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
------------------------------
Message: 6
Date: Fri, 17 Jul 2015 10:27:29 -0500
From: Alexander Vowinkel <vowinkel.alexander@gmail.com>
To: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] CloudMan auto-scaling - Use spot instance
for worker node?
Message-ID:
<CANPh1r_Hvb=M1J=xkw5pUA-_a0CyOCy9ckA_o1nhGiQM4RMpyQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I am referring to this:
http://dev.list.galaxyproject.org/CloudMan-auto-scaling-Use-spot-instance-for-worker-node-tp4665813p4665835.html
I would give an upvote for spot instances for auto-scaling.
Have there been more ideas about this since?
Best,
Alexander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150717/17ab92c6/attachment-0001.html>
------------------------------
Message: 7
Date: Fri, 17 Jul 2015 16:33:06 +0100
From: John Chilton <jmchilton@gmail.com>
To: Tobias Zimmermann
<Tobias.Zimmermann@computational.bio.uni-giessen.de>
Cc: "galaxy-dev@lists.galaxyproject.org"
<galaxy-dev@lists.galaxyproject.org>
Subject: Re: [galaxy-dev] Download data by data manager with non
generic ftp-proxy
Message-ID:
<CANwbokevw1PYDUkfvvRt21zQDuYHRYYTmCiVhXmfqNh32XO7Nw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Yeah - it seems like ftplib doesn't support proxies and that is what
is used by this script:
https://github.com/galaxyproject/tools-devteam/blob/master/data_managers/data_manager_fetch_genome_all_fasta/data_manager/data_manager_fetch_genome_all_fasta.py
Some relevant discussion here:
http://stackoverflow.com/questions/1293518/proxies-in-python-ftp-application
I might suggest reworking that script to use urllib2 which should
support ftp and proxies. Not sure if there is some feature in ftplib
though that is required for the data manager.
-John
On Mon, Jun 15, 2015 at 11:12 AM, Tobias Zimmermann
<Tobias.Zimmermann@computational.bio.uni-giessen.de> wrote:
> Dear All,
> I have some problems with the download of genome files by the data manager.
> I tried to download some genome files from the UCSC server with the data
> manager of galaxy ( name: data_manager_fetch_genome_all_fasta, revision:
> cca219f2b212), but it fails because of a timeout. Our galaxy server is in an
> internal network without connection to the Internet. To access the Internet
> we are required to go trough a http-proxy before any web-services can be
> accessed.
>
> The file
> /shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/data_manager_fetch_genome_all_fasta/data_manager/data_manager_fetch_genome_all_fasta.py
> manages the download of the file. Line 183 shows the problem. Here the FTP
> item from the python ftplib module tries to establishes the connection to
> the UCSC ftp server. This connection fails. So here is my question:
>
> How can i download data by the data manager with a http-proxy?
>
>
> Regards
> Tobias
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
------------------------------
Message: 8
Date: Fri, 17 Jul 2015 11:41:53 -0400
From: Enis Afgan <enis.afgan@irb.hr>
To: Alexander Vowinkel <vowinkel.alexander@gmail.com>
Cc: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] CloudMan auto-scaling - Use spot instance
for worker node?
Message-ID:
<CACJ_Meq4LO9nBJnyNyRvq8jgkN0T-vpPha2YwCi0YTWrZL0DSg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I've created a trello card for this, hence making it adding it to the
actual todo list: https://trello.com/c/cqEwunvl
Not sure when I'll get to this but it'll get done.
Thanks.
On Fri, Jul 17, 2015 at 11:27 AM, Alexander Vowinkel <
vowinkel.alexander@gmail.com> wrote:
> Hi,
>
> I am referring to this:
>
> http://dev.list.galaxyproject.org/CloudMan-auto-scaling-Use-spot-instance-for-worker-node-tp4665813p4665835.html
>
> I would give an upvote for spot instances for auto-scaling.
>
> Have there been more ideas about this since?
>
> Best,
> Alexander
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.galaxyproject.org/pipermail/galaxy-dev/attachments/20150717/ba774026/attachment-0001.html>
------------------------------
Subject: Digest Footer
_______________________________________________
galaxy-dev mailing list
galaxy-dev@lists.galaxyproject.org
https://lists.galaxyproject.org/listinfo/galaxy-dev
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
------------------------------
End of galaxy-dev Digest, Vol 109, Issue 17
*******************************************