- galaxy-dev - lists.galaxyproject.org

Tophat run stops
by McCully, Dwayne (NIH/NIAMS) [C] 20 Jul '15

20 Jul '15

When running a Tophat job for over a hour, I get the following message and the job stops. Would anyone know why and is there a way to stop Galaxy from cleaning up the job working directory so that I can check the error log files? Dwayne galaxy.jobs.runners.local DEBUG 2015-07-14 16:54:10,478 execution finished: /data/galaxy/galaxy_prod/database/job_working_directory/000/26/galaxy_26.sh galaxy.jobs.output_checker INFO 2015-07-14 16:54:10,595 Job 26: Log: tool progress galaxy.jobs.output_checker INFO 2015-07-14 16:54:10,595 Job 26: Log: tool progress galaxy.datatypes.metadata DEBUG 2015-07-14 16:54:10,675 setting metadata externally failed for HistoryDatasetAssociation 32: [Errno 2] No such file or directory galaxy.jobs.runners.local ERROR 2015-07-14 16:54:10,754 Job wrapper finish method failed Traceback (most recent call last): File "/data/galaxy/galaxy_prod/lib/galaxy/jobs/runners/local.py", line 129, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/data/galaxy/galaxy_prod/lib/galaxy/jobs/__init__.py", line 1126, in finish dataset.datatype.set_meta( dataset, overwrite=False ) # call datatype.set_meta directly for the initial set_meta call during dataset creation File "/data/galaxy/galaxy_prod/lib/galaxy/datatypes/binary.py", line 268, in set_meta proc = subprocess.Popen( args=command, stderr=open( stderr_name, 'wb' ) ) File "/opt/python/2.7.9/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/opt/python/2.7.9/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

1 0

Re: [galaxy-dev] Error sorting or uploading Bam File
by julie dubois 20 Jul '15

20 Jul '15

Hi, Thanks for your answers. I understand that permit to load or create "unsorted bam" (or sorted by name) can create bug with lot of tools and it's difficult to restrict this manipulation just for few tools. But I don't understand one thing. Why can I download from the toolshed a tool which permit me to sort bam file by name (from devteam) while it's not supported by galaxy ?! Other question. Anyone can help me to write a new entry for an ‘unsorted bam’ (or namesorted) file type in the file which describe the format file uploaded ? Or do you think that I can run this sorting during the processing of the tool which require this type of sorting : so run the tool of interest and internally to the process run the sorting by name before treating the bam file. In this way do you think that Galaxy will not create an index during the process ? Thanks. Julie 2015-07-17 21:51 GMT+02:00 Frederik Coppens <frcop(a)psb.vib-ugent.be>: > We also encountered this issue. > As I understood it: if Galaxy gets a BAM file (either through upload or > generated by a tool), it will generate metadata and indexing is part of > that. This only works if the BAM file is coordinate sorted, otherwise it > fails (although there is nothing wrong with the BAM file). > > As a lot of tools (and visualisation) require sorted BAM files, this avoids > a lot of problems for users (but does create this issue). > > An ‘unsorted bam’ (or namesorted) file type could be a solution although > this would cause the illogic situation where a ‘bam' is hierarchically not > above ‘unsorted bam’ as is the case for e.g. the types of fastq. I guess > changing what is now ‘bam’ to ‘sorted bam’ will cause problems for a lot of > users... > > Frederik > > On 17 Jul 2015, at 21:29, Alexander Vowinkel <vowinkel.alexander(a)gmail.com> > wrote: > > Indeed, this would be the right approach. > But so far the current strategy is quite imputed also in tools. > E.g. BWA from devteam automatically sorts the bam afterwards - wanted or > not. > > 2015-07-17 11:59 GMT-05:00 Ryan G <ngsbioinformatics(a)gmail.com>: >> >> I would think the correct approach would be to allow any bam file and if a >> tool requires a coordinate sorted bam and one is not provided it would make >> sense for the tool to fail with an error. >> >> Does galaxy itself require a coordinate sorted bam or this this so the >> majority of tools can function properly? >> >> Sent from my iPhone >> >> > On Jul 17, 2015, at 11:08 AM, John Chilton <jmchilton(a)gmail.com> wrote: >> > >> > I'm not the right person to respond to this but since no one else has >> > I will explain my limited (probably incorrect) understanding of the >> > problem and what needs to be done. I believe Galaxy assumes all bam >> > files are coordinated sorted by default - so Galaxy's bam datatype >> > would be better thought of as a coordinate-sorted bam. Certain people >> > do not want to allow non-coordinate sorted bam files because these >> > would be allowed by Galaxy as valid as inputs to tools that expect >> > sorted bams - and things would break. For the same reason I imagine >> > these people would argue the bai file needs to be generated because >> > certain tools will depend on its existence. >> > >> > I believe the proposed solution is to add an unsorted bam datatype to >> > Galaxy. I don't know if there is any work in progress on this but it >> > seems to be needed by many researchers. >> > >> > Hope this clarifies things somewhat, >> > -John >> > >> > >> >> On Mon, Jun 29, 2015 at 11:06 AM, julie dubois <dubjulie(a)gmail.com> >> >> wrote: >> >> Hi all, >> >> >> >> My galaxy is the last update and I work with samtools 1.2. >> >> I have a bug with the sorting and the upload of bam file. >> >> This bug appears only when I try to upload bam sorted by name or when >> >> I try to sort by name. >> >> When I work with bam sorted by coordinate, there is no problem. >> >> >> >> So, when I try to sort a bam file by name, this error appears under >> >> the "bug icon" : >> >> >> >> Tool execution generated the following error message: >> >> [bam_sort_core] merging from 20 files... >> >> *** Error in `python': double free or corruption (!prev): >> >> 0x0000000002897990 *** >> >> Aborted (core dumped) >> >> >> >> The tool produced the following additional output: >> >> >> >> [E::hts_idx_push] chromosome blocks not continuous >> >> Galaxy attempted to build the BAM index with samtools 1.0+ but failed: >> >> [Errno 2] No such file or directory: >> >> >> >> '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai' >> >> >> >> And the same type of error appears when I try to upload bam sorted by >> >> name. >> >> >> >> >> >> What I understand is that the bam can be sorted properly (The command >> >> line to sort is ok in a terminal with no bug and I can follow this >> >> step in the job_working_directory during the run) but Galaxy seems to >> >> fail to create a bai. >> >> >> >> For our analysis pipeline, I think the bai is not crucial, is it >> >> possible to avoid this creation of bai during uploading and sorting of >> >> bam file by name ? >> >> >> >> Is there another problem than this creation of bai ? >> >> >> >> Thanks. >> >> >> >> Julie >> >> ___________________________________________________________ >> >> Please keep all replies on the list by using "reply all" >> >> in your mail client. To manage your subscriptions to this >> >> and other Galaxy lists, please use the interface at: >> >> https://lists.galaxyproject.org/ >> >> >> >> To search Galaxy mailing lists use the unified search at: >> >> http://galaxyproject.org/search/mailinglists/ >> > ___________________________________________________________ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please use the interface at: >> > https://lists.galaxyproject.org/ >> > >> > To search Galaxy mailing lists use the unified search at: >> > http://galaxyproject.org/search/mailinglists/ >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > > ================================================================== > Frederik Coppens, Ph.D. > Project Leader Applied Bioinformatics & Biostatistics > IT manager > > Tel: 32 (0)9 3313692 Fax:32 (0)9 3313809 > VIB DEPARTMENT OF PLANT SYSTEMS BIOLOGY, UGent > Technologiepark 927, 9052 Gent, BELGIUM > > frcop(a)psb.ugent.be http://www.psb.ugent.be/abb > ================================================================== >

1 0

./run.sh segfault
by Martin Vickers 20 Jul '15

20 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi All, I have a weird issue that's just cropped up. After a new install of galaxy (checked out on Monday from github) on a ubuntu vm, using postgres rather than sqlite as well as a few other production recommendations, I started playing around with the Data Libraries functionality. I linked a bunch of fastq.gz files into galaxy (around 150 in total) and everything was working fine. I went home and the next day, it was down. I tried to start it up as usual (using an init.d script), it worked for less than a minute and then disappeared again. So I tried running it as the galaxy user using ./run.sh and I get a seg fault; Starting server in PID 23173. serving on http://144.124.110.39:8080 Segmentation fault Tried again with strace Starting server in PID 23552. serving on http://144.124.110.39:8080 [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 23552 - --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=23552, si_status=SIGKILL, si_utime=1590, si_stime=1930} --- rt_sigreturn() = 23552 write(2, "Killed\n", 7Killed ) = 7 read(10, "", 8192) = 0 exit_group(137) = ? +++ exited with 137 +++ I can't see anything odd in the log file and I've turned debugging on in galaxy.ini. I'm at a bit of a loss. Does anyone know what might be causing it? Cheers, - -- Dr. Martin Vickers Data Manager/HPC Systems Administrator Institute of Biological, Environmental and Rural Sciences IBERS New Building Aberystwyth University w: http://www.martin-vickers.co.uk/ e: mjv08(a)aber.ac.uk t: 01970 62 2807 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iQEcBAEBAgAGBQJVp80YAAoJEHa0a8GkKQgIGlIH/1VfAPbs/5ApDBdyoOV5qf1y oCOv93IojARyfI0ksSjF8NRNzw5fNp1R8AzZzomaR3SOUkBuZutre600sy0azTZw E6gjxtMuvaMyEsOTXtToVarVJT0wTG8+5DJRIYLxtYZm7kvbZK0WuzrN2zDT6663 Rnm7zI/zBpTAyp6uXwgmz0x5gpH6KFwRcEHEbU3JWy6nj1zithJShwYPlBuhT5IB OaPwOKflcZpZ8NBTEGsh038JrkU+eE50a9aEjQ2m/DpfM/TN9ujgEFm1dyy/iQS7 ewwQUpWJDkA/u0ZX602dsNdV2LvGuKVVMEHiQ25zaUQZ/iGTwKBQsFM2LlDybgA= =jzYG -----END PGP SIGNATURE-----

3 4

yaml import problem on startup
by Poole, Richard 20 Jul '15

20 Jul '15

Hi all, With the most recent update of Galaxy and also OS X 10.10.4 I am seeing this error message when first starting up the server: /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path self.check_version_conflict() This then is preventing certain tools from running correctly (such as samtools_filter (jjohnson) and they give the same error message when they fail: /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: Module yaml was already imported from /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, but /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg is being added to sys.path self.check_version_conflict() Any thoughts? Thanks, Richard Richard J Poole PhD Wellcome Trust Fellow Department of Cell and Developmental Biology University College London 21 University Street, London WC1E 6DE Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133)

2 4

Password-Reset
by Matthias Enders 20 Jul '15

20 Jul '15

Dear Galaxy Community, as we tried to establish the mail functionalities for our Galaxy instance, we managed to send email upon the "I forgot my password" user request. In the Email a Link is provided. Here two errors arise: 1. In the File "galaxy/lib/galaxy/webapps/galaxy/controllers/user.pyc" HTML Tags for this Email are generated, but the Email is send as plain/text, therefore the HTML Tags, as well as the (doubled) Link are visible for the user. 2. In our case, the provided link takes us to the normal user login page, where the user has no option of resetting his/her password. Can you provide me some assistance to tackle the problems? Kind Regards, Matthias Enders

1 0

Error sorting or uploading Bam File
by julie dubois 18 Jul '15

18 Jul '15

Hi all, My galaxy is the last update and I work with samtools 1.2. I have a bug with the sorting and the upload of bam file. This bug appears only when I try to upload bam sorted by name or when I try to sort by name. When I work with bam sorted by coordinate, there is no problem. So, when I try to sort a bam file by name, this error appears under the "bug icon" : Tool execution generated the following error message: [bam_sort_core] merging from 20 files... *** Error in `python': double free or corruption (!prev): 0x0000000002897990 *** Aborted (core dumped) The tool produced the following additional output: [E::hts_idx_push] chromosome blocks not continuous Galaxy attempted to build the BAM index with samtools 1.0+ but failed: [Errno 2] No such file or directory: '/data/galaxy-dist/database/job_working_directory/024/24109/__dataset_48619_metadata_temp_file_MqhjBN.bai' And the same type of error appears when I try to upload bam sorted by name. What I understand is that the bam can be sorted properly (The command line to sort is ok in a terminal with no bug and I can follow this step in the job_working_directory during the run) but Galaxy seems to fail to create a bai. For our analysis pipeline, I think the bai is not crucial, is it possible to avoid this creation of bai during uploading and sorting of bam file by name ? Is there another problem than this creation of bai ? Thanks. Julie

5 4

unsubscribe
by Ricardo Perez 18 Jul '15

18 Jul '15

On Jul 17, 2015 11:00 AM, <galaxy-dev-request(a)lists.galaxyproject.org> wrote: > Send galaxy-dev mailing list submissions to > galaxy-dev(a)lists.galaxyproject.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.galaxyproject.org/listinfo/galaxy-dev > or, via email, send a message with subject or body 'help' to > galaxy-dev-request(a)lists.galaxyproject.org > > You can reach the person managing the list at > galaxy-dev-owner(a)lists.galaxyproject.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of galaxy-dev digest..." > > > HEY! This is important! If you reply to a thread in a digest, please > 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to > the original subject for the thread. > 2. Strip out everything else in the digest that is not part of the thread > you are responding to. > > Why? > 1. This will keep the subject meaningful. People will have some idea from > the subject line if they should read it or not. > 2. Not doing this greatly increases the number of emails that match search > queries, but that aren't actually informative. > > Today's Topics: > > 1. Re: yaml import problem on startup (Marius van den Beek) > 2. Re: yaml import problem on startup (Marius van den Beek) > 3. Re: yaml import problem on startup (Poole, Richard) > 4. Re: Data Libraries: Unable to Link to Files When Paths > Contain Spaces (John Chilton) > 5. Re: Error sorting or uploading Bam File (John Chilton) > 6. Re: CloudMan auto-scaling - Use spot instance for worker > node? (Alexander Vowinkel) > 7. Re: Download data by data manager with non generic ftp-proxy > (John Chilton) > 8. Re: CloudMan auto-scaling - Use spot instance for worker > node? (Enis Afgan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 17 Jul 2015 12:23:06 +0200 > From: Marius van den Beek <m.vandenbeek(a)gmail.com> > To: "Poole, Richard" <r.poole(a)ucl.ac.uk> > Cc: "galaxy-dev(a)lists.galaxyproject.org" > <galaxy-dev(a)lists.galaxyproject.org> > Subject: Re: [galaxy-dev] yaml import problem on startup > Message-ID: > < > CACJqqiQo3uti7-doVVg_1L7-rzkwSxh9ccz2NSfJFUrg+kGsOQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Richard, > > Well, I do have the same problem now (running bamtools), after applying the > patch that should fix > metadata issue we discussed yesterday. > I'll dig into this. > > Marius > > On 17 July 2015 at 11:37, Poole, Richard <r.poole(a)ucl.ac.uk> wrote: > > > Hi all, > > > > With the most recent update of Galaxy and also OS X 10.10.4 I am seeing > > this error message when first starting up the server: > > > > /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: > > Module yaml was already imported from > > > /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, > > but > > > /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg > > is being added to sys.path > > self.check_version_conflict() > > > > This then is preventing certain tools from running correctly (such as > > samtools_filter (jjohnson) and they give the same error message when they > > fail: > > > > /Users/galaxy/galaxy-dist/lib/galaxy/__init__.py:63: UserWarning: > Module yaml was already imported from > /Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.10-intel.egg/yaml/__init__.pyc, > but > /Users/galaxy/galaxy-dist/eggs/PyYAML-3.10-py2.7-macosx-10.6-intel-ucs2.egg > is being added to sys.path > > self.check_version_conflict() > > > > Any thoughts? > > > > Thanks, > > Richard > > > > *Richard J Poole PhD* > > Wellcome Trust Fellow > > Department of Cell and Developmental Biology > > University College London > > 21 University Street, London WC1E 6DE > > Office (518 Rockefeller): +44 20 7679 6577 (int. 46577) > > Lab (529 Rockefeller): +44 20 7679 6133 (int. 46133) > > > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > >

1 0

Re: [galaxy-dev] CloudMan auto-scaling - Use spot instance for worker node?
by Alexander Vowinkel 17 Jul '15

17 Jul '15

Hi, I am referring to this: http://dev.list.galaxyproject.org/CloudMan-auto-scaling-Use-spot-instance-f… I would give an upvote for spot instances for auto-scaling. Have there been more ideas about this since? Best, Alexander

2 1

Download data by data manager with non generic ftp-proxy
by Tobias Zimmermann 17 Jul '15

17 Jul '15

Dear All, I have some problems with the download of genome files by the data manager. I tried to download some genome files from the UCSC server with the data manager of galaxy ( name: data_manager_fetch_genome_all_fasta, revision: cca219f2b212), but it fails because of a timeout. Our galaxy server is in an internal network without connection to the Internet. To access the Internet we are required to go trough a http-proxy before any web-services can be accessed. The file /shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_all_fasta/cca219f2b212/data_manager_fetch_genome_all_fasta/data_manager/data_manager_fetch_genome_all_fasta.py manages the download of the file. Line 183 shows the problem. Here the FTP item from the python ftplib module tries to establishes the connection to the UCSC ftp server. This connection fails. So here is my question: How can i download data by the data manager with a http-proxy? Regards Tobias

2 1

Data Libraries: Unable to Link to Files When Paths Contain Spaces
by Joseph Brent Greer 17 Jul '15

17 Jul '15

Hi All, We're having problems adding files to data libraries. We'd like to create links to files instead of copying them into Galaxy. We can create links if there aren't spaces in the paths. Also, we can copy files into libraries even if there are spaces in the paths. Unfortunately, we cannot create links to files when their paths contain spaces. The job standard error and miscellaneous information are: Miscellaneous information:uploaded RAW file Traceback (most recent call last): File "/share/galaxy/working/002/2615/set_metadata_hnsR90.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/home/galaxy/Galaxy/galaxy-dist Job Standard Error Traceback (most recent call last): File "/share/galaxy/working/002/2615/set_metadata_hnsR90.py", line 1, in <module> from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/home/galaxy/Galaxy/galaxy-dist/lib/galaxy_ext/metadata/set_metadata.py", line 80, in set_metadata filename_out = fields.pop( 0 ) IndexError: pop from empty list error We've set allow_library_path_paste = True in galaxy.ini. We've also tried adding "" or '' around the path and get the following error message: Invalid paths: "home/galaxy/Galaxy/galaxy-dist/"/share/projects/2014 Test"" Any help or suggestions would be great. Thanks, Joe Greer Proteomics Center of Excellence Northwestern University

2 1