October 2013 - galaxy-dev - lists.galaxyproject.org

/dev/xvda1 filling up when downloading large files via ftp
by Stanislas Werfel 16 Oct '13

16 Oct '13

Dear all, this problem has been mentioned before by McDowell(a)vims.ed, but there was no solution so far posted. I'm using the current cloudman on Amazon AWS. When I download a fastq.gz file via an EBI-SRA ftp link using the "Upload File" function, although the file itself is loading to the /mnt/galaxy drive, at the same time /dev/xvda1 is filling up and at some point the download stops and says "[Errno 28] No space left on device". I've checked in the universe_wsgi.ini and the ftp upload path is set correctly to /mnt/galaxy/tmp/ftp. Here's the df output: Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 10321208 9797012 0 100% / udev 8736124 8 8736116 1% /dev tmpfs 3497960 232 3497728 1% /run none 5120 0 5120 0% /run/lock none 8744896 0 8744896 0% /run/shm cgroup 8744896 0 8744896 0% /sys/fs/cgroup /dev/xvdb 423135208 207440 401433728 1% /mnt /dev/xvdf 104847360 22055572 82791788 22% /mnt/galaxy /dev/xvdg 733900800 685188288 48712512 94% /mnt/galaxyIndices I'm also having another issue: when I delete files in the history in galaxy, they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder. Could the two issues be connected? Are there maybe some rights missing for the galaxy instance to delete files? Thanks in advance for any help Best Stanislas

3 4

Can not find hg_g1k_v37.fa.
by shenwiyn 16 Oct '13

16 Oct '13

Hi everyone, I want to use the hg_g1k_v37.fa on my local Galaxy,but I can not find this data.When I use the command : rsync -avzP rsync://datacache.g2.bx.psu.edu/indexes/hg_g1k_v37/seq/hg_g1k_v37.fa the error occurred: receiving incremental file list rsync: link_stat "/hg_g1k_v37/seq/hg_g1k_v37.fa" (in indexes) failed: No such file or directory (2) sent 4 bytes received 6 bytes 4.00 bytes/sec total size is 0 speedup is 0.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1532) [receiver=3.0.6] Please any suggestions would be most welcome. Thanks. shenwiyn

3 4

Default values for repeat sections
by Eric Rasche 16 Oct '13

16 Oct '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, I've been digging through the available galaxy tool config files and have been unable to find a solution, so I bring my problem to you. I have the following (pretty generic) repeat section in a tool config I'm working on <repeat min="1" name="rpt_0" title="Colour Group"> <param value="EDHK" name="match" optional="False" type="text" label="String of letters that should be matched " /> <param value="red" name="fg" optional="False" type="text" label="Foreground colour for a group " /> <param value="blue" name="bg" optional="False" type="text" label="Background colour for a group " /> </repeat> However, for this particular tool, the default ought to be the repeat section three times, with different defaults each time. Is there any way to specify a repeat section, with a default of being repeated N times with an appropriate number of defaults set for each repeat? Cheers, Eric Rasche -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iQIcBAEBAgAGBQJSXgHgAAoJEMqDXdrsMcpVxUIQAM3UC3CZl//N47YPm+/Y5OVm Im2A9Y784rf6nk8Guq0wz4UOMSvnpgJnjlDew49Ly6UGgTdpZqBlbv+o0VQlk/Zz zs90TLVwQ8SMt5x4cSrEF8CJY7y4MlUDCxlaqSeGxmrH7FV3Ph+aRyfhsm32JJx0 duasl4WlJb3jtsBUhMBEQksuhIXTz8SgI0pfTnZ/EIp9ryWNRJiOqeLvCoHq0eNT othEkHmQn7G/o303fPGOcjpkM7SI+uFDtJLowSYBQeds+nIq9iYgcqeEBe7SmsUJ o9Fl4/duQ5aztx5Mg858U4y3lWRJoMwK9MxUqii9zacyepWMCrDFo7ZSyBgik5q8 RXgzr/ehifbMfPH04VaOI2rVtmQVlEQUVhok6JnG92V/Pu1tvZr9PQs5csNVl+ov 2xerZjKeUILeMkP2uOwCm4kPVlVwBW0mOMaSPEUQyFdsz/0HKV3h7RDLB7YFDcc8 DxIeHBY3mvBCd30RcniN42HZUes0bq+cg5eFej9zEwZxEinQKzEWRQFtCR+Mt+z9 4mBUE2gDO5OVL+m6iIt+k8lMevKx3cHom9say+jTNKiK1zbXfwqTda4vYYKxfQY8 1Nv/XPs1bIoFPFJJJIkQALeweENsjZ6iY0aJ5kGR4ZLRAMXVt34g5+r2PmXiphbT eZvA/INKRhwwiPBUf3Hg =RJCY -----END PGP SIGNATURE-----

2 4

Consistently denied server access
by Rhea Datta 16 Oct '13

16 Oct '13

Hello, My account is under rrdatta.nyu(a)gmail.com. I have been able to successfully transfer files and performed FTP uploads using Filezilla in the past. However in the last few days I have been consistently denied server access. The report I get is: Status: Connection attempt failed with "ECONNREFUSED - Connection refused by server". Please let me know if you have a solution to this problem. Thank you so much Rhea -- *Rhea R Datta, PhD* Postdoctoral Associate New York University rrd5(a)nyu.edu rrdatta.nyu(a)gmail.com (917) 613 4719

2 4

colaborativ pages
by Philippe Moncuquet 16 Oct '13

16 Oct '13

Hi, Is there any way to make collaboration on pages possible. I would like user to be able to modify an existing and publicly shared page when the creator of the page wants to offer this opportunity. It might be a bit complicated and out of the scope of Galaxy right now but I wanted to hear what you think about that ? Philip

2 1

Barcode splitter configuration
by UMD Bioinformatics 16 Oct '13

16 Oct '13

Hello Devteam, I've read through the fastx_readme regarding the barcode splitter. I know that i'm supposed to modify the "publicurl" and "basepath" variables in order to produce the necessary table. I've set the base path to a local folder but I don't know what to use for the public url. I've tried file://path/to/folder but that failed. I've also tried using a dropbox folder and it's public url but that also failed. error An error occurred with this dataset:sed: illegal option -- r sed: : sed script [-Ealn] [-i extension] [file ...] sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file …] The files are generated in both cases so as long as I know where to look I can add them manually but this is hardly a work around. I know that usegalaxy.org places the barcode split files in https://usegalaxy.org/dataset/path/to/file but how do I set that up locally? Thanks Ian

1 0

Re: [galaxy-dev] launching CloudMan 2.0 on AWS
by Enis Afgan 15 Oct '13

15 Oct '13

Hi Vipin, It seems there are too many volumes (or too much volume storage) in your AWS account and AWS is preventing any more from being created. You can either delete some volumes if you don't need them or send a request to AWS to increase your volume limit: https://aws.amazon.com/contact-us/ebs_volume_limit_request/ Cheers, Enis > > On Mon, Oct 14, 2013 at 10:06 PM, Vipin TS <vipin.ts(a)gmail.com> wrote: > >> Hi dev-team, >> >> I am trying to launch cloudman on AWS us-east-1a and I am getting the >> message >> *All cluster services started; the cluster is ready for use. >> (2013-10-14 19:45:52)* >> >> I am seeing some error message in the log. >> * 19:05:33 - Master starting* >> * 19:05:35 - Could not find service class matching userData service >> entry: PSS* >> * 19:05:35 - Completed the initial cluster startup process. This is a >> new cluster; waiting to configure the type.* >> * 19:06:11 - Migration service prerequisites OK; starting the service* >> * 19:06:11 - SGE service prerequisites OK; starting the service* >> * 19:06:26 - Setting up SGE...* >> * 19:06:40 - HTCondor service prerequisites OK; starting the service* >> * 19:06:48 - Hadoop service prerequisites OK; starting the service* >> * 19:07:04 - Done adding Hadoop service; service running.* >> * 19:08:13 - Initializing 'Galaxy' cluster type. Please wait...* >> * 19:08:14 - Error creating volume: EC2ResponseError: 400 Bad Request >> VolumeLimitExceededMaximum number of active volumes bytes, 20, >> exceeded.66659429-b0dd-4f88-b763-36fd9232d3c4* >> * 19:08:17 - Adding volume vol-f9c4728e (FS object for galaxy)...* >> * 19:08:42 - Successfully grew file system FS object for galaxy* >> * 19:08:42 - Successfully mounted file system /mnt/galaxy from >> /dev/xvdf* >> >> It seems that the galaxyIndices filesytem is not mounted to the instance. >> Also I am not able to start the postgres or Galaxy service through cloudman >> interface, both says that Cannot find the service. >> >> Please find the attached image. I think I am missing the galaxyIndices >> volume from the image,. any idea about the failure to launch the instance. >> >> >> Thanks, >> Vipin >> http://galaxy.cbio.mskcc.org >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > >

2 1

Fastx toolkit broken pipe
by UMD Bioinformatics 15 Oct '13

15 Oct '13

Hello, I've got a local install of galaxy with fastx tools installed. I'm getting a broken pipe error but I'm not sure why. I install fastx using the compiler outside galaxy and it was working fine inside galaxy previously. Fastx works fine via the command line. I've checked the xml files and I don't see how galaxy cannot find the commands. I am running this off and external volume, but I don't think that matters. Cheers Ian

5 12

Occasional job failure with /scripts/set_metadata.py ElementTree.parse(fname)
by Peter Cock 15 Oct '13

15 Oct '13

Hi all, Running galaxy-central default branch, I've seen a couple of cases where what otherwise appears to be a successful job has failed with: Traceback (most recent call last): File "./scripts/set_metadata.py", line 122, in <module> __main__() File "./scripts/set_metadata.py", line 72, in __main__ datatypes_registry.load_datatypes( root_dir=config_root, config=datatypes_config ) File "/mnt/galaxy/galaxy-central/lib/galaxy/datatypes/registry.py", line 67, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/mnt/galaxy/galaxy-central/lib/galaxy/util/__init__.py", line 135, in parse_xml tree = ElementTree.parse(fname) File "/mnt/galaxy/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py", line 859, in parse File "/mnt/galaxy/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 2] No such file or directory: '/mnt/galaxy/galaxy-central/database/tmp/tmpWJZ1Aw' I'm not quite sure what was in the temporary XML file which no longer exists, perhaps there is a race condition with the removal of the temporary files? Has anyone else observed this? Thanks, Peter One minor change which might help give a clearer error message here (and in related issues like the functional tests where not all the tools listed in tool_conf.xml.sample always exists on disk) is: $ hg diff /mnt/galaxy/galaxy-central/lib/galaxy/util/__init__.py diff -r 01b94ad2c377 lib/galaxy/util/__init__.py --- a/lib/galaxy/util/__init__.py Tue Oct 15 00:12:08 2013 -0400 +++ b/lib/galaxy/util/__init__.py Tue Oct 15 14:20:01 2013 +0100 @@ -132,6 +132,8 @@ def parse_xml(fname): """Returns a parsed xml tree""" + if not os.path.isfile(fname): + raise IOError("Asked to parse non-existant XML file %r" % fname) tree = ElementTree.parse(fname) root = tree.getroot() ElementInclude.include(root)

1 0

/dev/xvda1 filling up when downloading large files via ftp
by Stanislas Werfel 15 Oct '13

15 Oct '13

Dear all, this problem has been mentioned before by McDowell(a)vims.ed, but there was no solution so far posted. I'm using the current cloudman on Amazon AWS. When I download a fastq.gz file via an EBI-SRA ftp link using the "Upload File" function, although the file itself is loading to the /mnt/galaxy drive, at the same time /dev/xvda1 is filling up and at some point the download stops and says "[Errno 28] No space left on device". I've checked in the universe_wsgi.ini and the ftp upload path is set correctly to /mnt/galaxy/tmp/ftp. Here's the df output: Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 10321208 9797012 0 100% / udev 8736124 8 8736116 1% /dev tmpfs 3497960 232 3497728 1% /run none 5120 0 5120 0% /run/lock none 8744896 0 8744896 0% /run/shm cgroup 8744896 0 8744896 0% /sys/fs/cgroup /dev/xvdb 423135208 207440 401433728 1% /mnt /dev/xvdf 104847360 22055572 82791788 22% /mnt/galaxy /dev/xvdg 733900800 685188288 48712512 94% /mnt/galaxyIndices I'm also having another issue: when I delete files in the history in galaxy, they don't acually get deleted in the files/000/ folder. And galaxy is still showing them under "Using ..." in the total space used. So if I download a 3GB fastq file, then delete everything in the history, Using is still showing 3GB and the file is still there in the folder. Could the two issues be connected? Are there maybe some rights missing for the galaxy instance to delete files? Thanks in advance for any help Best Stanislas

1 0