May 2010 - galaxy-dev - lists.galaxyproject.org

Relaoding tools and tool menu in multiprocess setup
by Dennis Gascoigne 26 May '10

26 May '10

Hi; We have a multiprocess setup (job runner and multi web runners) and are frequently customising tools/adding new ones. When we were single threading at first, we could reload tools from the admin menu but that does not update the job runner version when multi thread. Is there any way to; - Reload tools without stopping galaxy when in multi thread - Update the tool menu This is critical because if you have a long running job, stopping the server will kill it - we have seq jobs that take days to complete and our setup is very fluent at the moment. I dont mind hacking if it works. Cheers DG

3 2

question about bam and bai
by Davide Cittaro 26 May '10

26 May '10

Just a silly one, Are bam indexes (.bai) automagically created when a bam file is imported (either from or into a library)? If not, how can I do it? d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

3 2

ldap integration
by Davide Cittaro 26 May '10

26 May '10

Hi all, AFAIK one can put galaxy behind a http proxy and let this to manage the authentication binding do LDAP. This morning I've been looking at the universe_wsgi.ini file and I've seen this: # Use user provided in an upstream server's $REMOTE_USER variable #use_remote_user = False # If use_remote_user is enabled and your external authentication # method just returns bare usernames, set a default mail domain #remote_user_maildomain = example.org I suppose I should set use_remote_user = True to enable proxied authentication... nevertheless the email passed to galaxy looks like a $REMOTE_USER + REMOVE_USER_MAILDOMAIN set in the unverse_wsgi.ini... does that mean that I will not able to authenticate as my ldap username is dcittaro and my email is davide.cittaro(a)ifom-ieo-campus.it? thanks d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

3 15

Select from Database as tool option
by Dennis Gascoigne 25 May '10

25 May '10

We have a tool which needs to select its options from a database table. Is it possible to do this with Galaxy i.e. have a drop down list (like with the loc files or genome selector); Can we use the cheetah code or otherwise smash an sql query from the tool? Cheers Dennis -

1 0

Re: [galaxy-dev] need help on extra servers
by Davide Cittaro 25 May '10

25 May '10

On May 22, 2010, at 3:28 AM, Edward Kirton wrote: > Hi Davide, > > I'm trying to get this working too but applying your patch gives me an error > Nevermind, I think galaxy team should work on this :-) Even if you had the patch working you may get some errors when a tool tries to read the config file. Apparently this is defined somewhere else too... I've been trying to import some data from UCSC and this is what I've got (after my patch): Traceback (most recent call last): File "/data/galaxy_dist/tools/data_source/data_source.py", line 6, in from galaxy.util import gzip_magic File "/data/galaxy_dist/lib/galaxy/util/__init__.py", line 21, in pkg_resources.require( 'docutils' ) File "/data/galaxy_dist/lib/galaxy/eggs/__init__.py", line 337, in require c = Crate() File "/data/galaxy_dist/lib/galaxy/eggs/__init__.py", line 197, in __init__ self.galaxy_config = GalaxyConfig() File "/data/galaxy_dist/lib/galaxy/eggs/__init__.py", line 299, in __init__ raise Exception( "error: unable to read Galaxy config from %s" % GalaxyConfig.config_file ) Exception: error: unable to read Galaxy config from /data/galaxy_dist/0 As you can see the config file name is passed here as "0", that means that the tool is reading it somewhere where is not defined (or it should be defined as universe_wsgi.ini)... I'll ask the audience :-) d > Traceback (most recent call last): > File "./scripts/check_eggs.py", line 20, in <module> > from galaxy.eggs import Crate > File "/house/tooldirs/galaxy/galaxy-jgi/lib/galaxy/eggs/__init__.py", line 290, in <module> > class GalaxyConfig( object ): > File "/house/tooldirs/galaxy/galaxy-jgi/lib/galaxy/eggs/__init__.py", line 291, in GalaxyConfig > config_file = os.path.join( galaxy_dir, sys.argv[2] ) > IndexError: list index out of range > Some eggs are out of date, attempting to fetch... > Traceback (most recent call last): > File "./scripts/fetch_eggs.py", line 21, in <module> > from galaxy.eggs import Crate, EggNotFetchable > File "/house/tooldirs/galaxy/galaxy-jgi/lib/galaxy/eggs/__init__.py", line 290, in <module> > class GalaxyConfig( object ): > File "/house/tooldirs/galaxy/galaxy-jgi/lib/galaxy/eggs/__init__.py", line 291, in GalaxyConfig > config_file = os.path.join( galaxy_dir, sys.argv[2] ) > IndexError: list index out of range > Fetch failed. > > I've got IT coming by to help me next week; I'll be sure to pass-along anything I learn. > > Cheers! > Ed > > > On Fri, May 21, 2010 at 1:32 AM, Davide Cittaro <davide.cittaro(a)ifom-ieo-campus.it> wrote: > > On May 21, 2010, at 10:13 AM, Davide Cittaro wrote: >> I'm now looking where and how the config file is passed to galaxy from command line... >> > > It works after this: > > $ hg diff lib/galaxy/eggs/__init__.py > diff -r 48432330228e lib/galaxy/eggs/__init__.py > --- a/lib/galaxy/eggs/__init__.py Fri May 14 10:37:36 2010 -0400 > +++ b/lib/galaxy/eggs/__init__.py Fri May 21 10:30:59 2010 +0200 > @@ -288,7 +288,7 @@ > raise EggNotFetchable( missing ) > > class GalaxyConfig( object ): > - config_file = os.path.join( galaxy_dir, "universe_wsgi.ini" ) > + config_file = os.path.join( galaxy_dir, sys.argv[2] ) > always_conditional = ( 'GeneTrack', 'pysam' ) > def __init__( self ): > self.config = ConfigParser.ConfigParser() > > Of course, this assumes that the config file is always the second argument (which is seems mandatory to me...) > > > d > > /* > Davide Cittaro > > Cogentech - Consortium for Genomic Technologies > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro(a)ifom-ieo-campus.it > */ > > > > > _______________________________________________ > galaxy-dev mailing list > galaxy-dev(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-dev > > /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

2 1

Error on save large file - out of memory
by Dennis Gascoigne 25 May '10

25 May '10

When we try to save a large file from galaxy (local instance) we get the following. Server should have plenty of resources. Cheers Dennis MemoryError: out of memory clear this clear this URL: http://genome.imb.qfab.org:8080/datasets/3197f0f893217878/display?to_ext=fa… Module weberror.evalexception.middleware:364 in respond view << try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter)>> app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ view << try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred>> environ, self.app) Module paste.wsgilib:544 in intercept_output view << try: for item in app_iter: output.write(item) finally: if hasattr(app_iter, 'close'):>> output.write(item) MemoryError: out of memory URL: http://genome.imb.qfab.org:8080/datasets/3197f0f893217878/display?to_ext=fa… '/home/d.gascoigne/Galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 544 in intercept_output output.write(item) MemoryError: out of memory <?xml version="1.0" ?> <traceback> <sysinfo> <language version="2.6.1"> Python </language> </sysinfo> <stack> <frame> <module> weberror.evalexception.middleware </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py </filename> <line> 364 </line> <function> respond </function> <operation> app_iter = self.application(environ, detect_start_response) </operation> <operation_context> try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter) </operation_context> </frame> <frame> <module> paste.debug.prints </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py </filename> <line> 98 </line> <function> __call__ </function> <operation> environ, self.app) </operation> <operation_context> try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred </operation_context> </frame> <frame> <module> paste.wsgilib </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py </filename> <line> 544 </line> <function> intercept_output </function> <operation> output.write(item) </operation> <operation_context> try: for item in app_iter: output.write(item) finally: if hasattr(app_iter, 'close'): </operation_context> </frame> </stack> <exception> <type> MemoryError </type> <value> out of memory </value> </exception> </traceback> <?xml version="1.0" ?> <traceback> <sysinfo> <language version="2.6.1"> Python </language> </sysinfo> <stack> <frame> <module> weberror.evalexception.middleware </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py </filename> <line> 364 </line> <function> respond </function> <operation> app_iter = self.application(environ, detect_start_response) </operation> <operation_context> try: __traceback_supplement__ = errormiddleware.Supplement, self, environ app_iter = self.application(environ, detect_start_response) try: return_iter = list(app_iter) </operation_context> </frame> <frame> <module> paste.debug.prints </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py </filename> <line> 98 </line> <function> __call__ </function> <operation> environ, self.app) </operation> <operation_context> try: status, headers, body = wsgilib.intercept_output( environ, self.app) if status is None: # Some error occurred </operation_context> </frame> <frame> <module> paste.wsgilib </module> <filename> /home/d.gascoigne/Galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py </filename> <line> 544 </line> <function> intercept_output </function> <operation> output.write(item) </operation> <operation_context> try: for item in app_iter: output.write(item) finally: if hasattr(app_iter, 'close'): </operation_context> </frame> </stack> <exception> <type> MemoryError </type> <value> out of memory </value> </exception> </traceback> -- Dennis Gascoigne 0407 639 995 dennis.gascoigne(a)gmail.com

2 1

Re: [galaxy-dev] Amazon EC2 install dataset Upload problems
by Enis Afgan 25 May '10

25 May '10

We'll have to take a look at the data library upload on the cloud - I have not played with that yet so I don't really have an answer for you just yet. It should operate normally because the Galaxy running on the cloud is stock Galaxy but we'll have to see why it's not working. Now as far as customizing the cloud instance - at this time, there is no additional documentation about the cloud other than the how-to-get started wiki page. This is something on the to-do list however. In the mean time, if you'd like to customize your cloud instance: all of the galaxy tools and galaxy application itself reside in /mnt/galaxyTools, which is an EBS volume. You can manipulate those tools and Galaxy itself there as you would like. Note that you should make symlinks to any added tools in directory /mnt/galaxyTools/tools/bin so that Galaxy can find needed tools (you can use stow to do this). Once you're done modifying Galaxy and adding tools, you'd have to unmount that file system, detach given EBS volume, and then create a snapshot of it (through AWS console). Next, in your S3 account, there is a bucket that corresponds to the given cluster (it's name will be gc-<hash> and if you have more than one cluster, let me know and we'll work on figuring out which bucket you should modify). In that bucket, there is a file persistent-volumes-latest.txt that contains a line TOOLS=snap-<ID>|<volume size>. Edit that file to change the reference to the snapshot so that it lists the snapshot you just created. Then, terminate the cluster (including the master instance) and start it back up. You should now have access to your tools and customized Galaxy. Hope this helps. Let us know if you have any more questions and we'll look into the data library upload issue. Enis On Mon, May 24, 2010 at 4:51 PM, <dennis.gascoigne(a)gmail.com> wrote: > I am using the DataLibrary upload from directory function. The reason I am > using this function is because I have a stack of files already on the server > for other reasons. Also, this is the only way you can work on files without > copying them to the database file directory. > > Basically, it works with a non-cloud instance, I expect it to work on the > cloud too. > > Is there any better documentation on the cloud install as opposed to the > single wiki how to install page? For example, we have a heavily cusomtised > install (changed tools and config) - is it correct to say that we would have > to reconfigure the cloud install by making new ami's of our changed configs > and instantiating them? Or will the cloud instances be able to deal with > changes to tools and configs across the cluster? > > Cheers > Dennis > > > On , Enis Afgan <eafgan(a)emory.edu> wrote: > > Hi Dennis, Could you please elaborate a bit on what you are trying to do. > When you say you're trying to get a dataset from the file system - did you > manually scp given dataset to the master EC2 instance and are trying to > register it with Galaxy now or something else? If you're first manually > copying the datasets, is there a reason why you're not using the upload tool > directly from a remote site - that approach worked for us without trouble > thus far. > > > > > > > > > > > > In principle, /mnt/galaxyData is NFS mounted across all worker instances > so they should all have access to the given file system. > > > > > > Enis > > > > On Mon, May 24, 2010 at 5:36 AM, Dennis Gascoigne > dennis.gascoigne(a)gmail.com> wrote: > > > > > > > > Hi; > > > > We are working our way through the cloud install and cannot get datasets > to upload from the file system. Should this functionality work? Or is there > a problem with cross instance access to the file system - we made sure we > used galaxyData mount but got a python script error. > > > > > > > > > > > > > > /mnt/galaxyData/upload_store/ > > > > > > > > > > > > > > Traceback (most recent call last): File > "/mnt/galaxyTools/galaxy-central/tools/data_source/upload.py", line 343, > in __main__() File > "/mnt/galaxyTools/galaxy-central/tools/data_source/upload.py", line 335, in > __main__ add_file( dataset > > > > > > > > > > > > error > > > > > > > > -- > > > > > > > > > > > > _______________________________________________ > > > > galaxy-dev mailing list > > > > galaxy-dev(a)lists.bx.psu.edu > > > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > > > > > > > > > > > >

1 0

Problem with import to data library from history
by Dennis Gascoigne 25 May '10

25 May '10

1 1

Extract Workflow Error
by Dennis Gascoigne 24 May '10

24 May '10

3 2

Amazon EC2 install dataset Upload problems
by Dennis Gascoigne 24 May '10

24 May '10

Hi; We are working our way through the cloud install and cannot get datasets to upload from the file system. Should this functionality work? Or is there a problem with cross instance access to the file system - we made sure we used galaxyData mount but got a python script error. /mnt/galaxyData/upload_store/ Traceback (most recent call last): File "/mnt/galaxyTools/galaxy-central/tools/data_source/upload.py", line 343, in __main__() File "/mnt/galaxyTools/galaxy-central/tools/data_source/upload.py", line 335, in __main__ add_file( dataset error --

2 1