- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Question
by Greg Von Kuster 04 Nov '09

04 Nov '09

Hello Rodolfo, I've cc'd the galaxy-dev mailing list as others may benefit from this exchange. On Nov 4, 2009, at 12:12 PM, Rodolfo Aramayo wrote: > Hi Greg, > > Thanks for replying. No we are not running into any problems with > Galaxy. In fact we LOVE IT! And students are asking to learn more > about it. In fact we had a little coup in here because students > refused to accept yet another Genomics Unix project, instead they > requested projects in Galaxy. This is great - so glad to hear this! > > Anyway, the reason I was contacting you guys is to ask guidance into > how to go about to load genomes that are not part of the current > Galaxy set up into Galaxy. For example I would like to know what > tables I would need to create for the genomes in question and how to > link them to MySQL. We are in the process of streamlining the process for supporting new genomes, but the current process is fairly simple. New database changes are needed. Since you are running your own local installation of Galaxy, new genomes can be specified by modifing the ${GALAXY_ROOT}/tool-data/shared/ucsc/builds.txt file. This file is generated using a script found at ${GALAXY_ROOT}/cron/updateucsc.sh. If this script is executed on a regular basis ( via cron perhaps ), it will be useful to instead add the required information to $ {GALAXY_ROOT}/tool-data/shared/ucsc/manual_builds.txt before running the script (otherwise each time the script is run, the changes to build.txt will be lost). See ${GALAXY_ROOT}/cron/README.txt for more information as well as the headers of each data file. > I have installed Galaxy locally without problems > and activated the EMBOSS part BUT, it is not clear to me how exactly > go about making Galaxy talk to MySQL. Any help in this area would be > GREATLY appreciated. Hooking up Galaxy to and installed postgres or mysql database is simply a config setting in universe_wsgi.ini ( look for the following section in this file in your galaxy root install directory ). The URL you'll use depends on the name and location of your database and the security you've implemented, but for mysql could look something like the following. See the mysql docs for details [app:main] # Specifies the factory for the universe WSGI application paste.app_factory = galaxy.web.buildapp:app_factory # By default, Galaxy uses a SQLite database found here #database_file = database/universe.sqlite # You may use a SQLAlchemy connection string to specify an external database # instead. PostgreSQL and MySQL are supported. database_connection = mysql://greg_test?unix_socket=/var/run/mysqld/ mysqld.sock > > Among other things, I am the director of the Laboratory for Genome > Bioinformatics here at Texas A&M University and I have been training > students in Bioinformatics/Genomics. I have access to very skillful > undergraduate students who are very computer savvy and are itching to > help Galaxy. We know a bit about Unix and MySQL but are not real > experts... > > Any help in this area would be great and would allow us to load the > fungal genomes we want to study into Galaxy > > Thanks for your help > > Rodolfo > > > > On Wed, Nov 4, 2009 at 09:12, Greg Von Kuster <greg(a)bx.psu.edu> wrote: >> Hello Rodolfo, >> Can you clarify what you would like to do? Are you running your >> own local >> Galaxy instance, or are you using our main public instance? Are >> you having >> problems uploading the files? >> Greg Von Kuster >> Galaxy Development Team >> >> On Oct 26, 2009, at 4:29 PM, Rodolfo Aramayo wrote: >> >> Hi, >> I teach Genomics and want to use Galaxy as a tool. I would like to >> upload a series of fungal genomes for this analysis. Who should I >> contact to talk about this? >> In other words, I need the name of a person who might help us or gude >> us through this process. >> Thank You for your help >> Rodolfo Aramayo, PhD >> Biology/Genetics >> Texas A&M University >> >> Greg Von Kuster >> Galaxy Development Team >> greg(a)bx.psu.edu >> >> >> Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Re: [galaxy-dev] Experience with Loading NGS data on standalone instance of galaxy
by Nate Coraor 04 Nov '09

04 Nov '09

Hi Matthias, I've moved this thread over to galaxy-dev (I probably should've done this a while ago). Matthias Dodt wrote: > One more question according change set 2812: > > Add a new option, 'allow_library_path_paste' that adds a new upload page > ("Upload files from file system paths") to the admin-side library upload > pages. > > I am not sure where to add this option- i added to universe_wsgi.ini: > > allow_library_path_paste = True > allow_library_path_paste = /somepath/ > > both didnt work- how can i enable the "allow_library_path_paste" option? It goes in universe_wsgi.ini and 'True' is the correct value. Once restarting, you should see this link over on the admin side only, not the user side data libraries page. Is this not the case? --nate

1 0

galaxy data libraries
by Matthias Dodt 04 Nov '09

04 Nov '09

Hi there! Galaxy provides the upload of files from user directories via the user_library_import_dir tag in universe_wsgi.ini. 1. If i choose "upload from directory" in the data-libraries menu, first i have a list of folders to choose. How can i assign different folders to the same user? Normally it is only possible to upload files from the users home directory, which matches the username i guess- 2. There is an option: copy files into galaxy: yes/no. If i choose "no" it seems to me that the file gets copied still, because it takes ages to finish? 3. Assuming the "linking" of files into galaxy would work: If i choose once the users home directory for uploading files and link those into galaxy, and later i add more files to the folder and want to upload those, ALL files would be uplinked into galaxy again, no matter if they already exist? Is there a way to choose which files of the folder should be uploaded? Thanks in advance!

2 1

Re: [galaxy-dev] Current Logged in User
by Greg Von Kuster 03 Nov '09

03 Nov '09

Hello Raghav, "trans" is the object defined by the UniverseWebTransaction class in ~/lib/galaxy/__init__.py, and is available in any of the mako templates and controller methods, and many other methods as well. In the future, please send questions like this to the galaxy- dev(a)bx.psu.edu mail list. Thanks very much, On Nov 3, 2009, at 5:51 AM, Raghav Bhatnagar wrote: > Hi Greg > I have a very basic question but somehow I could not figure it out. > I am trying to get the details of current logged in user. I can get > the workflows made by the user by mapping it from galaxy_user and > storedworkflow table from universe.sqlite database. But I am having > problem fetching the current logged in user. > > Now I saw the python scripts and almost every file calls the > function trans.get_user(). > I could not figure out where the function is defined. > Please help me getting the id of the current logged in user. > > Regards, > > -- > Raghav Bhatnagar > Graduate Student (CS) > Viterbi School of Engineering > University of Southern California Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Dynamic GUI for genome-browser custom tracks (another long email)
by Assaf Gordon 03 Nov '09

03 Nov '09

Hello, I'd like to suggest a feature that would make uploading custom tracks from Galaxy to the UCSC genome browser friendlier (IMHO). I don't have the code for it, this is just a suggestion, open for discussion. The way it works now - there are two options to upload custom tracks: 1. click on "Main" link - send the file as is to UCSC's genome browser. Track attributes (color, name, visibility, score, etc) are not sent. Track name is "User Track". If another track is uploaded - it overrides the previously uploaded track. 2. Run the "Build Custom Track" in the "Graph/Display Data" category. This allows setting track options, but it duplicates the existing file, and the options as statically stored in the file - they can't be changed, and the file can't be further processed (unless one removes the custom track line). My suggestion has four parts: Part 1 ------ Each viewable file (BED, intervals, PSL, Wiggle, BedGraph and in the future: SAM/BAM, bigWig, bigBed) will have another 'button' called "custom track attributes" (or a shorter, better name). Clicking on this button will show the user a page in which the user can set the common custom track options. See a mock-up here: http://cancan.cshl.edu/labmembers/gordon/custom_tracks_ui/genome_browser_cu… This page is purely client side - code is javascript. The "output" of this page is just one text line - the custom track attributes line. Part 2 ------ A new database table in galaxy will store the "custom track line" for each eligible dataset. When the user clicks on the above mentioned button, the line is read from the table. When the user clicks "OK" on the above page, the line is saved back to galaxy. Part 3 ------ A new method in the datasets controller will "inject" the dataset's custom line before sending the file's content. For example, if my dataset is a BED file, and contains: chr1 100 3000 chr2 200 3000 And my custom-track-line for this dataset is: name="Hello World" visibility=4 color=255,0,255 The new method will first send the custom-track-line, followed by the actual dataset content: track name="Hello World" visibility=4 color=255,0,255 chr1 100 3000 chr2 200 3000 This method will require just a couple of tricks: adding the "track" keyword at the beginning of the line, and adding the "type" keyword if this is not a standard BED file (e.g. for Wiggle/PSL/BedGraph) Part 4 ------- When click on the "main" button (to display the track in the genome browser), Galaxy will send this new method to the UCSC genome browser, instead of the raw data file. Outcome ------- Uploading custom tracks (with more attributes) will be much easier - no need to run an extra tool. One could them change display options are 'reload' a track (or by using the genome browser's reload button). No need for duplicated data files in galaxy, and tracks can be further processed, filtered, etc (if the custom-track-line) is copied from one dataset to another like other metadata. Further improvement ------------------- When (if?) galaxy supports large binary files (BAM, bigWig, bigBed), it could take advantage of the genome browers's 'bigDataUrl' feature: The new method (in part 3) doesn't need to transmit the entire dataset content into the genome browser. Instead can return the custom track line, and add the 'bigDataUrl' key, with the URL pointing to the real "display" method of the dataset. Something like: track name="Hello World" type=bigBed visibility=3 color=255,255,0 bigDataUrl=http://main.g2.bx.psu.edu/datasets/f8bca8dcf7f5c1d8/display/?sho… And then binary tracks are uploaded almost immediately, and the genome browser will query galaxy just for the parts of the file that are needed. As always, comments are welcomed. -gordon.

1 0

Latest download from repository
by Dijk, F van 02 Nov '09

02 Nov '09

Hi, I have two questions to ask, firstly: We've experienced some errors with Galaxy lately so we decided to download the latest copy from your repository. After installing it, it seems like some tools like "mapping with Bowtie and BWA" are missing. When running run.sh it looks like they aren't loading properly. Yesterday we did also download from these repositories and we didn't experience these errors, so do you know what the problem can be? Second: What version of python do you use for Galaxy, because the older python versions we use seem to give some problems. I hope to hear from you soon. Sincerely Freerk van Dijk De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

2 2

About cluster computing support
by sondos seif 30 Oct '09

30 Oct '09

Dear sir, I am Sondos Seif El-Din, a student in biomedical engineering department, Cairo University. I wanted to know if Galaxy supports cluster computing. Thanks in advance, Sondos Seif El-Din

2 1

toolpane info links
by Gunnar Rätsch 30 Oct '09

30 Oct '09

Hi there, I just talked with Anton at the Genome Informatics meeting about the possibility of including links to galaxy pages in the tools menu. This can be useful for describing a set of tools or even explaining the whole system. It would be useful to have these links on the top level as well as on the tool level. Would it be very hard to have more than one hierarchy in the tool menu? Thanks a lot! Gunnar

1 0

Error uploading file
by Dijk, F van 29 Oct '09

29 Oct '09

Dear, We are running Galaxy on our local server(linux 64 bit) and try to upload a file (fasta, fastq) to the application. This returns the following error: Traceback (most recent call last): File "/data/home/joris/galaxy_dist/tools/data_source/upload.py", line 10, in import galaxy.model File "/data/home/joris/galaxy_dist/lib/galaxy/model/__init__.py", line 13, in import galaxy.datatypes.registry File "/data/home/joris/galaxy_dist/lib/galaxy/datatypes/registry.py", line 6, in import data, tabular, interval, images, sequence, qualityscore, genetics, xml, coverage, tracks, chrominfo File "/data/home/joris/galaxy_dist/lib/galaxy/datatypes/data.py", line 6, in import metadata File "/data/home/joris/galaxy_dist/lib/galaxy/datatypes/metadata.py", line 5, in from galaxy.web import form_builder File "/data/home/joris/galaxy_dist/lib/galaxy/web/__init__.py", line 5, in from framework import expose, json, require_login, require_admin, url_for, error, form, FormBuilder File "/data/home/joris/galaxy_dist/lib/galaxy/web/framework/__init__.py", line 30, in pkg_resources.require( "SQLAlchemy >= 0.4" ) File "/data/home/joris/galaxy_dist/lib/galaxy/eggs/__init__.py", line 544, in require sys.path.remove(entry) ValueError: list.remove(x): x not in list Do you know how this error can be solved? I hope to hear from you soon. Sincerely, Freerk van Dijk De inhoud van dit bericht is vertrouwelijk en alleen bestemd voor de geadresseerde(n). Anderen dan de geadresseerde(n) mogen geen gebruik maken van dit bericht, het niet openbaar maken of op enige wijze verspreiden of vermenigvuldigen. Het UMCG kan niet aansprakelijk gesteld worden voor een incomplete aankomst of vertraging van dit verzonden bericht. The contents of this message are confidential and only intended for the eyes of the addressee(s). Others than the addressee(s) are not allowed to use this message, to make it public or to distribute or multiply this message in any way. The UMCG cannot be held responsible for incomplete reception or delay of this transferred message.

3 5

[hg] galaxy 2926: Fix a bug whereby PYTHONPATH would grow endles...
by Greg Von Kuster 28 Oct '09

28 Oct '09

details: http://www.bx.psu.edu/hg/galaxy/rev/ea0342f2df03 changeset: 2926:ea0342f2df03 user: Nate Coraor <nate(a)bx.psu.edu> date: Tue Oct 27 15:11:41 2009 -0400 description: Fix a bug whereby PYTHONPATH would grow endlessly in the local runner 1 file(s) affected in this change: lib/galaxy/jobs/runners/local.py diffs (47 lines): diff -r c2bed70a9b58 -r ea0342f2df03 lib/galaxy/jobs/runners/local.py --- a/lib/galaxy/jobs/runners/local.py Tue Oct 27 14:31:59 2009 -0400 +++ b/lib/galaxy/jobs/runners/local.py Tue Oct 27 15:11:41 2009 -0400 @@ -20,6 +20,12 @@ """Start the job runner with 'nworkers' worker threads""" self.app = app self.sa_session = app.model.context + # put lib into the PYTHONPATH for subprocesses + if 'PYTHONPATH' in os.environ: + os.environ['PYTHONPATH'] = '%s:%s' % ( os.environ['PYTHONPATH'], os.path.abspath( 'lib' ) ) + else: + os.environ['PYTHONPATH'] = os.path.abspath( 'lib' ) + # start workers self.queue = Queue() self.threads = [] nworkers = app.config.local_job_queue_workers @@ -54,12 +60,6 @@ return # If we were able to get a command line, run the job if command_line: - env = os.environ - if job_wrapper.galaxy_lib_dir is not None: - if 'PYTHONPATH' in os.environ: - env['PYTHONPATH'] = "%s:%s" % ( os.environ['PYTHONPATH'], job_wrapper.galaxy_lib_dir ) - else: - env['PYTHONPATH'] = job_wrapper.galaxy_lib_dir try: log.debug( 'executing: %s' % command_line ) proc = subprocess.Popen( args = command_line, @@ -67,7 +67,7 @@ cwd = job_wrapper.working_directory, stdout = subprocess.PIPE, stderr = subprocess.PIPE, - env = env, + env = os.environ, preexec_fn = os.setpgrp ) job_wrapper.set_runner( 'local:///', proc.pid ) job_wrapper.change_state( model.Job.states.RUNNING ) @@ -110,7 +110,7 @@ log.debug( 'executing external set_meta script for job %d: %s' % ( job_wrapper.job_id, external_metadata_script ) ) external_metadata_proc = subprocess.Popen( args = external_metadata_script, shell = True, - env = env, + env = os.environ, preexec_fn = os.setpgrp ) job_wrapper.external_output_metadata.set_job_runner_external_pid( external_metadata_proc.pid, self.sa_session ) external_metadata_proc.wait()

1 0