May 2013 - galaxy-dev - lists.galaxyproject.org

Issue with set_user_disk_usage.py and Postgres 8.x
by Lance Parsons 06 Jun '13

06 Jun '13

The recent updates to set_user_disk_usage.py for Postgres users have an issue with Postgres 8.x. The SQL in the pgcalc method (line 51) leads to the following error: sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "d.total_size" must appear in the GROUP BY clause or be used in an aggregate function LINE 4: FROM ( SELECT d.total_siz... ^ The problem is that version of Postgres before 9.x were a bit more restrictive in the use of GROUP BY. This can be fixed using DISTINCT ON instead. See this StackOverflow post for more info: http://stackoverflow.com/questions/1769361/postgresql-group-by-different-fr… I've included a patch below. Let me know if a pull request would be preferred. --- a/scripts/set_user_disk_usage.py +++ b/scripts/set_user_disk_usage.py @@ -52,7 +52,7 @@ sql = """ UPDATE galaxy_user SET disk_usage = (SELECT COALESCE(SUM(total_size), 0) - FROM ( SELECT d.total_size + FROM ( SELECT DISTINCT ON (d.id) d.total_size, d.id FROM history_dataset_association hda JOIN history h ON h.id = hda.history_id JOIN dataset d ON hda.dataset_id = d.id @@ -62,7 +62,7 @@ AND d.purged = false AND d.id NOT IN (SELECT dataset_id FROM library_dataset_dataset_association) - GROUP BY d.id) sizes) + ) sizes) WHERE id = :id RETURNING disk_usage; """ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

3 4

Re: [galaxy-dev] Creating multiple datasets in a libset
by Neil.Burdett＠csiro.au 05 Jun '13

05 Jun '13

Hi Guys, Did you manage to get multiple datasets working? I can't seem to upload multiple files. Only the last file appears in the history. I changed my code as mentioned in the thread below in "example_watch_folder.py" to add multiple files separated by a new line and increased the sleep time: for fname in os.listdir(in_folder): fullpath = os.path.join(in_folder, fname) print ' fullpath is [%s] ' % fullpath if os.path.isfile(fullpath): data = {} data['folder_id'] = library_folder_id data['file_type'] = 'auto' data['dbkey'] = '' data['upload_option'] = 'upload_paths' data['filesystem_paths'] = "/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz\n /home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz" print ' data is [%s] ' % str(data['filesystem_paths']) data['create_type'] = 'file' libset = submit(api_key, api_url + "libraries/%s/contents" % library_id, data, return_formatted = False) #TODO Handle this better, but the datatype isn't always # set for the followup workflow execution without this # pause. time.sleep(65) However, I get the following crash: ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "This One" f2db41e1fa331b3e fullpath is [/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz] data is [/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz /home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz] url is : http://barium-rbh/csiro/api/libraries/33b43b4e7093c91f/contents?key=64f3209… data is : {'file_type': 'auto', 'dbkey': '', 'create_type': 'file', 'folder_id': 'F33b43b4e7093c91f', 'upload_option': 'upload_paths', 'filesystem_paths': '/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T1_Screening.nii.gz\n /home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz'} url is : http://barium-rbh/csiro/api/workflows?key=64f3209856a3cf4f2d034a1ad5bf851c data is : {'workflow_id': 'f2db41e1fa331b3e', 'ds_map': {'14': {'src': 'ld', 'id': 'ff5476bcf6c921fa'}}, 'history': '141_S_0851_MRI_T2_Screening.nii.gz - apiFullCTE'} {'outputs': ['daecbdd824e1c349', '358eb58cd5463e0d', 'c0279aab05812500'], 'history': '3cc0effd29705aa3'} url is : http://barium-rbh/csiro/api/workflows?key=64f3209856a3cf4f2d034a1ad5bf851c data is : {'workflow_id': 'f2db41e1fa331b3e', 'ds_map': {'14': {'src': 'ld', 'id': '79966582feb6c081'}}, 'history': '141_S_0851_MRI_T2_Screening.nii.gz - apiFullCTE'} {'outputs': ['19c51286b777bc04', '0f71f1fc170d4ab9', '256444f6e7017e58'], 'history': 'b701da857886499b'} Traceback (most recent call last): File "./example_watch_folder.py", line 89, in <module> main(api_key, api_url, in_folder, out_folder, data_library, workflow ) File "./example_watch_folder.py", line 75, in main shutil.move(fullpath, os.path.join(out_folder, fname)) File "/usr/lib/python2.7/shutil.py", line 299, in move copy2(src, real_dst) File "/usr/lib/python2.7/shutil.py", line 128, in copy2 copyfile(src, dst) File "/usr/lib/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/home/galaxy/galaxy-drop/input/141_S_0851_MRI_T2_Screening.nii.gz' It says there is no such file, but this file has already been copied from the input to the output directory. Any help much appreciated Neil ------------------------------ Message: 2 Date: Mon, 29 Apr 2013 16:11:39 -0400 From: Rob Leclerc <robert.leclerc(a)gmail.com> To: Dannon Baker <dannon.baker(a)gmail.com> Cc: "galaxy-dev(a)bx.psu.edu" <galaxy-dev(a)bx.psu.edu> Subject: Re: [galaxy-dev] Creating multiple datasets in a libset Message-ID: <CAGkd85fHSgO2YC1T+Frctyso9G5rfQb=_mLyHGSdxPM+s3=8pg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Hi Dannon, I've written some code to (i) query a dataset to ensure that it's been uploaded after a submit and (ii) to ensure a resulting dataset has been written to the file. *#Block until all datasets have been uploaded* libset = submit(api_key, api_url + "libraries/%s/contents" % library_id, data, return_formatted = False) for ds in libset: while True: uploaded_file = display(api_key, api_url + 'libraries/%s/contents/%s' %(library_id, ds['id']), return_formatted=False) if uploaded_file['misc_info'] == None: time.sleep(1) else: break *#Block until all result datasets have been saved to the filesystem* result_ds_url = api_url + 'histories/' + history_id + '/contents/' + dsh['id']; while True: result_ds = display(api_key, result_ds_url, return_formatted=False) if result_ds["state"] == 'ok': break else: time.sleep(1) Rob Leclerc, PhD <http://www.linkedin.com/in/robleclerc> <https://twitter.com/#!/robleclerc> P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.leclerc(a)aya.yale.edu On Mon, Apr 29, 2013 at 11:18 AM, Dannon Baker <dannon.baker(a)gmail.com>wrote: > Yep, that example filesystem_paths you suggest should work fine. The > sleep() bit was a complete hack from the start, for simplicity in > demonstrating a very basic pipeline, but what you probably want to do for a > real implementation is query the dataset in question via the API, verify > that the datatype/etc have been set, and only after that execute the > workflow; instead of relying on sleep. > > > On Mon, Apr 29, 2013 at 9:24 AM, Rob Leclerc <robert.leclerc(a)gmail.com>wrote: > >> Hi Dannon, >> >> Thanks for the response. Sorry to be pedantic, but just to make sure that >> I understand the interpretation of this field on the other side of the API, >> I would need to have something like the following: >> >> data['filesystem_paths'] = "/home/me/file1.vcf \n /home/me/file2.vcf /n >> /home/me/file3.vcf" >> >> I assume I should also increase the time.sleep() to reflect the uploading >> of extra files? >> >> Cheers, >> >> Rob >> >> Rob Leclerc, PhD >> <http://www.linkedin.com/in/robleclerc><https://twitter.com/#!/robleclerc> >> P: (US) +1-(917)-873-3037 >> P: (Shanghai) +86-1-(861)-612-5469 >> Personal Email: rob.leclerc(a)aya.yale.edu >> >> >> On Mon, Apr 29, 2013 at 9:15 AM, Dannon Baker <dannon.baker(a)gmail.com>wrote: >> >>> Hey Rob, >>> >>> That example_watch_folder.py does just submit exactly one at a time, >>> executes the workflow, and then does the next all in separate transactions. >>> If you wanted to upload multiple filepaths at once, you'd just append more >>> to the ''filesystem_paths' field (newline separated paths). >>> >>> -Dannon >>> >>> >>> On Fri, Apr 26, 2013 at 11:54 PM, Rob Leclerc <robert.leclerc(a)gmail.com>wrote: >>> >>>> I'm looking at example_watch_folder.py and it's not clear from the >>>> example how you submit multiple datasets to a library. In the example, the >>>> first submit returns a libset [] with only a single entry and then proceeds >>>> to iterate through each dataset in the libset in the following section: >>>> >>>> data = {} >>>> >>>> data['folder_id'] = library_folder_id >>>> >>>> data['file_type'] = 'auto' >>>> >>>> data['dbkey'] = '' >>>> >>>> data['upload_option'] = 'upload_paths' >>>> >>>> >>>> >>>> *data['filesystem_paths'] = fullpath* >>>> >>>> data['create_type'] = 'file' >>>> >>>> libset = submit(api_key, api_url + "libraries/%s/contents" % >>>> library_id, data, return_formatted = False) >>>> >>>> time.sleep(5) >>>> >>>> for ds in libset: >>>> >>>> if 'id' in ds: >>>> >>>> wf_data = {} >>>> >>>> wf_data['workflow_id'] = workflow['id'] >>>> >>>> wf_data['history'] = "%s - %s" % (fname, >>>> workflow['name']) >>>> >>>> wf_data['ds_map'] = {} >>>> >>>> for step_id, ds_in in workflow['inputs' >>>> ].iteritems(): >>>> >>>> wf_data['ds_map'][step_id] = {'src':'ld', >>>> 'id':ds['id']} >>>> >>>> res = submit( api_key, api_url + 'workflows', >>>> wf_data, return_formatted=False) >>>> >>>> >>>> >>>> Rob Leclerc, PhD >>>> <http://www.linkedin.com/in/robleclerc><https://twitter.com/#!/robleclerc> >>>> P: (US) +1-(917)-873-3037 >>>> P: (Shanghai) +86-1-(861)-612-5469 >>>> Personal Email: rob.leclerc(a)aya.yale.edu >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> http://lists.bx.psu.edu/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>>> >>> >>> >> >

4 7

Internal Server Error during installation of ncbi_blast_plus
by Keilwagen, Jens 05 Jun '13

05 Jun '13

Hi guys, we are currently estalishing our own local Galaxy instance. However, when we try to install "Repository 'ncbi_blast_plus'" from the Galaxy main tool shed via the admin pages, we obtain the following error "Internal Server Error Galaxy was unable to sucessfully complete your request An error occurred. This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. The error has been logged to our team." We tried to install on three different days. Hence, we believe there might be another problem. Thanks for any help. Best regards, Jens

3 10

Information for Galaxy Event Horizon
by Sarah Maman 03 Jun '13

03 Jun '13

Dear all, Four days of training in genomics are occurring on our local instance of Galaxy (27-30 May). This training session takes place at INRA, Toulouse Auzeville, France, as part of a collaboration between Sigenae (http://www.sigenae.org/) and GenoToul bioinformatics platform (http://bioinfo.genotoul.fr/) This session is composed of four training days: * Galaxy first step * Reads alignment and SNP calling * RNAseq alignment and transcripts assemblies * sRNAseq Do you think it is possible to list our training session in your "Galaxy Event Horizon"? Sincerely, Sarah Maman -- --*-- Sarah Maman INRA - LGC - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--

2 1

Run galaxy on a specific SGE queue
by Sarah Maman 03 Jun '13

03 Jun '13

Hi everyone, Galaxy is set up on our cluster, and so far everything is working just fine. However, we'd like to change the submission queue from the default to a specific one. To do so, we tryed to set on the universe_wsgi.ini file the default_cluster_job_runner parameter as following: default_cluster_job_runner = drmaa://-q test.q -l mem=8G -l h_vmem=10G/ When doing so, an error occured telling us the job has been rejected because 2 queues were specified: the workq (default one) and the test.q. Should be the queue setting this way ? What are we doing wrong ? thanks for your help, Sarah -- Sarah Maman INRA - LGC - SIGENAE Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53

3 2

BWA installation
by מירב . 03 Jun '13

03 Jun '13

Hello all, I am trying to install BWA , I used the command: " bwa index -a bwtsw /global_data/hg18full.fa " and I got only 5 files insted of 8 The missing files are - rbwt - rpac - .rsa I didn't recieve any error message while running. thanks for your help merav

3 3

problem with current galaxy to submit job to drmaa/sge
by tin h 01 Jun '13

01 Jun '13

Hello galaxy-dev gurus, I was trying to upgrade my galaxy server... I removed the old galaxy-dist and ran hg clone https://bitbucket.org/galaxy/galaxy-dist/ restored universe_wsgi.ini file and various tool-data config, and tried to restart galaxy. After some twiddling, I see the error message at the end of this email. The strangest thing I see is this /usr/prog/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/wrappers.py on my current system, drmaa-0.4b3-py2.6.egg is a file and not a directory... is the latest code that I just downloaded corrupted or something? Much thanks in advance for your help in this matter. -Tin PS. Relevant entry in universe_wsgi.ini on cluster config: start_job_runners = drmaa default_cluster_job_runner = drmaa:/// galaxy.tools.genome_index DEBUG 2013-05-24 08:42:49,150 Loaded genome index tool: __GENOME_INDEX__ galaxy.jobs.manager DEBUG 2013-05-24 08:42:49,153 Starting job handler galaxy.jobs.runners DEBUG 2013-05-24 08:42:49,155 Starting 4 LocalRunner workers galaxy.jobs DEBUG 2013-05-24 08:42:49,156 Loaded job runner 'galaxy.jobs.runners.local:LocalJobRunner' as 'local' Traceback (most recent call last): File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line 35, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/app.py", line 159, in __init__ self.job_manager = manager.JobManager( self ) File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/manager.py", line 31, in __init__ self.job_handler = handler.JobHandler( app ) File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 29, in __init__ self.dispatcher = DefaultJobDispatcher( app ) File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 543, in __init__ self.job_runners = self.app.job_config.get_job_runner_plugins() File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 486, in get_job_runner_plugins rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get( 'kwds', {} ) ) File "/usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 75, in __init__ self.ds.initialize() File "/usr/prog/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py", line 274, in initialize _w.init(contactString) File " /usr/prog/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/wrappers.py", line 59, in init return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer)) File "/usr/prog/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value)) DrmCommunicationException: code 2: range_list containes no elements

2 2

Re: [galaxy-dev] [galaxy-user] List of genomes
by Jennifer Jackson 01 Jun '13

01 Jun '13

Hello Yongde, The list of genomes is gathered from many sources and is comprehensive to facilitate external display functionality (at UCSC - main and microbial-, Ensembl, Wormbase, etc.). When assigning a dataset in a standard format to one of these sources, those available will appear as links within the dataset's box. Trackster (Galaxy's native visualization tool) is available to most common data formats, even in the absence of an assigned database, through the use of the Custom Reference Genome function (aka "Custom Build"). We think this is a great advantage, in particular for cases such as yours - since you don't have restrict yourself to external applications that happen to host your genome. Click on the link here and select "Trackster" to give it a test run: The Custom Reference Genome function is also intended to be used for smaller genomes such as this one when performing alignments and most other jobs - no pre-indexing of the genome is necessary. Simply load the genome in fasta format as a dataset and use it with tools, using a "reference genome from the history". The rational is that these are many, small, easily indexed during the course of job processing, and provides immediate access to genomes that are either newly published, or not widely used, or simply too numerous as a whole class for us to practically process in full and keep current. We have detailed help about how to use the Custom Reference Genome method, including troubleshooting help should you need it, although in practice you will likely find this to be fairly simple with 2-3 preparatory steps, depending on the source. Most if not all of these can be done within Galaxy. http://wiki.galaxyproject.org/Support#Custom_reference_genome Hopefully this helps. If you do need more guidance, please let us know, Best, Jen Galaxy team On 4/29/13 9:01 AM, YBao wrote: > Hi All, > > I was trying to map a set of data to a genome, Klebsiella pneumoniae > subsp. Pneumoniae MGH 78578(31). While uploading the reads, I was able > to find the reference genome as listed above. However, when I tried to > map the data using wither bowtie or BWA, the pull down list did not > include this genome. Can someone help or enlighten me as why it did > not make into the list? > > Thanks > > Yongde > > -- > Yongde Bao > DNA Sciences Core > Dept. of Microbiology, Immunology, > and Cancer Biology > UVA > > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org

2 1

uploading large datasets into a new instance of Galaxy on a Mac Pro
by Elwood Linney 01 Jun '13

01 Jun '13

Hello, I installed Galaxy on a new Mac Pro with 64gb of ram and a solid state first drive and 3 , 2tb normal drives. I can upload from my desktop a smaller file like a reference list or a cuffdiff file but when I try to load a 16gb fastq file it just does not load. Given that I have my files on the same computer that has Galaxy, and given that I am not well versed in unix, would anyone have a suggestion as to ways I could easily upload these large files. I would like to think I am just naive and that this has a simple solution. Hopefully this is the correct list to pose this question. Thank you for any help, el linney For those in the no, I did see this appear on the terminal after I tried to upload the file through the browser mechanism for upload: "Overflow Error: signed integer is greater than maximum"

3 2

Storage question for Galaxy-Admins
by Raj Ayyampalayam 01 Jun '13

01 Jun '13

Hello, I am trying to come up with a number for our local galaxy storage requirements when we go live (This is an installation for research at our university). I checked out the survey that was done a while back for the GalaxyAdmins. It seems that the bigger installations have allocated about 200TB for their installation. It would be of great help to me if I can get some feedback on storage used by local galaxy installations. If possible please use the following format: Total capacity: Used capacity: Storage Technology: Comments: Thank you all very much. -Raj

3 2