April 2013 - galaxy-dev - lists.galaxyproject.org

Tools declaring Python dependencies
by Peter Cock 18 Apr '13

18 Apr '13

Hi all, Galaxy has long had two very simple (but largely unused) dependency mechanisms in a tool's XML (pre-dating the tool shed), for binaries and Python modules, e.g. <requirements> <requirement type="python-module">numpy</requirement> <requirement type="binary">taxBuilder</requirement> </requirements> http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#A.3Crequirement.… This has since been supplemented by the Tool Shed centric package requirement mechanism. How can a tool declare a Python dependency, and have it installed automatically via the Tool Shed? Presumably it would have to end up as an egg in the Galaxy eggs folder, and not be installed under the default Python's PYTHONPATH? There is nothing about Python modules mentioned here: http://wiki.galaxyproject.org/ToolShedToolFeatures What I would suggest is that the the existing <requirement type="python-module"> is compared to the curated modules on http://eggs.g2.bx.psu.edu/ and if available the relevant egg be download from there. In which was, I have a request for additional eggs to be hosted there ;) Regards, Peter

4 7

create a folder inside a library folder using API
by Hagai Cohen 18 Apr '13

18 Apr '13

Hi, How can I create a folder inside another library folder? I tried: lib_create_data = { 'name':'mm9', 'create_type': 'folder', } submit(api_key, api_url + "libraries/%s/contents" % library_id, lib_create_data, return_formatted=False) Where library_id is the upper library folder id. But this failed with the following error: Error. "Missing requred 'folder_id' parameter." Does someone know how to do this? Thanks, Hagai

2 1

Server Error: UnicodeDecodeError: 'utf8' codec can't decode byte 0xdd in position 29: invalid continuation byte
by Ramon Tiburski 18 Apr '13

18 Apr '13

Hi, I have this problem: I uploaded some data and when I click to display data in browser (the eye) I get this error: Server Error: URL: http://localhost:8080/datasets/d413a19dec13d11e/display/?preview=True Module paste.exceptions.errormiddleware:144 in __call__ >> app_iter = self.application(environ, sr_checker) Module paste.debug.prints:106 in __call__ >> environ, self.app) Module paste.wsgilib:543 in intercept_output >> app_iter = application(environ, replacement_start_response) Module paste.recursive:84 in __call__ >> return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ >> return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ >> return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:194 in handle_request >> return body( environ, start_response ) Module galaxy.web.framework:876 in render >> template.render_context( context ) Module mako.template:319 in render_context >> **kwargs) Module mako.runtime:692 in _render_context >> _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template >> callable_(context, *args, **kwargs) Module _base_mako:42 in render_body >> __M_writer(unicode(next.body())) Module _dataset_large_file_mako:45 in render_body >> __M_writer(filters.html_escape(unicode( unicode( truncated_data, 'utf-8' ) ))) UnicodeDecodeError: 'utf8' codec can't decode byte 0xdd in position 29: invalid continuation byte

1 0

about the permission of stdout and stderr
by BingXin Lu 18 Apr '13

18 Apr '13

Hi all, I updated my galaxy yesterday, after that every time I clicked stdout and stderr in the history panel, it shows the following message: 'Invalid dataset ID or you are not allowed to access this dataset'. I have tried to change the permission for the history, but it does not work. Have you come across this problem, and do you know how to see the original stdout and stderr? Thank you very much! Best regards, --Bingxin

2 1

Re: [galaxy-dev] symlink to june_2007_style
by Peter Cock 17 Apr '13

17 Apr '13

On Wed, Apr 17, 2013 at 10:30 PM, Robert Baertsch wrote: > hg update stable --clean is not an option. I lose all of my changes. > > I'm not sure why hg checkout tools does anything. the issue is with the > static directory. In the other thread, 'tools' was the name of my local branch where I'd checked in my local changes. Are you making changes locally but not committing them to source code control? That sounds very risky since you could easily loose them with the wrong hg command. Peter

1 0

symlink to june_2007_style
by Robert Baertsch 17 Apr '13

17 Apr '13

When merging in the latest galaxy-central, I'm getting this error: merging lib/galaxy/webapps/galaxy/controllers/library_common.py merging lib/galaxy/webapps/galaxy/controllers/page.py merging lib/galaxy/webapps/galaxy/controllers/user.py merging run.sh merging scripts/manage_db.py abort: could not symlink to 'style': File exists: /Users/robertbaertsch/src/medbook-merge/medbook-galaxy-central/static/june_2007_style Should I do the following: $ mv june_2007_style style $ ln -s style june_2007_style and then $ hg merge

2 1

snpEff on Galaxy
by Rob Leclerc 17 Apr '13

17 Apr '13

Does anyone have any experience running snpEff on Galaxy? We installed snpEff from the toolshed and now have the latest changeset (c052639fa666). We modified the config files to correct the pathnames but now we're running into a series of errors when we try to run '*Compute SNP effect*' with a file using the hg19 dataset *The first error was:* java.lang.RuntimeException: Property: 'hg37.genome' not found at ca.mcgill.mcb.pcingola.interval.Genome.<init>(Genome.java:78) at ca.mcgill.mcb.pcingola.snpEffect.Config.readGenomeConfig(Config.java:334) at ca.mcgill.mcb.pcingola.snpEffect.Config.read(Config.java:301) at ca.mcgill.mcb.pcingola.snpEffect.Config.<init>(Config.java:67) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:582) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77 This first error was confusing because the input file was for hg19, so it's not clear why there was a reference to hg37. Next, we then edited snpEff.config and added the line *hg37.genome : Homo_sapiens* * * with the other hg37.XX : Homo_sapiens keypairs. *This led to the second error:* ERROR: Cannot read file '/home/galaxy//snpEff/data//hg37/snpEffectPredictor.bin'. You can try to download the database by running the following command: java -jar snpEff.jar download hg37 java.lang.RuntimeException: java.io.FileNotFoundException: /home/galaxy/snpEff/data/hg37/snpEffectPredictor.bin (No such file or directory) at ca.mcgill.mcb.pcingola.snpEffect.SnpEffectPredictor.load(SnpEffectPredictor.java:58) at ca.mcgill.mcb.pcingola.snpEffect.Config.loadSnpEffectPredictor(Config.java:240) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:595) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77) Caused by: java.io.FileNotFoundException: /home/galaxy/snpEff/data/hg37/snpEffectPredictor.bin (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:137) at java.io.FileInputStream.<init>(FileInputStream.java:96) at ca.mcgill.mcb.pcingola.util.Gpr.readFileSerializedGzThrow(Gpr.java:468) at ca.mcgill.mcb.pcingola.snpEffect.SnpEffectPredictor.load(SnpEffectPredictor.java:55) ... 4 more The problem now is that is is looking for the hg37 directory, which doesn't exist but GRCh37.65 does exist. So we then change the directory name to hg37 and reran, resulting in the third error: VcfFileIterator.parseVcfLine(276): Fatal error reading file '/mnt/galaxyData/files/000/dataset_432.dat' (line: 1): 1 2338204 2338204 G/T + java.lang.RuntimeException: java.lang.RuntimeException: WARNING: Unkown IUB code for SNP '+' at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.parseVcfLine(VcfFileIterator.java:277) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.readNext(VcfFileIterator.java:331) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.readNext(VcfFileIterator.java:1) at ca.mcgill.mcb.pcingola.fileIterator.FileIterator.hasNext(FileIterator.java:69) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.iterateVcf(SnpEffCmdEff.java:215) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.runAnalysis(SnpEffCmdEff.java:698) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:648) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.run(SnpEff.java:261) at ca.mcgill.mcb.pcingola.snpEffect.commandLine.SnpEff.main(SnpEff.java:77) Caused by: java.lang.RuntimeException: WARNING: Unkown IUB code for SNP '+' at ca.mcgill.mcb.pcingola.vcf.VcfEntry.parseAlts(VcfEntry.java:530) at ca.mcgill.mcb.pcingola.vcf.VcfEntry.parse(VcfEntry.java:445) at ca.mcgill.mcb.pcingola.vcf.VcfEntry.<init>(VcfEntry.java:66) at ca.mcgill.mcb.pcingola.fileIterator.VcfFileIterator.parseVcfLine(VcfFileIterator.java:274) ... 8 more I'm going to assume that this is all related to the first issue, but I'm not sure how to fix the issue. Cheers! Rob Rob Leclerc, PhD <http://www.linkedin.com/in/robleclerc> <https://twitter.com/#!/robleclerc> P: (US) +1-(917)-873-3037 P: (Shanghai) +86-1-(861)-612-5469 Personal Email: rob.leclerc(a)aya.yale.edu

1 0

Re: [galaxy-dev] Improving Administrative Data Clean Up (pgcleanup.py vs cleanup_datasets.py)
by Lance Parsons 17 Apr '13

17 Apr '13

I have implemented a basic script to perform administrative dataset deletion and email notification. Right now it's limited to the history_dataset_assocation update_time and an optional tool_id string. I have pushed it to a galaxy-central fork and will issue a pull request once I've tested it a bit more. I'm open to comments or suggestions, it could certainly be extended. Hopefully people find this useful. See https://bitbucket.org/lance_parsons/galaxy-central-dataset-cleanup/commits/… admin_dataset_cleanup.py Documentation --------------------------------------------------------- Mark datasets as deleted that are older than specified cutoff and (optionaly) with a tool_id that matches the specified search string. This script is useful for administrators to cleanup after users who leave many old datasets around. It was modeled after the cleanup_datasets.py script originally distributed with Galaxy. Basic Usage: admin_cleanup_datasets.py universe_wsgi.ini -d 60 \ --template=email_template.txt Required Arguments: config_file - the Galaxy configuration file (universe_wsgi.ini) Optional Arguments: -d --days - number of days old the dataset must be (default: 60) --tool_id - string to search for in dataset tool_id --template - Mako template file to use for email notification -i --info_only - Print results, but don't email or delete anything -e --email_only - Email notifications, but don't delete anything Useful for notifying users of pending deletion --smtp - Specify smtp server If not specified, use smtp settings specified in config file --fromaddr - Specify from address If not specified, use error_email_to specified in config file Email Template Variables: cutoff - the cutoff in days email - the users email address datasets - a list of tuples containing 'dataset' and 'history' names xiandongmeng(a)lbl.gov wrote: > Hi Lance, > > I have same questions as yours about cleanup data in Galaxy. > > We maintain a local instance of the Galaxy system. I am thinking a way to delete datasets which are not accessed/updated for a certain period of time, no matter if users deleted them. In addition, sending an email to users before deleting their dataset. It looks like the current scripts for clean up only purges deleted histories/datasets. I was trying to find a way to use API functions to delete old files which are not deleted, but am not successful. I found the update_time is not access time in the dataset table. A reference file may be used frequently but the update_time is pretty old. This would be a problem if deleting file by the update_time. I think files should be deleted by the access time. Is it enough to resolve this problem by checking the update_time of the HistoryDatasetAssociation table? > > Thanks for your help, > > X. Meng > Joint Genome Institute, LBNL > > <quote author='Lance Parsons'> > Nate Coraor wrote: >> On Mar 22, 2013, at 11:56 AM, Lance Parsons wrote: >> >>> I have been running a Galaxy server for our sequencing researchers for a >>> while now and it's become increasingly successful. The biggest resource >>> challenge for us has been, and continues to be disk space. As such, I'd >>> like to implement some additional cleanup scripts. I thought I run a few >>> questions by this list before I got too far into things. >>> >>> In general, I'm wondering how to implement updates/additions to the >>> cleanup system that will be in line with the direction that the Galaxy >>> project is headed. The pgcleanup.py script is the newest piece of code >>> in this area (and even adds cleanup of exported histories, which are >>> absent from the older cleanup scripts). Also, the pgcleanup.py script >>> uses a "cleanup_event" table that I don't believe is used by the older >>> cleanup_datasets.py script. However, the new pgcleanup.py script only >>> works for Postgres, and worse, only for version 9.1+. I run my system on >>> RedHat (CentOS) and thus we use version 8.4 of Postgres. Are there plans >>> to support other databases or older versions of Postgres? >> Hi Lance, >> >> pgcleanup.py makes extensive use of Writable CTEs, so there is not really >> a way to port it to older versions. For 8.4 or MySQL, you can still use >> the older cleanup_datasets.py. > After looking at it a bit more, I see what you mean. Are there plans to > implement and additional cleanup scripts for non-postgres 9.1 users? > Just curious so I don't reinvent the wheel, I'd be happy to help with > existing efforts. >>> I'd like to implement a script to delete (set the deleted flag) for >>> certain datasets (e.g. raw data imported from our archive, for old, >>> inactive users, etc.). I'm wondering if it would make sense to try and >>> extend pgcleanup.py or cleanup_datasets.py. Or perhaps it would be best >>> to just implement a separate script, though that seems like I'd have to >>> re-implement a lot of boilerplate code for configuration reading, >>> connections, logging, etc. Any tips on generally acceptable (supported) >>> procedures for marking a dataset as deleted? >> You could probably reuse a lot of the code from either of the cleanup >> scripts for this. > Right. It seems to make sense to me to focus on the cleanup_datasets.py > since that will work for everyone. I would like to essentially mimic > the user deleting a dataset. I'd then email them to let them know that > some old data had been marked for deletion and let the rest of the > scripts proceed as normal, cleaning that up if they don't undelete it. > > It looks like I would want to mark the HistoryDatasetAssociations as > deleted? Is that correct? Would I need to do anything else to simulate > the user deleting the dataset? > > Thanks for the help, > Lance >> Thanks, >> --nate >> > -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

1 0

Fwd: [Gmod-announce] 2013 GMOD Summer School: Apply now!
by Dave Clements 17 Apr '13

17 Apr '13

Hello all, The 2013 GMOD Summer School is now accepting applications. Galaxy will again be included in the topics covered. And Amelia says: Applications are competitive, so we encourage you to apply well before the > deadline, June 10th. So, if you are interested, act soon. Dave C ---------- Forwarded message ---------- From: Amelia Ireland <amelia.ireland(a)gmod.org> Date: Wed, Apr 17, 2013 at 11:15 AM Subject: [Gmod-announce] 2013 GMOD Summer School: Apply now! To: gmod-announce <gmod-announce(a)lists.sourceforge.net>, gmod-devel(a)lists.sourceforge.net We are now accepting applications for the 2013 GMOD Summer School, to be held NESCent, Durham, North Carolina from July 19th to 23rd. The GMOD Summer School is the best way to learn how to install, configure, and use popular GMOD tools, including GBrowse, JBrowse, Galaxy, MAKER, Tripal, WebApollo, and Chado; courses are taught by the tool developers, and there will be evening sessions for those who want to work on their own data or troubleshoot issues with the developers. More information and online application form: http://gmod.org/wiki/2013_GMOD_Summer_School Applications are competitive, so we encourage you to apply well before the deadline, June 10th. If you have any questions, please contact help(a)gmod.org and we will be happy to answer them. -- Amelia Ireland GMOD Community Support http://gmod.org || @gmodproject ------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter _______________________________________________ Gmod-announce mailing list Gmod-announce(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gmod-announce -- http://galaxyproject.org/GCC2013 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/

1 0

Reports Web Application fails to start (unable to import TracksterConfig)
by Lance Parsons 17 Apr '13

17 Apr '13

After upgrading to the latest distribution release (4.1.2013) I can no longer start the reports web application. The reports-webapp.log shows the following exception. I've created a bug report on Trello for reference: https://trello.com/c/o0S3klAF Traceback (most recent call last): File "./scripts/paster.py", line 33, in <module> serve.run() File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/serve.py", line 1056, in run invoke(command, command_name, options, args[1:]) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/serve.py", line 1062, in invoke exit_code = runner.run(args) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/serve.py", line 227, in run result = self.command() File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/serve.py", line 650, in command app = loadapp( app_spec, name=app_name, relative_to=base, global_conf=vars) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/loadwsgi.py", line 350, in loadapp return loadobj(APP, uri, name=name, **kw) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/loadwsgi.py", line 375, in loadobj return context.create() File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/loadwsgi.py", line 813, in create return self.object_type.invoke(self) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/loadwsgi.py", line 249, in invoke return fix_call(context.object, context.global_conf, **context.local_conf) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/util/pastescript/loadwsgi.py", line 97, in fix_call val = callable(*args, **kw) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/webapps/reports/buildapp.py", line 61, in app_factory add_ui_controllers( webapp, app ) File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/webapps/reports/buildapp.py", line 33, in add_ui_controllers from galaxy.web.base.controller import BaseUIController File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/web/base/controller.py", line 23, in <module> from galaxy.visualization.genome.visual_analytics import get_tool_def File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/visualization/genome/visual_analytics.py", line 3, in <module> from galaxy.tools.parameters.basic import IntegerToolParameter, FloatToolParameter, SelectToolParameter File "/Users/lparsons/Documents/projects/sequencing/galaxy/galaxy-test1/lib/galaxy/tools/__init__.py", line 57, in <module> from galaxy.visualization.genome.visual_analytics import TracksterConfig ImportError: cannot import name TracksterConfig Removing PID file reports_webapp.pid -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 1