April 2014 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] [galaxy-user] Trackster error on dbkeys
by Jeremy Goecks 12 Apr '14

12 Apr '14

(Moved to galaxy-dev mailing list because it concerns server admin rather than Galaxy usage.) If everything worked fine with a human build but not for PF3d7, then my guess is that the len file for PF3d7 isn’t correctly formatted. Right now, the file parser is very unforgiving—no extra spaces or lines are allowed. If you check the len file and don’t spot an issue, please send to it to me and I’ll take a look. Best, J. -- Jeremy Goecks Assistant Professor of Computational Biology George Washington University On Apr 10, 2014, at 2:30 AM, Aarthi Mohan <aarthimohan11(a)gmail.com> wrote: > Hi all, > > I am setting up galaxy on a server running CentOS 6.5. > I found that Trackster application hits me with an error > "Could not load chroms for this dbkey: PF3D7", when I try "add new track" from Visualization. > > I am working with Plasmodium falciparum (PF3d7) genome, and I followed the instruction for setting visualization from "https://wiki.galaxyproject.org/Visualization%20Setup" and placed the 2bit, len files in the appropriate path. > > I have also made sure the genome build is present in the builds.txt under tool-data/shared/ucsc. > > For a test, I have the human chromosomes len files and Trackster loads it without any problem. > > Any suggestion would greatly help me. > > Thanks > RT > > Log file: > ----------- > 127.0.0.1 - - [10/Apr/2014:12:29:44 +0800] "GET /visualization/trackster HTTP/1.1" 200 - "http://127.0.0.1:8081/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > 127.0.0.1 - - [10/Apr/2014:12:29:45 +0800] "GET /api/genomes?chrom_info=True HTTP/1.1" 200 - "http://127.0.0.1:8081/visualization/trackster" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > 127.0.0.1 - - [10/Apr/2014:12:29:51 +0800] "GET /api/genomes/PF3D7?low=0&num=100 HTTP/1.1" 500 - "http://127.0.0.1:8081/visualization/trackster" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > Error - <type 'exceptions.IndexError'>: list index out of range > URL: http://127.0.0.1:8081/api/genomes/PF3D7?low=0&num=100 > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ > app_iter = self.application(environ, sr_checker) > File '/data/maarthi/Galaxy/Test/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ > return self.application(environ, start_response) > File '/data/maarthi/Galaxy/Test/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ > return self.application(environ, start_response) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ > return self.handle_request( environ, start_response ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request > body = method( trans, **kwargs ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 78 in decorator > return to_json_string( func( self, trans, *args, **kwargs ) ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/webapps/galaxy/api/genomes.py', line 42 in show > rval = self.app.genomes.chroms( trans, dbkey=id, num=num, chrom=chrom, low=low ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/visualization/genomes.py', line 292 in chroms > rval = genome.to_dict( num=num, chrom=chrom, low=low ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/visualization/genomes.py', line 153 in to_dict > chroms[ fields[0] ] = int( fields[1] ) > IndexError: list index out of range > > > CGI Variables > ------------- > CONTENT_LENGTH: '0' > HTTP_ACCEPT: 'application/json, text/javascript, */*; q=0.01' > HTTP_ACCEPT_ENCODING: 'gzip, deflate' > HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5' > HTTP_CONNECTION: 'keep-alive' > HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351' > HTTP_HOST: '127.0.0.1:8081' > HTTP_REFERER: 'http://127.0.0.1:8081/visualization/trackster' > HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0' > HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' > PATH_INFO: '/api/genomes/PF3D7' > QUERY_STRING: 'low=0&num=100' > REMOTE_ADDR: '127.0.0.1' > REQUEST_METHOD: 'GET' > SERVER_NAME: '0.0.0.0' > SERVER_PORT: '8081' > SERVER_PROTOCOL: 'HTTP/1.1' > > > WSGI Variables > -------------- > application: <paste.recursive.RecursiveMiddleware object at 0x6b37cd0> > is_api_request: True > paste.cookies: (<SimpleCookie: galaxysession='c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351'>, 'galaxysession=c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351') > paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] > paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x748eed0> > paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x795c310> > paste.parsed_querystring: ([('low', '0'), ('num', '100')], 'low=0&num=100') > paste.recursive.forward: <paste.recursive.Forwarder from /> > paste.recursive.include: <paste.recursive.Includer from /> > paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> > paste.recursive.script_name: '' > paste.throw_errors: True > request_id: 'c1be491ec06811e39ed60025905a9e10' > webob._parsed_query_vars: (MultiDict([('low', '0'), ('num', '100')]), 'low=0&num=100') > wsgi process: 'Multithreaded' > ------------------------------------------------------------ > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/

2 1

api dataset id problem
by Pandori n 11 Apr '14

11 Apr '14

Hi! What do we want : Using the API (bioblend 0.4.3, Galaxy-dist ( 22 jan 2014 ) ): upload files from user's FTP directory to Galaxy and launch workflow How we do it : 1. Determine the user's *libraryID* and *historyID* 2. Upload files : *gi.libraries.upload_file_from_server (libraryID, server_dir)* and get the *datasetID* in user's library (*libraryID*). 3. Check the status of the upoaded file : *gi.datasets.show_dataset (datasetID)*. 4. If state of dataset is *'ok'* (*gi.datasets.show_dataset (datasetID) ['state']*), we import it to the history (*historyID*): *gi.histories.upload_dataset_from_library (historyID, datasetID, 'library')* 5. Now we can run workflow Problem: For the new user the command *gi.datasets.show_dataset (datasetID)* works properly and displays information about the required dataset, but after a while (after some upload iterations) the command *gi.datasets.show_dataset(datasetID)* displays information about wrong dataset ( with identical *datasetID*, but not the current member of user's library (*libraryID*). But in bioblend docs: " *show_dataset(dataset_id, deleted=False)* Display information about and/or content of a dataset. This can be a history or a library dataset." Example: libraryID = gi.libraries.get_libraries()[ 0 ][ 'id' ] u'*417e33144b294c21*' lib = gi.libraries.show_library(libraryID, True) [ { u'url': u'/api/libraries/417e33144b294c21/contents/c8990a1b374598b0', u'type': u'file', u'id': u*'c8990a1b374598b0*', u'name': u'/opo_opo.csfasta' } ] datasetID = *c8990a1b374598b0* http://<server>/api/datasets/c8990a1b374598b0?key=<api_key> gi.datasets.show_dataset('c8990a1b374598b0') { u'accessible': True, u'visible': True, u'copied_from_ldda_id': u'911dde3ddb677bcd', u'file_size': 0, u'file_ext': u'data', *u'id': u'c8990a1b374598b0', * u'misc_info': u'failure running job', u'hda_ldda': u'hda', u'download_url': u'/api/histories/5969b1f7201f12ae/contents/c8990a1b374598b0/display', u'state': u'error', u'display_types': [], u'display_apps': [], u'misc_blurb': u'tool error', u'peek': u'<table cellspacing="0" cellpadding="3"></table>', u'data_type': u'data', u'tags': [], u'deleted': True, u'history_id': u'5969b1f7201f12ae', u'genome_build': u'?', u'hid': 220, u'visualizations': [], u'metadata_dbkey': u'?', *u'name': u'dataset_1.dat', *<-????? u'uuid': None, u'model_class': u'HistoryDatasetAssociation', u'purged': False, u'api_type': u'file' } BUT --- http:// <server>/api/libraries/417e33144b294c21/contents/c8990a1b374598b0?key=<api_key> gi.libraries.show_dataset(librarypID, '*c8990a1b374598b0*') { "data_type": "csfasta", "date_uploaded": "2014-04-08T12:07:14.357356", "file_name": "/home/pandorin/galaxy-dist/database/files/000/dataset_228.dat", "file_size": 82283303, "folder_id": "417e33144b294c21", "genome_build": "?", "id": "c8990a1b374598b0", "ldda_id": "*c8990a1b374598b0*", "message": "", "metadata_data_lines": null, "metadata_dbkey": "?", "metadata_sequences": null, "misc_blurb": "78.5 MB", "misc_info": "\nuploaded csfasta file", "model_class": "LibraryDataset", "name": "*opo_opo.csfasta*", "template_data": {}, "uploaded_by": "newnew(a)mail.ru", "uuid": null } In *libraries.show_dataset() *no '*state*' attribute. Best regards, Anatoly

1 0

Uploading a large number of files to a library creates duplicates
by Yves Gagnon 11 Apr '14

11 Apr '14

Hello fellow Galaxy devs. I stumbled upon this old thread from 2011. http://dev.list.galaxyproject.org/uploading-a-large-number-of-files-to-a-li… We still have the issue here. When we try to upload a large number of files to a library, it duplicates each files 5 times. Does anybody knows what could be causing this or how to prevent it? I wonder if this is related to another problem that we have when executing workflows with a relatively huge (50~100) amount of steps. When we click the button to send to a new history, it duplicates the new history 5 times also. http://dev.list.galaxyproject.org/Job-execution-order-mixed-up-td4662488.ht… The fact that I wonder if both are related is that the amount of duplicate is the same with both issues. It happens whether we have multiple web and job handlers or not. We tried both. Thanks for your time in advance! Yves Gagnon

1 0

understanding change of colours in history
by Neil.Burdett＠csiro.au 11 Apr '14

11 Apr '14

Hi, Can someone point me in the direction of where the code in Galaxy is called to change the history background for example when a job changes from Yellow to red on error, or yellow to green on success. I've been going around in circles now trying to figure out, how this is implemented and how the labels are updated on the history. Also, If I change history_common.mako file and I restart the system my changes aren't seen (i.e. add a new icon if the system goes into error) Thanks for any insight Neil

2 3

Can a tool upload a .loc file that is then reused by subsequent versions of the tool?
by Dooley, Damion 10 Apr '14

10 Apr '14

Hey folks! I'm trying to revise a tool that allows the user to select from an extendable list of template files for generating blast report output. The idea is that if some reporting needs aren't covered by a generic template, then they can develop their own template which can then show up on the list. And, wanting this to work with a tool that is versioned in the toolshed! So: 1) is there a way to include a .loc.sample in the tool that gets uploaded, and then can be manually revised over time? A .loc file that subsequent versions of the tool would reuse (instead of using their own minimal one that comes with toolshed install)? I can simply have the default template in the tools code folder, since it is apt to be upgraded over time. Custom reports would have their own absolute file paths defined in the .loc file. Thanks for your input! Damion

2 4

example_watch_folder.py problem importing file
by Liisa Koski 10 Apr '14

10 Apr '14

Hello, I am testing the example_watch_folder.py script on my local instance of Galaxy (Feb.10 distribution). I have set up a simple workflow with the Fasta-to-Tabular tool. It takes a single input fasta file. Works in Galaxy UI. I have created input and output folders and followed the steps here http://gmod.827538.n3.nabble.com/Trouble-Shooting-example-watch-folder-py-t… But when I run the script and put a simple fasta file in the input folder I get the following error in the logs: galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) command is: python galaxy_dist_dev/tools/data_source/upload.py galaxy_dist_dev galaxy_dist_dev/database/tmp/tmpK40iV3galaxy_dist_dev/database/tmp/tmpx_5QyU 3802:galaxy_dist_dev/database/job_working_directory/001/1899/dataset_3802_files:galaxy_dist_dev/database/files/003/dataset_3802.dat; return_code=$?; cd galaxy_dist_dev; galaxy_dist_dev/set_metadata.sh ./database/files galaxy_dist_dev/database/job_working_directory/001/1899 . galaxy_dist_dev/universe_wsgi.ini galaxy_dist_dev/database/tmp/tmpK40iV3 galaxy_dist_dev/database/job_working_directory/001/1899/galaxy.json; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) native specification is: -V -q all.q -l hostname="hostname.ca" galaxy.jobs.runners.drmaa INFO 2014-04-10 11:11:03,133 (1899) queued as 678390 galaxy.jobs DEBUG 2014-04-10 11:11:03,180 (1899) Persisting job destination (destination id: name_hwew) galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,579 (1899/678390) state change: job is running 10.1.1.111 - - [10/Apr/2014:11:11:07 -0400] "POST /api/workflows?key=dd3916dfb37dffc08f070f1e3503015a HTTP/1.1" 200 - "-" "Python-urllib/2.6" galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:07,778 (1899/678390) state change: job finished normally galaxy.jobs DEBUG 2014-04-10 11:11:08,082 setting dataset state to ERROR galaxy.jobs DEBUG 2014-04-10 11:11:08,227 job 1899 ended galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,227 Cleaning up external metadata files galaxy.jobs DEBUG 2014-04-10 11:11:08,652 (1900) Working directory for job is: galaxy_dist_dev/database/job_working_directory/001/1900 galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,770 Cleaning up external metadata files galaxy.jobs.handler INFO 2014-04-10 11:11:08,795 (1900) Job unable to run: one or more inputs in error state 10.202.22.186 - - [10/Apr/2014:11:11:31 -0400] "GET /history/list HTTP/1.1" 200 - "http://galaxy.server.ca:8080/root" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0" And in the dataset itself in the resulting Galaxy history UI WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'asn1', using datatype from galaxy_dist_dev/database/tmp/tmpK40iV3. Yesterday, before I did the upgrade to Feb.10. The input files would import (as a green dataset in the History UI) put be empty and say 'no peak' Thanks in advance for any help, Liisa

1 0

enforcing minimum length on text params
by Sanka, Ravi 10 Apr '14

10 Apr '14

Greetings, I am adding a new tool to our in-house Galaxy, which has a few string inputs. I have made each one a text param in the XML, but want to ensure that the user does not leave any empty when he executes the tool. How can I do this? I tried setting the optional field of each text param to false, but that doesn't work since, technically, an empty string still counts as a string. Is there a minimum length setting I can enforce? ---------------------------------------------- Ravi Sanka ICS – Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 ----------------------------------------------

2 4

LWR Support
by Alexandre Defelicibus 10 Apr '14

10 Apr '14

Hi all, I have some doubts about how to configure a LWR server on Linux. I have read the LWR documentation, however how to configure the LWR is not clear to me. This is my scenario: On my Galaxy server I have some tools that I would like to send their jobs to the LWR server. These tools are at the same folder (xml and py). The LWR server is installed on Linux, and it is accessible through the URL. This is my job_conf.xml: <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="lwr" type="runner" load="galaxy.jobs.runners.lwr:LwrJobRunner"> </plugin> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> <destination id="lwr" runner="lwr"> <param id="url">http://200.144.255.42:8005/</param> </destination> </destinations> <tools> <tool id="2pg" destination="lwr"/> </tools> <limits> <limit type="registered_user_concurrent_jobs">2</limit> <limit type="unregistered_user_concurrent_jobs">1</limit> <limit type="job_walltime">24:00:00</limit> <limit type="concurrent_jobs" id="local">1</limit> </limits> </job_conf> When I execute the tool, I get this error: ValueError: No JSON object could be decoded At the LWR server, I use a nginx server, and I see on the log file that the server receive this URL: GET /setup?tool_id=2pg&job_id=24&tool_version=1.0.0 So, I concluded that the Galaxy server is accessing the LWR server and creating a job, but something wrong happens on the LWR side, but I couldn't realize what is missing. So, I have some questions to ask: - Do I need to have a Galaxy instance running at the same server of LWR? - Are there other variables that I should have set on Galaxy or on LWR server? - I know it is better if I use the secure LWR, and I will do, but is there something wrong on my job_conf.xml file? I hope I have shown all the needed details about my installation. Best regards, Alexandre -- Alexandre Defelicibus Mestrando em Bioengenharia Programa de Pós-Graduação em Bioengenharia Universidade de São Paulo - USP

2 1

Re: [galaxy-dev] Launching a workflow via API, multiple params for one tool
by Nicola Soranzo 10 Apr '14

10 Apr '14

Il 2014-04-09 22:18 John Marmaduke Eppley ha scritto: > As far as I can figure out, the parameter map for running a workflow > via the API is supposed to look like this: > > {'blastn': {'param': 'evalue', 'value': '1e-06’}} > > This doesn’t seem to allow for multiple parameters to a single tool. > Is there a way to submit multiple parameters that I’m missing? For > example: > > {'blastn': {'param': 'evalue', 'value': '1e-06’, 'param': > ‘identity_cutoff’, ‘value’: ’70’}} > > …will not work. The dict() container doesn’t allow duplicate keys, > it just overwrites the values. Hi John, you are using the old and deprecated syntax, you can use instead: {'blastn': {'evalue': '1e-06', 'identity_cutoff': 70}} For more details look at _update_step_parameters() function documentation: https://bitbucket.org/galaxy/galaxy-dist/src/tip/lib/galaxy/webapps/galaxy/… Or look at this complete example: https://github.com/crs4/bioblend/blob/objects/docs/examples/w5_galaxy_api.py Moreover, you may consider using the more high-level bioblend API: http://bioblend.readthedocs.org/en/latest/ Best, Nicola

2 1

Bowse button on upload data tool page...
by Neil.Burdett＠csiro.au 10 Apr '14

10 Apr '14

Hi, how is the "Browse..." button implemented on the data upload tool page? When you click on it you get a new window appearing so you can select files to upload. Does anyone know where is the "Browse..." button is defined/implemented in the code? I thought it would be in "./templates/webapps/galaxy/tool_form.mako" but can't see any references to it Thanks foir any help Neil

2 1