- galaxy-dev - lists.galaxyproject.org

Manually installing a Data Manager & running its tests
by Peter Cock 14 Apr '14

14 Apr '14

Hi all, I'm currently trying to manually install Dan's example Data Manager for the NCBI BLAST databases - but without using the Tool Shed: http://testtoolshed.g2.bx.psu.edu/view/blankenberg/data_manager_example_bla… To be precise, I'm currently using a slightly modified version which we've moved into the Galaxy BLAST+ wrapper repository (thus far, only slight changes to folder names have been made from Dan's original): https://github.com/peterjc/galaxy_blast/tree/master/data_managers/ncbi_blas… https://github.com/peterjc/galaxy_blast/commit/21d7cff3e8dca13c4bd0f716a79a… My goal is to be able to run the functional tests, $ ./run_functional_tests.sh -id data_manager_blast_db As with any other manually installed tool, I have added this data-manager to my tool_conf.xml.sample file (and for local testing via the GUI, the tool_conf.xml file as well): <tool file="ncbi_blastdb/blastdb.xml" /> Specific to data-managers, I have manually incorporated the additions to the central data_manager_conf.xml file (this would normally be merged at runtime if installed from the ToolShed): $ more data_manager_conf.xml <?xml version="1.0"?> <data_managers> <data_manager tool_file="ncbi_blastdb/blastdb.xml" id="data_manager_blast_db"> <data_table name="blastdb"> <output> <column name="value" /> <column name="name" /> <column name="path" output_ref="out_file" > <move type="directory"> <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">blastdb/${path}</target> </move> <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/blastdb/${path}/${nucleotide_alias_name}</value_translation> <value_translation type="function">abspath</value_translation> </column> </output> </data_table> </data_manager> </data_managers> Note the tool_file setting points correctly at the relative path within the tools folder, and the tool does define a test: $ more tools/ncbi_blastdb/blastdb.xml <tool id="data_manager_blast_db" name="Blast DB" version="0.0.2" tool_type="manage_data"> <description>Downloader</description> ... <tests> <test> <param name="blastdb_name" value="est"/> <param name="advanced_selector" value="basic"/> <output name="out_file" file="est_out.json"/> </test> </tests> ... $ ./run_functional_tests.sh -list | grep manager data_manager_blast_db Blast DB $ ./run_functional_tests.sh -id data_manager_blast_db ... galaxy.tools DEBUG 2014-04-11 15:59:00,780 Loaded tool id: data_manager_blast_db, version: 0.0.2 into tool panel. ... galaxy.tools.data_manager.manager DEBUG 2014-04-11 15:59:00,864 Loaded Data Manager: data_manager_blast_db ... Failure: ValueError (No such test TestForTool_data_manager_blast_db) ... ERROR ====================================================================== ERROR: Failure: ValueError (No such test TestForTool_data_manager_blast_db) ---------------------------------------------------------------------- Traceback (most recent call last): File "/mnt/galaxy/galaxy-central/eggs/nose-0.11.1-py2.6.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_data_manager_blast_db ---------------------------------------------------------------------- I get the same error via TravisCI, e.g. https://travis-ci.org/peterjc/galaxy_blast/builds/22782979 (This suggests using Twill vs the API does not matter) Am I missing something simple here? Thanks, Peter

2 3

Add a private genome in database/build , only visible for a specific user connected on a local instance.
by Sarah Maman 14 Apr '14

14 Apr '14

Hello, I would like to know if it's possible to limit the access to a private reference genome, for a specific user, in database/build field ? In fact, I would like that the reference genome is visible/shown only for a specific user connected to your local Galaxy instance, and hidden for others users. Should I specify something in galaxy-dist/tool-data/shared/ucsc/builds.txt file ? Thanks in advance, Sarah Maman -- --*-- Sarah Maman INRA - GenPhySE - SIGENAE http://www.sigenae.org/ Chemin de Borde-Rouge - Auzeville - BP 52627 31326 Castanet-Tolosan cedex - FRANCE Tel: +33(0)5.61.28.57.08 Fax: +33(0)5.61.28.57.53 --*--

1 0

Re: [galaxy-dev] [galaxy-user] Trackster error on dbkeys
by Jeremy Goecks 12 Apr '14

12 Apr '14

(Moved to galaxy-dev mailing list because it concerns server admin rather than Galaxy usage.) If everything worked fine with a human build but not for PF3d7, then my guess is that the len file for PF3d7 isn’t correctly formatted. Right now, the file parser is very unforgiving—no extra spaces or lines are allowed. If you check the len file and don’t spot an issue, please send to it to me and I’ll take a look. Best, J. -- Jeremy Goecks Assistant Professor of Computational Biology George Washington University On Apr 10, 2014, at 2:30 AM, Aarthi Mohan <aarthimohan11(a)gmail.com> wrote: > Hi all, > > I am setting up galaxy on a server running CentOS 6.5. > I found that Trackster application hits me with an error > "Could not load chroms for this dbkey: PF3D7", when I try "add new track" from Visualization. > > I am working with Plasmodium falciparum (PF3d7) genome, and I followed the instruction for setting visualization from "https://wiki.galaxyproject.org/Visualization%20Setup" and placed the 2bit, len files in the appropriate path. > > I have also made sure the genome build is present in the builds.txt under tool-data/shared/ucsc. > > For a test, I have the human chromosomes len files and Trackster loads it without any problem. > > Any suggestion would greatly help me. > > Thanks > RT > > Log file: > ----------- > 127.0.0.1 - - [10/Apr/2014:12:29:44 +0800] "GET /visualization/trackster HTTP/1.1" 200 - "http://127.0.0.1:8081/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > 127.0.0.1 - - [10/Apr/2014:12:29:45 +0800] "GET /api/genomes?chrom_info=True HTTP/1.1" 200 - "http://127.0.0.1:8081/visualization/trackster" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > 127.0.0.1 - - [10/Apr/2014:12:29:51 +0800] "GET /api/genomes/PF3D7?low=0&num=100 HTTP/1.1" 500 - "http://127.0.0.1:8081/visualization/trackster" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" > Error - <type 'exceptions.IndexError'>: list index out of range > URL: http://127.0.0.1:8081/api/genomes/PF3D7?low=0&num=100 > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ > app_iter = self.application(environ, sr_checker) > File '/data/maarthi/Galaxy/Test/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ > return self.application(environ, start_response) > File '/data/maarthi/Galaxy/Test/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ > return self.application(environ, start_response) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in __call__ > return self.handle_request( environ, start_response ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/base.py', line 190 in handle_request > body = method( trans, **kwargs ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 78 in decorator > return to_json_string( func( self, trans, *args, **kwargs ) ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/webapps/galaxy/api/genomes.py', line 42 in show > rval = self.app.genomes.chroms( trans, dbkey=id, num=num, chrom=chrom, low=low ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/visualization/genomes.py', line 292 in chroms > rval = genome.to_dict( num=num, chrom=chrom, low=low ) > File '/data/maarthi/Galaxy/Test/galaxy-dist/lib/galaxy/visualization/genomes.py', line 153 in to_dict > chroms[ fields[0] ] = int( fields[1] ) > IndexError: list index out of range > > > CGI Variables > ------------- > CONTENT_LENGTH: '0' > HTTP_ACCEPT: 'application/json, text/javascript, */*; q=0.01' > HTTP_ACCEPT_ENCODING: 'gzip, deflate' > HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5' > HTTP_CONNECTION: 'keep-alive' > HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351' > HTTP_HOST: '127.0.0.1:8081' > HTTP_REFERER: 'http://127.0.0.1:8081/visualization/trackster' > HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0' > HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' > PATH_INFO: '/api/genomes/PF3D7' > QUERY_STRING: 'low=0&num=100' > REMOTE_ADDR: '127.0.0.1' > REQUEST_METHOD: 'GET' > SERVER_NAME: '0.0.0.0' > SERVER_PORT: '8081' > SERVER_PROTOCOL: 'HTTP/1.1' > > > WSGI Variables > -------------- > application: <paste.recursive.RecursiveMiddleware object at 0x6b37cd0> > is_api_request: True > paste.cookies: (<SimpleCookie: galaxysession='c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351'>, 'galaxysession=c6ca0ddb55be603abc74375a63eca6a53211ad8efeb084140a6e95d9de0824b221b12b4bbc465351') > paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] > paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x748eed0> > paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x795c310> > paste.parsed_querystring: ([('low', '0'), ('num', '100')], 'low=0&num=100') > paste.recursive.forward: <paste.recursive.Forwarder from /> > paste.recursive.include: <paste.recursive.Includer from /> > paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /> > paste.recursive.script_name: '' > paste.throw_errors: True > request_id: 'c1be491ec06811e39ed60025905a9e10' > webob._parsed_query_vars: (MultiDict([('low', '0'), ('num', '100')]), 'low=0&num=100') > wsgi process: 'Multithreaded' > ------------------------------------------------------------ > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/

2 1

api dataset id problem
by Pandori n 11 Apr '14

11 Apr '14

Hi! What do we want : Using the API (bioblend 0.4.3, Galaxy-dist ( 22 jan 2014 ) ): upload files from user's FTP directory to Galaxy and launch workflow How we do it : 1. Determine the user's *libraryID* and *historyID* 2. Upload files : *gi.libraries.upload_file_from_server (libraryID, server_dir)* and get the *datasetID* in user's library (*libraryID*). 3. Check the status of the upoaded file : *gi.datasets.show_dataset (datasetID)*. 4. If state of dataset is *'ok'* (*gi.datasets.show_dataset (datasetID) ['state']*), we import it to the history (*historyID*): *gi.histories.upload_dataset_from_library (historyID, datasetID, 'library')* 5. Now we can run workflow Problem: For the new user the command *gi.datasets.show_dataset (datasetID)* works properly and displays information about the required dataset, but after a while (after some upload iterations) the command *gi.datasets.show_dataset(datasetID)* displays information about wrong dataset ( with identical *datasetID*, but not the current member of user's library (*libraryID*). But in bioblend docs: " *show_dataset(dataset_id, deleted=False)* Display information about and/or content of a dataset. This can be a history or a library dataset." Example: libraryID = gi.libraries.get_libraries()[ 0 ][ 'id' ] u'*417e33144b294c21*' lib = gi.libraries.show_library(libraryID, True) [ { u'url': u'/api/libraries/417e33144b294c21/contents/c8990a1b374598b0', u'type': u'file', u'id': u*'c8990a1b374598b0*', u'name': u'/opo_opo.csfasta' } ] datasetID = *c8990a1b374598b0* http://<server>/api/datasets/c8990a1b374598b0?key=<api_key> gi.datasets.show_dataset('c8990a1b374598b0') { u'accessible': True, u'visible': True, u'copied_from_ldda_id': u'911dde3ddb677bcd', u'file_size': 0, u'file_ext': u'data', *u'id': u'c8990a1b374598b0', * u'misc_info': u'failure running job', u'hda_ldda': u'hda', u'download_url': u'/api/histories/5969b1f7201f12ae/contents/c8990a1b374598b0/display', u'state': u'error', u'display_types': [], u'display_apps': [], u'misc_blurb': u'tool error', u'peek': u'<table cellspacing="0" cellpadding="3"></table>', u'data_type': u'data', u'tags': [], u'deleted': True, u'history_id': u'5969b1f7201f12ae', u'genome_build': u'?', u'hid': 220, u'visualizations': [], u'metadata_dbkey': u'?', *u'name': u'dataset_1.dat', *<-????? u'uuid': None, u'model_class': u'HistoryDatasetAssociation', u'purged': False, u'api_type': u'file' } BUT --- http:// <server>/api/libraries/417e33144b294c21/contents/c8990a1b374598b0?key=<api_key> gi.libraries.show_dataset(librarypID, '*c8990a1b374598b0*') { "data_type": "csfasta", "date_uploaded": "2014-04-08T12:07:14.357356", "file_name": "/home/pandorin/galaxy-dist/database/files/000/dataset_228.dat", "file_size": 82283303, "folder_id": "417e33144b294c21", "genome_build": "?", "id": "c8990a1b374598b0", "ldda_id": "*c8990a1b374598b0*", "message": "", "metadata_data_lines": null, "metadata_dbkey": "?", "metadata_sequences": null, "misc_blurb": "78.5 MB", "misc_info": "\nuploaded csfasta file", "model_class": "LibraryDataset", "name": "*opo_opo.csfasta*", "template_data": {}, "uploaded_by": "newnew(a)mail.ru", "uuid": null } In *libraries.show_dataset() *no '*state*' attribute. Best regards, Anatoly

1 0

Uploading a large number of files to a library creates duplicates
by Yves Gagnon 11 Apr '14

11 Apr '14

Hello fellow Galaxy devs. I stumbled upon this old thread from 2011. http://dev.list.galaxyproject.org/uploading-a-large-number-of-files-to-a-li… We still have the issue here. When we try to upload a large number of files to a library, it duplicates each files 5 times. Does anybody knows what could be causing this or how to prevent it? I wonder if this is related to another problem that we have when executing workflows with a relatively huge (50~100) amount of steps. When we click the button to send to a new history, it duplicates the new history 5 times also. http://dev.list.galaxyproject.org/Job-execution-order-mixed-up-td4662488.ht… The fact that I wonder if both are related is that the amount of duplicate is the same with both issues. It happens whether we have multiple web and job handlers or not. We tried both. Thanks for your time in advance! Yves Gagnon

1 0

understanding change of colours in history
by Neil.Burdett＠csiro.au 11 Apr '14

11 Apr '14

Hi, Can someone point me in the direction of where the code in Galaxy is called to change the history background for example when a job changes from Yellow to red on error, or yellow to green on success. I've been going around in circles now trying to figure out, how this is implemented and how the labels are updated on the history. Also, If I change history_common.mako file and I restart the system my changes aren't seen (i.e. add a new icon if the system goes into error) Thanks for any insight Neil

2 3

Can a tool upload a .loc file that is then reused by subsequent versions of the tool?
by Dooley, Damion 10 Apr '14

10 Apr '14

Hey folks! I'm trying to revise a tool that allows the user to select from an extendable list of template files for generating blast report output. The idea is that if some reporting needs aren't covered by a generic template, then they can develop their own template which can then show up on the list. And, wanting this to work with a tool that is versioned in the toolshed! So: 1) is there a way to include a .loc.sample in the tool that gets uploaded, and then can be manually revised over time? A .loc file that subsequent versions of the tool would reuse (instead of using their own minimal one that comes with toolshed install)? I can simply have the default template in the tools code folder, since it is apt to be upgraded over time. Custom reports would have their own absolute file paths defined in the .loc file. Thanks for your input! Damion

2 4

example_watch_folder.py problem importing file
by Liisa Koski 10 Apr '14

10 Apr '14

Hello, I am testing the example_watch_folder.py script on my local instance of Galaxy (Feb.10 distribution). I have set up a simple workflow with the Fasta-to-Tabular tool. It takes a single input fasta file. Works in Galaxy UI. I have created input and output folders and followed the steps here http://gmod.827538.n3.nabble.com/Trouble-Shooting-example-watch-folder-py-t… But when I run the script and put a simple fasta file in the input folder I get the following error in the logs: galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) command is: python galaxy_dist_dev/tools/data_source/upload.py galaxy_dist_dev galaxy_dist_dev/database/tmp/tmpK40iV3galaxy_dist_dev/database/tmp/tmpx_5QyU 3802:galaxy_dist_dev/database/job_working_directory/001/1899/dataset_3802_files:galaxy_dist_dev/database/files/003/dataset_3802.dat; return_code=$?; cd galaxy_dist_dev; galaxy_dist_dev/set_metadata.sh ./database/files galaxy_dist_dev/database/job_working_directory/001/1899 . galaxy_dist_dev/universe_wsgi.ini galaxy_dist_dev/database/tmp/tmpK40iV3 galaxy_dist_dev/database/job_working_directory/001/1899/galaxy.json; sh -c "exit $return_code" galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) native specification is: -V -q all.q -l hostname="hostname.ca" galaxy.jobs.runners.drmaa INFO 2014-04-10 11:11:03,133 (1899) queued as 678390 galaxy.jobs DEBUG 2014-04-10 11:11:03,180 (1899) Persisting job destination (destination id: name_hwew) galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,579 (1899/678390) state change: job is running 10.1.1.111 - - [10/Apr/2014:11:11:07 -0400] "POST /api/workflows?key=dd3916dfb37dffc08f070f1e3503015a HTTP/1.1" 200 - "-" "Python-urllib/2.6" galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:07,778 (1899/678390) state change: job finished normally galaxy.jobs DEBUG 2014-04-10 11:11:08,082 setting dataset state to ERROR galaxy.jobs DEBUG 2014-04-10 11:11:08,227 job 1899 ended galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,227 Cleaning up external metadata files galaxy.jobs DEBUG 2014-04-10 11:11:08,652 (1900) Working directory for job is: galaxy_dist_dev/database/job_working_directory/001/1900 galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,770 Cleaning up external metadata files galaxy.jobs.handler INFO 2014-04-10 11:11:08,795 (1900) Job unable to run: one or more inputs in error state 10.202.22.186 - - [10/Apr/2014:11:11:31 -0400] "GET /history/list HTTP/1.1" 200 - "http://galaxy.server.ca:8080/root" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0" And in the dataset itself in the resulting Galaxy history UI WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'asn1', using datatype from galaxy_dist_dev/database/tmp/tmpK40iV3. Yesterday, before I did the upgrade to Feb.10. The input files would import (as a green dataset in the History UI) put be empty and say 'no peak' Thanks in advance for any help, Liisa

1 0

enforcing minimum length on text params
by Sanka, Ravi 10 Apr '14

10 Apr '14

Greetings, I am adding a new tool to our in-house Galaxy, which has a few string inputs. I have made each one a text param in the XML, but want to ensure that the user does not leave any empty when he executes the tool. How can I do this? I tried setting the optional field of each text param to false, but that doesn't work since, technically, an empty string still counts as a string. Is there a minimum length setting I can enforce? ---------------------------------------------- Ravi Sanka ICS – Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 ----------------------------------------------

2 4

LWR Support
by Alexandre Defelicibus 10 Apr '14

10 Apr '14

Hi all, I have some doubts about how to configure a LWR server on Linux. I have read the LWR documentation, however how to configure the LWR is not clear to me. This is my scenario: On my Galaxy server I have some tools that I would like to send their jobs to the LWR server. These tools are at the same folder (xml and py). The LWR server is installed on Linux, and it is accessible through the URL. This is my job_conf.xml: <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="lwr" type="runner" load="galaxy.jobs.runners.lwr:LwrJobRunner"> </plugin> </plugins> <handlers default="handlers"> <handler id="handler0" tags="handlers"/> </handlers> <destinations default="local"> <destination id="local" runner="local"/> <destination id="lwr" runner="lwr"> <param id="url">http://200.144.255.42:8005/</param> </destination> </destinations> <tools> <tool id="2pg" destination="lwr"/> </tools> <limits> <limit type="registered_user_concurrent_jobs">2</limit> <limit type="unregistered_user_concurrent_jobs">1</limit> <limit type="job_walltime">24:00:00</limit> <limit type="concurrent_jobs" id="local">1</limit> </limits> </job_conf> When I execute the tool, I get this error: ValueError: No JSON object could be decoded At the LWR server, I use a nginx server, and I see on the log file that the server receive this URL: GET /setup?tool_id=2pg&job_id=24&tool_version=1.0.0 So, I concluded that the Galaxy server is accessing the LWR server and creating a job, but something wrong happens on the LWR side, but I couldn't realize what is missing. So, I have some questions to ask: - Do I need to have a Galaxy instance running at the same server of LWR? - Are there other variables that I should have set on Galaxy or on LWR server? - I know it is better if I use the secure LWR, and I will do, but is there something wrong on my job_conf.xml file? I hope I have shown all the needed details about my installation. Best regards, Alexandre -- Alexandre Defelicibus Mestrando em Bioengenharia Programa de Pós-Graduação em Bioengenharia Universidade de São Paulo - USP

2 1