- galaxy-dev - lists.galaxyproject.org

Authentication failed when pushing to tool shed
by Erwan DELAGE 02 Mar '15

02 Mar '15

Hello, I’m trying to install a tool shed and I’m having an issue when pushing a tool on it. The authentication to both Galaxy and Galaxy tool shed is handled by Apache through a LDAP. I can access the tool shed interface, create repository and clone it without problem. I can even upload files through the web interface But when I try to push a tool, is asks me twice for authentication, and it failed the second time as you can see below : pushing to http://edelage@galaxy-bird.univ-nantes.fr/toolshed/repos/edelage/test_01 http authorization required realm: Galaxy-Bird user: edelage password: searching for changes 1 changesets found http authorization required realm: hgweb in the tool shed user: edelage password: abort: authorization failed Here are the apache logs : 193.52.103.60 - - [02/Mar/2015:16:13:36 +0100] "GET /toolshed/repos/edelage/test_01?cmd=capabilities HTTP/1.1" 401 493 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:13:56 +0100] "GET /toolshed/repos/edelage/test_01?cmd=capabilities HTTP/1.1" 200 130 "-" "mercurial/proto-1.0" 193.52.103.60 - - [02/Mar/2015:16:13:56 +0100] "GET /toolshed/repos/edelage/test_01?cmd=batch HTTP/1.1" 401 493 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:13:56 +0100] "GET /toolshed/repos/edelage/test_01?cmd=batch HTTP/1.1" 200 43 "-" "mercurial/proto-1.0" 193.52.103.60 - - [02/Mar/2015:16:13:56 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 493 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:13:56 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:14:13 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:14:13 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:14:13 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:14:13 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0" 193.52.103.60 - edelage [02/Mar/2015:16:14:13 +0100] "POST /toolshed/repos/edelage/test_01?cmd=unbundle HTTP/1.1" 401 260 "-" "mercurial/proto-1.0 » and here the tool_shed_webapp.log : 193.52.105.131, 193.52.103.60 - - [02/Mar/2015:16:13:56 +0200] "GET /toolshed/repos/edelage/test_01?cmd=capabilities HTTP/1.1" 200 130 "-" "mercurial/proto-1.0" 193.52.105.131, 193.52.103.60 - - [02/Mar/2015:16:13:56 +0200] "GET /toolshed/repos/edelage/test_01?cmd=batch HTTP/1.1" 200 43 "-" "mercurial/proto-1.0 » I probably missed something. I would appreciate any help. Thanks Erwan

1 0

Please post a Galaxy vacancy at the University of Oslo
by Nikolay Aleksandrov Vazov 02 Mar '15

02 Mar '15

Opening @ University of Oslo<https://wiki.galaxyproject.org/News/UiODeveloperBioinformatician> The University Center for Information Technology (USIT)<http://www.usit.uio.no/english/> at the University of Oslo has an opening for a developer / bioinformatician<http://uio.easycruit.com/vacancy/1347839/69316?iso=no> (announcement: Norwegian<http://uio.easycruit.com/vacancy/1347839/69316?iso=no>, English<https://translate.google.com/translate?sl=no&tl=en&u=http%3A%2F%2Fuio.easyc…>). The position includes the development and management of services aimed at life science, with particular focus on bioinformatics and our research portals that offer supercomputing via a simpler user interface. The position will maintain and implement solutions on a local Galaxy server. The position is permanent, and group communication is in English. N.B. The application deadline is March 12, 2015. The link to the application is here http://uio.easycruit.com/vacancy/1347839/69316?iso=no Sorry, the text is only in Norwegian, please use google translator :) Don't hesitate to get in touch with the contact persons in the announcement or Nikolay Vazov (n.a.vazov(a)usit.uio.no) from the DevTeam at University in Oslo Nikolay Vazov<http://www.usit.uio.no/english/about/organisation/bps/rc/rss/staff/nikolaiv…> =============== Nikolay Vazov, PhD Department for Research Computing, University of Oslo

1 0

Tool Testing: field inputs that occur inside repeat tag
by Dooley, Damion 01 Mar '15

01 Mar '15

Helop aI've been testing (using planemo) some tools scheduled for toolshed publication. My first test works, but Galaxy reports that the second one fails with: "Parameter %s requires a value, but has no legal values defined" % self.name AssertionError: Parameter filter_column requires a value, but has no legal values defined requests.packages.urllib3.connectionpool: DEBUG: "POST /api/tools HTTP/1.1" 500 None I presume its because of extra stuff needed to name variables have to be referred-to when they are inside a repeat section? Or is it just something pecular to do with data_table_sources? <test> <param name="blastxml_file" value="blast_reporting_1.blastxml"/> <output name="tabular_file" file="blast_reporting_1a.tabular"/> <param name="out_format" value="std"/> <param name="column_labels" value="" /> <param name="drop_redundant_hits" value="true"/> </test> <test> <param name="blastxml_file" value="blast_reporting_1.blastxml"/> <output name="tabular_file" file="blast_reporting_1b.tabular"/> <param name="out_format" value="std"/> <param name="column_labels" value="" /> <param name="drop_redundant_hits" value="true"/> <param name="filter_column" value="pident"/> <param name="filter_comparison" value="gte"/> <param name="filter_value" value="97"/> </test> The form repeat area: <repeat name="filter_num" title="Numeric Filter" min="0" max="4"> <param name="filter_column" type="select" label="Col"> <options from_data_table="blast_reporting_fields"> <filter type="static_value" value="numeric" column="type" /> <filter type="sort_by" column="name"/> </options> </param> <repeat name="constraint" title="Constraint" min="1" max="3"> <param name="filter_comparison" type="select" label="Comparison"> <option value="gte">>= </option> <option value="gt">></option> <option value="lt">< </option> <option value="lte"><= </option> <option value="==">equal to </option> <option value="!=">not equal to </option> </param> <param name="filter_value" type="text" value="" label="Value"> <validator type="regex" message="Please input a number">([0-9]+|[0-9]*\.[0-9]+)</validator> </param> </repeat> </repeat> The working command section that sends the filter field info is: #for $i, $my_repeat in enumerate( $filter_num ) $my_repeat.filter_column: #for $i, $my_repeat2 in enumerate( $my_repeat.constraint ) $my_repeat2.filter_comparison $my_repeat2.filter_value, #end for ; #end for Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

3 7

problems installing tool dependency
by Sajdak, Doris 27 Feb '15

27 Feb '15

Hi all, I am trying to install the Genome Diversity tools from the Miller Lab that are in the Galaxy toolshed. I've been able to get all dependencies installed except Quicktree. Whenever I try to install it, I get the error: Error downloading from URL ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz: <urlopen error ftp error: 550 pub4: No such file or directory> This is the tool_dependencies.xml file for installing quicktree: <?xml version="1.0"?> <tool_dependency> <package name="quicktree" version="1.1"> <install version="1.0"> <actions>  <action type="download_by_url" target_filename="quicktree_1.1.tar.gz">ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz</action>  <action type="shell_command">make quicktree</action>  <action type="shell_command">cp -R bin $INSTALL_DIR</action>  <action type="set_environment"> <environment_variable name="PATH" action="prepend_to">$INSTALL_DIR/bin</environment_variable> </action> </actions> </install> </package> </tool_dependency> I know this is not their file I'm downloading but the thing is, if I copy and paste the URL, I can download it with no problem. I'm not sure why it's reporting that it can't be found but the xml looks right to me. I've tried contacting the Miller Lab about this but haven't heard from anyone so I'm hoping someone here can tell me how to move past this problem. Is there a way to install this tool manually and have the main Genome Diversity tools recognize it on the Galaxy server? Also, I noticed when looking at the Tool Dependency Definitions at https://toolshed.g2.bx.psu.edu/ there are errors going back many versions for this tool (see Test runs - Installation errors - Tool dependencies). Thank you for any help you can provide. Dori ********************************** Dori Sajdak Senior Systems Administrator State University of NY at Buffalo Center for Computational Research 701 Ellicott St Buffalo, NY 14203 Phone: (716) 881-8934 Fax: (716) 849-6656 Web: http://ccr.buffalo.edu **********************************

3 7

Database deadlock with large workflows + dataset collections
by Aaron Petkau 27 Feb '15

27 Feb '15

Hey, I wanted to know if anyone else has had experience with database deadlock when using dataset collections and running a large number of samples through a workflow. Traceback (most recent call last): File "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 565, in finish_job job_state.job_wrapper.finish( stdout, stderr, exit_code ) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1250, in finish self.sa_session.flush() File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py", line 114, in do return getattr(self.registry(), name)(*args, **kwargs) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1718, in flush self._flush(objects) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py", line 1789, in _flush flush_context.execute() File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 331, in execute rec.execute(self) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py", line 475, in execute uow File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 59, in save_obj mapper, table, update) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py", line 485, in _emit_update_statements execute(statement, params) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1449, in execute params) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) DBAPIError: (TransactionRollbackError) deadlock detected DETAIL: Process 25859 waits for ShareLock on transaction 144373; blocked by process 25858. Process 25858 waits for ShareLock on transaction 144372; blocked by process 25859. HINT: See server log for query details. 'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE workflow_invocation.id = %(workflow_invocation_id)s' {'update_time': datetime.datetime(2015, 2, 27, 3, 51, 57, 81403), 'workflow_invocation_id': 48} I saw this post http://dev.list.galaxyproject.org/data-collections-workflow-bug-td4666496.h… with a similar issue and the solution was to make sure not to use a sqlite database, but I'm using a postgres database and still encountered this issue. This was after running a very large number of samples (~200) using dataset collections. Just wondering if anyone else was running into this issue? Thanks, Aaron

2 2

$GALAXY_TEST_DB_TEMPLATE failing with KeyError: <VerNum(127)>
by Peter Cock 27 Feb '15

27 Feb '15

Hi all, I've been using the SQLite template database feature to speed up functional testing. Rather than starting for the initial schema and applying all the incremental changes (as done when installing a fresh Galaxy instance), this lets the test suite start with a recent version of the database, by setting the environment variable $GALAXY_TEST_DB_TEMPLATE to a URL to fetch a template database, e.g. from: https://github.com/jmchilton/galaxy-downloads Database schema version 120 was working, changing to 127 broke: https://github.com/peterjc/pico_galaxy/commit/412d3fe88b8011f0b690956d129fd… TravisCI using v120, worked: https://travis-ci.org/peterjc/pico_galaxy/builds/52308064 TravisCI using v127, broke: https://travis-ci.org/peterjc/pico_galaxy/builds/52398778 galaxy.model.migrate.check DEBUG 2015-02-27 10:05:33,274 pysqlite>=2 egg successfully loaded for sqlite dialect Traceback (most recent call last): File "./scripts/functional_tests.py", line 602, in <module> sys.exit( main() ) File "./scripts/functional_tests.py", line 409, in main app = UniverseApplication( **kwargs ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/app.py", line 49, in __init__ self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/config.py", line 779, in _configure_models create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 61, in create_or_verify_database migrate() File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 57, in migrate migrate_to_current_version( engine, db_schema ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py", line 117, in migrate_to_current_version changeset = schema.changeset( None ) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py", line 80, in changeset changeset = self.repository.changeset(database, start_ver, version) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py", line 225, in changeset changes = [self.version(v).script(database, op) for v in versions] File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py", line 189, in version return self.versions.version(*p, **k) File "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/version.py", line 140, in version return self.versions[VerNum(vernum)] KeyError: <VerNum(127)> This is now testing against the (beta) Galaxy on GitHub: https://github.com/galaxyproject/galaxy Is it possible this is out of synch with the SQLite templates John posts here? https://github.com/jmchilton/galaxy-downloads Peter

2 2

problem with latest toolshed devteam cuffmerge wrapper multi-select
by Nikhil Joshi 26 Feb '15

26 Feb '15

Hi all, We run our own custom galaxy instance in the Amazon Cloud and I am updating our instance with the latest toolshed files. For cuffmerge, in past revisions, we would just have a button to add another GTF input. However, this latest one (10:b6e3849293b1 <https://toolshed.g2.bx.psu.edu/repository/view_changelog?id=84bbebf45888f374>) that has been replaced with a multi-select. We also have a few built-in genomes that we use with cuffmerge. When I choose the first GTF file (using locally cached sequence data) it will populate the reference genome correctly with our built-in genomes. However, once I choose more than one GTF file, I get one of two errors: No reference genome is available for the build associated with the selected input dataset OR An invalid option was selected, please verify The built-in genomes are definitely there and available for cuffmerge, but for some reason choosing more than one GTF file in the multi-select causes an error. Is this a bug or am I doing something wrong? - Nik. -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

3 4

custom java tools.xml : problem with classpath
by Pierre Lindenbaum 26 Feb '15

26 Feb '15

Hi Galaxy, I'm trying to put one of my java tool in galaxy. I've copied the required files in my tool directory under /tools/jvarkit (I can see it galaxy) $ ls commons-jexl-2.1.1.jar commons-logging-1.1.1.jar htsjdk-1.129.jar snappy-java-1.0.3-rc3.jar vcffilterjs.jar vcffilterjs.xml Here is the content of vcffilterjs.xml <?xml version="1.0"?> <tool hidden="false" id="com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS" version="1.0.0" name="vcffilterjs"> <requirements> <requirement type="binary">java</requirement> </requirements> <command><![CDATA[java -cp $INSTALL_DIR/commons-jexl-2.1.1.jar:$INSTALL_DIR/commons-logging-1.1.1.jar:$INSTALL_DIR/htsjdk-1.129.jar:$INSTALL_DIR/snappy-java-1.0.3-rc3.jar:$INSTALL_DIR/vcffilterjs.jar com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS -e '${expression}' -o ${output} ${input} 2> /tmp/error ]]></command> <inputs> <param format="vcf" name="input" type="data" label="VCF input"/> <param name="expression" type="text" label="javascript expression"/> </inputs> <outputs> <data format="vcf" name="output"/> </outputs> <stdio> <exit_code range="1:"/> <exit_code range=":-1"/> </stdio> <help>(...)</help> </tool> When I run this tool, the log says: NotFound: cannot find 'INSTALL_DIR' If I replace $INSTALL_DIR with \$INSTALL_DIR , the error on stderr is : Error: Could not find or load main class com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS How can I make this work ? Bonus question: is there a way to catch the error file to display it in galaxy when a process fails ? Thanks; Pierre

3 12

Importing file via FTP can silently fail, [Errno ftp error] 550
by Peter Cock 26 Feb '15

26 Feb '15

Hi all, Using either the legacy upload tool, or the new upload widget, the following FTP URL *appears* to load into Galaxy correctly giving a green history entry. However, on closer inspection it is an empty file with this peep text: Unable to fetch ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz [Errno ftp error] 550 pathogens: No such file or directory Clearly this should have been treated as a failure (red history entry). Should I file this as a bug on Trello? It seems there is a problem with the Sanger folder permissions which trips up Galaxy and curl (but does not bother Firefox and wget): $ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (9) Server denied you to change to the given directory Using curl -v for verbose mode it tries to walk the path (as per the https://www.ietf.org/rfc/rfc1738.txt standard) and fails with the command "CWD pathogens" matching the Galaxy error. The alternative curl FTP methods work: $ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz --ftp-method nocwd % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.8M 100 21.8M 0 0 7656k 0 0:00:02 0:00:02 --:--:-- 7654k and: $ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz --ftp-method singlecwd % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.8M 100 21.8M 0 0 7767k 0 0:00:02 0:00:02 --:--:-- 7766k You can also trick curl by replacing (some of the) slashes with the escaped version %2f which has the result of combining the CWD commands, e.g. $ curl -O ftp://ftp.sanger.ac.uk/pub%2fpathogens%2fBursaphelenchus%2fxylophilus%2fAssembly-v1.2/BurXv1.2.supercontigs.fa.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.8M 100 21.8M 0 0 6827k 0 0:00:03 0:00:03 --:--:-- 6829k This encoding trick does not seem to work within Galaxy. I plan to report this to Sanger tomorrow. Should they fix this quickly additional test cases might be useful (does Galaxy have an FTP server which might be used for testing data importing like this?). Is this a common issue? Should Galaxy automatically use an FTP CWD workaround in this situation? Regards, Peter

3 6

Subject: Serving Galaxy at a sub directory - symptoms without "filter-with = proxy-prefix"
by Dooley, Damion 25 Feb '15

25 Feb '15

This info is definitely useful! I ran into this exact same problem last week, trying to set up a new instance of galaxy from the mercurial repo. I was re-using a previous install's apache config; saw the same web page results. Had to go to the galaxy web docs to track down that filter-with setting. Damion

1 0