April 2012 - galaxy-dev - lists.galaxyproject.org

update_metadata.sh references missing get_python.sh
by Peter Cock 26 Apr '12

26 Apr '12

Hello all, The shell script scripts/cleanup_datasets/update_metadata.sh tries to call non-existent file scripts/get_python.sh What is this intended to do? Thanks, Peter

1 0

Creating a galaxy tool in R - "You must not use 8-bit bytestrings"
by Dan Tenenbaum 26 Apr '12

26 Apr '12

Apologies for originally posting this to galaxy-user; now I realize it belongs here. Hello, I'm a galaxy newbie and running into several issues trying to adapt an R script to be a galaxy tool. I'm looking at the XY plotting tool for guidance (tools/plot/xy_plot.xml), but I decided not to embed my script in XML, but instead have it in a separate script file, that way I can still run it from the command line and make sure it works as I make incremental changes. (So my script starts with args <- commandArgs(TRUE)). Also, if it doesn't work, this suggests to me that there is a problem with my galaxy configuration. First, I tried using the r_wrapper.sh script that comes with the XY plotting tool, but it threw away my arguments: An error occurred running this job: ARGUMENT '/Users/dtenenba/dev/galaxy-dist/database/files/000/dataset_4.dat' __ignored__ ARGUMENT '/Users/dtenenba/dev/galaxy-dist/database/files/000/dataset_3.dat' __ignored__ ARGUMENT 'Fly' __ignored__ ARGUMENT 'Tagwise' __ignored__ etc. So then I tried just switching to Rscript: <command interpreter="bash">Rscript RNASeq.R $countsTsv $designTsv "$organism" $dispersion $minimumCountsPerMillion $minimumSamplesPerTranscript $out_file1 $out_file2</command> (My script produces as output a csv file and a pdf file. The final two arguments I'm passing are the names of those files.) But then I get an error that Rscript can't be found. So I wrote a little wrapper script, Rscript_wrapper.sh: #!/bin/sh Rscript $* And called that: <command interpreter="bash">Rscript_wrapper.sh RNASeq.R $countsTsv $designTsv "$organism" $dispersion $minimumCountsPerMillion $minimumSamplesPerTranscript $out_file1 $out_file2</command> Then I got an error that RNASeq.R could not be found. So then I added the absolute path to my R script to the <command> tag. This seemed to work (that is, it got me further, to the next error), but I'm not sure why I had to do this; in all the other tools I'm looking at, the directory to the script to run does not have to be specified; I assumed that the command would run in the appropriate directory. So now I've specified the full path to my R script: <command interpreter="bash">Rscript_wrapper.sh /Users/dtenenba/dev/galaxy-dist/tools/bioc/RNASeq.R $countsTsv $designTsv "$organism" $dispersion $minimumCountsPerMillion $minimumSamplesPerTranscript $out_file1 $out_file2</command> And I get the following long error, which includes all of the output of my R script: Traceback (most recent call last): File "/Users/dtenenba/dev/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 133, in run_job job_wrapper.finish( stdout, stderr ) File "/Users/dtenenba/dev/galaxy-dist/lib/galaxy/jobs/__init__.py", line 725, in finish self.sa_session.flush() File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/scoping.py", line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py", line 1356, in flush self._flush(objects) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py", line 1434, in _flush flush_context.execute() File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 261, in execute UOWExecutor().execute(self, tasks) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 768, in execute_save_steps self.save_objects(trans, task) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 759, in save_objects task.mapper._save_obj(task.polymorphic_tosave_objects, trans) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/mapper.py", line 1413, in _save_obj c = connection.execute(statement.values(value_params), params) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/Users/dtenenba/dev/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE job SET update_time=?, stdout=?, stderr=? WHERE job.id = ?' ['2012-04-24 18:55:45.791417', '', 'BiocInstaller version 1.5.7, ?biocLite for help\nWarning message:\nNAs introduced by coercion \nLoading required package: methods\nLoading required package: limma\nLoading required package: BiasedUrn\nLoading required package: geneLenDataBase\nLoading required package: org.Dm.eg.db\nLoading required package: AnnotationDbi\nLoading required package: BiocGenerics\n\nAttaching package: \xe2\x80\x98BiocGenerics\xe2\x80\x99\n\nThe following object(s) are masked from \xe2\x80\x98package:stats\xe2\x80\x99:\n\n xtabs\n\nThe following object(s) are masked from \xe2\x80\x98package:base\xe2\x80\x99:\n\n anyDuplicated, cbind, colnames, duplicated, eval, Filter, Find,\n get, intersect, lapply, Map, mapply, mget, order, paste, pmax,\n pmax.int, pmin, pmin.int, Position, rbind, Reduce, rep.int,\n rownames, sapply, setdiff, table, tapply, union, unique\n\nLoading required package: Biobase\nWelcome to Bioconductor\n\n Vignettes contain introductory material; view with\n \'browseVignettes()\'. To cite Bioconductor, see\n \'citation("Biobase")\', and for packages \'citation("pkgname")\'.\n\nLoading required package: DBI\n\nCalculating library sizes from column totals.\nError in matrix(u, nrow = nrows, byrow = TRUE) : \n negative extents to matrix\nCalls: plotMDS.DGEList ... equalizeLibSizes -> splitIntoGroups -> lapply -> FUN -> matrix\nExecution halted\n', 15] Note that if I run my script from the command line: ./Rscript_wrapper.sh RNASeq.R /Users/dtenenba/dev/galaxy-dist/database/files/000/dataset_4.dat /Users/dtenenba/dev/galaxy-dist/database/files/000/dataset_3.dat Fly 1 1 Tagwise MDSPlot.pdf outputs.csv It works fine and does not produce a warning about "NAs introduced by coercion", nor does it fail with the "Error in matrix" above. So, can anyone tell me what is going wrong here? Why does R behave differently in galaxy than it does on the command line? (I'm using the same instance of R, same machine, for my galaxy and command-line efforts). Is this 8-bit bytestring error a red herring? Can I filter it so that galaxy is happy? Finally, one other curiosity. Every time I hit "Execute" in galaxy to run my tool, it is run twice--two jobs are created (which each fail in the same way). Why is this? My R script: https://gist.github.com/2482783 My XML file: https://gist.github.com/2482792 I can share more data (such as sample input files) if necessary. Thanks for your help. Dan

2 5

Re: [galaxy-dev] About an additional tool
by Jim Johnson 25 Apr '12

25 Apr '12

I put a samtools_filter tool in the toolshed that uses samtools view command. I called the samtools view command from the command section of the tool_config and redirected stderr to stdout to avoid having galaxy interpret it as an error. Jim Johnson Minnesota Supercomputing Institute University of Minnesota On Tue, Apr 24, 2012 at 10:03 AM, Ciara Ledero <lederoc(a)gmail.com> wrote: > Hi there, > > I know? Galaxy already has a SAM-to-BAM converter, but part of my > exercise/task is to incorporate a script that uses samtools' view command. I > get this error: > > [samopen] SAM header is present: 66338 sequences. > > according to Galaxy. But this might not be an error at all. Is? there any > way that I could tell Galaxy to ignore this and just continue with the > script? > > Thanks in advance! Any help would be greatly appreciated. > > CL

2 2

Adding New filetype
by Unknown 25 Apr '12

25 Apr '12

Hi Galaxy-Dev, I have a a special set of .txt files which contains some comments and tabular feilds. I want to make them as a different group so that I can list them without any other .txt files. I went through the Add New Data Type section.But I didnt get a clear idea of how to do it? The datatypes_conf.xml file already has an entry for txt. What is mime type and how we know to which class I have to add the new file type? Deepthi

2 1

How does one survive the updates?
by Michael Moore 25 Apr '12

25 Apr '12

I was running a single instance of galaxy on my own machine, playing with library_import and figuring why BAM files were getting errors on upload. (It is a path matter, when one drops into the subshell and the workaround is ln -s /usr/bin/samtools /home/galaxy/bin/samtools ) Anyway, I seemed to have it figured last eve so I shut down the notebook where galaxy was running (on RHEL6) and went home. This am when I started galaxy with sh run.sh I got some messages about egg,ini and replacing universe_wsgi. files from universe_wsgi.sample. files and nothing worked. My registration was gone. My admin_user was overwritten. I re-registered, and restored the file settings and restarted galaxy, but now no login sticks. It does not complain about a registered user, but it does not appear to hold the session--the user tab does not show me as logged in. Is there some different way it is handling cookies? Michael Moore

2 1

dynamic_options
by Unknown 25 Apr '12

25 Apr '12

Hi all, Is there any help available which describes about the usage of dynamic_options? Is the <code> file should be a python executable or can we use perl? Thanks and regards, Deepthi

3 4

One tool ,multiple set of outputs
by Unknown 25 Apr '12

25 Apr '12

Hi all, Can I implement a tool which run 2 different algorithms and providing different set of output? Eg:Suppose I am having a tool named mytool which is running 2 different programs for 2 different conditions. Output of first program is a text file and fasta file and output of second program is a text file only. The tool should produce either the outputset1 or outputset2 according to the conditions. Can we use any kind of filter in the <outputs> tag?

3 2

Tool development: how to pass parameters stored in an input dataset
by Joachim Jacob 25 Apr '12

25 Apr '12

Hi, Is it possible to pass the parameters of an input dataset? For example: the user selects a dataset of the history. I want to pass to my wrapperscript the input parameters of that dataset (the information you see when clicking the small 'i' on a dataset in the history). Regards, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

1 1

problem with "~/lib/galaxy/app.py" after code upgrade to "March 12, 2012" release ('6799:40f1816d6857')
by Hans-Rudolf Hotz 25 Apr '12

25 Apr '12

Hi We are in the process of migrating our Galaxy servers to the current "March 12, 2012" release ('6799:40f1816d6857'). We have encountered two issues, which I guess are related: If I restart the server, I get: galaxy.model.migrate.check INFO 2012-04-16 09:24:50,391 At database version 93 galaxy.tool_shed.migrate.check DEBUG 2012-04-16 09:24:50,407 MySQL_python egg successfully loaded for mysql dialect Traceback (most recent call last): File "/****/galaxy_dist/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/****/galaxy_dist/lib/galaxy/app.py", line 37, in __init__ verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) File "/****/galaxy_dist/lib/galaxy/tool_shed/migrate/check.py", line 42, in verify_tools db_schema = schema.ControlledSchema( engine, migrate_repository ) File "/****/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 24, in __init__ self._load() File "/****/galaxy_dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 42, in _load data = list(result)[0] IndexError: list index out of range I can 'temporary' fix this by commenting-out line 36 and 37 in ~/lib/galaxy/app.py 36# from galaxy.tool_shed.migrate.check import verify_tools 37# verify_tools( self, db_url, kwargs.get( 'global_conf', {} ).get( '__file__', None ), self.config.database_engine_options ) ...well, I don't like this fix, and I wonder if there is a better one? I have a suspicion, that this is related to problems I encountered during the database schema upgrade to version 93 for our MySQL database: at step "90to91" and "91to92" I get the following: 90 -> 91... Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table. 0091_add_tool_version_tables DEBUG 2012-04-16 08:55:49,719 Creating tool_version table failed: (OperationalError) (1050, "Table 'tool_version' already exists") u'\nCREATE TABLE tool_version (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tcreate_time DATETIME, \n\tupdate_time DATETIME, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' () 0091_add_tool_version_tables DEBUG 2012-04-16 08:55:49,719 Creating tool_version table failed: (OperationalError) (1050, "Table 'tool_version' already exists") u'\nCREATE TABLE tool_version (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tcreate_time DATETIME, \n\tupdate_time DATETIME, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' () 0091_add_tool_version_tables DEBUG 2012-04-16 08:55:49,720 Creating tool_version_association table failed: (OperationalError) (1050, "Table 'tool_version_association' already exists") u'\nCREATE TABLE tool_version_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' () 0091_add_tool_version_tables DEBUG 2012-04-16 08:55:49,720 Creating tool_version_association table failed: (OperationalError) (1050, "Table 'tool_version_association' already exists") u'\nCREATE TABLE tool_version_association (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' () Added 0 rows to the new tool_version table. done 91 -> 92... Migration script to create the migrate_tools table. 0092_add_migrate_tools_table DEBUG 2012-04-16 08:55:49,920 Creating migrate_tools table failed: (OperationalError) (1050, "Table 'migrate_tools' already exists") u'\nCREATE TABLE migrate_tools (\n\trepository_id VARCHAR(255), \n\trepository_path TEXT, \n\tversion INTEGER\n)\n\n' () 0092_add_migrate_tools_table DEBUG 2012-04-16 08:55:49,920 Creating migrate_tools table failed: (OperationalError) (1050, "Table 'migrate_tools' already exists") u'\nCREATE TABLE migrate_tools (\n\trepository_id VARCHAR(255), \n\trepository_path TEXT, \n\tversion INTEGER\n)\n\n' () done the bizarre thing is, all three tables: 'migrate_tools', 'tool_version_association','tool_version' are present.....well, this explains the error: 'OperationalError 1050, "Table 'migrate_tools' already exists" '....but why were the commands executed a second and third time? - what happened during the first execution? I can execute the three create statements without problems in a dummy database on the same MySQl server (Server version: 5.1.41-3ubuntu12.10) and if I compare the "describe migrate_tools" outcome for the real database (galaxy_xenon1) and the dummy database (galaxy_xenon1_old), I don't see a difference and it corresponds to the 'create statement' mysql> describe galaxy_xenon1.migrate_tools; +-----------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------------+--------------+------+-----+---------+-------+ | repository_id | varchar(255) | YES | | NULL | | | repository_path | text | YES | | NULL | | | version | int(11) | YES | | NULL | | +-----------------+--------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) mysql> describe galaxy_xenon1_old.migrate_tools; +-----------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------------+--------------+------+-----+---------+-------+ | repository_id | varchar(255) | YES | | NULL | | | repository_path | text | YES | | NULL | | | version | int(11) | YES | | NULL | | +-----------------+--------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) mysql> Has anybody seen similar issues or can shed some light on our problems? Thank you very much for your help Hans

3 7

More meaningful dataset names/easier method of identifying?
by Josh Nielsen 25 Apr '12

25 Apr '12

Hello, For a while now with the Galaxy mirror that we have I have found on many occasions a need to identify which dataset_*.dat files on the file system (in the "[galaxy_dist]/database/files/000/" directory) belong to which user, and even for the same user to distinguish between their various datasets. Files directly uploaded by the user will have a Galaxy job & dataset file name which match - like a Galaxy job name of "data 18" (for example) which actually is reflective of the file name 'dataset_18.dat' on the file system. However any analysis on that file thereafter that produces another dataset does not give you a clue of the corresponding file name. For example, a "Clip on data 18" run some time later may be called 'dataset_44.dat' on the filesystem, and a "Map with Bowtie on data 18" that runs on the clipped 'dataset_44.dat' may produce an output file of 'dataset_53.dat'. When debugging failed jobs, and after the user has rerun them for the umpteenth time, there may be dozens of identical or near-identical files to weed through, and the generic naming scheme is not helpful even though it is sequential (also not easy to keep track of/match up unless you are watching the file writes in the directory live). The current implementation makes sense for internal usage and the code that uses it, but it is difficult for a human to distinguish which files match the jobs in Galaxy. It would be useful to have more meaningful dataset file names or an easier way to identify them (a record that matches the "internal" and "external" names) for administrative maintenance reasons so that I can delete files, or possibly even export those .dat files to a network share where our users can perform manual analysis on them. Could anyone point me to where in the code I could look to make the dataset names more meaningful? Or perhaps I should request of the Galaxy developers (as a feature) a way for the users themselves to see under the "metadata name" of their job (like "Map with Bowtie on data 18") in the right side pane the *actual* corresponding file and location on the file system path to it (dataset_53.dat, for example). Or if not for users at least something for Administrators. Even a database that has four columns for the internal/filesystem dataset name, the job metadata name, the Galaxy job number (that the user sees), and the user that the dataset belongs to, would be helpful. A lot of our users are heavy into informatics though and would probably prefer that the user be able to see that information. Does anyone have any suggestions or thoughts about this? Thanks, Josh Nielsen

3 2