September 2011 - galaxy-dev - lists.galaxyproject.org

downloading bowtie indexed files
by Nikhil Joshi 29 Nov '11

29 Nov '11

Hi all, Just wondering if there is a way to download the bowtie indexed files after indexing. It seems that the indexed output is simply a meta-file that points to the directory where the indexed files are kept.... but what if I want to download all of the indexed files themselves? I wrote something to do this... basically I created an html page that is the output of bowtie-build which then points to the created files... but I'm wondering if there is an easier way...? - Nik.

2 1

Dataset Cleanup Question
by Lance Parsons 17 Nov '11

17 Nov '11

I am running a local instance of Galaxy and I've been trying to sort out some issue with dataset cleanup. For the most part, things are working OK running the shell scripts in the recommended order: delete_userless_histories.sh purge_histories.sh purge_libraries.sh purge_folders.sh delete_datasets.sh purge_datasets.sh I have the number of days set to 10. When I look at the reports webapp however, it reports that there are "62 datasets were deleted more than 15 days ago, but have not yet been purged, disk space: 12975717335." These have stuck around now for 45 days (and counting). I have even tried running the scripts with the -f option to force galaxy to re-evaluate the datasets to no avail. Any suggestions? Thanks. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University

2 2

Adding bowtie-build ONLY as its own tool
by Nikhil Joshi 10 Nov '11

10 Nov '11

Hi all, So I am trying to add a tool to our local Galaxy, for which it needs a bowtie-build indexed reference, but NOT the alignment. The alignment step is done by the tool itself (using bowtie). My question is two-fold: One, why does the bowtie tool do both indexing and mapping? It seems that you would want to have the indexing separate from the alignment so that you can index once and then align as many times as you need. Right now it seems like (correct me if I am wrong) that EVERY time you do an alignment, Galaxy will index the reference. This seems like it is adding a lot more time to the alignment step than is necessary. Is this correct, or am I missing something? Two, so if I were to make bowtie-build into its own tool, how could I access the files that it creates, i.e. since bowtie-build takes a reference and a prefix as input and then creates multiple files that use the prefix.... I guess I need to access the prefix name somehow.... any help would be highly appreciated. Thanks! - Nik.

2 1

manage_db.sh upgrade error
by Iry Witham 08 Nov '11

08 Nov '11

When I run sh manage_db.sh upgrade I get the following error: 2 -> 3... Traceback (most recent call last): File "./scripts/manage_db.py", line 55, in <module> main( repository=repo, url=db_url ) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0003_security_and_libraries.py", line 423, in upgrade metadata.create_all() File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/schema.py", line 1796, in create_all bind.create(self, checkfirst=checkfirst, tables=tables) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1129, in create self._run_visitor(self.dialect.schemagenerator, entity, connection=connection, **kwargs) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/engine/base.py", line 1158, in _run_visitor visitorcallable(self.dialect, conn, **kwargs).traverse(element) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 89, in traverse return traverse(obj, self.__traverse_options__, self._visitor_dict) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 200, in traverse return traverse_using(iterate(obj, opts), obj, visitors) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/visitors.py", line 194, in traverse_using meth(target) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/compiler.py", line 829, in visit_metadata collection = [t for t in sql_util.sort_tables(tables) if self._can_create(t)] File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/sql/util.py", line 22, in sort_tables return topological.sort(tuples, tables) File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/topological.py", line 32, in sort return [n.item for n in _sort(tuples, allitems, allow_cycles=False, ignore_self_cycles=True)] File "/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg/sqlalchemy/topological.py", line 209, in _sort raise CircularDependencyError("Circular dependency detected " + repr(edges) + repr(queue)) sqlalchemy.exc.CircularDependencyError: Circular dependency detected [(form_definition_current, form_definition), (library_item_info, library_folder_info_association), (library_item_info, library_item_info_element), (library_item_info, library_item_info_permissions), (library_item_info, library_dataset_dataset_info_association), (library_item_info, library_info_association), (library_item_info, library_dataset_info_association), (visualization_revision, visualization), (form_definition, external_service), (form_definition, form_values), (form_definition, request_type), (form_definition, run), (form_definition, form_definition_current), (history_dataset_association, job_to_input_dataset), (history_dataset_association, job_to_output_dataset), (history_dataset_association, implicitly_converted_dataset_association), (history_dataset_association, history_dataset_association_tag_association), (history_dataset_association, validation_error), (history_dataset_association, history_dataset_association_rating_association), (history_dataset_association, job_external_output_metadata), (history_dataset_association, history_dataset_association_annotation_association), (history_dataset_association, history_dataset_association_display_at_authorization), (history_dataset_association, metadata_file), (galaxy_user, stored_workflow_rating_association), (galaxy_user, stored_workflow_menu_entry), (galaxy_user, library_item_info), (galaxy_user, visualization_user_share_association), (galaxy_user, default_user_permissions), (galaxy_user, visualization_rating_association), (galaxy_user, dataset_tag_association), (galaxy_user, page_user_share_association), (galaxy_user, library_dataset_info_association), (galaxy_user, user_action), (galaxy_user, history), (galaxy_user, library_info_association), (galaxy_user, history_dataset_association_tag_association), (galaxy_user, workflow_step_tag_association), (galaxy_user, history_dataset_association_rating_association), (galaxy_user, page), (galaxy_user, history_rating_association), (galaxy_user, stored_workflow), (galaxy_user, history_annotation_association), (galaxy_user, user_role_association), (galaxy_user, api_keys), (galaxy_user, request), (galaxy_user, history_user_share_association), (galaxy_user, user_address), (galaxy_user, library_dataset_dataset_association), (galaxy_user, history_tag_association), (galaxy_user, page_rating_association), (galaxy_user, library_dataset_dataset_info_association), (galaxy_user, library_folder_info_association), (galaxy_user, visualization_tag_association), (galaxy_user, stored_workflow_user_share_connection), (galaxy_user, galaxy_session), (galaxy_user, visualization), (galaxy_user, history_dataset_association_display_at_authorization), (galaxy_user, job), (galaxy_user, galaxy_user_openid), (galaxy_user, event), (galaxy_user, stored_workflow_tag_association), (galaxy_user, stored_workflow_annotation_association), (galaxy_user, user_preference), (galaxy_user, workflow_step_annotation_association), (galaxy_user, page_annotation_association), (galaxy_user, history_dataset_association_annotation_association), (galaxy_user, page_tag_association), (galaxy_user, user_group_association), (galaxy_user, visualization_annotation_association), (sample, sample_dataset), (sample, sample_run_association), (sample, sample_event), (workflow_invocation, workflow_invocation_step), (history, job), (history, history_annotation_association), (history, job_export_history_archive), (history, history_tag_association), (history, history_dataset_association), (history, job_import_history_archive), (history, default_history_permissions), (history, history_user_share_association), (history, history_rating_association), (history, event), (history, galaxy_session_to_history), (history, galaxy_session), (history, sample), (run, sample_run_association), (run, request_type_run_association), (request_type, sample_state), (request_type, request_type_permissions), (request_type, request_type_run_association), (request_type, request_type_external_service_association), (request_type, request), (page, page_tag_association), (page, page_user_share_association), (page, page_revision), (page, page_rating_association), (page, page_annotation_association), (stored_workflow, workflow), (stored_workflow, stored_workflow_rating_association), (stored_workflow, stored_workflow_menu_entry), (stored_workflow, stored_workflow_tag_association), (stored_workflow, stored_workflow_annotation_association), (stored_workflow, stored_workflow_user_share_connection), (library_dataset_dataset_association, library_dataset_dataset_info_association), (library_dataset_dataset_association, job_to_output_library_dataset), (library_dataset_dataset_association, job_external_output_metadata), (library_dataset_dataset_association, library_dataset_dataset_info_template_association), (library_dataset_dataset_association, library_dataset_dataset_association_permissions), (library_dataset_dataset_association, implicitly_converted_dataset_association), (request, sample), (request, request_event), (post_job_action, post_job_action_association), (workflow, workflow_step), (workflow, workflow_invocation), (galaxy_session, job), (galaxy_session, galaxy_user_openid), (galaxy_session, user_action), (galaxy_session, galaxy_session_to_history), (galaxy_session, event), (visualization, visualization_user_share_association), (visualization, visualization_tag_association), (visualization, visualization_rating_association), (visualization, visualization_revision), (visualization, visualization_annotation_association), (workflow_step, workflow_invocation_step), (workflow_step, workflow_step_connection), (workflow_step, workflow_step_annotation_association), (workflow_step, workflow_step_tag_association), (workflow_step, workflow_output), (workflow_step, post_job_action), (sample_state, sample_event), (job, post_job_action_association), (job, job_to_output_library_dataset), (job, job_to_output_dataset), (job, job_export_history_archive), (job, job_to_input_dataset), (job, job_import_history_archive), (job, job_parameter), (job, job_external_output_metadata), (job, workflow_invocation_step), (job, task), (form_values, external_service), (form_values, run), (form_values, galaxy_user), (form_values, request), (form_values, sample), (external_service, sample_dataset), (external_service, request_type_external_service_association), (page_revision, page)][]

2 2

Re: [galaxy-dev] Inquiring
by Nate Coraor 07 Nov '11

07 Nov '11

Hi Yan, I've again moved this back to the galaxy-dev list. Please reply to this list and not galaxy-user. Yan Luo wrote: > Dear Nate, > > Thanks for your quick response. I appreciate, it is very important for us to > use it. We have problem recently. > > (1) In fact, I can't find the file "paster.log". The problem is that > recently we expanded our gluster(Linux sever) and did the rebalance that has > some bugs. Some of files permission have been changed. So we can't use > User/register righr now, when we tried, we got the sever error, do you have > any idea how to fix it? (we can change the ownships/read and write > permission for some files manually, but we don't know which files and where > they are?) If you are executing run.sh without the --daemon flag, the output will go to whatever terminal window you started Galaxy in, not to a file. You'll want to make sure that all of the files under Galaxy's root directory are owned by a single user, which is the same user which starts and runs the Galaxy process. > (2) We want to reboot the Galaxy, should we first stop and start as > follows? > $ sh run.sh --stop > $ sh run.sh The full command to stop would be `sh run.sh --stop-daemon`, but that only applies if you originally started Galaxy with `run.sh --daemon`. > Is there any difference if I use "-daemon" flag? Last times, my colleague > started using "./run.sh". How to stop it? > If "$ sh run.sh --stop" doesn't work (I didn't try yet, ), how can I find > the process (Linux) that is running by Galaxy and kill it? The --daemon flag runs Galaxy in the background and redirects its output to paster.log. Without the --daemon flag, the process stays connected to the terminal in which you start it. Since Galaxy wasn't originally started with the --daemon flag, you'll need to find the process, either by locating the terminal window in which it was started (possibly by your colleague) and then hitting Ctrl-C, or by using the `ps` command (e.g. `ps auxwww | grep python`) and killing the Galaxy process. It usually looks something like: nate 18213 0.9 0.4 438844 146260 pts/7 Sl+ 10:47 2:30 python ./scripts/paster.py serve universe_wsgi.ini So you would do: % kill 18213 > (3) Could you please let me know if there is a default administrator > user/password for galaxy. I want to add an administrator user, how can I do > that? > According to the instruction, I will change a line of "universe_wsgi.ini" as > follows. How can I set/get my password? > admin_users = yan.luo(a)email.com There is no default. Any users set in admin_uesrs will be administrators. If the account specified in admin_users does not yet exist, you can simply create it. Once created, that account will be an administrator. > (4) I found our "universe_wsgi.ini" contains the following setting, should I > remove "#" before this line, stop and start galaxy? > #allow_user_creation = True No, True is the default setting so with it commented it is still set to True. > I plan to restart our galaxy, if it is possible, could you please let me > know your phone #? I can call you sometime today, or if you prefer, I can > give you my phone #. Unfortunately, we don't have the resources for phone support. We'll be happy to help via email as much as possible. --nate > > Looking forward to hearing from you. > > Best Wishes, > > Yan Luo, Ph.D. > NIH > > > On Thu, Feb 10, 2011 at 10:01 AM, Nate Coraor <nate(a)bx.psu.edu> wrote: > > > Hi Yan, > > > > I've moved this discussion to the galaxy-dev list since it pertains to a > > local installation of Galaxy. > > > > Responses to your questions follow, in-line. > > > > Yan Luo wrote: > > > Dear Sir, > > > > > > (1)We installed Galaxy, but recently the user can't registered and got > > the > > > following error, how can we fix it? > > > > > > Sever error > > > An error occurred. See the error logs for more information.(To turn debug > > on > > > to display ...). > > > > Since debug = False in universe_wsgi.ini, you should be able to find a > > more detailed error message in the log file. If starting Galaxy with: > > > > % sh run.sh --daemon > > > > The default log file is 'paster.log' in Galaxy's root directory. > > > > > (2) Could you please let me know if there is any command to stop galaxy? > > > > If starting with the --daemon flag (as above), you can use: > > > > % sh run.sh --stop-daemon > > > > If running in the foreground, you can use Ctrl-C to terminate the > > process. There is a recent bug whereby Ctrl-C is ineffective on some > > platforms under Python 2.6 - in this case you will have to kill/pkill > > the process manually. We are working on a fix for the latter. > > > > > (3) If I reset universe_wsgi.ini file and want to set an administrator > > > user(I can add a line in the above file), how can I get the password? > > Should > > > I stop galaxy(See question 2) first? then run "./setup.sh" and > > "./run.sh". > > > > setup.sh would have only been necessary prior to running Galaxy the > > first time, however, this step has recently been removed. If you are > > referencing documentation that still refers to setup.sh, please let us > > know so we can update it - I did notice this was still on the > > "Production Server" page, so I removed it from there. > > > > You no longer need to run setup.sh at all. > > > > > (4) If I run "setup.sh", will a new file "universe_wsgi.ini" be > > generated? > > > if I want to change this file,should I edit it before "run.sh" and after > > > "setup.sh". Is it right? > > > > setup.sh and its replacements in run.sh and the Galaxy application > > itself never overwrite files, they only create files from sample files > > if they do not exist. > > > > > (5) I read some of your docs, command "sh setup.sh"(sh run.sh) and > > > "./setup.sh"(./run.sh), which one is correct under Linux? > > > > Both syntaxes are effectively the same in most cases. > > > > --nate > > > > > > > > Looking forward to hearing from you. > > > > > > Best Wises, > > > > > > Yan Luo, Ph.D. > > > NIH > > > < > > http://int.ask.com/web?siteid=10000861&webqsrc=999&l=dis&q=By%20the%20way, > > > > > > > > _______________________________________________ > > > galaxy-user mailing list > > > galaxy-user(a)lists.bx.psu.edu > > > http://lists.bx.psu.edu/listinfo/galaxy-user > > > >

3 38

Error moving tool-data dir ... bowite indexes missing
by Bossers, Alex 26 Oct '11

26 Oct '11

On our up-to-date galaxy_central version I have successfully moved the database/files directory to mounted storage. After adapting the universe_wsgi.ini it works fine after galaxy restart. Now the tool-data dir becomes bigger and bigger and I wanted to move that to mounted storage as well. I moved the tool-data dir to the other location, made sure file permissions were ok and adapted the universe_wsgi.ini tool_data_path to match the new tool-data location. Galaxy restarts and the loc files are read (since blast recognises its databases). However, the bowtie indexes are not read since they do not show up in the NGS bowtie mapper for illumina..... what could be wrong? When I restore the universe_wsgi.ini it works all fine. I am puzzled. Alex

2 1

BWA wrapper error
by dhivya arasappan 26 Oct '11

26 Oct '11

Hi all, I'm seeing the following error when I try to use the galaxy BWA wrapper for SOLID data : An error occurred running this job:Could not determine BWA version The alignment failed. Error aligning sequence. /bin/sh: bwa: not found Any suggestions in figuring out what this error has to do with would be appreciated. thanks Dhivya

2 1

Re: [galaxy-dev] HOW TO RETRIEVE DATA FROM HISTORY??!!
by colin molter 20 Oct '11

20 Oct '11

> > > Is there a way to directly move/copy data from your galaxy history to a > given location in the filesystem of the same galaxy server? > Said differently, there is a nice way to import data from the server to > galaxy, is it possible to do the reverse? > > So far, I am obliged to download the file from galaxy to my client machine > and then back to the server!!!! with huge bam files of 3Gb it is not so > convenient!! > OK, I found a better way by (a) go to the admin panel, push the 'add a new dataset in the library' button and selecting the one needed from the current history (b) move the selected dataset from the library to a location mounted to galaxy. that is ok for me. However, if someone has a better solution, any advices are fine for me thnks colin

5 5

The need for wrappers
by Timothy Wu 13 Oct '11

13 Oct '11

Hi, Looking under the tools directory I realized that some tools comes with *_wrapper.py. I wonder under what circumstances is this needed? I'm trying to do a quickie for RepeatMasker. It looks to me like the tool does not allow one to specify the name of the output files. It also appears to me that tools should have an $output variable specifies within the <command> tag. So I was wondering if this is one case where a wrapper.py is needed. Thanks. Timothy

3 2

Re: [galaxy-dev] Some changes to tophat and cufflinks wrappers for indexing loc files
by Chorny, Ilya 12 Oct '11

12 Oct '11

I made some changes to the cufflinks and tophat wrapper to pull gtf files from an indexed loc file as opposed to from history. The diff's are attached. I made it an optional feature. Please let me know if this makes its way into galaxy-central. Best, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny(a)illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

2 4