March 2014 - galaxy-dev - lists.galaxyproject.org

Torque Exec Exception
by ngsflow＠hygenomics.com 08 Mar '14

08 Mar '14

hi all: galaxy.jobs.runners.cli_job.torque WARNING 2014-03-07 11:23:33,764 No valid qstat XML return from `qstat -x`, got the following: galaxy.jobs.runners ERROR 2014-03-07 11:23:33,765 Unhandled exception checking active jobs Traceback (most recent call last): File "/share/home/ldapuser/main/lib/galaxy/jobs/runners/__init__.py", line 339, in monitor self.check_watched_items() File "/share/home/ldapuser/main/lib/galaxy/jobs/runners/cli.py", line 150, in check_watched_items job_states = self.__get_job_states() File "/share/home/ldapuser/main/lib/galaxy/jobs/runners/cli.py", line 202, in __get_job_states job_states.update(job_interface.parse_status(cmd_out.stdout, job_ids)) TypeError: 'NoneType' object is not iterable #qmgr -c "p q batch" create queue batch set queue batch queue_type = Execution set queue batch enabled = True set queue batch started = True Finish the torque tasks ， throw exceptions in the log file。 ngsflow(a)hygenomics.com

2 1

tool_shed question on multiple tool versions
by ruiwang.sz 07 Mar '14

07 Mar '14

Hi Guys, Sorry for the duplicate posting...nobody saw my original one seems and I'm still wondering what the proper way is. I searched around the wiki but didn't find much. Originally, I installed macs dependency for galaxy myself. The version is 1.4.2. It worked well. Now, after galaxy upgrading removed it, and I installed it back from tool_shed, things broke. In the galaxy tool panel, it looks the same like before(as expected), but when I click on macs, it will show an option of MACS 'version 1.0.0' and 'version 1.0.1'. Be default it is 1.0.1, which is defined in migrated_tools_conf.xml: <section id="peak_calling" name="NGS: Peak Calling" version=""> <tool file=" toolshed.g2.bx.psu.edu/repos/devteam/macs/ae2ec275332a/macs/macs_wrapper.xml" guid="toolshed.g2.bx.psu.edu/repos/devteam/macs/peakcalling_macs/1.0.1"> <tool_shed>toolshed.g2.bx.psu.edu</tool_shed> <repository_name>macs</repository_name> <repository_owner>devteam</repository_owner> <installed_changeset_revision>ae2ec275332a</installed_changeset_revision> <id>toolshed.g2.bx.psu.edu/repos/devteam/macs/peakcalling_macs/1.0.1 </id> <version>1.0.1</version> </tool> </section> if we dive in this file toolshed.g2.bx.psu.edu/repos/devteam/macs/ae2ec275332a/macs/macs_wrapper.xml, we could see that in this dir /home/bioinfoadmin/app/shed_tools/ toolshed.g2.bx.psu.edu/repos/devteam/macs/ae2ec275332a/macs, there is a tool_dependency xml file that has following content: <?xml version="1.0"?> <tool_dependency> <package name="macs" version="1.3.7.1"> <repository changeset_revision="a7ea583a35d2" name="package_macs_1_3_7_1" owner="devteam" prior_installation_required="False" toolshed="http://toolshed.g2.bx.psu.edu" /> </package> <package name="R" version="2.15.0"> <repository changeset_revision="8ab0d08a3da1" name="package_r_2_15_0" owner="devteam" prior_installation_required="False" toolshed=" http://toolshed.g2.bx.psu.edu" /> </package> </tool_dependency> These versions are both older than what we installed before(1.4.2 and 3.0.2). could we manually modify that to use our own? wouldn't that be a bad practice? while in the same dir, macs_wrapper.xml has: <requirements> <requirement type="package" version="1.3.7.1">macs</requirement> <requirement type="package" version="2.15.0">R</requirement> </requirements> Seems that this is the tool_shed version of macs(which is identical to the one on main galaxy server, I think?). However this one does not work with our input data. Additionally, when I click to switch from 1.0.1 to 1.0.0 of macs, it immediately reports error. So, what if I want to keep using the one I installed myself? I know that tool_shed is a cleaner way to manage tools, but in our situation the upgrade and elimination actually broke the thing that worked before. I want to ask the proper way to handle this kind of situation. could I simply drop the entry from migrated_tools_conf.xml about macs, and restore the old tools_conf.xml entry? in that way we could continue using the 1.4.2 we have. However next time when we do the upgrade, things will break again I guess? I think that I shouldn't be the only person that has this concern? how to handle multiple version/dependency version for a tool? is there an option to not use the tool_shed but keep own version? Thanks, Rui

1 0

new data manager can't build bowtie or sam fasta, interface differences, how to fetch ncbi stuff
by Langhorst, Brad 07 Mar '14

07 Mar '14

I’m slowly wrapping my head around the new data manager stuff… but I don’t think I quite have it yet. I was able to download hg19 using the reference genome tool I see where it puts downloaded files now - took me a while to figure that out. When I run the sam fa build tool it fails like this: Traceback (most recent call last): File "/mnt/ngswork/galaxy/galaxy-toolshed-tools.0214/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/data_manager_sam_fa_index_builder/data_manager/data_manager_sam_fa_index_builder.py<http://bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e5…>", line 71, in <module> if __name__ == "__main__": main() File "/mnt/ngswork/galaxy/galaxy-toolshed-tools.0214/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/data_manager_sam_fa_index_builder/data_manager/data_manager_sam_fa_index_builder.py<http://bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e5…>", line 66, in main build_sam_index( data_manager_dict, options.fasta_filename, target_directory, options.fasta_dbkey, data_table_name=options.data_table_name or DEFAULT_DATA_TABLE_NAME ) File "/mnt/ngswork/galaxy/galaxy-toolshed-tools.0214/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/data_manager_sam_fa_index_builder/data_manager/data_manager_sam_fa_index_builder.py<http://bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e5…>", line 25, in build_sam_index proc = subprocess.Popen( args=args, shell=False, cwd=target_directory, stderr=tmp_stderr.fileno() ) File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory What am I missing here? I can’t tell what file it’s referring to. The bowtie2 index builder also fails. ———— The old interface that allowed download, then building of chosen indices was more convenient. It’s not fun to download, wait, then click each builder to schedule a build. Is a replacement for the more unified interface planned? Am I supposed to just build a workflow that builds all the stuff i want? ———— I can’t tell what I’m supposed to feed as “dbkey” to the ncbi downloader… It’s also not easy for me to find the list of dbkeys available at ucsc. (eschColi_K12 does not seem to work - don’t know why, but i guess it doesn’t know to try to microbes) it would be very nice to allow users to select which genome they want by choosing from a list. I’m headed back for manual loc file territory for now - but I’d love some enlightenment about how this is supposed to work and or information about how others are using it. Brad -- Brad Langhorst, Ph.D. Applications and Product Development Scientist

2 1

Galaxy database size and location
by Ravi Alla 07 Mar '14

07 Mar '14

Hi fellow galaxy devs, I am trying to understand how to implement the galaxy database and get an idea of how big it could get. Currently we are running galaxy on a webserver, and want to have the postgresql db on locally mounted partition and not on an NFS partition. This limits us to around 100GB of storage for the db. We will create data libraries for users to load their data without copying to galaxy, so input files won't be duplicated. Is there anything we can do about the output files? Do these files need to end up in the database or can we put them on the NFS partition somewhere with the db holding information about their location? I noticed that on a routine small analysis I could easily have 20GB or more of output files and history and all this is in the database. If output files and history files are written to the database, are they cleaned up daily to avoid storage issues? Please advise. Thanks Ravi Alla

2 3

Galaxy reports
by Philippe Moncuquet 07 Mar '14

07 Mar '14

Hi, We have recently updated our instance, I have noticed a bug display in the report page. In the section 'Jobs per Month', the month column does not contain any month name but number instead. Has anybody experienced the same thing ? Regards, Philip

3 4

Unable to build pbs_python via scramble
by Pete Schmitt 07 Mar '14

07 Mar '14

2 11

Re: [galaxy-dev] No such file or directory: '/nextgen3/galaxy/galaxy-dist/database/job_working_directory/000/###/galaxy_624.o'
by Nate Coraor 07 Mar '14

07 Mar '14

Just a warning, -noac has a pretty severe impact on performance in my (and others' on this list) experience. You might also want to try messing with the 'lookupcache' mount option. --nate On Thu, Mar 6, 2014 at 2:20 PM, Pete Schmitt <Peter.R.Schmitt(a)dartmouth.edu>wrote: > Hello Nate, > > I had that parameter set to 1, but I up'd it to 5. I also added -noac to > the nfs mounts for /nextgen3 > > That appears to have fixed it. > > Thank you!!! > > > On 3/6/14, 1:57 PM, Nate Coraor wrote: > > Hi Pete, > > I'd suggest setting retry_job_output_collection > 0 in > universe_wsgi.ini. This is usually a symptom of attribute caching on > network filesystems. > > --nate > > > On Wed, Mar 5, 2014 at 8:06 PM, Pete Schmitt < > Peter.R.Schmitt(a)dartmouth.edu> wrote: > >> >> >> In trying something simple, using galaxy I downloaded data from USCS >> main. The data gets downloaded but the job errors out. I verified that >> the job actually ran, and completed successfully according to the scheduler >> but I get errors like this: >> >> galaxy.jobs.runners.drmaa DEBUG 2014-03-05 18:17:35,941 (624/ >> 46.dirigo.mdibl.org) state change: job finished normally >> galaxy.jobs.runners ERROR 2014-03-05 18:17:36,060 (624/ >> 46.dirigo.mdibl.org) Job output not returned from cluster: [Errno 2] No >> such file or directory: >> '/nextgen3/galaxy/galaxy-dist/database/job_working_directory/000/624/galaxy_624.o' >> >> There are no directories being created below the 000 directory. I >> verified that the directory tree is owned by galaxy and that the galaxy >> user can run jobs from the command line as a normal user. >> >> I set the parameter "cleanup_job = never". It was set to "always" which >> is probably why the files were never there. Now the files are there, >> including the galaxy_###.o file but galaxy still errors like above. >> >> I had set the parameter "cluster_files_directory = database/pbs", but >> that doesn't seem to work any longer. The .o and .e files used to end up >> there. >> >> Here is an example: >> >> (galaxyvenv)[galaxy@dirigo 630]$ ll >> total 16 >> -rw------- 1 galaxy galaxy 0 Mar 5 19:29 galaxy_630.e >> -rw-rw-r-- 1 galaxy galaxy 2 Mar 5 19:29 galaxy_630.ec >> -rw------- 1 galaxy galaxy 940 Mar 5 19:29 galaxy_630.o >> -rwxr-xr-x 1 galaxy galaxy 2429 Mar 5 19:29 galaxy_630.sh >> -rw-rw-r-- 1 galaxy galaxy 138 Mar 5 19:29 galaxy.json >> -rw-rw-r-- 1 galaxy galaxy 2139 Mar 5 19:29 >> metadata_in_HistoryDatasetAssociation_1182_o830e3 >> -rw-rw-r-- 1 galaxy galaxy 20 Mar 5 19:29 >> metadata_kwds_HistoryDatasetAssociation_1182_hOhPp7 >> -rw-rw-r-- 1 galaxy galaxy 55 Mar 5 19:29 >> metadata_out_HistoryDatasetAssociation_1182_Ynb70M >> -rw-rw-r-- 1 galaxy galaxy 2 Mar 5 19:29 >> metadata_override_HistoryDatasetAssociation_1182_HsMljG >> -rw-rw-r-- 1 galaxy galaxy 44 Mar 5 19:29 >> metadata_results_HistoryDatasetAssociation_1182_LxdsAZ >> (galaxyvenv)[galaxy@dirigo 630]$ pwd >> /nextgen3/galaxy/galaxy-dist/database/job_working_directory/000/630 >> >> Here is the error from this: >> >> galaxy.jobs.runners.drmaa DEBUG 2014-03-05 19:31:37,731 (630/ >> 51.dirigo.mdibl.org) state change: job is running >> galaxy.jobs.runners.drmaa DEBUG 2014-03-05 19:31:49,119 (630/ >> 51.dirigo.mdibl.org) state change: job finished normally >> galaxy.jobs.runners ERROR 2014-03-05 19:31:50,225 (630/ >> 51.dirigo.mdibl.org) Job output not returned from cluster: [Errno 2] No >> such file or directory: >> '/nextgen3/galaxy/galaxy-dist/database/job_working_directory/000/630/galaxy_630.o' >> galaxy.jobs DEBUG 2014-03-05 19:31:50,252 finish(): Moved >> /nextgen3/galaxy/galaxy-dist/database/job_working_directory/000/630/galaxy_dataset_856.dat >> to /nextgen3/galaxy/galaxy-dist/database/files/000/dataset_856.dat >> galaxy.jobs DEBUG 2014-03-05 19:31:50,351 job 630 ended >> >> On the galaxy page in the history you get in pink: >> 1 UCSC Main on Human: knownGene (chr22:1-51304566) >> error >> An error occurred with this dataset: >> Job output not returned from cluster >> >> But the dataset is there. >> >> On 3/5/14, 3:33 PM, Nate Coraor wrote: >> >> The old-style url syntax is supposed to continue to work, if you have any >> details on what's not working I can look in to it. That said, job_conf.xml >> is the way forward, and a job_conf.xml for the drmaa runner would be a >> pretty trivial change from the one you have for the pbs runner, e.g.: >> >> <?xml version="1.0"?> >> <job_conf> >> <plugins> >> <plugin id="pbs" type="runner" >> load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"/> >> </plugins> >> <handlers> >> <handler id="dirigo"/> >> </handlers> >> <destinations default="pbs_default"> >> <destination id="pbs_default" runner="pbs"/> >> <param id="nativeSpecification">-l walltime= >> 72:00:00,nodes=1:ppn=4</param> >> </destination> >> </destinations> >> </job_conf> >> >> Just make sure you set $DRMAA_LIBRARY_PATH in your environment to the >> correct libdrmaa.so. >> >> --nate >> >> >> On Wed, Mar 5, 2014 at 3:27 PM, Pete Schmitt < >> Peter.R.Schmitt(a)dartmouth.edu> wrote: >> >>> Hello Nate, >>> >>> I have that version installed and was using it in the older versions of >>> galaxy for a few years. Once I loaded this new version, it no longer >>> worked with the old >>> definitions in the universe file using: default_cluster_job_runner = >>> drmaa:/// >>> >>> Do I need a job_conf.xml that uses the drmaa runner? >>> >>> >>> >>> On 3/5/14, 3:21 PM, Nate Coraor wrote: >>> >>> Hi Pete, >>> >>> The latest error is pretty strange and not one I've encountered >>> before. It suggests that scramble is not loading setuptools in place of >>> distutils and thus does not have access to the setuptools extensions >>> (notably, egg-related functionality). Something abnormal still seems to be >>> going on with your python environment. >>> >>> You can use drmaa if you like (this is known to work well). You will >>> want to use the libdrmaa for Torque that's maintained by the Poznan >>> Supercomputing and Networking Center, rather than the libdrmaa that can be >>> built directly with the Torque source. PSNC libdrmaa for Torque/PBS can be >>> found here: http://apps.man.poznan.pl/trac/pbs-drmaa >>> >>> --nate >>> >>> >>> On Wed, Mar 5, 2014 at 3:10 PM, Pete Schmitt < >>> Peter.R.Schmitt(a)dartmouth.edu> wrote: >>> >>>> Is there any other alternatives to pbs_python for interfacing to a >>>> torque scheduler. This method appears to be a dead end. >>>> >>>> >>>> >>>> On 3/4/14, 9:51 AM, Nate Coraor wrote: >>>> >>>> Pete, >>>> >>>> Is it possible that `python` as the Galaxy user is calling a python >>>> other than /opt/python/2.7.6/bin/python (e.g. the system version without >>>> the -dev/-devel package installed)? The safest bet for ensuring you're >>>> using the right python and that it's not going to have conflicting modules, >>>> I'd suggest using a Python virtualenv. This is easy to set up, just make >>>> sure you run it with the correct python executable, for example: >>>> >>>> % wget >>>> https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.4.tar.… >>>> % tar zxf virtualenv-1.11.4.tar.gz >>>> % /opt/python/2.7.6/bin/python ./virtualenv-1.11.4/virtualenv.py >>>> galaxyvenv >>>> % . ./galaxyvenv/bin/activate >>>> % cd galaxy-dist >>>> % python ./scripts/fetch_eggs.py >>>> % LIBTORQUE_DIR=/opt/torque/active/lib python scripts/scramble.py -e >>>> pbs_python >>>> >>>> --nate >>>> >>>> >>>> On Mon, Mar 3, 2014 at 5:19 PM, Pete Schmitt < >>>> Peter.R.Schmitt(a)dartmouth.edu> wrote: >>>> >>>>> I uninstalled pbs_python 4.4.0 and reinstalled 4.3.5 as root (not >>>>> using scramble) >>>>> >>>>> When I try this method as the galaxy user: >>>>> LIBTORQUE_DIR=/opt/torque/active/lib python scripts/scramble.py -e >>>>> pbs_python >>>>> >>>>> I get the following output: >>>>> >>>>> src/pbs_wrap.c:2813: warning: function declaration isn't a prototype >>>>> gcc -pthread -shared build/temp.linux-x86_64-2.7/src/pbs_wrap.o >>>>> -L/opt/torque/4.2.7/lib -L. -ltorque -lpython2.7 -o >>>>> build/lib.linux-x86_64-2.7/_pbs.so -L/opt/torque/4.2.7/lib -ltorque >>>>> -Wl,-rpath -Wl,/opt/torque/4.2.7/lib >>>>> /usr/bin/ld: cannot find -lpython2.7 >>>>> >>>>> >>>>> LD_LIBRARY_PATH="/opt/python/2.7.6/lib:/opt/torque/active/lib:/usr/local/lib" >>>>> >>>>> I'm not sure why it can't find the libpython2.7.so file. When I >>>>> built it as root there is a -L/opt/python/2.7.6/lib in that gcc line. >>>>> >>>>> >>>>> >>>>> >>>>> On 3/3/14, 4:13 PM, Nate Coraor wrote: >>>>> >>>>> Hi Pete, >>>>> >>>>> Your subject says you are unable to build pbs_python using scramble. >>>>> Could you provide details on what's not working there? >>>>> >>>>> Galaxy is not going to work with a different version of pbs_python >>>>> unless a bit of hacking is done to make it attempt to do so. We test >>>>> Galaxy with specific versions of its dependencies, which is why we >>>>> control the versions of those dependencies and provide the scramble >>>>> script to (hopefully) make it painless to build them yourself, should >>>>> it be necessary to do so, as is always the case with pbs_python. >>>>> >>>>> --nate >>>>> >>>>> On Mon, Mar 3, 2014 at 3:57 PM, Pete Schmitt<Peter.R.Schmitt(a)dartmouth.edu> <Peter.R.Schmitt(a)dartmouth.edu> wrote: >>>>> >>>>> Following the directions from here:https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#PBS >>>>> >>>>> I'm trying to get pbs_python to work as I'm using torque for scheduling >>>>> galaxy jobs. >>>>> >>>>> Note: This is a fresh install of galaxy from galaxy-dist on CentOS 5.10 >>>>> >>>>> I have pbs_python 4.4.0 module installed into a source-built version of >>>>> python/2.7.6 >>>>> >>>>> I get the following error in the output of run.sh: >>>>> galaxy.jobs INFO 2014-03-03 15:46:45,485 Handler 'main' will load all >>>>> configured runner plugins >>>>> Traceback (most recent call last): >>>>> File >>>>> "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/webapps/galaxy/buildapp.py", >>>>> line 39, in app_factory >>>>> app = UniverseApplication( global_conf = global_conf, **kwargs ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/app.py", line 130, in >>>>> __init__ >>>>> self.job_manager = manager.JobManager( self ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/jobs/manager.py", line >>>>> 31, in __init__ >>>>> self.job_handler = handler.JobHandler( app ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/jobs/handler.py", line >>>>> 30, in __init__ >>>>> self.dispatcher = DefaultJobDispatcher( app ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/jobs/handler.py", line >>>>> 568, in __init__ >>>>> self.job_runners = self.app.job_config.get_job_runner_plugins( >>>>> self.app.config.server_name ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/jobs/__init__.py", line >>>>> 449, in get_job_runner_plugins >>>>> module = __import__( module_name ) >>>>> File "/nextgen3/galaxy/galaxy-dist-py27/lib/galaxy/jobs/runners/pbs.py", >>>>> line 31, in <module> >>>>> raise Exception( egg_message % str( e ) ) >>>>> Exception: >>>>> >>>>> The 'pbs' runner depends on 'pbs_python' which is not installed or not >>>>> configured properly. Galaxy's "scramble" system should make this >>>>> installation >>>>> simple, please follow the instructions found at: >>>>> >>>>> http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster >>>>> >>>>> Additional errors may follow: >>>>> pbs-python==4.3.5 >>>>> >>>>> This is the job_conf.xml file: >>>>> >>>>> <?xml version="1.0"?> >>>>> <job_conf> >>>>> <plugins> >>>>> <plugin id="pbs" type="runner" >>>>> load="galaxy.jobs.runners.pbs:PBSJobRunner"/> >>>>> </plugins> >>>>> <handlers> >>>>> <handler id="dirigo"/> >>>>> </handlers> >>>>> <destinations default="pbs_default"> >>>>> <destination id="pbs_default" runner="pbs"/> >>>>> <param >>>>> id="Resource_List">walltime=72:00:00,nodes=1:ppn=4</param> >>>>> </destinations> >>>>> </job_conf> >>>>> >>>>> I did not use the scramble system to install the pbs_python module. I >>>>> downloaded the latest version >>>>> available and installed it from the root account. >>>>> >>>>> >>>>> -- >>>>> >>>>> Pete Schmitt >>>>> >>>>> >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid >>>>> Computational Genetics Lab Institute for Quantitative Biomedical Sciences >>>>> Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: >>>>> 603-646-8109 <603-646-8109> http://discovery.dartmouth.edu >>>>> <http://discovery.dartmouth.edu> http://columbia.dartmouth.edu/grid >>>>> <http://columbia.dartmouth.edu/grid> http://www.epistasis.org >>>>> <http://www.epistasis.org> http://iQBS.org <http://iQBS.org> * >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid >>>> Computational Genetics Lab Institute for Quantitative Biomedical Sciences >>>> Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: >>>> 603-646-8109 <603-646-8109> http://discovery.dartmouth.edu >>>> <http://discovery.dartmouth.edu> http://columbia.dartmouth.edu/grid >>>> <http://columbia.dartmouth.edu/grid> http://www.epistasis.org >>>> <http://www.epistasis.org> http://iQBS.org <http://iQBS.org> * >>>> >>> >>> >>> -- >>> >>> >>> * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid >>> Computational Genetics Lab Institute for Quantitative Biomedical Sciences >>> Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: >>> 603-646-8109 <603-646-8109> http://discovery.dartmouth.edu >>> <http://discovery.dartmouth.edu> http://columbia.dartmouth.edu/grid >>> <http://columbia.dartmouth.edu/grid> http://www.epistasis.org >>> <http://www.epistasis.org> http://iQBS.org <http://iQBS.org> * >>> >> >> >> -- >> >> >> * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid >> Computational Genetics Lab Institute for Quantitative Biomedical Sciences >> Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: >> 603-646-8109 <603-646-8109> http://discovery.dartmouth.edu >> <http://discovery.dartmouth.edu> http://columbia.dartmouth.edu/grid >> <http://columbia.dartmouth.edu/grid> http://www.epistasis.org >> <http://www.epistasis.org> http://iQBS.org <http://iQBS.org> * >> > > > -- > > > * Pete Schmitt Technical Director: Discovery Cluster NH INBRE Grid > Computational Genetics Lab Institute for Quantitative Biomedical Sciences > Dartmouth College, HB 6203 L12 Berry/Baker Library Hanover, NH 03755 Phone: > 603-646-8109 <603-646-8109> http://discovery.dartmouth.edu > <http://discovery.dartmouth.edu> http://columbia.dartmouth.edu/grid > <http://columbia.dartmouth.edu/grid> http://www.epistasis.org > <http://www.epistasis.org> http://iQBS.org <http://iQBS.org> * >

1 0

Nightly testing status on the (Test) Tool Shed
by Peter Cock 07 Mar '14

07 Mar '14

Hi all, Are both the main and test Tool Sheds currently meant to be running nightly tests again? I've not really got back into Galaxy tool testing since Christmas. Most of my tools' test results look sensible, but there seem to be some issues still, e.g. http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler Not tested since 2014-01-04 http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr 2014-01-29 missing blast_datatypes dependency (why?) 2014-01-28 showing failed tests for sambamba_filter (wrong tool) Regards, Peter

4 32

Workflows with Conditionals and Loops
by Michael E. Cotterell 07 Mar '14

07 Mar '14

Is there a way to have conditionals and loops for the various steps in a Galaxy workflow? For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output. Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results. Any ideas? Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Instructor of Record, Graduate RA & TA, University of Georgia Department Liaison, CS Graduate Student Association, University of Georgia mepcotterell(a)gmail.com (mailto:mepcotterell@gmail.com) mepcott(a)uga.edu (mailto:mepcott@uga.edu) mec(a)cs.uga.edu (mailto:mec@cs.uga.edu) http://michaelcotterell.com/

5 8

Thread nucleotides onto a protein alignment (back-translation)
by Peter Cock 07 Mar '14

07 Mar '14

Hi all, If anyone is interested, I've just wrapped a little Python script I'd prepared early for use in Galaxy for a fairly common need - "back translating" a protein alignment into a nucleotide alignment by threading the unaligned sequence. We're testing this locally, and barring any major issues I would expect to release this to the main Tool Shed in a week or so - but if any of you want to have a play with it now, and pass on feedback, please do so: Development repository: https://github.com/peterjc/pico_galaxy/tree/master/tools/align_back_trans Test Tool Shed release: http://testtoolshed.g2.bx.psu.edu/view/peterjc/align_back_trans Planned Tool Shed release: http://toolshed.g2.bx.psu.edu/view/peterjc/align_back_trans (pending) This uses Biopython's AlignIO [*] module and so can handle a range of alignment file formats - for now I have restricted this to "fasta", "clustal" and "phylip" which I believe are all already in use on the tool shed - there is probably scope here for a more coordinated effort to define Galaxy datatypes in this area, including things like the PFAM/Stockholm format and strict/relaxed variants of PHYLIP. Regards, Peter [*] http://biopython.org/wiki/AlignIO

1 1