March 2013 - galaxy-dev - lists.galaxyproject.org

Broken pipe in server
by Thomas van Gurp 05 Mar '13

05 Mar '13

Dear all, Our galaxy suffers from frequent hickups, which apparently are due to broken pipes. It will give these errors in paster.log ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 53234) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/httpse rver.py", line 1068, in process_request_in_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.7/SocketServer.py", line 641, in __init__ self.finish() File "/usr/lib/python2.7/SocketServer.py", line 694, in finish self.wfile.flush() File "/usr/lib/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) error: [Errno 32] Broken pipe ---------------------------------------- Any idea why this might be an what I would need to check? Cheers, THomas

2 1

errors with run_functional_tests.sh
by Florence Maurier 05 Mar '13

05 Mar '13

Hi, We are currently installing galaxy on a server. By reading the galaxy wiki, we found the script run_functional_tests.sh (the principle is really handy!). Unfortunately, we encounter errors when we use it. I hope someone can help us. In reality, we encounter several types of errors ... The first is present in each run, for example for the execution follows: ./run_functional_tests.sh -id fastqc ... FastQC:Read QC ( fastqc ) > Test-1 ... galaxy.web.framework DEBUG 2013-03-01 09:42:10,466 ... galaxy.tools.actions.upload_common INFO 2013-03-01 09:42:13,357 tool upload1 created job id 1 ... galaxy.jobs DEBUG 2013-03-01 09:42:27,436 Tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2013-03-01 09:42:27,757 job 1 ended galaxy.tools.actions.upload_common INFO 2013-03-01 09:42:40,883 tool upload1 created job id 2 ... galaxy.jobs DEBUG 2013-03-01 09:42:52,000 Tool did not define exit code or stdio handling; checking stderr for success galaxy.jobs DEBUG 2013-03-01 09:42:52,311 job 2 ended galaxy.jobs.manager DEBUG 2013-03-01 09:42:55,285 (3) Job assigned to handler 'main' galaxy.jobs DEBUG 2013-03-01 09:43:00,477 (3) Working directory for job is: /var/local/galaxy/galaxy-dist/database/job_working_directory/000/3 galaxy.jobs.handler DEBUG 2013-03-01 09:43:00,478 dispatching job 3 to local runner galaxy.jobs.handler INFO 2013-03-01 09:43:00,622 (3) Job dispatched galaxy.jobs.runners.local DEBUG 2013-03-01 09:43:00,727 Local runner: starting job 3 galaxy.jobs.runners.local DEBUG 2013-03-01 09:43:01,193 executing: python /var/local/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py -i "/tmp/tmp_YcL4J/database/files/000/dataset_1.dat" -d "/var/local/galaxy/galaxy-dist/database/job_working_directory/000/3/dataset_3_files" -o "/tmp/tmp_YcL4J/database/files/000/dataset_3.dat" -n "fastqcout" -f "fastq" -j "1000gsample.fastq" -e "/var/local/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc" -c "/tmp/tmp_YcL4J/database/files/000/dataset_2.dat" galaxy.jobs.runners.local DEBUG 2013-03-01 09:43:04,515 execution finished: python /var/local/galaxy/galaxy-dist/tools/rgenetics/rgFastQC.py -i "/tmp/tmp_YcL4J/database/files/000/dataset_1.dat" -d "/var/local/galaxy/galaxy-dist/database/job_working_directory/000/3/dataset_3_files" -o "/tmp/tmp_YcL4J/database/files/000/dataset_3.dat" -n "fastqcout" -f "fastq" -j "1000gsample.fastq" -e "/var/local/galaxy/galaxy-dist/tool-data/shared/jars/FastQC/fastqc" -c "/tmp/tmp_YcL4J/database/files/000/dataset_2.dat" galaxy.jobs DEBUG 2013-03-01 09:43:04,578 Tool did not define exit code or stdio handling; checking stderr for success _galaxy.tools DEBUG 2013-03-01 09:43:04,814 Error opening galaxy.json file: [Errno 2] No such file or directory: '/var/local/galaxy/galaxy-dist/database/job_working_directory/000/3/galaxy.json'_ galaxy.jobs DEBUG 2013-03-01 09:43:04,899 job 3 ended ok ---------------------------------------------------------------------- Ran 1 test in 60.889s OK ... Exception OSError: (2, 'No such file or directory', '/tmp/tmp_YcL4J/database/tmp/tmpn5w0Ln') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w' at 0x18ffd5d0>> ignored According to other messages from the mailing list, my understanding is that this message should be ignored and we only have to consider the "OK" that says the test went well. Is that correct ? The other type of error we find is the following: ./run_functional_tests.sh -id cshl_fastx_clipper ... ERROR: Failure: ValueError (No such test TestForTool_cshl_fastx_clipper) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/local/galaxy/galaxy-dist/eggs/nose-0.11.1-py2.6.egg/nose/failure.py", line 39, in runTest raise self.exc_class(self.exc_val) ValueError: No such test TestForTool_cshl_fastx_clipper ---------------------------------------------------------------------- ... For some of these tools, the error seems logical given that the xml configuration tool does not contain test. However, for others like fastx_clipper, the test is described... So, why is there a problem with this kind of tools ? Finally, we have one last type of error, like this: ./run_functional_tests.sh -id sam_to_bam ... SAM-to-BAM ( sam_to_bam ) > Test-2 ... galaxy.web.framework DEBUG 2013-03-05 18:02:15,115 Error: this request returned None from get_history(): http://localhost:9051/user/logout galaxy.web.framework DEBUG 2013-03-05 18:02:15,159 Error: this request returned None from get_history(): http://localhost:9051/ ERROR ====================================================================== ERROR: SAM-to-BAM ( sam_to_bam ) > Test-2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/local/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 171, in test_tool self.do_it( td, shed_tool_id=shed_tool_id ) File "/var/local/galaxy/galaxy-dist/test/functional/test_toolbox.py", line 43, in do_it shed_tool_id=shed_tool_id ) File "/var/local/galaxy/galaxy-dist/test/base/twilltestcase.py", line 193, in upload_file tc.fv("1","dbkey", dbkey) File "/var/local/galaxy/galaxy-dist/eggs/twill-0.9-py2.6.egg/twill/commands.py", line 450, in formvalue set_form_control_value(control, value) File "/var/local/galaxy/galaxy-dist/eggs/twill-0.9-py2.6.egg/twill/utils.py", line 190, in set_form_control_value raise ClientForm.ItemNotFoundError('cannot find value/label "%s" in list control' % (val,)) ItemNotFoundError: cannot find value/label "chrM" in list control -------------------- >> begin captured stdout << --------------------- --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- galaxy.web.framework: DEBUG: Error: this request returned None from get_history(): http://localhost:9051/user/logout galaxy.web.framework: DEBUG: Error: this request returned None from get_history(): http://localhost:9051/ --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 2 tests in 56.139s FAILED (errors=1) The first test is ok, but the second is failed. So, the dependence seems to be correctly installed, but the test doesn't pass, and I don't understand why... Thank you in advance for your help ! Florence

2 1

Authentication: email addresses are not validated prior to account creation
by Vipin TS 05 Mar '13

05 Mar '13

Hello members, I believe currently there is no process to validate email address provided during user account creation. We are experiencing a huge fake account creation attack on our public facing galaxy instance<http://galaxy.raetschlab.org> . Does anybody who has been managing a public instance, implemented an on-demand account creation activation by sending an email containing a link, which when clicked, validate the account creation request. Or any plans from dev-team to add this in future release? thanks in advance, --/Vipin

4 4

NCBI rpsblast and rpstblastn for CDD searches etc
by Peter Cock 05 Mar '13

05 Mar '13

On Mon, Feb 4, 2013 at 6:14 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > On Mon, Feb 4, 2013 at 5:43 PM, Luobin Yang <yangluob(a)isu.edu> wrote: >> Hi, Peter, >> >> What difficulties did you see to include psi-blast and phi-blast when you >> were working on the other programs in the NCBI+ suite? I would be >> interested in creating a tool configuration file for each of these two. >> >> Thanks, >> Luobin > > Hi Luobin, > > At first glance, the binaries for psiblast, rpsblast, and rpstblastn don't > look too different for wrapping. We would need to define a third loc > file for the RPS BLAST domain databases (like the existing loc files > for nucleotide and protein databases). I've got wrappers working for rpsblast and rpstblastn with a new blastdb_d.loc file for the associated domain databases, currently in my bitbucket repository and live on our internal Galaxy server. Barring any unexpected problems, I'll include this in an update to the Tool Shed in a week or so - so watch this space: http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/ Meanwhile work on wrapping psiblast is ongoing, with several contributors. We'll need to have ASN.1 base formats defined in Galaxy first (discussed on another thread). Regards, Peter

1 0

GALAXY LOCAL INSTALLATION: Tools and Data
by Rafael Hernández 05 Mar '13

05 Mar '13

Hi, I've installed Galaxy in local and following the Tool dependencies<http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies> manual I installed all necessary tools. However for some of then such as BWA, Bowtie,... it's usually needed to create indexes. Where should I store those indexes? In the galaxy directory or in each tool directory? How can I tell galaxy where the indexes are? I find a manual's page talking about it / http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup) suggesting to organize the various index files for each build, but I don't know if these folders should be stored into Galaxy directory (eg. in tool-data directory) or what. Could you please explain this a little? Thank you and regards, Rafa Hernández de Diego Genomics of Gene Expression Lab. Bioinformatics and Genomics Department Centro de Investigación Príncipe Felipe (CIPF) C/ Eduardo Primo Yúfera 3 46012 Valencia, Spain

3 4

Re: [galaxy-dev] [galaxy-user] Setting up galaxy local server
by Nate Coraor 05 Mar '13

05 Mar '13

On Mar 3, 2013, at 5:21 PM, Anusha Nagari wrote: > Hi, > > I am a new user of galaxy and am trying to set up local version of galaxy on my mac. I followed the instructions in this link and successfully installed the local version using the following steps: > > mkdir galaxy-python/ > mkdir galaxy-dist/ > ln -s /usr/local/bin/ galaxy-python/python > export PATH=~/galaxy-python:$PATH > hg clone https://bitbucket.org/galaxy/galaxy-dist/ > cd galaxy-dist > sh run.sh > > I could load galaxy fine for the first time. However after I closed it and tried to run it by sh run.sh, it is giving the following error and am unable to startup galaxy: > __import__( "pkg_resources" ).declare_namespace( __name__ ) > Traceback (most recent call last): > File "./scripts/manage_tools.py", line 12, in <module> > from migrate.versioning.shell import main > ImportError: No module named migrate.versioning.shell Hi Anusha, I've moved this thread over to the galaxy-dev list since it concerns a local installation. Could you check to see if you've set the PYTHONPATH environment variable? It seems that something odd is going on with your environment. Also, if you opened a new shell for the second time you executed run.sh, you'll need to set the same PATH variable as you did the first time. That export does not persist unless you place it in one of your shell's startup files. The symlink method is quick and easy, but if you are going to have a persistent Galaxy server that you'll be installing python modules in to, I would suggest using a virtualenv as described here: http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Use… --nate > > However, I am not able to upload any datasets into galaxy either from UCSC genome browser or from my computer with the following error: > An error occurred with this dataset: /Users/anushanagari/Desktop/Softwares/galaxy-dist/lib/galaxy/__init__.py:5: UserWarning: Module site was already imported from /Library/Frameworks/Python.framework/Versions/4.2.30201/lib/python2.5/site.pyc, but /Library/Frameworks/Python.framework/Version > > Tool execution generated the following error message: > /Users/anushanagari/Desktop/Softwares/galaxy-dist/lib/galaxy/__init__.py:5: UserWarning: Module site was already imported from /Library/Frameworks/Python.framework/Versions/4.2.30201/lib/python2.5/site.pyc, but /Library/Frameworks/Python.framework/Versions/4.2.30201/lib/python2.5/site-packages/Enstaller-3.0.9-py2.5.egg is being added to sys.path > __import__( "pkg_resources" ).declare_namespace( __name__ ) > /Users/anushanagari/Desktop/Softwares/galaxy-dist/lib/galaxy/__init__.py:5: UserWarning: Module pkg_resources was already imported from /Users/anushanagari/Desktop/Softwares/galaxy-dist/lib/pkg_resources.pyc, but /Library/Frameworks/Python.framework/Versions/4.2.30201/lib/python2.5/site-packages/Enstaller-3.0.9-py2.5.egg is being added to sys.path > __import__( "pkg_resources" ).declare_namespace( __name__ ) > > > Could you please help me with this and also provide any additional documentation that could help me in getting to know galaxy better? Thanks in advance! Hoping to hear from you soon! > > Best, > Anusha > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/

1 0

Manage API keys in admin panel
by Rémy Dernat 05 Mar '13

05 Mar '13

Hi, I would like to create a tool or a template that could list and (re)generate key for the user you want if you are login as an admin. I found the api_keys function in the user controller but I don't know how to use it. It seems that the decorators before the function checked if you are login as the good user... I also found the mako template that used this api_keys function but it is still unclear for me. Here is my sql function (FULL OUTER JOIN compatibility for MySQL) that list every keys even if they are empty : SELECT `galaxy_user`.`id`, `api_keys`.`user_id`, `api_keys`.`key`, `galaxy_user`.`email`, `galaxy_user`.`username` FROM `galaxy_user` LEFT JOIN `api_keys` on `api_keys`.`user_id` = `galaxy_user`.`id` UNION SELECT `galaxy_user`.`id`, `api_keys`.`user_id`, `api_keys`.`key`, `galaxy_user`.`email`, `galaxy_user`.`username` FROM `galaxy_user` RIGHT JOIN `api_keys` on `api_keys`.`user_id` = `galaxy_user`.`id` I know how to do that in PHP but it won't use python framework sqlalchemy (= only for mysql or pgsql...) ! Morevoer I don't know how do you generate your api_keys. Is it a md5 sum based on username / password ? Or is it just a random key with fix length ? Any help would be gracefull, Regards, Remy

1 2

Support Question
by Jennifer Jackson 04 Mar '13

04 Mar '13

Hi Tomas, I am going to move your question over to the galaxy-dev(a)bx.psu.edu mailing list, which give it better exposure to the development community. This is where you will also want to post questions about local installs going forward. Someone here may have advice about this tool - but really your best bet is with the tool/wrapper author. I checked the Galaxy Main Tool Shed, and the repository there for this tool shows it is inactive/never downloaded in the current tool shed version. So, this was not your source or you obtained it some time ago (~ 2 yrs). Still, if this was your source/version, log into the Tool Shed, click on "Repository Actions", and contact the repo owner: http://toolshed.g2.bx.psu.edu/view/petr-novak/repeatrxplorer http://wiki.galaxyproject.org/ToolShedRepositoryFeatures#Contacting_the_own… This is another link to the tool author for the Main Tool Shed wrapper and the primary tool itself, and perhaps for the wrapper version you are working with as well (from some other source, but you will have to check): http://repeatexplorer.umbr.cas.cz/static/html/help/manual.html Good luck! Jen Galaxy team On 2/28/13 12:34 AM, Thomas Muench wrote: > Hi, > > i have installed RepeatExplorer on our local galaxy instance, but with every try, i see only this error message and i don't know why? All installation steps of RepeatExplorer was correctly. > > Thank you for your help > > Thomas Münch > leibniz-institute for plant genetics and crop plant research > Gatersleben, Germany > > > Traceback (most recent call last): > File "/opt/galaxy/lib/galaxy/jobs/runners/local.py", line 56, in run_job > job_wrapper.prepare() > File "/opt/galaxy/lib/galaxy/jobs/__init__.py", line 368, in prepare > self.command_line = self.tool.build_command_line( param_dict ) > File "/opt/galaxy/lib/galaxy/tools/__init__.py", line 1524, in build_command_line > command_line = fill_template( self.command, context=param_dict ) > File "/opt/galaxy/lib/galaxy/util/template.py", line 9, in fill_template > return str( Template( source=template_text, searchList=[context] ) ) > File "/opt/galaxy/eggs/Cheetah-2.2.2-py2.4-linux-x86_64-ucs4.egg/Cheetah/Template.py", line 1004, in __str__ > return getattr(self, mainMethName)() > File "cheetah_DynamicallyCompiledCheetahTemplate_1362033265_04_79640.py", line 92, in respond > NotFound: cannot find '__root_dir__' > > > failure preparing job > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org

1 0

Search software on public Galaxy servers
by Bernardo Bello 04 Mar '13

04 Mar '13

Hi community, Due to the increasing Galaxy public servers PublicGalaxyServers<http://wiki.galaxyproject.org/PublicGalaxyServers> it would be very useful to have a search tool in order to look for software ready-to-use in the public servers. Since I know this is not available. Will it be feasible? Here is the Trello Card https://trello.com/c/FdF5h17c Regards, Bernardo -- *Bernardo Bello Ortí* PhD student CReSA-IRTA Campus de Bellaterra-Universitat Autònoma de Barcelona Edifici CReSA 08193 Bellaterra (Barcelona, Spain) Tel.: 647 42 52 63 *www.cresa.es * * *

4 4

Re: [galaxy-dev] [External] Re: data from galaxy
by Dannon Baker 04 Mar '13

04 Mar '13

I'm a little confused about the state of this instance. In the cloud admin panel, you have a user listed. And, in galaxy, you can log in as that user and act as an admin? But when you go to "Manage users" there are no users listed? Lastly, I've cc'd galaxy-dev here. Please do try to email the list with these questions, and not individual developers directly. -Dannon On Wed, Feb 27, 2013 at 4:38 PM, Sun, Wenping [USA] <Sun_Wenping(a)bah.com>wrote: > Hi Dannon, > I just found that the problem of user and data history from below email > that happened again today. The galaxy cloud cluster was terminated 1-2 > weeks ago. I have tried once 1 weeks ago to re-launch and it seemed > everything was fine, both with users and data history. > > However, I just tried to re-launch galaxy and found that all the users not > there. the admin email still on the galaxy launch page, however, on "manage > users" from galaxy interface, there was no user there (it suppose 2 user, > same as the admin I added in the first place). All the data history was not > available because of that as well. > > I really appreciate your help on this at your early convenience!! I need > to retrieve them for tomorrow morning meeting. > Kathryn > > -----Original Message----- > From: Dannon Baker [mailto:dannonbaker@me.com] > Sent: Monday, January 21, 2013 9:51 AM > To: Sun, Wenping [USA] > Subject: Re: [External] Re: data from galaxy > > Different -- the point is to create a *new* admin user since the password > was lost for the other one. > > > On Jan 21, 2013, at 9:50 AM, "Sun, Wenping [USA]" <Sun_Wenping(a)bah.com> > wrote: > > > Use different email address or the same? > > > > -----Original Message----- > > From: Dannon Baker [mailto:dannonbaker@me.com] > > Sent: Monday, January 21, 2013 9:50 AM > > To: Sun, Wenping [USA] > > Subject: Re: [External] Re: data from galaxy > > > > Add a new user as an admin in the cloudman interface, and then register > and log in as that new user. Then look at the Admin section of Galaxy and > you can reset the password for your other user there. > > > > On Jan 21, 2013, at 9:47 AM, "Sun, Wenping [USA]" <Sun_Wenping(a)bah.com> > wrote: > > > >> No, I tried and it said not recognized. I even cannot reset the > password, here is the error for reset password "Mail is not configured for > this Galaxy instance. Please contact an administrator." > >> I have lot of data and don't have time to reproduce them. Please help > me with this. > >> Thanks a lot! > >> Kathryn > >> > >> -----Original Message----- > >> From: Dannon Baker [mailto:dannonbaker@me.com] > >> Sent: Monday, January 21, 2013 9:45 AM > >> To: Sun, Wenping [USA] > >> Subject: [External] Re: data from galaxy > >> > >> You should be able to log back in with the same username and password > you used before. Is it not working? > >> > >> -Dannon > >> > >> > >> On Jan 21, 2013, at 9:43 AM, "Sun, Wenping [USA]" <Sun_Wenping(a)bah.com> > wrote: > >> > >>> Hi Dannon, > >>> > >>> I have the whole processed data from galaxy on cloudman and they are > on during the weekend. However, I found out that I was logged out this > morning. My user name is still in admin. I tried to log in but I could not. > This data takes time and would you please direct me how to get back to my > data? > >>> > >>> Thank you in advance! > >>> Kathryn > >> > > > >

2 3