May 2012 - galaxy-dev - lists.galaxyproject.org

galaxy cluster integration
by Anne Pajon 07 May '12

07 May '12

Hi, I am struggling at integrating galaxy with our cluster. I am currently running galaxy on a separate server machine from our LSF cluster. The DRMAA library has been installed on the cluster and I've re-installed galaxy into a shared file system visible and accessible from both the server and the cluster. I've read the wiki page: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster and tried to follow the "unified method" without much success so far. After spending some time digging/reading the mailing list, I would like to be sure that I understand the problem well enough before changing our plans. Am I right to think that there is no way to make the system works because I cannot currently configure galaxy to ssh into the cluster head node for submitting jobs to the scheduler? Is the only way forward to have a dedicated cluster node where I could run galaxy on it? Thanks in advance for your help. Kind regards, Anne. -- Anne Pajon, Ph.D. Cancer Research UK - Cambridge Research Institute Li Ka Shing Centre, Robinson Way, Cambridge CB2 0RE anne.pajon(a)cancer.org.uk | +44 (0)1223 404 334 NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose. We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you. Cancer Research UK Registered in England and Wales Company Registered Number: 4325234. Registered Charity Number: 1089464 and Scotland SC041666 Registered Office Address: Angel Building, 407 St John Street, London EC1V 4AD.

3 8

Returned mail: see transcript for details
by nate＠bx.psu.edu 07 May '12

07 May '12

Dear user of lists.bx.psu.edu, Mail server administrator of lists.bx.psu.edu would like to inform you that: Your account was used to send a large amount of junk e-mail messages during the recent week. Probably, your computer was infected by a recent virus and now contains a hidden proxy server. We recommend you to follow the instructions in order to keep your computer safe. Virtually yours, The lists.bx.psu.edu team.

1 0

Error when using the "Compare Two Datasets" tool
by Antonio Ribeiro 07 May '12

07 May '12

Hi Galaxy Community, I have recently setup a small local instance of Galaxy and, when trying to replicate the quickie 15 (about 454 Mapping with Lastz) steps, but using real 454 data, got this error in the execution of the "Comparing two datasets" tool. Would anyone have any hint about how to solve it*? Apparently, there was a similar report of this issue to the list (in March, 2012), but I could not find the answer for it. *Thanks, in advance, for any help or advice! Antonio **************************************************************************************************************** Traceback (most recent call last): File "/home/acbellorib/galaxy-dist/lib/galaxy/jobs/runners/local.py", line 133, in run_job job_wrapper.finish( stdout, stderr ) File "/home/acbellorib/galaxy-dist/lib/galaxy/jobs/__init__.py", line 684, in finish self.sa_session.flush() File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/scoping.py", line 127, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py", line 1356, in flush self._flush(objects) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/session.py", line 1434, in _flush flush_context.execute() File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 261, in execute UOWExecutor().execute(self, tasks) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 773, in execute_save_steps self.execute_cyclical_dependencies(trans, task, False) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 789, in execute_cyclical_dependencies self.execute(trans, [t], isdelete) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 753, in execute self.execute_save_steps(trans, task) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 768, in execute_save_steps self.save_objects(trans, task) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/unitofwork.py", line 759, in save_objects task.mapper._save_obj(task.polymorphic_tosave_objects, trans) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/orm/mapper.py", line 1413, in _save_obj c = connection.execute(statement.values(value_params), params) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 824, in execute return Connection.executors[c](self, object, multiparams, params) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement return self.__execute_context(context) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 896, in __execute_context self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 950, in _cursor_execute self._handle_dbapi_exception(e, statement, parameters, cursor, context) File "/home/acbellorib/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'UPDATE history_dataset_association SET update_time=?, info=?, blurb=?, peek=?, tool_version=?, metadata=? WHERE history_dataset_association.id = ?' ['2012-05-07 03:35:33.977226', 'join (GNU coreutils) 8.13\nCopyright (C) 2011 Free Software Foundation, Inc.\nLicen\xc3\xa7a GPLv3+: GNU GPL vers\xc3\xa3o 3 ou posterior <http://gnu.org/licenses/gpl.html>.\nEste \xc3\xa9 um software livre: voc\xc3\xaa \xc3\xa9 livre para modific\xc3\xa1-lo ou redistribu\xc3\xad-lo.\nN\xc3\xa3o h\xc3\xa1 NENHUM', '~610,000 lines', u'GKJUVNG01A0005\t0\tGeneDB\t1511989\t255\t236M1I99M\t*\t0\t0\tCTTGAACTCTTCTGACGTTACCCTGTCCCCGGCACCGCCGACACGACCGCCTCCTGTTTCTCTGGCCGCTGCAGGCGTCAAGGTCGACGCTGAGTTGTCACTGCTCGCGACGACTCCACCGAGAACAAGCGAGCAGGAGAACGGCGCAGACACGCGGTCCTTGAGTGCTCTGTCGACACTGGTTGCAGCGATGGAAGGCGGTGA\nGTCGTCGACGCTGCTTCCACGGGCTCCTCCTGCCCCCGCGCACCAACACTGGTAAGCCCAGCGTGGCGTCAAGGCGGGCTGTAGCCTCGGCGGACAAGGGTGCCGTGAAAAGNGAGTCGCCAACGTCTTCCA\t*\n\nGKJUVNG01A0016\t0\tGeneDB\t71861\t255\t79M\t*\t0\t0\tTTCTGCATCCCGCTCCTGGGCACACATCGTGAGAGTAGCACCTGTGCACTCGCTCTCGCTGTCGCACCACGCTTCGCGT\t*\n\nGKJUVNG01A001C\t16\tGeneDB\t1482878\t255\t7H48M1D229M1I15M\t*\t0\t0\tATCAGCCGCCTCGAGGCACACGAGAGCGACGTCACCGAGCGCCTTGAGCACAGTACCAGCGAACGACAGCGCAGCTGACGTCGTTGGAGAAGGACATGCGCTTTCGCCTTCAGCAACTAGCCGACGCAGATCGCAAGGAGCGGGCGGCAGCGGCGGACCGGCAAACGGCCTTGCTCGAGTCCCTGGCCGCCCTCCG\nCGGCGAGCCGGCTGCCGTGTCTGCGGAGGGCATGACGGAGACGATGCTGCGACAGCTCATACATCTCATAGAGGCGGGGATGGGGGAAGGTACAGGC\t*\n\nGKJUVNG01A001K\t16\tGeneDB\t296529\t255\t286M3H\t*\t0\t0\tATTCCATCCAGTCGGCTCTTGCAGATTCTCAGTCGCGATCAGGAAGTGGTAGATCGCGCCCTTGGCGAGTCGGCTCAGCTCATCTTCCTGACAATTCTCCAGCTCTTCGGCATGGTGTGCTTCAACGCTATCCACTTCGGCCCTTTCATCGCTGTGCTTCCGATATCCGCGCTTCTCTTCTACCATCTGACGGTTCGCTTTCTATCC', '', <read-only buffer for 0x7f1958038e88, size -1, offset 0 at 0x7f19580565f0>, 516]

1 0

Running Galaxy in a production environment
by CHEBBI Mohamed Amine 05 May '12

05 May '12

Hi Galaxy Team, I'am trying to put Galaxy into production on a dedicated server. And I have some questions please : 1) how can I integrate galaxy behind an Apache server (or nginx)? 2) how to switch the SQL lite database to the Postgresql one (would you have an example to do it)? 3) What should I do to link the Galaxy session management for an outside application like Django I thank you in advance. Regards, Amine.

2 1

job terminating on warnings from software
by Michael Moore 05 May '12

05 May '12

I have a long string of steps dying on the last step because GNUPLOT sends a warning to the log which galaxy faithfully records 11 faithful times in my history files. The 10 other files have downloadable content, and in fact, outside of galaxy the plot works. It is simply changing some intervals. How can I tell galaxy to keep going unless I get an actual error? Michael

2 1

Spooky behavior
by Michael Moore 04 May '12

04 May '12

OK data library, (brand new). I upload one BAM file using galaxy's "Add Dataset" I start a new history, titling it, "testing multiple use". I import to current history. I run a job that reads the BAM file and produces some output. Everything works. I have 13 output data sets in the history, all showing green. Now I create a second history, called "2nd use same dataset same library" Again I check the box and click "import to current history" Then I try to run the same job as before... [bam_header_read] bgzf_check_EOF: Invalid argument [bam_header_read] invalid BAM binary header (this is not a BAM file). [main_samview] fail to read the header from "/.../galaxy/database/files/000/dataset_961.dat" (Ellipsis is mine, not Python's) Oh my, looking in that file space, I discover that my BAM file is at dataset_962.dat. Was gibt? Michael Moore .

1 0

SAM to BAM tool for custom genome build
by Unknown 04 May '12

04 May '12

Hi Galaxy Team, I am using custom genome build for creating a sam file. When I am trying to use the SAM to BAM tool inside Sam Tools with option locally cached I am getting an error like "Unspecified genome build, click the pencil icon in the history item to set the genome build". I added the custom genome .fa and .fai files path to sam_fa_indices.loc file. I could not find the locally cached genome in the drop down list when I clicked on the pencil icon. What should I do to specify the genome build on my sam file? Regards, Deepthi.

1 0

Tracking .bai files
by Unknown 04 May '12

04 May '12

Hai Galaxy Team, I have created a .sam file using my own tool. I used a custom genome that I added to the bowtie_indices.loc. I want to create another tool which will contain 3 steps, 1. samtool view (sam to bam conversion) 2. samtool sort (sorting bam file) 3. samtool index (creating .bai file) My problem is how the galaxy can know about the index(.bai) file I am creating? >From one of the mailing list I learn that a bai file is automatically created when adding a bam file to the history. http://dev.list.galaxyproject.org/question-about-bam-and-bai-td4136797.html So can I skip the 3rd step(samtool index)? ie,Whether a .bai file will create automatically when the tool create the sorted bam file? If yes,where will be the .bai file? Regards, Deepthi.

1 0

ImportError: No module named NSISSoft.NSISBase
by Kleo Achilleos 04 May '12

04 May '12

Dear all, I have a local installation of Galaxy. I need to create a new galaxy tool. My python program runs outside Galaxy. When I call it through Galaxy it does not find my custom libraries. * * *ImportError: No module named NSISSoft.NSISBase* * * *I understand it is a PYTHONPATH problem. How can I fix it?* * * *Kleo Achilleos* *MSc student at UCY, Cyprus*

2 3

Problem about configuring Proftpd with Mysql and galaxy
by Misharl mon 04 May '12

04 May '12

Hi everyone, I have a problem about configuring Proftpd and Mysql with galaxy, i have followed the steps from the galaxy wiki about ftp. I have created a database 'galaxy' and a user mmonsoor which has all the rights on the database. I think that the reason is about the proftpd.conf, below is a copy of my proftpd.conf: # This is a basic ProFTPD configuration file (rename it to # 'proftpd.conf' for actual use. It establishes a single server # and a single anonymous login. It assumes that you have a user/group # "nobody" and "ftp" for normal operation and anon. ServerName "Galaxy Server" ServerType standalone DefaultServer on # Port 21 is the standard FTP port. Port 21 # Don't use IPv6 support by default. UseIPv6 off # Umask 022 is a good standard umask to prevent new dirs and files # from being group and world writable. Umask 077 SyslogFacility DAEMON SyslogLevel debug # To prevent DoS attacks, set the maximum number of child processes # to 30. If you need to allow more than 30 concurrent connections # at once, simply increase this value. Note that this ONLY works # in standalone mode, in inetd mode you should use an inetd server # that allows you to limit maximum number of processes per service # (such as xinetd). MaxInstances 30 # Set the user and group under which the server will run. User nobody Group nogroup PassivePorts 30000 40000 # To cause every FTP user to be "jailed" (chrooted) into their home # directory, uncomment this line. DefaultRoot ~ # Automatically create home directory if it doesn't exist CreateHome on dirmode 700 # Allow users to overwrite their files AllowOverwrite on # Allow users to resume interrupted uploads AllowStoreRestart on # Normally, we want files to be overwriteable. AllowOverwrite on # Bar use of SITE CHMOD by default <Limit SITE_CHMOD> DenyAll </Limit> # Bar use of RETR (download) since this is not a public file drop <Limit RETR> DenyAll </Limit> # A basic anonymous configuration, no upload directories. If you do not # want anonymous users, simply delete this entire <Anonymous> section. <Anonymous ~ftp> User ftp Group ftp # We want clients to be able to login with "anonymous" as well as "ftp" UserAlias anonymous ftp # Limit the maximum number of anonymous logins MaxClients 10 # We want 'welcome.msg' displayed at login, and '.message' displayed # in each newly chdired directory. DisplayLogin welcome.msg DisplayChdir .message # Limit WRITE everywhere in the anonymous chroot <Limit WRITE> DenyAll </Limit> </Anonymous> # Do not authenticate against real (system) users #AuthPAM off # Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 SQLPasswordEngine on SQLPasswordEncoding hex # Set up mod_sql to authenticate against the Galaxy database SQLEngine on SQLBackend sql SQLConnectInfo galaxy@localhost mmonsoor MyPassword SQLAuthTypes SHA1 SQLAuthenticate users # An empty directory in case chroot fails SQLDefaultHomedir /var/opt/local/proftpd # Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user. SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT "email,password,'512','512','/home/mmonsoor/galaxy-dist/database/files/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" I don't know what is wrong about my proftpd.conf? Thanks in advance to all. Mish

1 0