December 2011 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy with PBS runner Setting
by Nate Coraor 20 Dec '11

20 Dec '11

On Dec 19, 2011, at 4:17 PM, Le-Shin Wu wrote: > Nate, > > Thanks a lot for your information. After checking the log file, I found the following error message: > > galaxy.jobs ERROR 2011-12-19 16:01:33,648 Job runner is not loadable: galaxy.jobs.runners.'pbs' > Traceback (most recent call last): > File "/N/soft/mason/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1120, in _load_plugin > module = __import__( module_name ) > ImportError: No module named 'pbs' Oh, sorry I missed it the first time around. You'll need to remove the single quotes from around pbs in universe_wsgi.ini. --nate > > So it looks like pbs module is not loaded. We do have pbs_python-4.1.0-py2.6-linux-x86_64-ucs4.egg under galaxy-dist/eggs, but , based on the log file, it seems not within the python path. Could you please let us know how to add this egg file to the python path in galaxy? > > Le-Shin > > > > On Dec 19, 2011, at 3:51 PM, Nate Coraor wrote: > >> On Dec 19, 2011, at 3:03 PM, Le-Shin Wu wrote: >> >>> We are trying to get galaxy to work on our cluster. Unfortunately, there are some questions I still cannot find the answer to and I'm hoping someone here can help me out. >>> >>> We would like to use the unified approach with TPRQUE PBS to support running galaxy tools on a cluster. Our questions are listed as below. >>> >>> (1) How to configure the universe_wsgi.ini file? Based on the galaxy wiki page, we changed the following parameters in universe_wsgi.ini >>> new_file_path = /N/dc/projects/galaxy/new_files # this directory is accessible both to galaxy server and cluster nodes >>> start_job_runners = 'pbs' >>> default_cluster_job_runner = pbs://cluster.hostname.edu/ #cluster.hostname.edu is the hostname for our cluster >>> >>> Also under [galaxy:tool_runners] section, we add a new tool named mytool for testing as below. >>> mytool = pbs://cluster.hostname.edu/ >>> >>> Does this configuration look correct and enough? >>> >>> (2) We created a new python egg for pbs by using >>> LIBTORQUE_DIR=/usr/local/lib/libtorque python scripts/scramble.py -e pbs_python --config=universe_wsgi.ini >>> >>> A new file pbs_python-4.1.0-py2.6-linux-x86_64-ucs4.egg was then created under eggs. >>> >>> Do we need to do other things or only this operation is enough? >>> >>> (3) We created a new directory named mytool under tools and added a new file named newTool.xml in it. The content of newTool.xml is as below. Note that, the /N/soft/mason/test_tool is a executable binary file at our cluster. >>> >>> <tool id="mytool" name="Testing tool" version="0.12.7"> >>> <description>do the pbs testing</description> >>> <command> >>> /N/soft/mason/test_tool -p 8 hg18 -f $inputseq $output >>> </command> >>> <inputs> >>> <param name="db" type="inputseq" label="input file" /> >>> </inputs> >>> <outputs> >>> <data format="sam" name="output" label="result" /> >>> </outputs> >>> <tests> >>> <test> >>> <param name="inputseq" value="a.fa" /> >>> <output name="output" file="b.sam" /> >>> >>> </test> >>> </tests> >>> >>> <help> >>> **What it does** >>> This tool uses Bowtie to produce an alignment. >>> .. _Bowtie: http: >>> >>> </help> >>> </tool> >>> >>> Does this xml file look correct? Do we also need a python wrapper file to somehow invoking PBS support? >>> >>> (4) Add the following text into tool_conf.xml. >>> <section name="My Tools" id="mt"> >>> <tool file="mytool/newTool.xml" /> >>> </section> >>> >>> (5) After doing the above modification and restarting our galaxy server , we can see "My Tools" showing on the tools panel of our galaxy web interface. But after we select an input file and click execute button, the job status shows an error "Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator". We also check the log file, it shows " Invalid job runner: bps". >> >> Hi Le-Shin, >> >> It would appear that the PBS job runner failed to start. You should be able to see a traceback with more details about this displayed during the server's startup. >> >> --nate >> >>> >>> We appreciate very much for your kind help. >>> >>> >>> Le-Shin >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> > >

1 0

Re: [galaxy-dev] Hidden history item
by Jeremy Goecks 20 Dec '11

20 Dec '11

Alex, I just implemented this functionality in galaxy-central changset 303741fda0ec and documented it on the wiki. https://bitbucket.org/galaxy/galaxy-central/changeset/303741fda0ec http://wiki.g2.bx.psu.edu/Admin/Tools/Tool%20Config%20Syntax#A.3Coutputs.3E… Best, J. > Hi all, > > by default I let my tools produce ("the old way") a logfile of the console and additional running statistics. This is convenient at tool setup and ocassionally for troubleshooting. Anyway I would like to make the corresponding history item hidden. I tried including hidden="true" in the output tag but that does not make any difference. > > Is this possible at all or only applicable to running workflows? > > Any pointers appreciated. > Alex

2 1

Galaxy with PBS runner Setting
by Le-Shin Wu 20 Dec '11

20 Dec '11

We are trying to get galaxy to work on our cluster. Unfortunately, there are some questions I still cannot find the answer to and I'm hoping someone here can help me out. We would like to use the unified approach with TPRQUE PBS to support running galaxy tools on a cluster. Our questions are listed as below. (1) How to configure the universe_wsgi.ini file? Based on the galaxy wiki page, we changed the following parameters in universe_wsgi.ini new_file_path = /N/dc/projects/galaxy/new_files # this directory is accessible both to galaxy server and cluster nodes start_job_runners = 'pbs' default_cluster_job_runner = pbs://cluster.hostname.edu/ #cluster.hostname.edu is the hostname for our cluster Also under [galaxy:tool_runners] section, we add a new tool named mytool for testing as below. mytool = pbs://cluster.hostname.edu/ Does this configuration look correct and enough? (2) We created a new python egg for pbs by using LIBTORQUE_DIR=/usr/local/lib/libtorque python scripts/scramble.py -e pbs_python --config=universe_wsgi.ini A new file pbs_python-4.1.0-py2.6-linux-x86_64-ucs4.egg was then created under eggs. Do we need to do other things or only this operation is enough? (3) We created a new directory named mytool under tools and added a new file named newTool.xml in it. The content of newTool.xml is as below. Note that, the /N/soft/mason/test_tool is a executable binary file at our cluster. <tool id="mytool" name="Testing tool" version="0.12.7"> <description>do the pbs testing</description> <command> /N/soft/mason/test_tool -p 8 hg18 -f $inputseq $output </command> <inputs> <param name="db" type="inputseq" label="input file" /> </inputs> <outputs> <data format="sam" name="output" label="result" /> </outputs> <tests> <test> <param name="inputseq" value="a.fa" /> <output name="output" file="b.sam" /> </test> </tests> <help> **What it does** This tool uses Bowtie to produce an alignment. .. _Bowtie: http: </help> </tool> Does this xml file look correct? Do we also need a python wrapper file to somehow invoking PBS support? (4) Add the following text into tool_conf.xml. <section name="My Tools" id="mt"> <tool file="mytool/newTool.xml" /> </section> (5) After doing the above modification and restarting our galaxy server , we can see "My Tools" showing on the tools panel of our galaxy web interface. But after we select an input file and click execute button, the job status shows an error "Unable to run job due to a misconfiguration of the Galaxy job running system. Please contact a site administrator". We also check the log file, it shows " Invalid job runner: pbs". We appreciate very much for your kind help. Le-Shin

2 1

Executing a binary file from Galaxy job
by Roger Ngo 19 Dec '11

19 Dec '11

Hi, I have a perl script which executes a Java JAR file located in tool-data/shared/jars/ and this Jar file at one point executes 2 other binary programs during its job process. These 2 programs are located in the same directory as the JAR file. I am getting errors in Galaxy saying "No such file or directory" when these 2 external programs are called. I am assuming this is the case because they are not located in current working directory of the current Galaxy job process. Is there a way to somehow redirect Galaxy into the correct directory where the binary files are located? Thanks.

2 1

Re: [galaxy-dev] Error viewing BAM files in IGV
by Nate Coraor 19 Dec '11

19 Dec '11

On Dec 14, 2011, at 4:22 PM, Alexander Graf wrote: > Hi nate, > I have tried it with several BAM files, resulting in the same error. > If I download the BAM and bai files from Galaxy and load it into IGV manually, everything works like charm. > Up to now I could not figure out why it is not working. > Could I have better success switching to the nginx-server? Hi Alex, It should work with Apache as well. I don't have an environment set up here to test, but could you take a look at the Apache access and error logs to determine whether the file is being found and read properly? It's possible that the request is returning something other than a 200 code and the file data. --nate > > Alex > > Am 12.12.2011 um 16:11 schrieb Nate Coraor: > >> On Dec 9, 2011, at 6:45 AM, Alexander Graf wrote: >> >>> Hello, >>> I have recently updated our Galaxy dist and I'm running into problems viewing BAM files in IGV (v2.0.22), saying: Invalid GZIP header. >> >> Hi Alex, >> >> Your config below looks okay at first glance. Can you verify that the file in question is a valid BAM? Or is this happening with all BAMs? >> >> --nate >> >>> I have configured the Apache as explained in the Wiki using this http.conf: >>> >>> >>> ------------------------------------------------------------------------------------------------------------------------------------ >>> <VirtualHost *:80> >>> ServerName 127.0.0.1 >>> RewriteEngine on >>> >>> RewriteRule ^/galaxy$ /galaxy/ [R] >>> RewriteRule ^/galaxy/static/style/(.*) /opt/galaxy/static/june_2007_style/blue/$1 [L] >>> RewriteRule ^/galaxy/static/scripts/(.*) /opt/galaxy/static/scripts/packed/$1 [L] >>> RewriteRule ^/galaxy/static/(.*) /opt/galaxy/static/$1 [L] >>> RewriteRule ^/galaxy/favicon.ico /opt/galaxy/static/favicon.ico [L] >>> RewriteRule ^/galaxy/robots.txt /opt/galaxy/static/robots.txt [L] >>> RewriteRule ^/galaxy(.*) http://localhost:8081$1 [P] >>> >>> <Proxy http://localhost:8081> >>> Order deny,allow >>> Allow from all >>> </Proxy> >>> >>> <Location "/galaxy"> >>> # Define the authentication method >>> XSendFile on >>> XSendFilePath / >>> # Compress all uncompressed content.http.conf >>> SetOutputFilter DEFLATE >>> SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary >>> SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary >>> </Location> >>> <Directory "/galaxy/static"> >>> ExpiresActive On >>> ExpiresDefault "access plus 6 hours" >>> </Directory> >>> </VirtualHost> >>> ---------------------------------------------------------------------------------------------------------------------------------------- >>> >>> >>> By having this parts changed in universe_wsgi.ini: >>> ---------------------------------------------------------------------------------------------------------------------------------------- >>> [server:main] >>> use = egg:Paste#http >>> port = 8081 >>> host = 0.0.0.0 >>> use_threadpool = True >>> >>> [filter:gzip] >>> use = egg:Paste#gzip >>> >>> [filter:proxy-prefix] >>> use = egg:PasteDeploy#prefix >>> prefix = /galaxy >>> >>> [app:main] >>> paste.app_factory = galaxy.web.buildapp:app_factory >>> filter-with = proxy-prefix >>> cookie_path = /galaxy >>> apache_xsendfile = True >>> upstream_gzip = False >>> ---------------------------------------------------------------------------------------------------------------------------------------- >>> >>> The resulting Galaxy error log is: >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> galaxy.web.framework DEBUG 2011-12-09 12:32:08,825 Error: this request returned None from get_history(): http://10.153.182.203/galaxy/root >>> 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] "GET /galaxy/root?app_action=data&user_id=c9a3f3a19e75965d&app_name=igv_bam&link_name=local_default&action_param=galaxy_9b0f702d0207cd78.bam.bai&dataset_id=9b0f702d0207cd78 HTTP/1.1" 200 - "-" "IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22" >>> 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] "HEAD /galaxy/display_application/9b0f702d0207cd78/igv_bam/local_default/c9a3f3a19e75965d/data/galaxy_9b0f702d0207cd78.bam HTTP/1.1" 302 - "-" "IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22" >>> galaxy.web.framework DEBUG 2011-12-09 12:32:08,915 Error: this request returned None from get_history(): http://10.153.182.203/galaxy/root >>> 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] "HEAD /galaxy/root?app_action=data&user_id=c9a3f3a19e75965d&app_name=igv_bam&link_name=local_default&action_param=galaxy_9b0f702d0207cd78.bam&dataset_id=9b0f702d0207cd78 HTTP/1.1" 200 - "-" "IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22" >>> ---------------------------------------- >>> Exception happened during processing of request from ('127.0.0.1', 52683) >>> Traceback (most recent call last): >>> File "/opt/galaxy/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 1053, in process_request_in_thread >>> self.finish_request(request, client_address) >>> File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request >>> self.RequestHandlerClass(request, client_address, self) >>> File "/usr/lib/python2.6/SocketServer.py", line 618, in __init__ >>> self.finish() >>> File "/usr/lib/python2.6/SocketServer.py", line 661, in finish >>> self.wfile.flush() >>> File "/usr/lib/python2.6/socket.py", line 297, in flush >>> self._sock.sendall(buffer(data, write_offset, buffer_size)) >>> error: [Errno 32] Broken pipe >>> ---------------------------------------- >>> >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>> >>> >>> Thanks in advance for your help >>> >>> Alex >>> >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ > >

1 0

concurrent job run parameter problem
by Oleksandr Moskalenko 19 Dec '11

19 Dec '11

Scenario: Running two velvet jobs with the same parameters except for the "Conditional (contig_lgth) either 0 or 1 (with minimum contig length = 500) If the scheduler runs these jobs separately they produce expected results. However, if they happen to start running simultaneously then one of the jobs looses its parameters and both jobs produce identical results corresponding to the contig_lgth not set. Galaxy config is load-balanced with jobs tracked in the database and metadata set externally in postgresql. the .sh files that are generated appear to be correct and job directories are created appropriately. Whatever fouls up the resuts seems to happen on the rear end of the job I guess. I wonder if anyone ran into this issue? It should be a common scenario where people try out different config parameters for the same tool running on the same dataset, so I wonder if it's more likely to be my galaxy configuration problem then an actual galaxy bug. My job runner config: http://pastebin.com/vMWDHAQm Thanks, Alex

2 1

Job runner out of memory problem
by Oleksandr Moskalenko 19 Dec '11

19 Dec '11

It seems that even under very light i.e. just a couple of long-running jobs present load the job runner slowly, but steadily (1-3Mb per sec) grows its memory consumption until it's killed by linux OOM killer. My job runner config: http://pastebin.com/vMWDHAQm I'm currently restarting the runner out of crontab when it is killed by OOM, but it's not a sensible solution by any means. I wonder if anyone encountered this and how it was solved. Thanks, Alex

2 1

abolut ftp setup
by Richard Liao 19 Dec '11

19 Dec '11

Dear all, I'm a student at Fudan University in China. I was trying to enable ftp uploading in my local version of GALAXY. I tried the setting in http://wiki.g2.bx.psu.edu/Admin/Config/Upload%20via%20FTP, but I got the following problems: 1. The galaxyftp seems to be a pasql database, shall I build it myself? or does it already exists in galaxy directory? 2. the galaxy_user seems to be a table, shall I build it myself? and how shall I update it and integrate it with galaxy? Thanks for your attention, and looking forward to your reply! Best wishes Liao Ruiqi

2 1

Re: [galaxy-dev] [galaxy-user] Help need in troubleshooting cufflinks
by Carlos Borroto 19 Dec '11

19 Dec '11

Hi Vijay, I ran into this error also, which by the way is not a galaxy issue but cufflinks and more specific is a Boost's bug. In my case it was caused because I was using the last version of Boost and there is a bug on that version causing the problem. You could apply this patch to boost: https://svn.boost.org/trac/boost/changeset/75540 If you are by any chance using Mac OS X, I would recommend using Homebrew (http://mxcl.github.com/homebrew/) to install cufflinks. Homebrew will take care of installing any dependency needed by cufflinks, for example Boost. In Homebrew Boost have been patched to resolve this issue. Hope it helps, Carlos PS: I'm copying galaxy-dev list, as this is an issue with a local install. On Wed, Dec 14, 2011 at 4:15 AM, Vijay T <vijay.t(a)elogic.co.in> wrote: > Hi Team, > > This is Vijay from ELogic Technologies Pvt Ltd, Bangalore, India. > > 1. Using Galaxy local > 2. NA > 3. Galaxy was downloaded using the tar files > 4. NA > 5. When i tried to install Cufflinks into my system i am getting the following error. > > In file included from hits.h:21:0, > from abundances.h:20, > from differential.cpp:17: > common.h: At global scope: > common.h:25:25: error: ‘boost::BOOST_FOREACH’ has not been declared > In file included from differential.h:29:0, > from differential.cpp:18: > replicates.h: In member function ‘bool ReplicatedBundleFactory::next_bundle(HitBundle&)’: > replicates.h:141:50: warning: unused variable ‘s2’ [-Wunused-variable] > differential.cpp: In member function ‘void TestLauncher::perform_testing(std::vector<boost::shared_ptr<SampleAbundances> >&)’: > differential.cpp:111:31: warning: unused variable ‘s2’ [-Wunused-variable] > differential.cpp: In function ‘void sample_abundance_worker(const string&, SampleAbundances&, HitBundle*, bool, bool)’: > differential.cpp:704:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] > make[2]: *** [differential.o] Error 1 > make[2]: Leaving directory `/home/binet/Downloads/cufflinks-1.2.0/src' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/home/binet/Downloads/cufflinks-1.2.0' > make: *** [all] Error 2 > > > Even i have tried with various versions of cufflinks but i am getting the same error. > Please kindly help me in getting out of this error. > > > Thanks and Regards > > Vijay > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/

1 0

uname: write error: Broken pipe
by Peter Cock 19 Dec '11

19 Dec '11

Dear all, While updating my MIRA wrapper for MIRA v3.4.0, with a test case that takes about half an hour to run, I was sometimes getting the following error message (on stderr): uname: write error: Broken pipe My hunch is this is a side effect of adding a version_command tag, <version_command>mira --version | head -n 1</version_command> https://bitbucket.org/peterjc/galaxy-central/changeset/d2c80e51d1fc This isn't down to the dataset or very long run times - I've tried a small phage which assembles in about 5 minutes with the same error on stdout. When I have repeated the exact same command at the terminal, it has run without issue, e.g. $ mira --version | head -n 1 &> /home/pjcock/repositories/galaxy-central/database/tmp/GALAXY_VERSION_STRING_192; python /home/pjcock/repositories/galaxy-central/tools/sr_assembly/mira.py mira /home/pjcock/repositories/galaxy-central/database/files/000/dataset_403.dat /home/pjcock/repositories/galaxy-central/database/files/000/dataset_404.dat /home/pjcock/repositories/galaxy-central/database/files/000/dataset_406.dat /home/pjcock/repositories/galaxy-central/database/files/000/dataset_405.dat /home/pjcock/repositories/galaxy-central/database/files/000/dataset_407.dat /home/pjcock/repositories/galaxy-central/database/files/000/dataset_408.dat mira --job=denovo,genome,accurate 454_SETTINGS -LR:lsd=yes -LR:mxti=no -LR:ft=fastq -FN:fqi=/home/pjcock/repositories/galaxy-central/database/files/000/dataset_402.dat COMMON_SETTINGS -OUT:rrot=yes -OUT:rtd=yes MIRA took 4.98 minutes Done One point which may be relevant is that the MIRA binary produces quite a lot of output on stdout (often megabytes), which my Python wrapper scripts is redirecting to a file (to collect it as a Galaxy output). Perhaps this issue is related to lots of data on stdout/stderr? For the moment I am removing the version_command tag, which appears to solve this problem, and I will report back if the broken pipe error still happens. Has anyone else noticed anything similar? Peter

1 0