hello Newbie questions about galaxy and clustering
hello I have installed an galaxy instance for purpose testing on a 8 CPU server without problem all run perfectly. I have made some galaxy xml wrapper to understant how galaxy calls program and other little things. Now I want to install a Galaxy server with a cluster configuration. Before I have tested HTCondor but I have no Idea how I can mix Galaxy and HTCondor. The perfect thing was somebody have already tested this type of condor. But I can satisfy myself with any other cluster type. What I don't understant in fact ( maybe my worst problem ) where lie all parts of this puzzle. I explain. I know that I need Galaxy. I know too I need a cluster ( in my case HTCondor ) I understand I need something between these 2 parts ... I think something like PBS or SGE but it's not totaly clear for me I'm search on galaxy mailling list archive ... but I just found 2 mails in 2008 :( has anybody made this conf or have a configuration ? greetings PS : sorry for my poor english :)
hello, My subject is at the border between Galaxy and Condor I'found method to use Galaxy with condor cluster solution. In fact I have wrote a script when Galaxy call clustalw2 , it calls in fact a script called clustalw2. All run exactly as I want excepting ... but Galaxy get in error and said there was a problem I turn my script to debug mode and it puts followings output + prog=/usr/bin/clustalw2 + err=0 + infile=-infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat + outfile=-outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat + outorder=-OUTORDER=ALIGNED + seqnos=-SEQNOS=OFF + typeseq=-TYPE=DNA + var6= + var7= + var8= + var9= + echo ++ echo -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat ++ sed -e s/-infile=// + infile2=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat ++ sed -e 's/galaxy_dataset_\(.*\)\.dat//' ++ sed -e s/-outfile=// ++ echo -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat + WorkingDir=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + [[ -n /usr/bin/clustalw2==clustalw2 ]] + echo 'universe = vanilla' + echo 'executable = /usr/bin/mpirun ' + echo 'arguments = "-np 10 -hostfile /home/galaxy/hostfile /usr/bin/clustalw-mpi -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat -OUTORDER=ALIGNED -SEQNOS=OFF -TYPE=DNA "' + echo 'Getenv = True machine_count = 1 transfer_input_files = /home/galaxy/hostfile,/usr/bin/clustalw-mpi, /home/galaxy/galaxy-dist/database/files/000/dataset_11.dat should_transfer_files = yes Log = /home/galaxy/logs/normal.$(cluster).$(process).log Output = /home/galaxy/logs/normal.$(cluster).$(process).out Error = /home/galaxy/logs/normal.$(cluster).$(process).error notification = never queue 1' + /usr/bin/condor_submit /home/galaxy/submit Submitting job(s). 1 job(s) submitted to cluster 179. I suspect the method I use, doesn't return properly information to galaxy ... but I don't know what I can do ... Any Idea ...
Hello, I've moved this over to the galaxy-dev list since it concerns a local installation. You'll want to use the 'condor' job running plugin, which is provided with the Galaxy distribution. Have a look at the job_conf.xml.sample_advanced file to see how this would be configured. A very basic job_conf.xml that should submit all jobs to HTCondor: <?xml version="1.0"?> <job_conf> <plugins workers="4"> <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" /> </plugins> <destinations default="condor"> <destination id="condor" runner="condor"/> </destinations> </job_conf> --nate On Apr 19, 2013, at 8:32 AM, leconte wrote:
hello,
My subject is at the border between Galaxy and Condor
I'found method to use Galaxy with condor cluster solution.
In fact I have wrote a script when Galaxy call clustalw2 , it calls in fact a script called clustalw2.
All run exactly as I want excepting ... but Galaxy get in error and said there was a problem
I turn my script to debug mode and it puts followings output
+ prog=/usr/bin/clustalw2 + err=0 + infile=-infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat + outfile=-outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat + outorder=-OUTORDER=ALIGNED + seqnos=-SEQNOS=OFF + typeseq=-TYPE=DNA + var6= + var7= + var8= + var9= + echo
++ echo -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat ++ sed -e s/-infile=// + infile2=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat ++ sed -e 's/galaxy_dataset_\(.*\)\.dat//' ++ sed -e s/-outfile=// ++ echo -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat + WorkingDir=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/ + [[ -n /usr/bin/clustalw2==clustalw2 ]] + echo 'universe = vanilla' + echo 'executable = /usr/bin/mpirun ' + echo 'arguments = "-np 10 -hostfile /home/galaxy/hostfile /usr/bin/clustalw-mpi -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat -OUTORDER=ALIGNED -SEQNOS=OFF -TYPE=DNA "' + echo 'Getenv = True machine_count = 1 transfer_input_files = /home/galaxy/hostfile,/usr/bin/clustalw-mpi, /home/galaxy/galaxy-dist/database/files/000/dataset_11.dat should_transfer_files = yes Log = /home/galaxy/logs/normal.$(cluster).$(process).log Output = /home/galaxy/logs/normal.$(cluster).$(process).out Error = /home/galaxy/logs/normal.$(cluster).$(process).error notification = never queue 1' + /usr/bin/condor_submit /home/galaxy/submit Submitting job(s). 1 job(s) submitted to cluster 179.
I suspect the method I use, doesn't return properly information to galaxy ... but I don't know what I can do ...
Any Idea ... ___________________________________________________________ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using "reply all" in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists, please use the interface at:
To search Galaxy mailing lists use the unified search at:
participants (2)
-
leconte -
Nate Coraor