- galaxy-dev - lists.galaxyproject.org

How to configure samtools in galaxy on cluster
by michael xu 07 Aug '12

07 Aug '12

Recently, I am trying to configure our galaxy on our cluster with PBS submission. I run galaxy locally very well on the head node, but I met one issue when I tried to configure galaxy on our cluster. In universe_wsgi.ini file, I added the following entries, start_job_runners = pbs default_cluster_job_runner = pbs:/// And then I run the command to generate an egg, LIBTORQUE_DIR=/usr/lib64/libtorque python scripts/scramble.py -e pbs_python --config=universe_wsgi.ini Then I run galaxy and submitted two jobs into the PBS queue. I checked the galaxy log and PBS queue and found these two jobs were submitted successfully. But after a while, galaxy told me that the two jobs were failed. I suspected I did not modify the galaxy:tool_runners section in the universe_wsgi.ini file. The node machines cannot find the 3rd party tools. Can anyone give me some advices? Thanks a lot!

2 1

Different User with Torque
by Wood, James, D 06 Aug '12

06 Aug '12

I am a systems administrator for a cluster, and we use Torque and Moab as the resource manager and scheduler. I read that you can use drmaa to submit jobs as a different user other than the 'galaxy' user using the external scripts, but I haven't yet found a way to do it with torque. Is there a solution that you could suggest? I am guessing that the pbs runner process would have to be forked and use setuid to be able to run under the user. This would be great, because it would make it easier to keep track of user time utilization.

2 2

Galaxy output not showing on tool success, indicating job failure.
by John Patterson 06 Aug '12

06 Aug '12

Hello. I am an undergraduate working on a tool very similar to Tophat. My question is probably very basic, but I am having issues displaying the output. I am assuming is giving me the failed run boxes, even though my tool completes execution and generates its files. Note that I am not working on a cluster, as I am simply trying out galaxy with this tool. I have modified universe_wsgi.ini "cleanup_job = onsuccess" to keep the output from being deleted due to the failure. Every file that is queued up in the history pane for each run appears in the working directory after execution, and I confirm successful runs with stdout and logs. If you click on the boxes in the history pane, you can download the output files correctly. I just cannot see previews of them or see them in the middle screen. output in my tool XML is specified as follows: <data format="sam" name="alignments" label="${tool.name} on ${on_string}: alignments" from_work_dir="mapsplice_out/alignments.sam" /> where mapsplice out is a sub directory created to contain all output of my command line tool. The reason I have it contained is because it runs its own combine and clean-up jobs before it is done executing, and you are left with logs. If you do not specify 'from_work_dir' in the output, the output files are there, but if you do specify, galaxy seems to be moving them into the database. stderr prints: (also I am aware stderr causes galaxy failures, I have nothing printing to stderr) "/home/XXXX/galaxy-dist/database/files/000/dataset_387.dat is fasta format" which suggests to me that galaxy is trying to copy the contents of the output files into this particular file which is not what needs to happen. If I do not include the optional "from_work_dir" in the <data> tag I am not able to download the files that are created. On the wiki, the output tag set says that " You specify the number and type of your output files using the contained<data>tags. You must pass them to your tool executable through using line variables just like the parameters described in the previous sections." For this, I have created variables to hold the output and pass them in the command string, but my tool takes no parameters to specify output. It simply creates various output files based on the parameters passed in. However it still runs regardless. The preview I get after execution is the print to stdout (exact copy of the log) that indicate a successful run. I apologize for the length, I am just trying to give most the helpful information. If more is needed, let me know. I am running my own instance of galaxy. John P - Undergraduate Research University of Kentucky Bioniformatics

1 0

queue question
by Zwemer, Lillian 06 Aug '12

06 Aug '12

Hi There, I'm not sure if this is the correct address to email to, so apologies in advance if I am in error. I am using the Galaxy Main (publicly available) site. I am wondering if there is a way to tell where my job is in the queue - in other words, how long it will be until my job begins to run and also ,how long until the run is done. I submitted a Bowtie paired end alignment last night (8.5.12) and the history states that my job is still waiting to run (grey). Thank you, Lillian Zwemer Lillian Zwemer, PhD Mother Infant Research Institute Tufts Medical Center The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Tufts Medical Center HIPAA Hotline at (617) 636-4422. If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.

1 0

Emailing to all our Galaxy users
by Joachim Jacob 06 Aug '12

06 Aug '12

Dear devs, How can I extract a list of the email addresses of all users of our Galaxy? Cheers, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

3 3

Re: [galaxy-dev] Error in loading tools.
by Nikhil Joshi 04 Aug '12

04 Aug '12

On Fri, Aug 3, 2012 at 11:07 PM, Nikhil Joshi <najoshi(a)ucdavis.edu> wrote: > Hey Mo, > > So I'm figuring out the same issues, and I think I have been able to > figure it out.... but for sure you need to uncomment the > tool_config_file and tool_path lines in your universe_wsgi.ini file, > probably should create a new directory one above your install > directory called "shed_tools", and then restart galaxy. I also ran > into the problem where I needed to create a TMPDIR shell variable that > pointed to the correct tmp directory. I know you guys don't like the > command line, so if this sounds like gibberish to you, I'm hoping the > galaxy team can provide a simpler solution. > > - Nik. > > P.S. I'm working on creating a DESeq repository that can be > automatically installed from the toolbox. I will let you know when I > am successful. It was nice to meet you at the conference! :) > > On Wed, Aug 1, 2012 at 9:07 AM, Mohammad Heydarian <mheydarian(a)gmail.com> wrote: >> Thanks for this. >> >> I updated my local version of Galaxy Cloudman and tried loading a tool and >> got a new error: >> >> The tool_config_file setting in universe_wsgi.ini must include at least one >> shed tool configuration file name with a <toolbox> tag that includes a >> tool_path attribute value which is a directory relative to the Galaxy >> installation directory in order to automatically install tools from a Galaxy >> tool shed (e.g., the file name shed_tool_conf.xml whose <toolbox> tag is >> <toolbox tool_path="../shed_tools">). >> >> See the Automatic installation of Galaxy tool shed repository tools into a >> local Galaxy instance section of the Galaxy tool shed wiki for all of the >> details. >> >> >> Any help interpreting this would be greatly appreciated! >> >> >> >> Cheers, >> >> Mo Heydarian >> >> >> >> >> On Tue, Jul 31, 2012 at 4:11 PM, Dannon Baker <dannonbaker(a)me.com> wrote: >>> >>> The problem here is that the public toolshed interface has been updated >>> more recently than the galaxy install on that cloud instance. You should >>> be good to go if you update galaxy (possible through the admin interface at >>> <your_ec2_instance>/cloud/admin/) to the latest version. >>> >>> -Dannon >>> >>> On Jul 31, 2012, at 12:11 PM, Mohammad Heydarian <mheydarian(a)gmail.com> >>> wrote: >>> >>> > Hi Galaxy Community! >>> > Hope all is well. >>> > >>> > I've been trying to install some tools in my Galaxy Cloudman instance >>> > (using image: ami-da58aab3 ) from the main toolshed. When I try to "install >>> > to local Galaxy" I get the error: >>> > >>> > Not Found >>> > >>> > The resource could not be found. >>> > No action for /admin_toolshed/prepare_for_install >>> > >>> > >>> > >>> > I have tried to load tools on a couple of different Cloudman instances >>> > and tried to load a variety of tools and end up with the same error. Any >>> > help would be fantastic. >>> > >>> > Cheers, >>> > Mo Heydarian >>> > >>> > >>> > >>> > ___________________________________________________________ >>> > Please keep all replies on the list by using "reply all" >>> > in your mail client. To manage your subscriptions to this >>> > and other Galaxy lists, please use the interface at: >>> > >>> > http://lists.bx.psu.edu/ >>> >> >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > > > > -- > Nikhil Joshi > Bioinformatics Analyst/Programmer > UC Davis Bioinformatics Core > http://bioinformatics.ucdavis.edu/ > najoshi -at- ucdavis -dot- edu > 530.752.2698 (w) -- Nikhil Joshi Bioinformatics Analyst/Programmer UC Davis Bioinformatics Core http://bioinformatics.ucdavis.edu/ najoshi -at- ucdavis -dot- edu 530.752.2698 (w)

1 0

Toolshed: disabling registering
by Joachim Jacob 03 Aug '12

03 Aug '12

Hi, Can I disable the account creation by users in our Galaxy tool shed? Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 3

genbank data type
by Marc Logghe 03 Aug '12

03 Aug '12

Hi all, The last few days I have been experimenting with a local Galaxy installation. Via the tool shed, emboss and emboss_datatypes were installed. Next, a custom tool was added for which a rich sequence is needed as input. I was hoping to achieve this as such: <param name="input" type="data" format="genbank" label="Annotated sequences"/> As a test, a genbank was created using EMBOSS's newseq and output format genbank was chosen (the history indicates the format is 'genbank', so that's fine). However, it seems not to be recognized as valid input for our custom tool. Guess I'm overlooking something obvious ? Regards, Marc ________________________________________ THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at ablynx(a)ablynx.com. Thank you for your co-operation. "NANOBODY" and "NANOCLONE" are registered trademarks of Ablynx N.V. ________________________________________

1 1

Strange "Could not moveŠ." error
by Iry Witham 02 Aug '12

02 Aug '12

Hi Dev-Team, I am experiencing a strange error that I cannot replicate on two identical instances of my Galaxy server. I have a tool that runs fine on my test instance and fails on my development instance. The servers are identically configured and running the same build. The handler.log from the successful instance shows the following: galaxy.jobs DEBUG 2012-08-02 11:31:07,881 (295) Working directory for job is: /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295 galaxy.jobs.handler DEBUG 2012-08-02 11:31:07,881 dispatching job 295 to pbs runner galaxy.jobs.handler INFO 2012-08-02 11:31:08,083 (295) Job dispatched galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:08,382 (295) submitting file /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/pbs/295.sh galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:08,382 (295) command is: perl /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/tools/jax_cnv/cnv-seq.pl --out-basename cnv_seq_result --test /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/files/001/dataset_1504.dat --ref /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/files/001/dataset_1503.dat --genome-size 2716965481 --p-value 0.001 --bigger-window 1.5 --no-annotate --log2-threshold 0.6; cd /hpcdata/galaxy-test/galaxy-setup/galaxy-dist; /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/set_metadata.sh ./database/files /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295 . /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/universe_wsgi.ini /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmplTSKeF /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/galaxy.json /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/metadata_in_HistoryDatasetAssociation_689_Oh21hx,/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/metadata_kwds_HistoryDatasetAssociation_689_xgxlZy,/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/metadata_out_HistoryDatasetAssociation_689_H0claK,/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/metadata_results_HistoryDatasetAssociation_689_rdQBs8,,/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/metadata_override_HistoryDatasetAssociation_689_B0U0MV galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:08,395 (295) queued in default queue as 112605.scyld.localdomain galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:08,776 (295/112605.scyld.localdomain) PBS job state changed from N to R galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:27,117 (295/112605.scyld.localdomain) PBS job state changed from R to C galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:31:27,118 (295/112605.scyld.localdomain) PBS job has completed successfully galaxy.jobs DEBUG 2012-08-02 11:31:27,197 finish(): Moved /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/job_working_directory/000/295/cnv_seq_result.cnv to /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/files/001/dataset_1555.dat as directed by from_work_dir galaxy.datatypes.metadata DEBUG 2012-08-02 11:31:27,249 loading metadata from file for: HistoryDatasetAssociation 689 galaxy.jobs DEBUG 2012-08-02 11:31:27,382 job 295 ended galaxy.datatypes.metadata DEBUG 2012-08-02 11:31:27,382 Cleaning up external metadata files The same log from the failing instance shows the following: galaxy.jobs DEBUG 2012-08-02 11:54:29,234 (113) Working directory for job is: /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113 galaxy.jobs.handler DEBUG 2012-08-02 11:54:29,234 dispatching job 113 to pbs runner galaxy.jobs.handler INFO 2012-08-02 11:54:29,420 (113) Job dispatched galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:29,669 (113) submitting file /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/pbs/113.sh galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:29,669 (113) command is: perl /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/tools/jax_cnv/cnv-seq.pl --out-basename cnv_seq_result --test /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/files/000/dataset_131.dat --ref /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/files/000/dataset_132.dat --genome-size 2716965481 --p-value 0.001 --bigger-window 1.5 --no-annotate --log2-threshold 0.6; cd /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist; /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/set_metadata.sh ./database/files /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113 . /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/universe_wsgi.ini /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/tmp/tmpgTn7ey /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/galaxy.json /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/metadata_in_HistoryDatasetAssociation_136_VYG968,/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/metadata_kwds_HistoryDatasetAssociation_136_yo_4aK,/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/metadata_out_HistoryDatasetAssociation_136_D6HwmI,/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/metadata_results_HistoryDatasetAssociation_136_zunslF,,/hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/metadata_override_HistoryDatasetAssociation_136_FDUxso galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:29,680 (113) queued in default queue as 112609.scyld.localdomain galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:30,163 (113/112609.scyld.localdomain) PBS job state changed from N to R galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:33,298 (113/112609.scyld.localdomain) PBS job state changed from R to C galaxy.jobs.runners.pbs DEBUG 2012-08-02 11:54:33,298 (113/112609.scyld.localdomain) PBS job has completed successfully galaxy.jobs DEBUG 2012-08-02 11:54:33,354 finish(): Could not move /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/job_working_directory/000/113/cnv_seq_result.cnv to /hpcdata/galaxy-dev/galaxy-setup/galaxy-dist/database/files/000/dataset_140.dat as directed by from_work_dir galaxy.jobs DEBUG 2012-08-02 11:54:33,467 job 113 ended Why would one fail and the other succeed? Thanks, Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

3 4

Re: [galaxy-dev] [galaxy-user] Fwd: Local instance and storage space to provide
by Scott McManus 01 Aug '12

01 Aug '12

Sorry - I fouled up and should have kept this on galaxy-dev. -Scott ----- Original Message ----- > > I'm forwarding this to galaxy-user instead of galaxy-dev. > > -Scott > > ----- Forwarded Message ----- > From: "Scott McManus" <scottmcmanus(a)gatech.edu> > To: "Sarah Maman" <sarah.maman(a)toulouse.inra.fr> > Cc: galaxy-dev(a)lists.bx.psu.edu > Sent: Wednesday, August 1, 2012 4:18:29 PM > Subject: Re: [galaxy-dev] Local instance and storage space to provide > > > Hi Sarah- > > I'm sure others have experiences that could help, so I hope they will > chime in. Nate put together some slides two years ago on production > setups, > including some architecture for the server main.g2.bx.psu.edu: > http://wiki.g2.bx.psu.edu/Events/GDC2010?action=AttachFile&do=get&target=GD… > > There is also a wiki page describing production environments: > http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Production%20Server > > Again, I'm sure others have had different experiences with both more > and less > hardware on hand. Of course your mileage may vary based on the tools > being used > (which in your case I'm guessing is Tophat), the number of datasets > that will > need to be analyzed, and so on. > > I hope that helps! > > -Scott > > ----- Original Message ----- > > Dear all, > > > > To purchase equipment related to a local instance of Galaxy, I > > would > > like to have your feedback. > > > > Did you notice load problems when 10 to 20 users launch jobs at the > > same > > time and on a single instance of Galaxy, with mi-RNA or RNA seq > > data > > ? > > > > What do you recommend in terms of storage space, number of nodes, > > number > > of instances? > > > > Thank you in advance, > > Sarah Maman > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ >

1 0