- galaxy-dev - lists.galaxyproject.org

job runner issue
by shashi shekhar 19 Aug '11

19 Aug '11

Hi In my galaxy instance, whatever jobs i am submitting it goes into queued state. If I restart the server then the previous submitted jobs state changes to running. but the newly submitted jobs again goes to queued state. I am at a loss to understand this behaviour of galaxy and unable to debug it. The job submission uses a customized runner. How is it actually goes into the queued state automatically when all the workers thread are free? Does galaxy_session table is_valid attribute makes jobs state true? Or what all places in tables the queued states are getting stored. I can only see that the jobs table state attribute only stores the state. The server logs points error here: galaxy.jobs ERROR 2011-07-29 11:01:28,098 failure running job 2243 Traceback (most recent call last): File "/home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py", line 202, in __monitor_step self.dispatcher.put( JobWrapper( job, self ) ) File "/home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py", line 856, in put self.job_runners[runner_name].put( job_wrapper ) File "/home/gwadmin/galaxy-central/lib/galaxy/jobs/runners/gw.py", line 375, in put job_wrapper.change_state( model.Job.states.QUEUED ) File "/home/gwadmin/galaxy-central/lib/galaxy/jobs/__init__.py", line 437, in change_state self.sa_session.flush() Regards Karuna

2 1

Could anyone tell me why the result file in directary /galaxy-dist /database/files/000 do not return to the web page
by liyanhui607 19 Aug '11

19 Aug '11

Dear Sir, We have a program writen in perl. It can run nomally in linux environment with a file saved as result. After added it to galaxy system, we found that it can produce the right result file in directary "/var/we/galaxy-dist/database/files/000" , but the result file did not return to the web page where the state is always "Job is currently running". Could anyone tell me what is wrong ? why it does not work? The bagging-SVM.xml and the bagging-SVM.pl are attached. Thank you very much! Best Wishes! Yan-Hui Li

2 1

Galaxy: self-cleaning on file_path
by Sanka, Ravi 19 Aug '11

19 Aug '11

Hello, We have recently put a local installation of Galaxy on our network and connected it to a MYSQL database. In order to have more space available for uploaded and created files, we also changed the values of file_path and new_file_path in the config file. The file_path field was changed to a larger, communal location. The new_file_path was changed to a directory called "tmp" in that same communal location. This location, however, undergoes self-cleaning; deleting any files that reach 7 days old (and any directories that have no files less than 7 days old). This would include any datasets uploaded or created by Galaxy. Can the Galaxy framework and schema handle this occurrence? Or will it cause errors to occur? ----------------------------------------------- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -----------------------------------------------

2 3

database problem
by shashi shekhar 19 Aug '11

19 Aug '11

Hi ALL, I want to delete some data from history_dataset_association. Will it effect any other tables.and I want to delete records for last 3 days by sqlite query manually.How may tables will be effected by this and provide me tables name .How can i delete the records. Regards shashi shekhar

2 1

Cluster setup - shared temporary directory
by Peter Cock 19 Aug '11

19 Aug '11

Hi all, I'm reading http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster Could someone expand a little on this section please: > Create a shared temporary directory > > Some tools make use of temporary files created on the server, > but accessed on the nodes. For this, you'll need to make a > directory (galaxy_dist/database/tmp by default) ... I presume this is talking about the universe_wsgi.ini setting new_file_path = database/tmp (if so, could that be explicit)? I would like to know more about this from the tool author point of view. Could you at least give one example of a tool that uses this temporary folder? As a tool author I am unclear what the purpose is (and it would be a shock if I accidentally use this mapped folder instead of the local temp drive of a node). Thanks, Peter

4 6

Extract Genomic DNA insisting on build for GFF3 file
by Peter Cock 19 Aug '11

19 Aug '11

Hi all, Is this a bug, or have I misunderstood something? 1. Goto http://usegalaxy.org (or a local Galaxy running galaxy-dist) 2. Import this genomic FASTA file, ftp://ftp.ncbi.nih.gov/genomes/Bacteria/Nanoarchaeum_equitans_Kin4_M_uid580… 3. Import this GFF3 file, ftp://ftp.ncbi.nih.gov/genomes/Bacteria/Nanoarchaeum_equitans_Kin4_M_uid580… 4. Goto "Fetch Sequences", "Extract Genomic DNA" 5. Choose fetch sequence for intervals in NC_005213.gff, interpret yes, source history, NC_005213.ffn, output FASTA 6. Click execute. Expected result: Tool runs Actual result: Red error against the gff file, Unspecified genome build, click the pencil icon in the history item to set the genome build The fact I'm using a FASTA file from my history should mean the genome build is irrelevant as that only applies to "locally cached genomes" (right?). Thanks, Peter

2 18

Using data_fabfile.py to update data from s3
by Clare Sloggett 18 Aug '11

18 Aug '11

Hi, I am trying to use install_data_s3 (data_fabfile.py) to get data from the cloudbiolinux bucket into a virtualbox VM, as per the instructions at usegalaxy.org/vm . When I ran fab -f data_fabfile.py -H localhost install_data_s3:config/galaxy_default_biodata.yaml the first time, it worked. But when I try running it a second time, it dies with [localhost] run: echo 'index phix /mnt/galaxyIndices/genomes/phiX174/phix/seq/phix.fa >> sam_fa_indices.loc [localhost] out: /bin/bash: sam_fa_indices.loc: Permission denied This means that I can't update /opt/galaxy/pkg/cloudbiolinux/config/galaxy_default_biodata.yaml and run install_data_s3 again to get more genomes. The first time I ran it I only included phiX (just to test it, and because that's the default). fab fails the second time whether I edit the yaml file or not. I found a file called sam_fa_indices.loc under /mnt/galaxyTools/galaxy-central/tool-data and sure enough it has write permissions off unless you are user galaxy (not galaxyadmin, which you are at that point in the instructions). What am I doing wrong? Is this list the right venue for these questions? I also noticed a couple of minor things in earlier stages of the instructions: sudo mount /dev/sdb /mnt/galaxyData and sudo mount /dev/sdb /mnt/galaxyIndices are telling the user to mount the same device on both; sdb is only illustrative but should probably be sdc or something. Also those `sudo mount` commands mount galaxyData and galaxyIndices as root-only, so running fab will fail with permissions errors even the first time. I changed the permissions on /mnt/galaxyIndices to get round this, just to try it out, but in general this might have been a bad thing to do! Finally: something I don't understand happens if I try to redirect the stdout and/or stderr from fab into a file. So it's hard to keep records of what happened each time I ran it. If anyone understands what's going on there that would be helpful too! Sorry for what are probably basic questions, I'm new to python fabric. Thanks, Clare -- E: sloc(a)unimelb.edu.au P: 03 903 53357 M: 0414 854 759

2 1

Re: [galaxy-dev] galaxy-dev Digest, Vol 61, Issue 10 - Handling runner-specific (SLURM) DRMAA settings (Roman Valls)
by Mariusz Mamoński 18 Aug '11

18 Aug '11

> The general config file allows us to set a fixed project: > > default_cluster_job_runner = drmaa://-A a2010002 -p core > > And even set per-tool job settings. But we would like each user to have > the ability to change those settings. > > > What is the least intrusive way to set per-user native (site-specific) > job manager settings ? you may try to use an user's local DRMAA configuration file: ~/.slurm_drmaa.conf ". If multiple configuration sources are present then all configurations are merged with values from user-defined files taking precedence (in following order: $SLURM_DRMAA_CONF, ~/.slurm_drmaa.conf, /etc/slurm_drmaa.conf)" there you can put any user's specific settings, e.g.: job_categories: { default: "-A a2010002 -p core", } Cheers, -- Mariusz

3 4

Re: [galaxy-dev] [galaxy-user] tailoring backend SGE submission requests to specific applications
by Nate Coraor 18 Aug '11

18 Aug '11

Joseph Hargitai wrote: > > Hi, > > is there a best practices, to replace general SGE requests with requests specific to applications - Hi Joe, I've moved this question to the dev list since it concerns a local installation. You can tailor SGE options per-tool as shown at the bottom of the page here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster > did anyone actually build such request options to be part of the selectable items in the apps menus? These options are not user-selectable, although I think some people have added this in the past, so perhaps someone will reply with a patch. --nate > > > best, > joe > > ___________________________________________________________ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/

1 0

Re: [galaxy-dev] [galaxy-bugs] Some issues when running jobs through drmaa job runner.
by Luobin Yang 18 Aug '11

18 Aug '11

Hi, Nate, On Tue, Aug 16, 2011 at 12:38 PM, Nate Coraor <nate(a)bx.psu.edu> wrote: > Hi Luobin, > > Sorry for the delay in response. I would suggest moving this discussion > to the galaxy-dev mailing list since it does not contain any private > data. The wide audience on galaxy-dev may be able to come up with > additional ideas than what we on the Galaxy Team come up with. > > Please see my responses inline below. > > Luobin Yang wrote: > > Hi, > > > > Thanks for Martin Dahlo's excellent blog ( > > http://mdahlo.blogspot.com/2011/06/galaxy-on-uppmax.html) on making > SLURM > > work with GALAXY, I am able to run galaxy jobs on a cluster, but I've got > a > > couple of issues: > > > > 1. Sometimes a job is in waiting state and it won't start to run until I > > restart galaxy. > > When this happens, is the job stuck in Galaxy or in the queueing system > (PBS, SGE, something else?)? > When this happens, the job is stuck in Galaxy, it doesn't submit the job to SLURM queueing system. > In your config file (universe_wsgi.ini), is track_jobs_in_database = > True or False? Are you using SQLite or another database? If you watch > I am using PostgreSQL database system. track_jobs_in_database is False in my config file. Which option is better when I have a queueing system? > the Galaxy log, are any exceptions or other errors logged when a job > becomes stuck? Galaxy log doesn't show any exception or errors. > > 2. Sometimes a job is in running state even though it is already finished > > and restarting galaxy can make the job's state change from running to > > finished. > > Most likely, Galaxy is setting metadata on the job's outputs. You can > probably speed up this process by setting: > > set_metadata_externally = True > > in the config file. > > Thanks, > --nate > > > > > I am wondering what's causing those issues and how they can be fixed. > > > > Thanks, > > Luobin > > > > >

3 3