August 2012 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Installing Galaxy and Hooking into a SGE Cluster
by Assaf Gordon 27 Aug '12

27 Aug '12

(Moving to Galaxy-dev, seems more appropriate). John Jones wrote, On 08/23/2012 05:29 PM: > > His original question was about getting Galaxy to recognise LDAP authentication and personal storage space rather than shared storage space as is usual with Galaxy. > Licensing only came into it because Greg wanted LDAP authentication to track individual user usage (for billing) and I questioned the legality of billing for the software used by Galaxy, as I'm sure some components have a non-commercial usage licence clause. > 1. Technically (billing on an SGE cluster): There's a script in "./contrib" called "collect_sge_job_timings.sh" . details are here: http://dev.list.galaxyproject.org/Detailed-SGE-timing-information-about-gal… Assuming you're using PostgreSQL + SGE, it will give you what you want (break-down of SGE jobs timings and Galaxy users). With little adaptation, you can use it to exact SGE JobID and Galaxy Tool-ID / user ID. If you want to extract it directly from the galaxy database, then the following SQL will get you started: === select email as "user", tool_id as "tool", job_runner_external_id as "SGE_ID" from job, galaxy_user where job.user_id = galaxy_user.id and job_runner_name like 'drmaa%' ; === "job_runner_external_ID" will be the SGE-ID, and once you have the list (per each galaxy user), you can use "qacct" to get the information for the job, then calculate the charges. 2. Legally: IANAL, but to the best of my (limited) understanding: 1. Galaxy itself (server code, etc.) - completely legal to run it commercially, even charge money for it - it is licensed as a BSD/MIT style license. 2. Individual tools: if the tools are GPL'd (e.g. bowtie, tophat, bwa, cufflinks) or BSD/MIT - completely legal to run them commercially, and even charge money for them. if the tools use any special license that restrict commercial use (commonly stated as "free for personal, academic, non-commercial use", e.g. Jim Kent's UCSC Genome Browser tools) - then you can't use them in your commercial company without buying a special license (not even internally). It's really not complicated at all, and most (if not all) tools that you compile from source to install them will have a file called "license" or "copying" that will tell you what is the license. If the tools didn't come with a source code, they are probably not free/open-source for commercial use (e.g. - novoalign). 3. Morally: Anyone who thinks of free/open-source as "charity" is missing the point. There is no "charity", and it's perfectly legal, moral, and even recommended to charge money when offering services that are based on free/open-source tools - the requirement (when using GPL) is to give the source code to users when they ask for it (and some other complications, don't want to start a flame war here) - so if they don't want to pay $1,000 for your service - the are more than welcomed to buy their own cluster and storage and servers and run the program themselves - that is the whole point of free/open source. There is not charity. When people write (and publish) software - they explicitly decide upon the license they prefer, and they should know what are the affects of the license. For example, the Galaxy team released Galaxy under a permissive BSD/MIT style license, so they were completely aware of the fact that not only Galaxy can be incorporated into a commercial product, the license doesn't even require the commercial company to release any code changes they make to Galaxy. It's a conscious decision, not an after-thought, and not charity. (end of my rambling...) -gordon

4 6

Tool development without a local Tool Shed?
by Peter Cock 27 Aug '12

27 Aug '12

Dear all, Until recently I'd not needed to define new datatypes while developing tools for Galaxy - instead using (and sometimes modifying) existing datatypes in situ. I have been manually 'installing' my in-development tools by added their XML files to tool_conf.xml - and this has worked fine to date. However, I would now like to experiment with defining or modifying datatypes which would eventually be defined via a datatypes_conf.xml file in a ToolShed entry. My impression from reading the latest wiki documentation on http://wiki.g2.bx.psu.edu/Tool%20Shed is that I will have to install my own local tool shed. Is there a simpler alternative? e.g. Can I add the new datatypes_conf.xml file(s) to a configuration entry instead? Thanks, Peter

2 4

apache external auth and proftpd
by Laure 27 Aug '12

27 Aug '12

Hi galaxy users, I configured galaxy with external ldap authentification using apache proxy, so my galaxy users are automatically created in galaxy database. When I want to configure proftpd to work with galaxy as it is mentionned in galaxy wiki, authentification fails on the ftp because registered password in galaxy_user table mismatchs ldap user password encrypted in SHA1. It seems to be a randomly created password. If I manually translate the ldap password to SHA1, and modify it in galaxy_user table, ftp login works correctly. I wonder how passwords are inserted in galaxy database when users are automatically created ? Is there a way to go through this ? Thanks, Laure

2 1

Declaring dependencies on file format ToolShed entries
by Peter Cock 27 Aug '12

27 Aug '12

Dear all, Is there any way for a Tool Shed bundle (or individual tool) to declare a dependency on another Tool Shed bundle (and ideally a minimum version) which provides a datatype? For example, the 'emboss_5' suite depends on the 'emboss_datatypes' entry. Likewise the ncbi_blast_plus suite depends on 'blast_datatypes' to define the 'blastxml' format (and soon nucleotide and protein BLAST databases). As a specific case, I'd like to update my 'blast2go' wrapper to say it now also depends on blast_datatypes because it uses the 'blastxml' format. Similarly some of my sequence filtering/selection tools could be expanded to handle other formats like 'genbank', which is defined in the 'emboss_datatypes' repository. Regards, Peter

2 1

Changing the location of toolsheddb fails
by Joachim Jacob 27 Aug '12

27 Aug '12

Hi all, I want to change the location of the toolshed database. First I copied all contents to the new location, and next, I adjusted community_wsgi.ini: file_path = /mnt/toolsheddb/community_files However, the toolshed fails to start now, throwing this error: RepoError: repository /home/galaxy/toolshed/database/community_files/000/repo_1 not found It still checks the old location apparently. Thanks for the assistance, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib

2 2

Re: [galaxy-dev] How to speed up job execution
by Hans-Rudolf Hotz 27 Aug '12

27 Aug '12

On 08/24/2012 07:37 PM, kauerbach(a)comcast.net wrote: > Hi Hans, > > I added my email address to the admin_users in universe_wsgi.ini , > logged out of Galaxy, and logged back in but I don't see any 'admin' > tab. Could you tell me what the problem might be? > any change in the "universe_wsgi.ini" file requires a restart of the Galaxy server. Regards, Hans PS: Please keep all replies on the list by using "reply all" in your mail client. > Thank you! > > -Ken. > > ------------------------------------------------------------------------ > > *From: *"Hans-Rudolf Hotz" <hrh(a)fmi.ch> > *To: *kauerbach(a)comcast.net > *Cc: *"Galaxy Development" <galaxy-dev(a)bx.psu.edu> > *Sent: *Friday, August 24, 2012 1:07:20 PM > *Subject: *Re: [galaxy-dev] How to speed up job execution > > > > On 08/24/2012 06:59 PM, kauerbach(a)comcast.net wrote: > > That's very helpful. Thank you. I'm new to Galaxy. Can you tell me how > > to become an 'admin' user and/or change the current admin user? > > > > again, change the settings in "universe_wsgi.ini" by adding your e-mail > (the one you use to log into Galaxy): > > #admin_users = foo(a)bar.com > > > > Regards, Hans > > > > Thank you. > > > > -Ken. > > > > ------------------------------------------------------------------------ > > > > *From: *"Hans-Rudolf Hotz" <hrh(a)fmi.ch> > > *To: *kauerbach(a)comcast.net > > *Cc: *galaxy-dev(a)lists.bx.psu.edu > > *Sent: *Friday, August 24, 2012 12:39:07 PM > > *Subject: *Re: [galaxy-dev] How to speed up job execution > > > > > > > > On 08/24/2012 05:22 PM, kauerbach(a)comcast.net wrote: > > > Hello, > > > > > > On our local instance of Galaxy jobs are taking an extremely long time > > > to run, or they are not run at all. In the history it always says "job > > > is waiting to run", even simple jobs like reformatting text files > to be > > > tab delimited. Is there a way to 1) check the Galaxy job queue, and 2) > > > perhaps modify some parameters to speed up processing? > > > > > > Any suggestions would be greatly appreciated. > > > > > > > Hi Kenneth > > > > Usually, Galaxy is not slow, but the tools executed by Galaxy are slow > > (eg due to bad IO performance of the host server). > > > > "job is waiting to run" indicates there are unfinished jobs in the > > queue. There are two ways to check the queue: > > > > If you are an 'Admin', select the Admin tab -> "Manage jobs". This will > > list all running jobs. And you can kill jobs as well. > > > > Or use the galaxy report tools and have a look at "Today's jobs" > > > > > > > > Depending on your set-up, you might consider changing > "universe_wsgi.ini": > > > > # Number of concurrent jobs to run (local job runner) > > local_job_queue_workers = 5 > > > > to allow more parallel jobs > > > > > > > > Hope this helps > > Hans > > > > > > > > > Thank you. > > > > > > > > > > > > > > > ___________________________________________________________ > > > Please keep all replies on the list by using "reply all" > > > in your mail client. To manage your subscriptions to this > > > and other Galaxy lists, please use the interface at: > > > > > > http://lists.bx.psu.edu/ > > > > > >

1 0

Problem of global env $PATH when using galaxy in daemon mode in ubuntu
by Christophe Antoniewski 26 Aug '12

26 Aug '12

Hi everybody, I am getting troubles when I try running our galaxy instance as a daemon service in ubuntu, using the procedure provided by James Casbon and available in the /contrib galaxy directory (briefly, installing the galaxy.debian-init in the /etc/init.d directory) Everything is OK, *except* that third party softwares that are installed in /bin of the galaxy home directory fail. I suspect a problem of $PATH when the service is launched with the command : start-stop-daemon --chuid $USER --group $GROUP --start --make-pidfile \ --pidfile $PIDFILE --background --chdir $DIR --exec $PYTHON -- $OPTS; in the James bash script. However, R scripts for instance are run correctly. Taking this observation into account, we added /home/galaxy/bin in the /etc/environment unfortunately without success. Any advices for fixing this irritating bug is welcome ! Chris

2 3

Galaxy with LSF scheduler
by kauerbach＠comcast.net 24 Aug '12

24 Aug '12

Hello, Could you please tell me if it's possible to run Galaxy with an LSF scheduler, and if so, how? Thank you.

2 1

import from FTP directories - ownership and copy/move operations
by Shantanu Pavgi 24 Aug '12

24 Aug '12

I am seeing some odd behavior with BAM file import from FTP staging directory where ownership of imported file is retained instead of being changed to galaxy system user account. I should note that our site is not using galaxy specific FTP server configuration, but instead we have configured a directory namespace where cluster users can drop-off files using scp, sftp or even fetch using wget. The galaxy system account can access these files and hence it can import them into it's datasets staging directory using 'FTP' upload mechanism. We have been using this configuration for about a year now with ~ 15,000 files deposited using it. There have been few instances (22) where file ownership didn't change to galaxy system account after a file was imported into the galaxy. Normally when a file is imported from FTP directory to Galaxy's datasets staging directory, it's inode entry and ownership is changed. In these twenty-two cases, which appear to be specific with BAM files, the inode entry and ownership didn't change, but the file name is in galaxy's datase_nnnnn.dat format and file contents/size are legit. So it seems for 'successful' import operations the file is copied to galaxy space (ownership and inode changed) and for 'unsuccessful' import operations file is moved to galaxy space. I was wondering if someone could help me in understanding the upload tool operation, so that I can get better sense of underlying file ownership issues. I have seen it happening for a file where it had exactly same permissions/ACLs, as the file which was successfully imported. So are there any other factors that might be preventing file ownership/inode changes? Does galaxy have a different import mechanisms according to file type? We are using galaxy-dist revision 40f1816d6857 (will be updated soon!). This issue has happened even before this dist version was in place. Any pointers for debugging will be really helpful. -- Shantanu

1 0

email change
by Unknown 24 Aug '12

24 Aug '12

Hi galaxy team, I changed my email address in my personal data settings. But it seems not working. Deeps

1 0