https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote:

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know. Ilya -----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code Sorry, the link was wrong. https://bitbucket.org/ichorny/galaxy-central-ilya2/ -----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new everything but the tophat/cufflinks wrapper. Thanks, Ilya -----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote:

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya, I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code? Thanks, --nate

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours. Chorny, Ilya wrote:

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

Chorny, Ilya wrote:

Sweet!! Can't wait to see it.

Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours.

Chorny, Ilya wrote:

...

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy. BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write? Thanks, Ilya -----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine. -----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code Chorny, Ilya wrote:

Sweet!! Can't wait to see it.

Have a look: https://bitbucket.org/natefoo/galaxy-central-ichorny/ Caveats: * job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file. --nate

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours.

Chorny, Ilya wrote:

...

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/

Nate Coraor (nate@bx.psu.edu) wrote:

Chorny, Ilya wrote:

...

So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy.

Okay, this is pretty odd, I'll try to see what's going on here. If you figure anything out, please let me know.

Whoops, variable name collision in finish(). I've just committed a fix.

...

BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write?

I imagine for some environments, it's preferable to avoid world-writable directories. Also, going forward, containing all of a job's filesystem data to a single directory will be important for running on multiple clusters without access to a single shared filesystem mounted at the same path.

There's an issue in bitbucket somewhere regarding saving standard out and standard error for all jobs, which would solve the problem of it being in the working directory.

For debugging, if you comment out shutil.rmtree( self.working_directory ) in jobs.cleanup(), it'll leave everything behind.

--nate

...

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine.

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Sweet!! Can't wait to see it.

Have a look:

https://bitbucket.org/natefoo/galaxy-central-ichorny/

Caveats:

* job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True

One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file.

--nate

...

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours.

Chorny, Ilya wrote:

...

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

I am pulling and testing as we speak. I did a preliminary test on my VM but it does not have enough memory to run BWA. We pull into my dev version on the cluster and do further testing. Also need to test how it integrates with the splitting code. Should have some results by EOD. Ilya -----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Wednesday, November 02, 2011 9:15 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code Hi Ilya, I made a few more commits this morning. Could you test it out in the present state? If everything works properly in your environment, I think it's pretty close to being ready to integrate. The major outstanding issue is to try to avoid chmods in the upload action, which I have some ideas about. --nate Nate Coraor (nate@bx.psu.edu) wrote:

Nate Coraor (nate@bx.psu.edu) wrote:

...

Chorny, Ilya wrote:

...

So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy.

Okay, this is pretty odd, I'll try to see what's going on here. If you figure anything out, please let me know.

Whoops, variable name collision in finish(). I've just committed a fix.

...

...

BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write?

I imagine for some environments, it's preferable to avoid world-writable directories. Also, going forward, containing all of a job's filesystem data to a single directory will be important for running on multiple clusters without access to a single shared filesystem mounted at the same path.

There's an issue in bitbucket somewhere regarding saving standard out and standard error for all jobs, which would solve the problem of it being in the working directory.

For debugging, if you comment out shutil.rmtree( self.working_directory ) in jobs.cleanup(), it'll leave everything behind.

--nate

...

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine.

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Sweet!! Can't wait to see it.

Have a look:

https://bitbucket.org/natefoo/galaxy-central-ichorny/

Caveats:

* job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True

One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file.

--nate

...

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours.

Chorny, Ilya wrote:

...

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-reques t/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

Everything seems to be working correctly, including splitting. -----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Wednesday, November 02, 2011 9:25 AM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code I am pulling and testing as we speak. I did a preliminary test on my VM but it does not have enough memory to run BWA. We pull into my dev version on the cluster and do further testing. Also need to test how it integrates with the splitting code. Should have some results by EOD. Ilya -----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Wednesday, November 02, 2011 9:15 AM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code Hi Ilya, I made a few more commits this morning. Could you test it out in the present state? If everything works properly in your environment, I think it's pretty close to being ready to integrate. The major outstanding issue is to try to avoid chmods in the upload action, which I have some ideas about. --nate Nate Coraor (nate@bx.psu.edu) wrote:

Nate Coraor (nate@bx.psu.edu) wrote:

...

Chorny, Ilya wrote:

...

So it looks like for some reason galaxy is not picking up the information in the .e file. Not sure why. I even switched it back to writing the files to the pbs directory and it still did not pick up the .e files. BWA was core dumping but I was not given the error message in galaxy.

Okay, this is pretty odd, I'll try to see what's going on here. If you figure anything out, please let me know.

Whoops, variable name collision in finish(). I've just committed a fix.

...

...

BTW, I find it nice to be able to see the .o and .e files. The way you have it coded the .o and .e are written out to the job_working_directory and then disappear at the end of the run. But the way I had it coded the pbs directory is world read writable. Is there an issue with having the pbs directory be world read write?

I imagine for some environments, it's preferable to avoid world-writable directories. Also, going forward, containing all of a job's filesystem data to a single directory will be important for running on multiple clusters without access to a single shared filesystem mounted at the same path.

There's an issue in bitbucket somewhere regarding saving standard out and standard error for all jobs, which would solve the problem of it being in the working directory.

For debugging, if you comment out shutil.rmtree( self.working_directory ) in jobs.cleanup(), it'll leave everything behind.

--nate

...

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Monday, October 31, 2011 8:37 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

It seems to be working but when I run BWA or Bowtie it finishes green but the files are empty. Doesn't give me much information. I also tried fastqc and it ran just fine.

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Monday, October 31, 2011 1:21 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Sweet!! Can't wait to see it.

Have a look:

https://bitbucket.org/natefoo/galaxy-central-ichorny/

Caveats:

* job_working_directory has to be in a shared filesystem * outputs_to_working_directory must be True * set_metadata_externally must be True

One thing that would be good from a security standpoint would be if the external chown script took a Galaxy Job ID for its argument instead of a path, and then looked up the working directory path from the ID. That way the script couldn't be used maliciously to chown any file.

--nate

...

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Thursday, October 27, 2011 1:24 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Yup, cleaning up some things that were accidentally lost in merges. I added another bitbucket repo so as not to mess up yours.

Chorny, Ilya wrote:

...

Hi Nate,

Any updates?

Thanks,

Ilya

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:24 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

One more thing there are some changes in lib/galaxy/tools/__init__.py and lib/galaxy/model/__init__.py that are not related to actual user. They look like they are toolshed related and may have been changed because to version of the file that I copied over was older then galaxy-central. Probably should have done a merge first. I can redo it if you like. Let me know.

Ilya

-----Original Message----- From: Chorny, Ilya Sent: Friday, October 21, 2011 2:11 PM To: Chorny, Ilya; Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-----Original Message----- From: galaxy-dev-bounces@lists.bx.psu.edu [mailto:galaxy-dev-bounces@lists.bx.psu.edu] On Behalf Of Chorny, Ilya Sent: Friday, October 21, 2011 2:07 PM To: Nate Coraor (nate@bx.psu.edu) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-reques t/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya

-----Original Message----- From: Nate Coraor (nate@bx.psu.edu) [mailto:nate@bx.psu.edu] Sent: Friday, October 21, 2011 12:56 PM To: Chorny, Ilya Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: Re Acutal user code

Chorny, Ilya wrote:

...

Nate,

I have been trying to figure out how to make the changes you suggested (i.e. write output files to the job_working_directory) and then move them over to files but I am having a difficult time navigating the code. If you could point me in the right direction, that would be helpful. I know the lib/galaxy/models/__init__.py initializes the output files but I don't think that is the right place to make the change. This change probably needs to be done in the Job_Wrapper but most of the code is very cryptic to me. Any guidance would be greatly appreciated. These changes should not take very long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the other code in your fork. Could you create a fork with just this code?

Thanks, --nate

...

Thanks,

Ilya

Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: ichorny@illumina.com<mailto:ichorny@illumina.com> Website: www.illumina.com<http://www.illumina.com>

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:

http://lists.bx.psu.edu/

___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/