On Feb 7, 2013, at 2:29 PM, greg wrote:
I think I found the problem. The TMPDIR environment variable was set to /tmp/5393732.1.f03.q for jobs galaxy was running. (I guess the admins do this?)
SGE does this to create a job-specific temporary directory that it can clean up after each job terminates.
I updated /usr/local/galaxy/job_environment_setup_file and also /home/galaxy/.bashrc to set TMPDIR to /scratch/galaxy and it seems to work now.
Great, thanks for letting us know. --nate
Thanks for the help.
-Greg
On Thu, Feb 7, 2013 at 1:19 PM, greg <margeemail@gmail.com> wrote:
Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py to print out debug information like host, os.environ, tempfile.gettempdir(), etc?
Would I be able to see its stdout from galaxy or the log, or is there something special I need to do to retrieve the information?
On Thu, Feb 7, 2013 at 1:12 PM, greg <margeemail@gmail.com> wrote:
Update:
When I run as the Galaxy user, Python does have the right temp directory:
tempfile.gettempdir() '/scratch/galaxy'
So does that mean this upload job isn't running as galaxy, or is skipping the job_environment_setup_file? Or could something else be going on?
Any ideas, now I'm really stuck.
Thanks,
Greg
On Wed, Feb 6, 2013 at 3:35 PM, greg <margeemail@gmail.com> wrote:
Ok, when I ran Python in my last two emails I was running as myself, not the galaxy user, and only the galaxy user has write permission to /scratch/galaxy
So that's why Python was ignoring /scratch/galaxy for me. If it doesn't have write access it tries the next temp directory in its list.
I'm going to try debugging as the galaxy user next.
-Greg
On Wed, Feb 6, 2013 at 3:21 PM, greg <margeemail@gmail.com> wrote:
Hi Nate,
I don't see $TMPDIR being set on the cluster, in addition to my previous email I ran:
>> print os.environ.keys() ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', 'SGE_QMASTER_PORT']
But I think we've narrowed it down to something interfering with Python deciding the temp file location. I just can't figure out what.
On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Feb 6, 2013, at 3:00 PM, greg wrote:
> Thanks Nate, > > It turns out I already had this as the first line of my job setup file: > > export TEMP=/scratch/galaxy > > But when I look in that directory, there's plenty of free space, and I > also don't see any recent files there. So I'm wondering if the upload > jobs aren't seeing that for some reason. > > Any ideas on how I could diagnose this more?
Hi Greg,
The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96:
fd, temp_name = tempfile.mkstemp()
If you print temp_name, that will tell you what file the upload tool is writing to. You may also want to take a look at:
http://docs.python.org/2/library/tempfile.html#tempfile.tempdir
Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be used.
--nate
> > -Greg > > > Relevant info? > > >> grep env galaxy-dist/universe_wsgi.ini > environment_setup_file = /usr/local/galaxy/job_environment_setup_file > >> cat /usr/local/galaxy/job_environment_setup_file > export TEMP=/scratch/galaxy > #active Python virtual env just for galaxy > source /usr/local/galaxy/galaxy_python/bin/activate > ... path setup lines ... > > > > > > > > > > > On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor <nate@bx.psu.edu> wrote: >> On Feb 6, 2013, at 2:32 PM, greg wrote: >> >>> Hi guys, >>> >>> When I try to upload a directory of files from a server directory I'm >>> seeing the error below. >>> >>> It appears to be trying to write to a temp directory somewhere that >>> I'm guessing doesn't have enough space? Is there a way I can direct >>> where it writes to for temporary files like this? >> >> Hi Greg, >> >> There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this is not the case for the upload tool. >> >>> Am I understanding right, that these upload jobs are running on our >>> cluster? I think it would be a problem if its trying to use the >>> default temp directory on each cluster node since they aren't >>> provisioned with much space. >> >> This is correct. On the cluster, add something to your user's shell startup files (or see the environment_setup_file option in universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable to a suitable temp space. >> >> --nate >> >>> >>> Please advise. >>> >>> Thanks, >>> >>> Greg >>> >>> >>> >>> Miscellaneous information:Traceback (most recent call last): File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 384, in __main__() File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 373, in __main__ add_file( dataset, >>> Job Standard Error >>> >>> >>> Traceback (most recent call last): >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 384, in >>> __main__() >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 373, in __main__ >>> add_file( dataset, registry, json_file, output_path ) >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 270, in add_file >>> line_count, converted_path = sniff.convert_newlines( dataset.path, >>> in_place=in_place ) >>> File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", >>> line 99, in convert_newlines >>> fp.write( "%s\n" % line.rstrip( "\r\n" ) ) >>> IOError: [Errno 28] No space left on device >>> ___________________________________________________________ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >>