I think I found the problem. The TMPDIR environment variable was set to /tmp/5393732.1.f03.q for jobs galaxy was running. (I guess the admins do this?)
I updated /usr/local/galaxy/job_environment_setup_file and also /home/galaxy/.bashrc to set TMPDIR to /scratch/galaxy and it seems to work now.
Thanks for the help.
-Greg
On Thu, Feb 7, 2013 at 1:19 PM, greg margeemail@gmail.com wrote:
Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py to print out debug information like host, os.environ, tempfile.gettempdir(), etc?
Would I be able to see its stdout from galaxy or the log, or is there something special I need to do to retrieve the information?
On Thu, Feb 7, 2013 at 1:12 PM, greg margeemail@gmail.com wrote:
Update:
When I run as the Galaxy user, Python does have the right temp directory:
tempfile.gettempdir()
'/scratch/galaxy'
So does that mean this upload job isn't running as galaxy, or is skipping the job_environment_setup_file? Or could something else be going on?
Any ideas, now I'm really stuck.
Thanks,
Greg
On Wed, Feb 6, 2013 at 3:35 PM, greg margeemail@gmail.com wrote:
Ok, when I ran Python in my last two emails I was running as myself, not the galaxy user, and only the galaxy user has write permission to /scratch/galaxy
So that's why Python was ignoring /scratch/galaxy for me. If it doesn't have write access it tries the next temp directory in its list.
I'm going to try debugging as the galaxy user next.
-Greg
On Wed, Feb 6, 2013 at 3:21 PM, greg margeemail@gmail.com wrote:
Hi Nate,
I don't see $TMPDIR being set on the cluster, in addition to my previous email I ran:
> print os.environ.keys()
['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', 'SGE_QMASTER_PORT']
But I think we've narrowed it down to something interfering with Python deciding the temp file location. I just can't figure out what.
On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor nate@bx.psu.edu wrote:
On Feb 6, 2013, at 3:00 PM, greg wrote:
Thanks Nate,
It turns out I already had this as the first line of my job setup file:
export TEMP=/scratch/galaxy
But when I look in that directory, there's plenty of free space, and I also don't see any recent files there. So I'm wondering if the upload jobs aren't seeing that for some reason.
Any ideas on how I could diagnose this more?
Hi Greg,
The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96:
fd, temp_name = tempfile.mkstemp()If you print temp_name, that will tell you what file the upload tool is writing to. You may also want to take a look at:
http://docs.python.org/2/library/tempfile.html#tempfile.tempdirSome cluster environments set $TMPDIR, and if that is set, $TEMP will not be used.
--nate
-Greg
Relevant info?
> grep env galaxy-dist/universe_wsgi.ini environment_setup_file = /usr/local/galaxy/job_environment_setup_file
> cat /usr/local/galaxy/job_environment_setup_file export TEMP=/scratch/galaxy #active Python virtual env just for galaxy source /usr/local/galaxy/galaxy_python/bin/activate ... path setup lines ...
On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor nate@bx.psu.edu wrote: > On Feb 6, 2013, at 2:32 PM, greg wrote: > >> Hi guys, >> >> When I try to upload a directory of files from a server directory I'm >> seeing the error below. >> >> It appears to be trying to write to a temp directory somewhere that >> I'm guessing doesn't have enough space? Is there a way I can direct >> where it writes to for temporary files like this? > > Hi Greg, > > There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this is not the case for the upload tool. > >> Am I understanding right, that these upload jobs are running on our >> cluster? I think it would be a problem if its trying to use the >> default temp directory on each cluster node since they aren't >> provisioned with much space. > > This is correct. On the cluster, add something to your user's shell startup files (or see the environment_setup_file option in universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable to a suitable temp space. > > --nate > >> >> Please advise. >> >> Thanks, >> >> Greg >> >> >> >> Miscellaneous information:Traceback (most recent call last): File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 384, in __main__() File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 373, in __main__ add_file( dataset, >> Job Standard Error >> >> >> Traceback (most recent call last): >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 384, in >> __main__() >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 373, in __main__ >> add_file( dataset, registry, json_file, output_path ) >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 270, in add_file >> line_count, converted_path = sniff.convert_newlines( dataset.path, >> in_place=in_place ) >> File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", >> line 99, in convert_newlines >> fp.write( "%s\n" % line.rstrip( "\r\n" ) ) >> IOError: [Errno 28] No space left on device >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ >