Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py to print out debug information like host, os.environ, tempfile.gettempdir(), etc? Would I be able to see its stdout from galaxy or the log, or is there something special I need to do to retrieve the information? On Thu, Feb 7, 2013 at 1:12 PM, greg <margeemail@gmail.com> wrote:
Update:
When I run as the Galaxy user, Python does have the right temp directory:
tempfile.gettempdir() '/scratch/galaxy'
So does that mean this upload job isn't running as galaxy, or is skipping the job_environment_setup_file? Or could something else be going on?
Any ideas, now I'm really stuck.
Thanks,
Greg
On Wed, Feb 6, 2013 at 3:35 PM, greg <margeemail@gmail.com> wrote:
Ok, when I ran Python in my last two emails I was running as myself, not the galaxy user, and only the galaxy user has write permission to /scratch/galaxy
So that's why Python was ignoring /scratch/galaxy for me. If it doesn't have write access it tries the next temp directory in its list.
I'm going to try debugging as the galaxy user next.
-Greg
On Wed, Feb 6, 2013 at 3:21 PM, greg <margeemail@gmail.com> wrote:
Hi Nate,
I don't see $TMPDIR being set on the cluster, in addition to my previous email I ran:
print os.environ.keys() ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', 'SGE_QMASTER_PORT']
But I think we've narrowed it down to something interfering with Python deciding the temp file location. I just can't figure out what.
On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Feb 6, 2013, at 3:00 PM, greg wrote:
Thanks Nate,
It turns out I already had this as the first line of my job setup file:
export TEMP=/scratch/galaxy
But when I look in that directory, there's plenty of free space, and I also don't see any recent files there. So I'm wondering if the upload jobs aren't seeing that for some reason.
Any ideas on how I could diagnose this more?
Hi Greg,
The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96:
fd, temp_name = tempfile.mkstemp()
If you print temp_name, that will tell you what file the upload tool is writing to. You may also want to take a look at:
http://docs.python.org/2/library/tempfile.html#tempfile.tempdir
Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be used.
--nate
-Greg
Relevant info?
grep env galaxy-dist/universe_wsgi.ini environment_setup_file = /usr/local/galaxy/job_environment_setup_file
cat /usr/local/galaxy/job_environment_setup_file export TEMP=/scratch/galaxy #active Python virtual env just for galaxy source /usr/local/galaxy/galaxy_python/bin/activate ... path setup lines ...
On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Feb 6, 2013, at 2:32 PM, greg wrote:
> Hi guys, > > When I try to upload a directory of files from a server directory I'm > seeing the error below. > > It appears to be trying to write to a temp directory somewhere that > I'm guessing doesn't have enough space? Is there a way I can direct > where it writes to for temporary files like this?
Hi Greg,
There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this is not the case for the upload tool.
> Am I understanding right, that these upload jobs are running on our > cluster? I think it would be a problem if its trying to use the > default temp directory on each cluster node since they aren't > provisioned with much space.
This is correct. On the cluster, add something to your user's shell startup files (or see the environment_setup_file option in universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable to a suitable temp space.
--nate
> > Please advise. > > Thanks, > > Greg > > > > Miscellaneous information:Traceback (most recent call last): File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 384, in __main__() File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 373, in __main__ add_file( dataset, > Job Standard Error > > > Traceback (most recent call last): > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 384, in > __main__() > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 373, in __main__ > add_file( dataset, registry, json_file, output_path ) > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 270, in add_file > line_count, converted_path = sniff.convert_newlines( dataset.path, > in_place=in_place ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", > line 99, in convert_newlines > fp.write( "%s\n" % line.rstrip( "\r\n" ) ) > IOError: [Errno 28] No space left on device > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/