Chris Cole wrote:

Just checked it again (sorry I shouldn't have sent the previous email) with a directory and it still isn't working.

www-galaxy@ge-002: tmp> cd ~/data_import/ www-galaxy@ge-002: data_import> ls README www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH www-galaxy@ge-002: data_import> ls TOH/ 0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt Dd_inuc_1.txt Dmono_1.txt processed 0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt Dd_inuc_2.txt Dmono_2.txt www-galaxy@ge-002: data_import> ls -l total 96 -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH -> /homes/pschofield/data/TOH

I got the TOH option in the 'Upload directory of files', which I selected, but the upload failed. Again, this is the error I get when selecting one of the filenames in the dataset: Traceback (most recent call last): File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 326, in __main__() File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 318, in __main__ add_file( dataset, error

Hi Chris,

Chances are, the files in /homes/pschofield/data/TOH are not writable by Galaxy. Galaxy was trying to convert newlines to UNIX on those files as part of the import process. This behavior was not intentional.

I've just committed 3517:0c9e154e9176, which will prevent Galaxy from ever attempting to modify the import files. When copying in to Galaxy, they will still converted, but no longer "in-place" (a temp file is used). When not copying (i.e. symlinking), no newline conversion will be performed since doing so would require modifying the import file.

--nate

Thanks for help.

Chris

On 3/12/10 11:33 AM, "Chris Cole" chris@compbio.dundee.ac.uk wrote:

On 11/03/10 17:01, Nate Coraor wrote:

...

Chris Cole wrote:

...

Just checked it again (sorry I shouldn't have sent the previous email) with a directory and it still isn't working.

www-galaxy@ge-002: tmp> cd ~/data_import/ www-galaxy@ge-002: data_import> ls README www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH www-galaxy@ge-002: data_import> ls TOH/ 0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt Dd_inuc_1.txt Dmono_1.txt processed 0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt Dd_inuc_2.txt Dmono_2.txt www-galaxy@ge-002: data_import> ls -l total 96 -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH -> /homes/pschofield/data/TOH

I got the TOH option in the 'Upload directory of files', which I selected, but the upload failed. Again, this is the error I get when selecting one of the filenames in the dataset: Traceback (most recent call last): File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 326, in __main__() File "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 318, in __main__ add_file( dataset, error

Hi Chris,

Chances are, the files in /homes/pschofield/data/TOH are not writable by Galaxy. Galaxy was trying to convert newlines to UNIX on those files as part of the import process. This behavior was not intentional.

Yup. That's correct. Galaxy has read-only access to the original files. This is necessary as users have their own locations for their raw NGS data which they want to upload to Galaxy.

Hi Nate, Hi Chris

I was hoping this would also explain our weird problem (first mentioned in my e-mail to the list Jan 21st)...but even if the files are writable by galaxy it doesn't work, eg:

if I try to import the following files: (sorry, I have to blank out the full path)

galaxy@erbium:/****/test$ ls -l total 8 -rw-rw-rw- 1 haruhotz gbioinfo 841 2010-03-14 16:18 P51003.fasta -rw-r--r-- 1 galaxy galaxy 828 2010-03-14 16:18 P51004.fasta galaxy@erbium:/****/test$

it doesn't work, with the following error message (for both files):

" Traceback (most recent call last): File "/***/galaxy_dist/tools/data_source/upload.py", line 311, in __main__() File "/***/galaxy_dist/tools/data_source/upload.py", line 302, in __main__ add_fil

error "

Only if I delete the file P51003.fasta, eg as user galaxy:

galaxy@erbium:/****/test$ rm P51003.fasta galaxy@erbium:/****/test$ ls -l total 4 -rw-r--r-- 1 galaxy galaxy 828 2010-03-14 16:18 P51004.fasta galaxy@erbium:/****/test$

the remaining file get's imported (and I guess 'rewritten' by galaxy):

galaxy@erbium:/****/test$ ls -l total 4 -rw------- 1 galaxy galaxy 828 2010-03-14 16:26 P51004.fasta galaxy@erbium:/****/test

Nevertheless, I put my hope into the the next update (including "3517:0c9e154e9176") which I will do in the next few weeks.

Regard, Hans

...

I've just committed 3517:0c9e154e9176, which will prevent Galaxy from ever attempting to modify the import files. When copying in to Galaxy, they will still converted, but no longer "in-place" (a temp file is used). When not copying (i.e. symlinking), no newline conversion will be performed since doing so would require modifying the import file.

Great. I'll look out for the latest updates and try it again, then. Again, thanks very much for the fixes. Cheers,

Chris

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Chris Cole wrote:

Sorry for the delay, only just got back to looking at this.

Following the latest updates (tip is now 3560:4c95f1a101f1) I can now successfully update files not belonging to Galaxy. However, softlinks in the import directory aren't being dereferenced correctly.

e.g. This is the contents of the import directory: caterpillar: /homes/www-galaxy/data_import> ls -l total 96 -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README lrwxrwxrwx 1 ccole barton 65 Mar 25 15:47 solexa_files -> /homes/ccole/projects/Dicty_RNAseq/raw_data/NOBACK/SOLEXA results

I deleted the softlink following upload, but now the files are no longer accessible. FASTQ Groomer for example reports this error: Traceback (most recent call last): File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py", line 37, in ? if __name__ == "__main__": main() File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py", line 18, in main for read_count, fastq_read in enumerate( fastqReader( open( input_filename ), format = input_type ) ): IOError: [Errno 2] No such file or directory: '/homes/www-galaxy/data_import/solexa_files/Ax2.txt'

So, it looks like Galaxy didn't dereference the 'solexa_files' softlink (to me at least).

Hi Chris,

This wasn't part of the original design. We were handling when the contents of import directories were symlinks, but not when the import directories themselves were symlinks. This has been added in 3629:6b93e705c8a4.

--nate

Cheers,

Chris _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Chris Cole wrote:

Oh, right. I got the impression that all the symlinks were dereferenced once.

They were, but only the ones *in* the subdirectories of the import directory. It wasn't dereferencing if the actual subdirectories were symlinks.

--nate

Thanks for the update. I'll check it out when it gets to the main tree. Thanks again for your help. Cheers,

Chris