4:12 p.m.
On 09/03/10 15:42, Chris Cole wrote:
On 09/03/10 14:53, Nate Coraor wrote:
> Chris Cole wrote:
>> Anyone got any ideas on this?
>>
>> It's still not working (following the most recent updates) and we've
>> got some NGS data to import into Galaxy.
>
> Hi Chris,
>
> To use this feature, the contents of library_import_dir should
> themselves be directories (or symlinks to the same). Selecting that
> directory from the dropdown in the library upload form will then import
> the contents of those directories.
Ah, right. That's not clear in the docs.
How come it works with locally symlinked files, then?
Just checked it again (sorry I shouldn't have sent the previous email)
with a directory and it still isn't working.
www-galaxy@ge-002: tmp> cd ~/data_import/
www-galaxy@ge-002: data_import> ls
README
www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH
www-galaxy@ge-002: data_import> ls TOH/
0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt
Dd_inuc_1.txt Dmono_1.txt processed
0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt
Dd_inuc_2.txt Dmono_2.txt
www-galaxy@ge-002: data_import> ls -l
total 96
-rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH ->
/homes/pschofield/data/TOH
I got the TOH option in the 'Upload directory of files', which I
selected, but the upload failed. Again, this is the error I get when
selecting one of the filenames in the dataset:
Traceback (most recent call last): File
"/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 326,
in __main__() File
"/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 318,
in __main__ add_file( dataset,
error
Thanks for help.
Chris
5:01 p.m.
New subject: Weird problem with importing datasets
Chris Cole wrote:
Just checked it again (sorry I shouldn't have sent the previous
email)
with a directory and it still isn't working.
www-galaxy@ge-002: tmp> cd ~/data_import/
www-galaxy@ge-002: data_import> ls
README
www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH
www-galaxy@ge-002: data_import> ls TOH/
0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt
Dd_inuc_1.txt Dmono_1.txt processed
0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt
Dd_inuc_2.txt Dmono_2.txt
www-galaxy@ge-002: data_import> ls -l
total 96
-rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH ->
/homes/pschofield/data/TOH
I got the TOH option in the 'Upload directory of files', which I
selected, but the upload failed. Again, this is the error I get when
selecting one of the filenames in the dataset:
Traceback (most recent call last): File
"/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 326,
in __main__() File
"/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line 318,
in __main__ add_file( dataset,
error
Hi Chris,
Chances are, the files in /homes/pschofield/data/TOH are not writable by
Galaxy. Galaxy was trying to convert newlines to UNIX on those files as
part of the import process. This behavior was not intentional.
I've just committed 3517:0c9e154e9176, which will prevent Galaxy from
ever attempting to modify the import files. When copying in to Galaxy,
they will still converted, but no longer "in-place" (a temp file is
used). When not copying (i.e. symlinking), no newline conversion will
be performed since doing so would require modifying the import file.
--nate
Thanks for help.
Chris
10:33 a.m.
New subject: Weird problem with importing datasets
On 11/03/10 17:01, Nate Coraor wrote:
Chris Cole wrote:
> Just checked it again (sorry I shouldn't have sent the previous email)
> with a directory and it still isn't working.
>
> www-galaxy@ge-002: tmp> cd ~/data_import/
> www-galaxy@ge-002: data_import> ls
> README
> www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH
> www-galaxy@ge-002: data_import> ls TOH/
> 0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt
> Dd_inuc_1.txt Dmono_1.txt processed
> 0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt
> Dd_inuc_2.txt Dmono_2.txt
> www-galaxy@ge-002: data_import> ls -l
> total 96
> -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
> lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH ->
> /homes/pschofield/data/TOH
>
> I got the TOH option in the 'Upload directory of files', which I
> selected, but the upload failed. Again, this is the error I get when
> selecting one of the filenames in the dataset:
> Traceback (most recent call last): File
> "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line
> 326, in __main__() File
> "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line
> 318, in __main__ add_file( dataset,
> error
Hi Chris,
Chances are, the files in /homes/pschofield/data/TOH are not writable by
Galaxy. Galaxy was trying to convert newlines to UNIX on those files as
part of the import process. This behavior was not intentional.
Yup. That's correct. Galaxy has read-only access to the original files.
This is necessary as users have their own locations for their raw NGS
data which they want to upload to Galaxy.
I've just committed 3517:0c9e154e9176, which will prevent Galaxy
from
ever attempting to modify the import files. When copying in to Galaxy,
they will still converted, but no longer "in-place" (a temp file is
used). When not copying (i.e. symlinking), no newline conversion will be
performed since doing so would require modifying the import file.
Great. I'll look out for the latest updates and try it again, then.
Again, thanks very much for the fixes.
Cheers,
Chris
3:56 p.m.
New subject: Weird problem with importing datasets
On 3/12/10 11:33 AM, "Chris Cole" <chris(a)compbio.dundee.ac.uk> wrote:
On 11/03/10 17:01, Nate Coraor wrote:
> Chris Cole wrote:
>
>> Just checked it again (sorry I shouldn't have sent the previous email)
>> with a directory and it still isn't working.
>>
>> www-galaxy@ge-002: tmp> cd ~/data_import/
>> www-galaxy@ge-002: data_import> ls
>> README
>> www-galaxy@ge-002: data_import> ln -s /homes/pschofield/data/TOH
>> www-galaxy@ge-002: data_import> ls TOH/
>> 0min_1.txt 30m_1.txt 60min_1.txt Cd_inuc_1.txt Cmono_1.txt
>> Dd_inuc_1.txt Dmono_1.txt processed
>> 0min_2.txt 30m_2.txt 60min_2.txt Cd_inuc_2.txt Cmono_2.txt
>> Dd_inuc_2.txt Dmono_2.txt
>> www-galaxy@ge-002: data_import> ls -l
>> total 96
>> -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
>> lrwxrwxrwx 1 www-galaxy barton 26 Mar 9 15:59 TOH ->
>> /homes/pschofield/data/TOH
>>
>> I got the TOH option in the 'Upload directory of files', which I
>> selected, but the upload failed. Again, this is the error I get when
>> selecting one of the filenames in the dataset:
>> Traceback (most recent call last): File
>> "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line
>> 326, in __main__() File
>> "/homes/www-galaxy/galaxy_devel/tools/data_source/upload.py", line
>> 318, in __main__ add_file( dataset,
>> error
>
> Hi Chris,
>
> Chances are, the files in /homes/pschofield/data/TOH are not writable by
> Galaxy. Galaxy was trying to convert newlines to UNIX on those files as
> part of the import process. This behavior was not intentional.
Yup. That's correct. Galaxy has read-only access to the original files.
This is necessary as users have their own locations for their raw NGS
data which they want to upload to Galaxy.
Hi Nate, Hi Chris
I was hoping this would also explain our weird problem (first mentioned in
my e-mail to the list Jan 21st)...but even if the files are writable by
galaxy it doesn't work, eg:
if I try to import the following files:
(sorry, I have to blank out the full path)
galaxy@erbium:/****/test$ ls -l
total 8
-rw-rw-rw- 1 haruhotz gbioinfo 841 2010-03-14 16:18 P51003.fasta
-rw-r--r-- 1 galaxy galaxy 828 2010-03-14 16:18 P51004.fasta
galaxy@erbium:/****/test$
it doesn't work, with the following error message (for both files):
"
Traceback (most recent call last): File
"/***/galaxy_dist/tools/data_source/upload.py", line 311, in __main__()
File "/***/galaxy_dist/tools/data_source/upload.py", line 302, in __main__
add_fil
error
"
Only if I delete the file P51003.fasta, eg as user galaxy:
galaxy@erbium:/****/test$ rm P51003.fasta
galaxy@erbium:/****/test$ ls -l
total 4
-rw-r--r-- 1 galaxy galaxy 828 2010-03-14 16:18 P51004.fasta
galaxy@erbium:/****/test$
the remaining file get's imported (and I guess 'rewritten' by galaxy):
galaxy@erbium:/****/test$ ls -l
total 4
-rw------- 1 galaxy galaxy 828 2010-03-14 16:26 P51004.fasta
galaxy@erbium:/****/test
Nevertheless, I put my hope into the the next update (including
"3517:0c9e154e9176") which I will do in the next few weeks.
Regard, Hans
> I've just committed 3517:0c9e154e9176, which will prevent
Galaxy from
> ever attempting to modify the import files. When copying in to Galaxy,
> they will still converted, but no longer "in-place" (a temp file is
> used). When not copying (i.e. symlinking), no newline conversion will be
> performed since doing so would require modifying the import file.
Great. I'll look out for the latest updates and try it again, then.
Again, thanks very much for the fixes.
Cheers,
Chris
_______________________________________________
galaxy-dev mailing list
galaxy-dev(a)lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-dev
4:11 p.m.
New subject: Weird problem with importing datasets
Sorry for the delay, only just got back to looking at this.
Following the latest updates (tip is now 3560:4c95f1a101f1) I can now
successfully update files not belonging to Galaxy. However, softlinks in
the import directory aren't being dereferenced correctly.
e.g. This is the contents of the import directory:
caterpillar: /homes/www-galaxy/data_import> ls -l
total 96
-rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
lrwxrwxrwx 1 ccole barton 65 Mar 25 15:47 solexa_files ->
/homes/ccole/projects/Dicty_RNAseq/raw_data/NOBACK/SOLEXA results
I deleted the softlink following upload, but now the files are no longer
accessible. FASTQ Groomer for example reports this error:
Traceback (most recent call last):
File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
line 37, in ?
if __name__ == "__main__": main()
File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
line 18, in main
for read_count, fastq_read in enumerate( fastqReader( open(
input_filename ), format = input_type ) ):
IOError: [Errno 2] No such file or directory:
'/homes/www-galaxy/data_import/solexa_files/Ax2.txt'
So, it looks like Galaxy didn't dereference the 'solexa_files' softlink
(to me at least).
Cheers,
Chris
9:26 p.m.
New subject: Weird problem with importing datasets
Chris Cole wrote:
Sorry for the delay, only just got back to looking at this.
Following the latest updates (tip is now 3560:4c95f1a101f1) I can now
successfully update files not belonging to Galaxy. However, softlinks in
the import directory aren't being dereferenced correctly.
e.g. This is the contents of the import directory:
caterpillar: /homes/www-galaxy/data_import> ls -l
total 96
-rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
lrwxrwxrwx 1 ccole barton 65 Mar 25 15:47 solexa_files ->
/homes/ccole/projects/Dicty_RNAseq/raw_data/NOBACK/SOLEXA results
I deleted the softlink following upload, but now the files are no longer
accessible. FASTQ Groomer for example reports this error:
Traceback (most recent call last):
File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
line 37, in ?
if __name__ == "__main__": main()
File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
line 18, in main
for read_count, fastq_read in enumerate( fastqReader( open(
input_filename ), format = input_type ) ):
IOError: [Errno 2] No such file or directory:
'/homes/www-galaxy/data_import/solexa_files/Ax2.txt'
So, it looks like Galaxy didn't dereference the 'solexa_files' softlink
(to me at least).
Hi Chris,
This wasn't part of the original design. We were handling when the
contents of import directories were symlinks, but not when the import
directories themselves were symlinks. This has been added in
3629:6b93e705c8a4.
--nate
Cheers,
Chris
_______________________________________________
galaxy-dev mailing list
galaxy-dev(a)lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-dev
8 a.m.
New subject: Weird problem with importing datasets
On 12/04/10 22:26, Nate Coraor wrote:
Chris Cole wrote:
> Sorry for the delay, only just got back to looking at this.
>
> Following the latest updates (tip is now 3560:4c95f1a101f1) I can now
> successfully update files not belonging to Galaxy. However, softlinks
> in the import directory aren't being dereferenced correctly.
>
> e.g. This is the contents of the import directory:
> caterpillar: /homes/www-galaxy/data_import> ls -l
> total 96
> -rw-r--r-- 1 www-galaxy barton 142 Mar 9 14:14 README
> lrwxrwxrwx 1 ccole barton 65 Mar 25 15:47 solexa_files ->
> /homes/ccole/projects/Dicty_RNAseq/raw_data/NOBACK/SOLEXA results
>
> I deleted the softlink following upload, but now the files are no
> longer accessible. FASTQ Groomer for example reports this error:
> Traceback (most recent call last):
> File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
> line 37, in ?
> if __name__ == "__main__": main()
> File "/homes/www-galaxy/galaxy_devel/tools/fastq/fastq_groomer.py",
> line 18, in main
> for read_count, fastq_read in enumerate( fastqReader( open(
> input_filename ), format = input_type ) ):
> IOError: [Errno 2] No such file or directory:
> '/homes/www-galaxy/data_import/solexa_files/Ax2.txt'
>
> So, it looks like Galaxy didn't dereference the 'solexa_files'
> softlink (to me at least).
Hi Chris,
This wasn't part of the original design. We were handling when the
contents of import directories were symlinks, but not when the import
directories themselves were symlinks. This has been added in
3629:6b93e705c8a4.
Oh, right. I got the impression that all the symlinks were dereferenced
once.
Thanks for the update. I'll check it out when it gets to the main tree.
Thanks again for your help.
Cheers,
Chris
1:15 p.m.
New subject: Weird problem with importing datasets
Chris Cole wrote:
Oh, right. I got the impression that all the symlinks were
dereferenced
once.
They were, but only the ones *in* the subdirectories of the import
directory. It wasn't dereferencing if the actual subdirectories were
symlinks.
--nate
Thanks for the update. I'll check it out when it gets to the main tree.
Thanks again for your help.
Cheers,
Chris
4417
days inactive
4452
days old
galaxy-dev@lists.galaxyproject.org
7 comments
3 participants
participants (3)
-
Chris Cole -
Hotz, Hans-Rudolf -
Nate Coraor