When you import to a library (uploading from preconfigured server directory) there is an option "No" to not copy, and instead use in place.

We played with this for a while then decided against it. moving the source fileor editing it etc. can cause havoc.

Cheers Dennis

On Wed, Jun 30, 2010 at 8:01 PM, Michael Siebauer michael_siebauer@eva.mpg.de wrote:

Hi,

is there a way/module or data Library that allows users to access there files WITHOUT the need of uploading them first? Cause our Galaxy instance is running within the intranet, all files could be accessed straight via NFS.

Thx, Michael ;-) _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Non-admin users can upload directories of files as well as long as the admin has set the following config in universe_wsgi.ini:

# The following can be configured to allow non-admin users to upload a directory of files. The # configured directory must contain sub-directories named the same as the non-admin user's Galaxy # login ( email ). The non-admin user is restricted to uploading files or sub-directories of files # contained in their directory. user_library_import_dir = <some directory on the Galaxy instance's file system> allow_library_path_paste = True

On Jun 30, 2010, at 8:35 AM, Michael Siebauer wrote:

Hmm but there is no way, for a user to upload (without copying) his files. Only the Admin User can create and fill data libraries, right?

Cheers, Michael

Am 30.06.2010 12:17, schrieb Dennis Gascoigne:

...

When you import to a library (uploading from preconfigured server directory) there is an option "No" to not copy, and instead use in place.

We played with this for a while then decided against it. moving the source fileor editing it etc. can cause havoc.

Cheers Dennis

On Wed, Jun 30, 2010 at 8:01 PM, Michael Siebauer michael_siebauer@eva.mpg.de wrote:

...

Hi,

is there a way/module or data Library that allows users to access there files WITHOUT the need of uploading them first? Cause our Galaxy instance is running within the intranet, all files could be accessed straight via NFS.

Thx, Michael ;-) _______________________________________________ galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Greg Von Kuster Galaxy Development Team greg@bx.psu.edu

If you want another option, I wrote a quick tool wrapper and bash script. It's a total hack but it works. * create two files server_copy.sh and server_copy.xml in <<GALAXYDIR>>/tools/data_source with the following content. * Add the following line somewhere in your tool_conf <tool file="data_source/server_copy.xml"/>

Cheers Dennis

##################<<bash script server_copy.sh>> ######################(this is ridiculously simple and does NO error check or validation - I have pretty good users) #!/bin/sh

cp $*

##################<<XML script>>###################### <?xml version="1.0"?> <tool name="Server Upload" id="server_upload"> <!-- This tool is used when the user needs to access files stored on galaxy's server without routing through the client machine--> <description></description> <command interpreter="bash">server_copy.sh $input $output </command> <inputs> <param name="input" type="text" area="true" size="2x200" label="Full path to source file" help="The file must be specified as the full path - i.e. /home/user.name/mydata/foo.bar"/> <param name="out_format" type="select" label="Output data type"> <option value="fasta">FASTA</option> <option value="fastqillumina">FASTQ Illumina (or other qual=64)</option> <option value="fastqsanger">FASTQ Sanger (or other qual=33)</option> <option value="sam">SAM</option> <option value="bed">BED</option> <option value="text">TXT</option> <option value="interval">Interval</option> </param> <param format="txt" name="fname" type="text" size="200" label="Enter a name for your file."/> </inputs> <outputs> <data format="text" name="output" size="200" label="Moved: ${fname.value} " > <change_format> <when input="out_format" value="fasta" format="fasta" /> <when input="out_format" value="fastqillumina" format="fastqillumina" /> <when input="out_format" value="fastqsanger" format="fastqsanger" /> <when input="out_format" value="sam" format="sam" /> <when input="out_format" value="interval" format="interval" /> <when input="out_format" value="bed" format="bed" /> <when input="out_format" value="text" format="text" /> </change_format> </data> </outputs> </tool>

On Wed, Jun 30, 2010 at 11:01 PM, Greg Von Kuster greg@bx.psu.edu wrote:

Non-admin users can upload directories of files as well as long as the admin has set the following config in universe_wsgi.ini: # The following can be configured to allow non-admin users to upload a directory of files.  The # configured directory must contain sub-directories named the same as the non-admin user's Galaxy # login ( email ).  The non-admin user is restricted to uploading files or sub-directories of files # contained in their directory. user_library_import_dir = <some directory on the Galaxy instance's file system> allow_library_path_paste = True

On Jun 30, 2010, at 8:35 AM, Michael Siebauer wrote:

Hmm but there is no way, for a user to upload (without copying) his files. Only the Admin User can create and fill data libraries, right?

Cheers, Michael

Am 30.06.2010 12:17, schrieb Dennis Gascoigne:

When you import to a library (uploading from preconfigured server

directory) there is an option "No" to not copy, and instead use in

place.

We played with this for a while then decided against it. moving the

source fileor editing it etc. can cause havoc.

Cheers

Dennis

On Wed, Jun 30, 2010 at 8:01 PM, Michael Siebauer

michael_siebauer@eva.mpg.de  wrote:

Hi,

is there a way/module or data Library that allows users to access there

files WITHOUT the need of uploading them first? Cause our Galaxy instance is

running within the intranet, all files could be accessed straight via NFS.

Thx, Michael ;-)

---

galaxy-dev mailing list

galaxy-dev@lists.bx.psu.edu

http://lists.bx.psu.edu/listinfo/galaxy-dev

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Greg Von Kuster Galaxy Development Team greg@bx.psu.edu

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

I forgot the add that in the XML file, One should also quote the two parameters, to make sure the $input variable always counts a single parameter, even if it contains spaces or tabs:

change from: <command interpreter="bash">server_copy.sh $input $output </command>

to <command interpreter="bash">server_copy.sh '$input' '$output' </command>

Assaf Gordon wrote, On 06/30/2010 06:47 PM:

Dennis,

Please note that your shell script is slightly dangerous and prone to malicious attacks. Not too bad if all your users anyhow have local accounts on your server, but generally speaking, it's risky because it would allow users without local access to get information on your system.

The simplest example is a user asking to copy a sensitive file, like "/etc/passwd". A denial-of-service would be to ask to copy special files like "/etc/urandom" - will fill up your disk.

But it gets more dangerous because you don't validate the options at all, so if I use this string as the input file: "--target-directory /tmp /etc/passwd"

You'll pass it directly to the "cp" program, and "cp" will treat all arguments as input files and copy them to "/tmp" - effectively allowing to write to ANY location on your disk that is writable by your galaxy user.

Not to mention that with a little bit of probing, one can find your "universe_wsgi.ini" and see secrets/passwords of your galaxy, and the database password.

I would recommend at the very minimum, to change the shell script as follows (this would disable arbitrary writing):

#!/bin/sh

INPUT="$1" OUTPUT="$2"

[ -f "$INPUT" ] || { echo "Error: input file '$INPUT' is not a valid file" >&2 ; exit 1 ; } cp -- "$INPUT" "$OUTPUT" || exit 1 ===

To be even more careful, I would check the absolute path of the input file against a list of locations you allow:

#!/bin/sh

INPUT="$1" OUTPUT="$2"

#the file must exist (-e) ABS=$(readlink -en -- "$INPUT") [ -f "$ABS" ] || { echo "Error: input file '$INPUT' is not a valid file" >&2 ; exit 1 ; }

# Extract the first 6 characters, make sure they are "/home/" PREFIX=${ABS:0:6} [ "x$PREFIX" = "x/home/" ] || { echo "Invalid input file path ($ABS)" >&2 ; exit 1 ; }

cp -- "$INPUT" "$OUTPUT" || exit 1

But even that would not stop users from getting other users' files (if they are readable by your galaxy user).

If you must allow users to get files directly from the server, I would recommend setting up a public place (just one directory) to which users can copy the files, and your script will allow importing files only from that folder.

-gordon

Dennis Gascoigne wrote, On 06/30/2010 05:53 PM:

...

If you want another option, I wrote a quick tool wrapper and bash script. It's a total hack but it works.

• create two files server_copy.sh and server_copy.xml in

<<GALAXYDIR>>/tools/data_source with the following content.

• Add the following line somewhere in your tool_conf <tool

file="data_source/server_copy.xml"/>

Cheers Dennis

##################<<bash script server_copy.sh>> ######################(this is ridiculously simple and does NO error check or validation - I have pretty good users) #!/bin/sh

cp $*

##################<<XML script>>######################

<?xml version="1.0"?>

<tool name="Server Upload" id="server_upload"> <!-- This tool is used when the user needs to access files stored on galaxy's server without routing through the client machine--> <description></description> <command interpreter="bash">server_copy.sh $input $output </command> <inputs> <param name="input" type="text" area="true" size="2x200" label="Full path to source file" help="The file must be specified as the full path - i.e. /home/user.name/mydata/foo.bar"/> <param name="out_format" type="select" label="Output data type"> <option value="fasta">FASTA</option> <option value="fastqillumina">FASTQ Illumina (or other qual=64)</option> <option value="fastqsanger">FASTQ Sanger (or other qual=33)</option> <option value="sam">SAM</option> <option value="bed">BED</option> <option value="text">TXT</option> <option value="interval">Interval</option> </param> <param format="txt" name="fname" type="text" size="200" label="Enter a name for your file."/> </inputs> <outputs> <data format="text" name="output" size="200" label="Moved: ${fname.value} " > <change_format> <when input="out_format" value="fasta" format="fasta" /> <when input="out_format" value="fastqillumina" format="fastqillumina" /> <when input="out_format" value="fastqsanger" format="fastqsanger" /> <when input="out_format" value="sam" format="sam" /> <when input="out_format" value="interval" format="interval" /> <when input="out_format" value="bed" format="bed" /> <when input="out_format" value="text" format="text" /> </change_format> </data> </outputs> </tool>

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

Thank you very much. That solves my problem :-)

Am 01.07.2010 03:10, schrieb Dennis Gascoigne:

Assaf;

Yes, I should have pointed that out. Thankyou for doing so for me.

Ours is a small group of users on a tightly locked down system with no public access, galaxy running in a restricted user account and where the users are both responsible and more importantly, i know where they live so I can hunt them down if they do anything bad!

Dennis

On Thu, Jul 1, 2010 at 8:51 AM, Assaf Gordongordon@cshl.edu wrote:

...

I forgot the add that in the XML file, One should also quote the two parameters, to make sure the $input variable always counts a single parameter, even if it contains spaces or tabs:

change from: <command interpreter="bash">server_copy.sh $input $output</command>

to <command interpreter="bash">server_copy.sh '$input' '$output'</command>

Assaf Gordon wrote, On 06/30/2010 06:47 PM:

...

Dennis,

Please note that your shell script is slightly dangerous and prone to malicious attacks. Not too bad if all your users anyhow have local accounts on your server, but generally speaking, it's risky because it would allow users without local access to get information on your system.

The simplest example is a user asking to copy a sensitive file, like "/etc/passwd". A denial-of-service would be to ask to copy special files like "/etc/urandom" - will fill up your disk.

But it gets more dangerous because you don't validate the options at all, so if I use this string as the input file: "--target-directory /tmp /etc/passwd"

You'll pass it directly to the "cp" program, and "cp" will treat all arguments as input files and copy them to "/tmp" - effectively allowing to write to ANY location on your disk that is writable by your galaxy user.

Not to mention that with a little bit of probing, one can find your "universe_wsgi.ini" and see secrets/passwords of your galaxy, and the database password.

I would recommend at the very minimum, to change the shell script as follows (this would disable arbitrary writing):

#!/bin/sh

INPUT="$1" OUTPUT="$2"

[ -f "$INPUT" ] || { echo "Error: input file '$INPUT' is not a valid file">&2 ; exit 1 ; } cp -- "$INPUT" "$OUTPUT" || exit 1 ===

To be even more careful, I would check the absolute path of the input file against a list of locations you allow:

#!/bin/sh

INPUT="$1" OUTPUT="$2"

#the file must exist (-e) ABS=$(readlink -en -- "$INPUT") [ -f "$ABS" ] || { echo "Error: input file '$INPUT' is not a valid file">&2 ; exit 1 ; }

# Extract the first 6 characters, make sure they are "/home/" PREFIX=${ABS:0:6} [ "x$PREFIX" = "x/home/" ] || { echo "Invalid input file path ($ABS)">&2 ; exit 1 ; }

cp -- "$INPUT" "$OUTPUT" || exit 1

But even that would not stop users from getting other users' files (if they are readable by your galaxy user).

If you must allow users to get files directly from the server, I would recommend setting up a public place (just one directory) to which users can copy the files, and your script will allow importing files only from that folder.

-gordon

Dennis Gascoigne wrote, On 06/30/2010 05:53 PM:

...

If you want another option, I wrote a quick tool wrapper and bash script. It's a total hack but it works.

• create two files server_copy.sh and server_copy.xml in

<<GALAXYDIR>>/tools/data_source with the following content.

• Add the following line somewhere in your tool_conf<tool

file="data_source/server_copy.xml"/>

Cheers Dennis

##################<<bash script server_copy.sh>> ######################(this is ridiculously simple and does NO error check or validation - I have pretty good users) #!/bin/sh

cp $*

##################<<XML script>>######################

<?xml version="1.0"?>

<tool name="Server Upload" id="server_upload"> <!-- This tool is used when the user needs to access files stored on galaxy's server without routing through the client machine--> <description></description> <command interpreter="bash">server_copy.sh $input $output</command> <inputs> <param name="input" type="text" area="true" size="2x200" label="Full path to source file" help="The file must be specified as the full path - i.e. /home/user.name/mydata/foo.bar"/> <param name="out_format" type="select" label="Output data type"> <option value="fasta">FASTA</option> <option value="fastqillumina">FASTQ Illumina (or other qual=64)</option> <option value="fastqsanger">FASTQ Sanger (or other qual=33)</option> <option value="sam">SAM</option> <option value="bed">BED</option> <option value="text">TXT</option> <option value="interval">Interval</option> </param> <param format="txt" name="fname" type="text" size="200" label="Enter a name for your file."/> </inputs> <outputs> <data format="text" name="output" size="200" label="Moved: ${fname.value} "> <change_format> <when input="out_format" value="fasta" format="fasta" /> <when input="out_format" value="fastqillumina" format="fastqillumina" /> <when input="out_format" value="fastqsanger" format="fastqsanger" /> <when input="out_format" value="sam" format="sam" /> <when input="out_format" value="interval" format="interval" /> <when input="out_format" value="bed" format="bed" /> <when input="out_format" value="text" format="text" /> </change_format> </data> </outputs> </tool>

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev

---

galaxy-dev mailing list galaxy-dev@lists.bx.psu.edu http://lists.bx.psu.edu/listinfo/galaxy-dev