How to import data from a non-local source using SCP?
I have a cluster (cluster A) set up with Galaxy. Our sequencing data gets mapped to hg19, then the resulting BAM files are placed on a SAN connected to a different cluster (cluster B) that cluster A does not have NFS access to. We cannot install an FTP server on cluster B either. The only way to get data from cluster B to cluster A is to use scp. Is there a way to set up a Data Library in Galaxy on cluster A that refers to non-local data and transfers the data from cluster B when needed? Or is it possible to have a Galaxy instance on cluster B share data with a Galaxy instance on cluster A?
On Jan 4, 2012, at 2:52 PM, Ryan wrote:
I have a cluster (cluster A) set up with Galaxy. Our sequencing data gets mapped to hg19, then the resulting BAM files are placed on a SAN connected to a different cluster (cluster B) that cluster A does not have NFS access to. We cannot install an FTP server on cluster B either. The only way to get data from cluster B to cluster A is to use scp. Is there a way to set up a Data Library in Galaxy on cluster A that refers to non-local data and transfers the data from cluster B when needed? Or is it possible to have a Galaxy instance on cluster B share data with a Galaxy instance on cluster A?
Hi Ryan, There is not presently a way to temporarily pull data from other filesystems using scp or share data without a common filesystem. You'll have to copy the data to cluster A and then load it into a library there. --nate
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Actually, I found a way! I can mount the remote filesystem using sshfs. On Wed, Feb 1, 2012 at 12:36 PM, Nate Coraor <nate@bx.psu.edu> wrote:
On Jan 4, 2012, at 2:52 PM, Ryan wrote:
I have a cluster (cluster A) set up with Galaxy. Our sequencing data gets mapped to hg19, then the resulting BAM files are placed on a SAN connected to a different cluster (cluster B) that cluster A does not have NFS access to. We cannot install an FTP server on cluster B either. The only way to get data from cluster B to cluster A is to use scp. Is there a way to set up a Data Library in Galaxy on cluster A that refers to non-local data and transfers the data from cluster B when needed? Or is it possible to have a Galaxy instance on cluster B share data with a Galaxy instance on cluster A?
Hi Ryan,
There is not presently a way to temporarily pull data from other filesystems using scp or share data without a common filesystem. You'll have to copy the data to cluster A and then load it into a library there.
--nate
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
participants (3)
-
Nate Coraor -
Ryan -
Ryan Golhar