Hi, I'm hoping that the list can help me out. I'm trying to write some tools to capture elements to ease migration from a course development server to a training server. I'm using the Bioblend API which I am a newbie at using.
In the code for extracting the data files from shared data libraries all is well except for 'Tabular' data (e.g. fasta and txt are OK). The error seems to be at the stage where the program calls the Bioblend .datasets.download_dataset() function. NB The tabular files in question can be downloaded fine from the server on the appropriate Galaxy page. (My code is at the end of this message - Please forgive the quality of my Python code).
If a tabular data file is present then I get the following output & errors from my program:- Root path: /usr/users/fr009/fernande/gal_exports/data Library: Intro Galaxy Training No. of items in library 11
/usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/BM361007.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/NB_domains.tabular <<
--------------------------------------------------------------------------- DatasetStateException Traceback (most recent call last) <ipython-input-36-8f38351db9f4> in <module>() 52 filepath = os.path.join(rootpathplus,fname.strip("/")) 53 print '>>',filepath,'<<' ---> 54 gi.datasets.download_dataset(items['id'],filepath,False)
/usr/local/lib/python2.7/dist-packages/bioblend/galaxy/datasets/__init__.pyc in download_dataset(self, dataset_id, file_path, use_default_filename, wait_for_completion, maxwait) 81 dataset = self.show_dataset(dataset_id) 82 if not dataset['state'] == 'ok': ---> 83 raise DatasetStateException("Dataset not ready. Dataset id: %s, current state: %s" % (dataset_id, dataset['state'])) 84 85 # Galaxy release_13.01 and earlier does not have file_ext in the dataset
DatasetStateException: u'Dataset not ready. Dataset id: 1cd8e2f6b131e891, current state: error'
Also note that : g_vers = gi.config.get_config() g_vers['version_major']
Outputs]: u'15.03'
Any help/elucidation gratefully received (I'm on a learning curve here) :-)
Thanks in anticipation. Mark Fernandes #####Code omitting the template files###### --------------------------------------------------------- import bioblend.galaxy import os # Ipython Notebook version - MAF Oct 2015 # Server specific bit. in Real code get this from environ & args (argv) server = 'http://N85460.nbi.ac.uk:8080/' api_key = 'secret_squirrel'
rootpath = os.path.join(os.environ['HOME'],'gal_exports/data') print 'Root path: ',rootpath
#connect to galaxy server instance gi = bioblend.galaxy.GalaxyInstance(url=server, key=api_key)
#fetch names info about data libraries dl = gi.libraries.get_libraries()
# for each library, get the folder names for each set & create the sub-dirs if they do not exist for libs in dl: print 'Library: ', libs['name'] rootpathplus = os.path.join(rootpath,libs['name'])
# Code to check directory pre-esistence and create it if it doesn't # Move this out to a support module in finished code.... try: os.makedirs(rootpathplus, 0755) except OSError: if not os.path.isdir(rootpathplus): raise
# for each library, get the folder names for each set & create the sub-dirs if they do not exist fold = gi.libraries.get_folders(libs['id']) print 'No. of items in library',len(fold)
if len(fold) > 1: for items in fold: dirpath = os.path.join(rootpathplus,os.path.normpath(items['name'].strip("/"))) try: os.makedirs(dirpath, 0755) except OSError: if not os.path.isdir(dirpath): raise # # for each folder create the sub-dir off of user rootdir cl = gi.libraries.show_library(libs['id'],True) for items in cl: if items['type']=='file': fname = os.path.normpath(items['name']) file_ext = os.path.splitext(fname)[1] # check for tabular data file and omit to avoid error message if file_ext != '.tabular': # Swap above if with this one to let the bad stuff happen... # if file_ext != '.untrapped': filepath = os.path.join(rootpathplus,fname.strip("/")) print '>>',filepath,'<<' gi.datasets.download_dataset(items['id'],filepath,False)
Root path: /usr/users/fr009/fernande/gal_exports/data Library: Intro Galaxy Training No. of items in library 11
/usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/BM361007.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVH9.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVRblb1.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVRblb2.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Test.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/SNPsDensity/UCSC_Human_chr22_SNPs.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/SNPsDensity/UCSC_Human_refGene_chr22_Exons.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/TAF1_ChIP << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/TAF1_ChIP_chr11 << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/UCSC_Human_RefGene_chr11_Promoters << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/UCSC_Human_refGene_chr11 << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/data_visualisation/www.compsysbio.org_bacteriome_dataset_functional_interactions.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/Local_data/truseq_exome_targeted_regions.hg19.bed.chr22.bed << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/BioMart_Homo_sapiens_genes_GRCh37p13_chr22.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/TAF1_ChIP.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/UCSC_Human_refGene_chr22_Exons.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/illumina_truseq_exome_targeted_regions_hg19_chr22.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/workflows/UCSC_Human_chrX_SNPs.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/workflows/UCSC_Human_refGene_chrX_Exons.txt <<
Library: New data library No. of items in library 1 ----------------------------------------------------------------------------------
T: +44 (0)1603 255000 Institute of Food Research, Norwich Research Park, Norwich, NR4 7UA, UK
www.ifr.ac.uk Facebook | Twitter | LinkedIn | YouTube
The Institute of Food Research receives strategic funding from BBSRC
Hi Mark, it seems that one dataset in the library (probably
/usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/NB_domains.tabular )
is in 'error' state, so BioBlend, as a precaution, refuses to download it, even if it is possible to download it in Galaxy.
Hope that helps, Nicola
On 08/10/15 15:48, mark fernandes (IFR) wrote:
Hi, I'm hoping that the list can help me out. I'm trying to write some tools to capture elements to ease migration from a course development server to a training server. I'm using the Bioblend API which I am a newbie at using.
In the code for extracting the data files from shared data libraries all is well except for 'Tabular' data (e.g. fasta and txt are OK). The error seems to be at the stage where the program calls the Bioblend .datasets.download_dataset() function. NB The tabular files in question can be downloaded fine from the server on the appropriate Galaxy page. (My code is at the end of this message - Please forgive the quality of my Python code).
If a tabular data file is present then I get the following output & errors from my program:- Root path: /usr/users/fr009/fernande/gal_exports/data Library: Intro Galaxy Training No. of items in library 11
/usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/BM361007.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/NB_domains.tabular <<
DatasetStateException Traceback (most recent call last) <ipython-input-36-8f38351db9f4> in <module>() 52 filepath = os.path.join(rootpathplus,fname.strip("/")) 53 print '>>',filepath,'<<' ---> 54 gi.datasets.download_dataset(items['id'],filepath,False)
/usr/local/lib/python2.7/dist-packages/bioblend/galaxy/datasets/__init__.pyc in download_dataset(self, dataset_id, file_path, use_default_filename, wait_for_completion, maxwait) 81 dataset = self.show_dataset(dataset_id) 82 if not dataset['state'] == 'ok': ---> 83 raise DatasetStateException("Dataset not ready. Dataset id: %s, current state: %s" % (dataset_id, dataset['state'])) 84 85 # Galaxy release_13.01 and earlier does not have file_ext in the dataset
DatasetStateException: u'Dataset not ready. Dataset id: 1cd8e2f6b131e891, current state: error'
Also note that : g_vers = gi.config.get_config() g_vers['version_major']
Outputs]: u'15.03'
Any help/elucidation gratefully received (I'm on a learning curve here) :-)
Thanks in anticipation. Mark Fernandes
#####Code omitting the template files######
import bioblend.galaxy import os # Ipython Notebook version - MAF Oct 2015 # Server specific bit. in Real code get this from environ & args (argv) server = 'http://N85460.nbi.ac.uk:8080/' api_key = 'secret_squirrel'
rootpath = os.path.join(os.environ['HOME'],'gal_exports/data') print 'Root path: ',rootpath
#connect to galaxy server instance gi = bioblend.galaxy.GalaxyInstance(url=server, key=api_key)
#fetch names info about data libraries dl = gi.libraries.get_libraries()
# for each library, get the folder names for each set & create the sub-dirs if they do not exist for libs in dl: print 'Library: ', libs['name'] rootpathplus = os.path.join(rootpath,libs['name'])
# Code to check directory pre-esistence and create it if it doesn't # Move this out to a support module in finished code.... try: os.makedirs(rootpathplus, 0755) except OSError: if not os.path.isdir(rootpathplus): raise
# for each library, get the folder names for each set & create the sub-dirs if they do not exist fold = gi.libraries.get_folders(libs['id']) print 'No. of items in library',len(fold)
if len(fold) > 1: for items in fold: dirpath = os.path.join(rootpathplus,os.path.normpath(items['name'].strip("/"))) try: os.makedirs(dirpath, 0755) except OSError: if not os.path.isdir(dirpath): raise# # for each folder create the sub-dir off of user rootdir cl = gi.libraries.show_library(libs['id'],True) for items in cl: if items['type']=='file': fname = os.path.normpath(items['name']) file_ext = os.path.splitext(fname)[1] # check for tabular data file and omit to avoid error message if file_ext != '.tabular': # Swap above if with this one to let the bad stuff happen... # if file_ext != '.untrapped': filepath = os.path.join(rootpathplus,fname.strip("/")) print '>>',filepath,'<<' gi.datasets.download_dataset(items['id'],filepath,False)
Root path: /usr/users/fr009/fernande/gal_exports/data Library: Intro Galaxy Training No. of items in library 11
/usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/NBARC/BM361007.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVH9.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVRblb1.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Pinfestans/PinfestansAVRblb2.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Getting_started/Test.fasta << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/SNPsDensity/UCSC_Human_chr22_SNPs.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/SNPsDensity/UCSC_Human_refGene_chr22_Exons.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/TAF1_ChIP << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/TAF1_ChIP_chr11 << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/UCSC_Human_RefGene_chr11_Promoters << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/Interval_operations/TAF1/UCSC_Human_refGene_chr11 << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/data_visualisation/www.compsysbio.org_bacteriome_dataset_functional_interactions.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/Local_data/truseq_exome_targeted_regions.hg19.bed.chr22.bed << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/BioMart_Homo_sapiens_genes_GRCh37p13_chr22.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/TAF1_ChIP.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/UCSC_Human_refGene_chr22_Exons.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/loading_data/illumina_truseq_exome_targeted_regions_hg19_chr22.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/workflows/UCSC_Human_chrX_SNPs.txt << /usr/users/fr009/fernande/gal_exports/data/Intro Galaxy Training/workflows/UCSC_Human_refGene_chrX_Exons.txt <<
Library: New data library No. of items in library 1
T: +44 (0)1603 255000 Institute of Food Research, Norwich Research Park, Norwich, NR4 7UA, UK
www.ifr.ac.uk Facebook | Twitter | LinkedIn | YouTube
The Institute of Food Research receives strategic funding from BBSRC ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
mark fernandes (IFR) -
Nicola Soranzo