October 2014 - galaxy-dev - lists.galaxyproject.org

Assertion errors when starting Galaxy
by Ricardo Perez 14 Oct '14

14 Oct '14

Hello, When starting Galaxy after updating it we get the assertion errors that follow the following format: galaxy.datatypes.registry ERROR 2014-10-14 15:51:40,004 Error loading display application (./display_applications/ucsc/bam.xml): Filename and id attributes required for dynamic_links Traceback (most recent call last): File "/home/galaxy/galaxy-dist.old/lib/galaxy/datatypes/registry.py", line 576, in load_display_applications display_app = DisplayApplication.from_file( config_path, self ) File "/home/galaxy/galaxy-dist.old/lib/galaxy/datatypes/display_applications/application.py", line 170, in from_file return cls.from_elem( parse_xml( filename ).getroot(), datatypes_registry ) File "/home/galaxy/galaxy-dist.old/lib/galaxy/datatypes/display_applications/application.py", line 183, in from_elem for link in DynamicDisplayApplicationBuilder( dynamic_links, rval, datatypes_registry.build_sites ): File "/home/galaxy/galaxy-dist.old/lib/galaxy/datatypes/display_applications/application.py", line 91, in __init__ assert filename is not None, 'Filename and id attributes required for dynamic_links' AssertionError: Filename and id attributes required for dynamic_links Any one else getting the same thing after updating their Galaxy instance? Thank You, --Ricardo

1 0

Galaxy database corrupted
by Thuy Linh Chu 14 Oct '14

14 Oct '14

Recently I tried to upload some annotation/reference files into Galaxy. After couple failed tries using the upload method, I was able to do it via the using the full file path. However, after these tries/retries, I believe Galaxy database got confused and was corrupted. Existing workflows with existing reference were used when datasets that previously were processed without any problems, now have funky result set names and parameters to tool wrappers reference funky paths. Here’s an example of the error I’m encountering. The highlighted part references the dbSNP file: python /home/svcbioinfo/galaxy/galaxy-reproduce/tools/gatkV2/gatk_wrapper.py --max_jvm_heap "16g" --max_jvm_heap_fraction "1" --stdout "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24218.dat" -d "-I" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24216.dat" "bam" "gatk_input_first" -d "" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/_metadata_files/001/metadata_1491.dat" "bam_index" "gatk_input_first" -p 'java -jar "/home/svcbioinfo/galaxy/galaxy-reproduce/tool-data/shared/jars/gatk/GenomeAnalysisTKv2.jar" -T "RealignerTargetCreator" --num_threads 4 -o "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24217.dat" -et "NO_ET" -K "/home/svcbioinfo/galaxy/galaxy-reproduce/tool-data/shared/jars/genomichealth.com.key" -R "/mnt/ngs/analysis/refData/processed/bwa/hg19_decoy_rcrs8001sp.fasta" ' -d "-known:dbsnp,%(file_type)s" "/mnt/ngs/analysis/tchu/Workflow_RWC/SEQUENCER99/140201_SNTEST_0021_TESTFC21/analysis_31/primary/fastq/99_21_3_ACAGTG_R2.fastq" "fastq" "input_dbsnp_0" -d "-known:indels,%(file_type)s" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/016/dataset_16596.dat" "vcf" "input_indels_0" -p '--pedigreeValidationType "STRICT"' -d "--intervals" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/022/dataset_22821.dat" "bed" "input_intervals_0" -p '--interval_padding "400"' -p '--interval_set_rule "UNION"' -p '--downsampling_type "NONE"' -p ' --validation_strictness "SILENT" --interval_merging "ALL" ' -p ' --unsafe "LENIENT_VCF_PROCESSING" The highlighted part should be the dbSNP file which typically has a Galaxy based name and type of “vcf” such as below: python /home/svcbioinfo/galaxy/galaxy-reproduce/tools/gatkV2/gatk_wrapper.py --max_jvm_heap "16g" --max_jvm_heap_fraction "1" --stdout "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24314.dat" -d "-I" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24312.dat" "bam" "gatk_input_first" -d "" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/_metadata_files/001/metadata_1492.dat" "bam_index" "gatk_input_first" -p 'java -jar "/home/svcbioinfo/galaxy/galaxy-reproduce/tool-data/shared/jars/gatk/GenomeAnalysisTKv2.jar" -T "RealignerTargetCreator" --num_threads 4 -o "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/024/dataset_24313.dat" -et "NO_ET" -K "/home/svcbioinfo/galaxy/galaxy-reproduce/tool-data/shared/jars/genomichealth.com.key" -R "/mnt/ngs/analysis/refData/processed/bwa/hg19_decoy_rcrs.fasta" ' -d "-known:dbsnp,%(file_type)s" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/016/dataset_16598.dat" "vcf" "input_dbsnp_0" -d "-known:indels,%(file_type)s" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/016/dataset_16596.dat" "vcf" "input_indels_0" -p '--pedigreeValidationType "STRICT"' -d "--intervals" "/mnt/ngs/analysis/svcbioinfo/DATA_reproduce/files/016/dataset_16617.dat" "bed" "input_intervals_0" -p '--interval_padding "400"' -p '--interval_set_rule "UNION"' -p '--downsampling_type "NONE"' -p ' --validation_strictness "SILENT" --interval_merging "ALL" ' -p ' --unsafe "LENIENT_VCF_PROCESSING" I wonder if there's a way to recover from this or I'll have to reinstall the database. Thanks for your help. Thuy-Linh Chu

1 0

CloudMan with HTCondor and Slurm & mounting pathes
by Abdularahman Azab 14 Oct '14

14 Oct '14

Dear all, I've installed a galaxy cloud cluster using usergalaxy.org/cloudlaunch It runs fine, but I need to know how to let the cluster run on THCondor or slurm instead of SGE. I can see that Condor is already installed on the galaxy-master node since the condor_status command is there. But how can I switch the default galaxy job-runner between SGE, HTCondor, and slurm Also regarding the mounting option of a e.g. s3 bucket on the master node. It works fine and it can be accessed from the command line. But how to let galaxy see this mounted path for e.g, reading input files and storing resulted history elements in it? Thank you, Yours sincerely, Abdulrahman Azab Head engineer, ELIXIR.NO / The Genomic HyperBrowser team Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway Email: azab(a)ifi.uio.no</owa/abdulara(a)ifi.uio.no>, Cell-phone: +47 46797339 ---- Senior Lecturer in Computer Engineering Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt Email: abdulrahman.azab(a)mans.edu.eg</owa/abdulrahman.azab(a)mans.edu.eg>

2 1

error running API with example_watch_folder.py
by Neil.Burdett＠csiro.au 14 Oct '14

14 Oct '14

Hi, when I use the example_watch_folder.py to check for files in an input directory and process them if they are present. The workflow runs to completion successfully, but the "example_watch_folder.py" task crashes with the following errors, thus it can only process one file at a time, as the example_watch_folder.py has to be constantly restarted. python example_watch_folder.py d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/ /home/galaxy/cte_input /home/galaxy/cte_output "API Imports" f597429621d6eb2b in_folder [/home/galaxy/cte_input] fullpath [/home/galaxy/cte_input/141_S_0851_MRI_T1_Screening.nii.gz] submit library_id [f2db41e1fa331b3e] libset [[{u'url': u'/new/api/libraries/f2db41e1fa331b3e/contents/82b264d8c3d11790', u'id': u'82b264d8c3d11790', u'name': u'141_S_0851_MRI_T1_Screening.nii.gz'}]] waits for several minutes .... Traceback (most recent call last): File "example_watch_folder.py", line 97, in <module> main(api_key, api_url, in_folder, out_folder, data_library, workflow ) File "example_watch_folder.py", line 77, in main res = submit( api_key, api_url + 'workflows', wf_data, return_formatted=False) File "common.py", line 117, in submit r = post( api_key, url, data ) File "common.py", line 51, in post return json.loads( urllib2.urlopen( req ).read() ) File "/usr/lib/python2.7/socket.py", line 351, in read data = self._sock.recv(rbufsize) File "/usr/lib/python2.7/httplib.py", line 541, in read return self._read_chunked(amt) File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked raise IncompleteRead(''.join(value)) httplib.IncompleteRead: IncompleteRead(176 bytes read) Due to the crash there is no outputs shown on the screen. Any ideas? Thanks Neil p.s. there is nothing in the log file.

2 2

failure to export workflows
by Wolfgang Maier 13 Oct '14

13 Oct '14

Dear all, our local Galaxy does not seem to be able to export workflows anymore, instead fails with, for example: Not Found The resource could not be found. No route for /u/w/myworkflow/json-download We are running latest_2014.10.06 . Has anyone else experienced this ? Thanks, Wolfgang

3 6

missing "fastq groomer" tool on galaxy cloudman
by NAMIT KUMAR 13 Oct '14

13 Oct '14

Hello, I am using galaxy on amazon web servers ( AMI: ami-a7dbf6ce; Name: Galaxy CloudMan 2.3). This is a setup I have used several time in the past with no issue, for suddenly since yesterday- there is *NO option of fastq groomer tool* on this ami. fastq groomer tool is still available on public server but not on aws cloudman. Anyone has any information on this? best, Namit

3 2

quick note of thanks
by Chris Dagdigian 13 Oct '14

13 Oct '14

Wanted to send Dannon & the list a quick note of thanks. Cloudman-managed Galaxy is pretty slick. I'm using it today to run 26 individual Galaxy instances running on big c3.8xlarge AWS nodes for a training class. Each of the 26 training systems is built from a shared cluster string config which means I can stand up tested systems that also contain a "hidden" history that students can access and refer to if they get stuck. We wanted to give each student their own individual Galaxy sandbox to play on and Cloudman made it very easy. Faster than custom configuring a personal AMI and using Chef/devOps tricks to clone and automate the deployment. All told it took me about 1.5 days to get my head around cloudman and make all the usual mistakes regarding breaking things, screwing things up and otherwise running through the learning curve. After that it took me about 1 day to load the master cluster image up with data, history and run through all of the lab exercises. After that it was about 2 hours to share the cluster and start firing up the clones One thing that would help (and perhaps this is something that I missed in the documentation) would be the ability to pass in the shared cluster string as user-data at launch time. That would have let me fire up the training nodes with a single shell script. Instead I used a shell script to fire up the nodes but then I had to hit the cloudman URL on each node to punch in the shared-cluster URI string. Thanks again! Regards, Chris

2 1

API : Max retries exceeded
by Gildas Le Corguille 13 Oct '14

13 Oct '14

Hi, I wrote a script which need to retrieve a list of all users on our production instance. It seems that we have too many users (325) :P bioblend.galaxy.client.ConnectionError: HTTPConnectionPool(host='galaxy.sb-roscoff.fr', port=8080): Max retries exceeded with url: /api/users?key=fd420zefff9a9df586210fd8c8452d98 (Caused by <class 'socket.error'>: [Errno 111] Connection refused): None My script is running on an test instance which have less users. Is there a way to set this threshold ? Thanks Gildas PS : don't try my api_key, little hacker, it's a fake :P

2 3

GATK2 java version related issues
by Iry Witham 10 Oct '14

10 Oct '14

Hi Team, I am sorry to be a pest, but I am under-the-gun to get this resolved. I have followed the instructions for installing GATK2 on my cloud VM. After doing that I discovered that the version of java running on the VM was v1.6.x and GATK2 required v1.7.0x. I removed version 6 and installed version 7, uninstalled GATK2 and reinstalled it and reset all of the settings that may have reverted with the uninstalls/reinstalls. Now when I run Unified Genotyper I get the following error: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/broadinstitute/gatk/engine/CommandLineGATK : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:643) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) at java.net.URLClassLoader.access$000(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:212) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:323) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:268) Could not find the main class: org.broadinstitute.gatk.engine.CommandLineGATK. Program will exit. Researching this I found a post at http://gatkforums.broadinstitute.org/discussion/3378/gatk2-error-when-i-tes… that refers to upgrading java as the solution. However, that did not fix the issue. In addition I copied the jar file to a temp directory and unzipped it to examine the MANAFEST.MF file for versioning issues. This is what I found: Manifest-Version: 1.0 Build-Jdk: 1.7.0_51 Built-By: gsa-engineering Created-By: Apache Maven Main-Class: org.broadinstitute.gatk.engine.CommandLineGATK Archiver-Version: Plexus Archiver This tells me that there is something else causing the problem. Does anyone have an idea where to look now? Regards,Iry The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

2 2

Why does Galaxy CloudMan 2.3 (ami-a7dbf6ce) run the boinc client?
by Chris Dagdigian 10 Oct '14

10 Oct '14

Just a paranoia check ... I manually started the cloudman galaxy AMI (ami-a7dbf6ce) and noticed that it is configured and running a boinc client Just wanted to check if this was legit and expected ... Regards, Chris

2 1