June 2010 - galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] Galaxy cloud problem
by Enis Afgan 16 Jun '10

16 Jun '10

Hi Martin, Deleting those volumes would be a problem because all of the data that you upload onto the cloud and analyze through Galaxy are stored on the external EBS volumes. That is the only way to persist the data across cluster instantiations. It looks like we should include a better error/log message if requested volume is not found indicating what happened... At this point, you will not be able to retrieve your data. The best thing to do now would be to go to the S3 tab in your AWS console and delete any buckets that were associated with the cluster(s) you created. The names of all of those buckets will start with 'gc-'. Then, you can instantiate a completely new cluster. BTW, the 2GB EBS volume you see as attached to the running master instance is the actual instance itself. Galaxy Cloud is using EBS AMI's and every running instance will have a volume like like that attached to it. Enis On Wed, Jun 16, 2010 at 1:50 PM, Martin Paradesi < p.martin.samuel.rao(a)gmail.com> wrote: > Hi Enis, > > Here is the cluster status log: > > - 17:38:19 - Master starting... > - 17:38:19 - No existing instances found; starting fresh. > - 17:38:23 - User's bucket 'gc-6658d20fda911b5847f71e33723ea78d' found; > retrieving previously used snapshots. > > I see that a 2GB EBS volume is attached to my cluster. Nothing gets > appended to this log even after I request Galaxy to start 1 node with 100GB > space. I must mention that I performed the following actions before this > whole problem started: > > 1. I terminated the main Galaxy instance last week on AWS. > 2. I came back this week to find that 2 EBS volumes were still listed in > "My Resources" on my dashboard. > 3. After seeing that my bill for these EBS volumes was around $50, I > immediately went to EBS instances and detached and deleted those volumes. > > Do you think that this could have caused some problems with starting the > Galaxy cloud? > > Martin > > > On Wed, Jun 16, 2010 at 1:22 PM, Enis Afgan <eafgan(a)emory.edu> wrote: > >> Hi Martin, >> In the cluster status log, do you maybe get a message like following: >> ... >> 17:14:28 - EC2 insufficient capacity error: BotoServerError: 500 Internal >> Server Error InsufficientInstanceCapacityWe currently do not have >> sufficient m1.large capacity in the Availability Zone you requested >> (us-east-1b). Our system will be working on provisioning additional >> capacity. You can currently get m1.large capacity by not specifying an >> Availability Zone in your request or choosing >> us-east-1a.b2bec3a6-28be-4650-9757-57c1c08c2606 >> 17:14:28 - Encountered problems when starting instances. Currently, >> keeping track of '0' instances >> ... >> >> If you are, as you can see, the problem is caused by the cloud (i.e., >> Amazon) and unfortunately there is nothing we can do about it but wait a few >> hours. >> At the same time, we have not made any changes to the public Galaxy Cloud >> code so it should be operating normally. >> Let us know if this was the problem or something else. If the insufficient >> capacity is not the problem, could you please send us the contents of the >> log so we can get a bit better of an idea what's wrong. You can access full >> instance log at http://ec2-xxxxx/cloud/log >> >> Enis >> >> >> On Wed, Jun 16, 2010 at 11:51 AM, Martin Paradesi < >> p.martin.samuel.rao(a)gmail.com> wrote: >> >>> Hi, >>> >>> I've been experimenting with Galaxy on the cloud for quite a while. >>> Recently, I faced this problem: >>> >>> Whenever I visit http;//ec2xxxxxxx.com/cloud, I can see the Galaxy >>> cloud console. When I click on "Start cluster" and fill out the number of >>> nodes and size of the cluster, nothing happens. More specifically, I still >>> see the "Start cluster" and do not see the "Access Galaxy" button or >>> whatever used to appear when Galaxy began running. When I go back and look >>> at the EC2 dashboard, I see that only 1 instance is running (the actual >>> Galaxy cloud instance). I refreshed the Galaxy cloud console page several >>> times and waited for about 15 minutes after I entered the required >>> parameters for starting Galaxy cluster. Can you please fix this? >>> >>> Thanks, >>> Martin >>> >>> _______________________________________________ >>> galaxy-dev mailing list >>> galaxy-dev(a)lists.bx.psu.edu >>> http://lists.bx.psu.edu/listinfo/galaxy-dev >>> >>> >> >

1 0

Galaxy cloud problem
by Martin Paradesi 16 Jun '10

16 Jun '10

Hi, I've been experimenting with Galaxy on the cloud for quite a while. Recently, I faced this problem: Whenever I visit http;//ec2xxxxxxx.com/cloud, I can see the Galaxy cloud console. When I click on "Start cluster" and fill out the number of nodes and size of the cluster, nothing happens. More specifically, I still see the "Start cluster" and do not see the "Access Galaxy" button or whatever used to appear when Galaxy began running. When I go back and look at the EC2 dashboard, I see that only 1 instance is running (the actual Galaxy cloud instance). I refreshed the Galaxy cloud console page several times and waited for about 15 minutes after I entered the required parameters for starting Galaxy cluster. Can you please fix this? Thanks, Martin

2 1

add eggs
by Karl Nordström 16 Jun '10

16 Jun '10

Hi, How would I in an easy fashion replace one of the current eggs with a custom one? Is there a any documentation? Cheers, Karl

2 1

Help: History panel of Galaxy interface shows traceback operational error
by wenqing zhang 14 Jun '10

14 Jun '10

Hi; I tried new install in linux centos at school and home. I make everything with the same setting. the difference is that galaxy at home is running well, but the galaxy at school shows the problem, which is the histroy panel of galaxy interface showing the traceback operational error. I didn't change anything else except for modifying the database_connection to local mysql in drupal in universe_wsgi.ini file. Grag mentioned the data in the galaxy database could be corruptted. I couldn't understand how the data was corruptted since we didn't change any data and setting in galaxy. if the data is really corruptted, why the galaxy at home is running well? I appreciate anyone could give any idea. Thanks. wenqing zhang _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:…

2 2

News briefs for Galaxy distribution updates
by Greg Von Kuster 14 Jun '10

14 Jun '10

Hello Galaxy community, Based on your input at the Galaxy Developers Conference at CSHL last month, we are beginning a new process to improve clarification of the contents of new change sets that are available for upgrading your local Galaxy instances. Our initial approach to accomplish this is to send out a news brief like this one on a regular basis ( about every 2 weeks or so ) to let you know what has been committed since the last news brief. Of course, since htis is the first new brief, it'll be a bit different from those that follow in that we'll just communicate the highlights of the new features / fixes over the past 2 weeks. This process will allow you to decide on which of the following options you'd like to use when updating your local Galaxy instance. 1) Use the same process you've always used ( i.e., update at will, or whenever you see email messages come from the http://lists.bx.psu.edu/listinfo/galaxy-commits distribution list, assuming you have subscribed ) 2) Wait for the next news brief, and decide whether you want to update at that time to the specific change set to which it refers. The news brief will provide instructions for updating to a specific change set. We're treating this process as the beginning of our improved communication to you about what's available in upgrades. Please don't hesitate to provide feedback to help improve this process. So now on to our first news brief! ============================ Here are the highlights of what you will get if you perform the following upgrade - the change set was made available to the distribution on June 8, 2010: hg pull -u -r 837aabec314e ------------------ New features ------------------ 1) From the Admin perspective, the library dataset information page ( the page displayed when you click on a dataset link in a library folder ) now includes all of the metadata associated with the dataset, as well as all active links to the dataset ( the undeleted history or library items that point to this dataset's disk file ). Here is an example of a page that displays this new information. Notice that you can click on the History or Library links to see the history or library that contains the item that points to same disk file. 2) The new and vastly improved version of the Galaxy reports webapp. The Galaxy reports are not supported with the sqlite db as they are intended for Galaxy production environments, but postgres will certainly work, and mysql should work as well. The old version of the reports webapp had it's own template path defined in the reports_webapp.ini file, but the new version uses the normal galaxy template location. Due to this change, you should copy reports_webapp.ini.sample to reports_webapp.ini, and change the following setting to be what you want: port = 9001 host = 127.0.0.1 ... # database_connection = postgres:///galaxy_test?user=postgres&password=postgres To start up the reports: %sh run_reports.sh and point your browse at the URL defined by the host and port setting in your config file. The reports are greatly improved, and use the Galaxy grid framework for displaying jobs. You can use the Advanced Search feature to search for a specific job id, and filter by job state, and the various links on the grids allow for other additional filtering and sorting. The new and improved Disk space maintenance report includes a link on each displayed dataset that (when clicked) will display all of the active history items and library datasets that use that specific dataset's disk file. 3) Galaxy now allows for importing and exporting histories. Options for exporting the current history to a file or importing a history from a file are now listed in the history options menu. 4) Add a jar directory / bin which is accessible to tool configurations by specifying e.g. '${GALAXY_DATA_INDEX_DIR}/shared/jars/FOO.jar', where FOO.jar is the desired jar which has been placed in / linked to 'GALAXY_ROOT/tool-data/shared/jars' (by default; $GALAXY_DATA_INDEX_DIR can be configured to point elsewhere [value of 'tool_data_path' configuration variable in universe_wsgi.ini]). The setup.sh script has been modified to create this directory when not present. ----------------------------------------- Existing feature improvements ----------------------------------------- 1) Improvements in Search+Select boxes. 2) Add support for the GFF format to the Galaxy Operation subtract tool 3) Trackster: - Implemented drag-zooming by letting you drag a region in the top label track (where chrom position is shown). Refactored to not use zoom levels anymore. - Can manually set display modes for tracks. - Add text search to trackster dataset selection grid and fix dbkey filtering. - respect block color for bam 4) When use_remote_user = True, still display the logged in user in the User menu 5) Enhancement to the data libraries grid: display the file size of each library dataset in the grid. 6) Enable form builder to build select inputs and (2) enable users to create visualizations via a button on the saved visualizations page. -------------- New tools -------------- 1) SAM indel filter 2) CBI Rice Mart data source 3) Add tools contributed by Chungoo Park for Principal Component Analysis, Linear Discriminant Analysis, and drawing Receiver Operating Characteristic plots. ------------- Bug fixes ------------- 1) Fix autocomplete select not closing after dragging text inside, and holding arrow keys not triggering multiple times in firefox. ============================ Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

2 2

Re: [galaxy-dev] [galaxy-bugs] How to make a data library in a local galaxy instance?
by Greg Von Kuster 14 Jun '10

14 Jun '10

Hello Terry, On Jun 14, 2010, at 8:42 AM, Terrence Barrette wrote: > Thanks Greg, > Had trouble getting this page from the wiki. The search function is focused on bitbucket repositories. How would I find this page in the wiki, to cut down on later questions? Is there a search for the galaxy-central wiki that I am missing? For now, it's best to watch our new features section links on our wiki home page - the section includes a site map link. New features! Read the details of newly introduced features. Check out our preliminary site map > > Looking at this page does it explain where the data going? I need the data to live on my repository shares. How do I direct galaxy to put them there? Better yet how do I direct galaxy to see them there because if possible, I don't want 2 copies of my data library for the sequence data, its over 1 TB. What happens if I paste a bunch of local file URLs in the URL text box? By default, Galaxy stores all data files in the location specified in the universe_wsgi.ini file in this [relative] setting: # Where dataset files are saved file_path = database/files Have you read our wiki discussing the various options for uploading files to a data library? You'll probably want to use the "Upload files from filesystem paths" option and check the "Copy data into Galaxy?" box so the files stay where they are. http://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFiles > > Is there a line by line breakdown of the universe_wsgi.ini file, or section by section? Many more comments will soon be added to the config file to better communicate the uses for each setting. This is on our "short list", so should be available within a couple of weeks. > > Some background context... > I am creating a galaxy instance for the entire lab. I would like to use the initial 8 core server as a starting point. I would like to use our NAS for the data libraries and multiple GB result storage (like mapping results as a secondary lib) and also the local array for most results, but I can direct attach 7+ TB if that is more logical and faster. I would like to direct analysis to dedicated analysis machines for the more intense work like mapping, etc. > > Thanks, > Terry > > > > > > > >>> Greg Von Kuster <greg(a)bx.psu.edu> 6/11/2010 1:10 PM >>> > Terrence, see our wiki at > > http://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/Libraries > > > On Jun 11, 2010, at 12:27 PM, Terrence Barrette wrote: > >> Hi, >> What is the process to make a local data library on my local galaxy server? >> Thanks >> Terry >> >> >> >> ********************************************************** >> Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues >> <Terrence Barrette2.vcf>_______________________________________________ >> galaxy-bugs mailing list >> galaxy-bugs(a)lists.bx.psu.edu >> http://lists.bx.psu.edu/listinfo/galaxy-bugs > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > ********************************************************** > Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues > <Terrence Barrette2.vcf> Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Again on display on external UCSC
by Davide Cittaro 14 Jun '10

14 Jun '10

Hi again... I still have problems in visualizing on external UCSC sites using the display_application... I've implemented bigWig data format, and it works (not really the sniffer but who cares at the moment?).... What is important is that the display_applications gives me a link to the genome browser that is correct: http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7… which points to http://genome.ifom-ieo-campus.it/cgi-bin/hgTracks?db=hg18&hgt.customText=ht… which has this link in hgt.customText http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7… which says track type=bigWig name="wgEncodeYaleChIPseqRawDataRep2Helas3Pol2_fs.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b9b902f807f2/ucsc_bigwig/campus/090e9746a4403c88/data/galaxy_cac7b9b902f807f2.bw db=hg18 Everything is correct, apparently... Needles to say, the bigWig file is fully operational meaning that: 1- if I download the bigDataUrl to another machine and show that file through another apache, the genome browser can read it 2- the original file can be seen and loaded. BTW, the genome browser complains: http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7… is not a chromosome id r-tree index file That probably means that there's something between the genome browser and the bigWig file (galaxy?) which passes wrong data/headers or possibly is passing something *before* the actual bigWig data stream... As I have issues with BAM files too, I believe the cause may be the same... Any help/hint/debug strategy is really appreciated! d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro(a)ifom-ieo-campus.it */

1 2

Re: [galaxy-dev] help: can't connect to external mysql atabase
by Greg Von Kuster 11 Jun '10

11 Jun '10

Again, make sure Galaxy has it's own dedicated database. Also make sure you are running the latest Galaxy code - see getgalaxy.org for instructions on getting the latest code as your traceback infers that you have database tables that are out of sync with your Galaxy code. If you are attempting a new install, drop all of the tables in the database, and start up your Galaxy instance with: %sh run.sh Galaxy will automatically create all of the tables for you, and everything should be fine. On Jun 11, 2010, at 12:03 PM, wenqing zhang wrote: > Hi > > we just modify the path of databse connection in file universe_wsgi.ini without to change anything else, the path connection is to the local database mydatabase, which is new one not without any table inside. the histoy panel shows error traceback, which is operational error (1054, "unknown column 'published' in 'field list' ") u 'insert into histroy (creat_time, update_time, and more ... > > does it mean we need to setup tables in our local database for history or something else? > > does "the data inyour database is corrupted" you mentioned mean data of database in galaxy engine is corrupted or the data in our local database is corrupted ? > > thanks again. > > Subject: Re: [galaxy-dev] help: can't connect to external mysql atabase > From: greg(a)bx.psu.edu > Date: Fri, 11 Jun 2010 11:11:58 -0400 > CC: galaxy-dev(a)bx.psu.edu > To: wqzhang71(a)hotmail.com > > Probably a data problem - the traceback is probably stating that some history or data cannot be located or something, which means that the data in your database is corrupted. > > On Jun 11, 2010, at 10:59 AM, wenqing zhang wrote: > > Hi, > thanks for your response. > > we changed the database path connection to mysql database in drupal as following > database_connection=mysql://tiger:goDB@localhost/mydatabase?unix_socket=/opt/drupal-6.16-0/mysql/tmp/mysql.sock > > it starts to run, but when we take a look at it through the web browser(127.0.0.1:8080), the right column (history panel) of galaxy is not work normally, showing error traceback. is it still connection problem, or database broplem? > > thanks in advance for your more help. > > > > > Subject: Re: [galaxy-dev] help: can't connect to external mysql atabase > From: greg(a)bx.psu.edu > Date: Thu, 10 Jun 2010 18:36:26 -0400 > CC: galaxy-dev(a)lists.bx.psu.edu > To: wqzhang71(a)hotmail.com > > Hello, > > The problem is that the connection string is a URL, it shouldn't have the path to the mysql binary in it (should just be "mysql://tiger:goDB@localhost/mydatabase") Also, Galaxy and drupal are different and unrelated pieces of software, and Galaxy should have its own dedicated database. Galaxy should work with mysql, but we highly recommend postgres. > > > On Jun 9, 2010, at 6:55 PM, wenqing zhang wrote: > > Hi; > > the mysql databse is installed inside drupal in local server. > the mysql path location is /opt/drupal-6.16-0/mysql/bin/mysql. > > when I tried to connect Galaxy to mysql database by modifying database_connection in file universe_wsgi.ini as following > database_connection = /opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase > I fialed to do it. and ArgumentError: could not parse rfc1738URL from string '/opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase' > > someone can help me? > > thanks. > > The New Busy is not the old busy. Search, chat and e-mail from your inbox. Get started. _______________________________________________ > galaxy-dev mailing list > galaxy-dev(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-dev > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > > Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. Learn more. > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > > The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail. Get busy. Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

Re: [galaxy-dev] help: can't connect to external mysql atabase
by Greg Von Kuster 11 Jun '10

11 Jun '10

Probably a data problem - the traceback is probably stating that some history or data cannot be located or something, which means that the data in your database is corrupted. On Jun 11, 2010, at 10:59 AM, wenqing zhang wrote: > Hi, > thanks for your response. > > we changed the database path connection to mysql database in drupal as following > database_connection=mysql://tiger:goDB@localhost/mydatabase?unix_socket=/opt/drupal-6.16-0/mysql/tmp/mysql.sock > > it starts to run, but when we take a look at it through the web browser(127.0.0.1:8080), the right column (history panel) of galaxy is not work normally, showing error traceback. is it still connection problem, or database broplem? > > thanks in advance for your more help. > > > > > Subject: Re: [galaxy-dev] help: can't connect to external mysql atabase > From: greg(a)bx.psu.edu > Date: Thu, 10 Jun 2010 18:36:26 -0400 > CC: galaxy-dev(a)lists.bx.psu.edu > To: wqzhang71(a)hotmail.com > > Hello, > > The problem is that the connection string is a URL, it shouldn't have the path to the mysql binary in it (should just be "mysql://tiger:goDB@localhost/mydatabase") Also, Galaxy and drupal are different and unrelated pieces of software, and Galaxy should have its own dedicated database. Galaxy should work with mysql, but we highly recommend postgres. > > > On Jun 9, 2010, at 6:55 PM, wenqing zhang wrote: > > Hi; > > the mysql databse is installed inside drupal in local server. > the mysql path location is /opt/drupal-6.16-0/mysql/bin/mysql. > > when I tried to connect Galaxy to mysql database by modifying database_connection in file universe_wsgi.ini as following > database_connection = /opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase > I fialed to do it. and ArgumentError: could not parse rfc1738URL from string '/opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase' > > someone can help me? > > thanks. > > The New Busy is not the old busy. Search, chat and e-mail from your inbox. Get started. _______________________________________________ > galaxy-dev mailing list > galaxy-dev(a)lists.bx.psu.edu > http://lists.bx.psu.edu/listinfo/galaxy-dev > > Greg Von Kuster > Galaxy Development Team > greg(a)bx.psu.edu > > > > > Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. Learn more. Greg Von Kuster Galaxy Development Team greg(a)bx.psu.edu

1 0

help: can't connect to external mysql atabase
by wenqing zhang 10 Jun '10

10 Jun '10

Hi; the mysql databse is installed inside drupal in local server. the mysql path location is /opt/drupal-6.16-0/mysql/bin/mysql. when I tried to connect Galaxy to mysql database by modifying database_connection in file universe_wsgi.ini as following database_connection = /opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase I fialed to do it. and ArgumentError: could not parse rfc1738URL from string '/opt/drupal-6.16-0/mysql/bin/mysql://tiger:godatabase@localhost/mydatabase' someone can help me? thanks. _________________________________________________________________ The New Busy is not the old busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:…

2 1