- galaxy-dev - lists.galaxyproject.org

Re: [galaxy-dev] iRODS
by Dan Bedard 13 May '16

13 May '16

The iRODS dev team is certainly interested in how people are using and want to use iRODS with Galaxy. Mikael, we would be happy to host the PDF and/or a guest blog post about making it work. And we can provide some assistance if there are difficulties on the iRODS side of this. Cheers, Dan Bedard — Dan Bedard Executive Director iRODS Consortium mobile: 1-919-346-3262 email: danb(a)renci.org twitter: @DataDanB iRODS User Group Meeting 2016 June 7-9 -- Chapel Hill, NC -- https://irods.org/ugm2016 > Message: 2 > Date: Wed, 9 Mar 2016 01:24:24 +0000 > From: "Kandalaft, Iyad" <Iyad.Kandalaft(a)AGR.GC.CA> > To: James Taylor <james(a)bx.psu.edu>, Mohamed Kassam > <k.mamoud(a)gmail.com> > Cc: "galaxy-dev(a)lists.galaxyproject.org" > <galaxy-dev(a)lists.galaxyproject.org> > Subject: Re: [galaxy-dev] iRODS > Message-ID: > <CA1994D763C7444CB37EFC823D43C3824933F672(a)ONOTTAXES1.AGR.GC.CA> > Content-Type: text/plain; charset="us-ascii" > > We were considering implementing data pull/push tools for Galaxy-iRODS integration but we haven't started work on that. > I recall that the Galaxy team looked at backing Galaxy with iRODS but moved away from the idea. There was a trello card with that information. > > > Iyad Kandalaft > Bioinformatics Systems Developer > Microbial Biodiversity Bioinformatics > Science & Technology Branch > Agriculture & Agri-Food Canada > Iyad.Kandalaft(a)agr.gc.ca | (613) 759-1228 > > ________________________________________ > From: galaxy-dev [galaxy-dev-bounces(a)lists.galaxyproject.org] on behalf of James Taylor [james(a)bx.psu.edu] > Sent: March 8, 2016 10:09 AM > To: Mohamed Kassam > Cc: galaxy-dev(a)lists.galaxyproject.org > Subject: Re: [galaxy-dev] iRODS > > Mohamed, > > We implemented but never deployed the iRODS object store, so I suspect > it has not been tested much. If anyone out there is using it please > let us know. > > On Mon, Mar 7, 2016 at 10:30 AM, Mohamed Kassam <k.mamoud(a)gmail.com> wrote: >> Dear all, >> I would like to use our interlan iRODs to pull data. >> I searched and i find that there is an object store for that. >> How it works ? DO I need to configure something else ? >> >> Thanks in advance, >> >> >> Mohamed >> >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > > ------------------------------ > > Message: 3 > Date: Wed, 9 Mar 2016 12:01:38 +0100 > From: Mikael Loaec <mikael.loaec(a)versailles.inra.fr> > To: galaxy-dev(a)lists.galaxyproject.org > Cc: Olivier Inizan <Olivier.Inizan(a)versailles.inra.fr> > Subject: [galaxy-dev] iRODS > Message-ID: <56E00292.5070604(a)versailles.inra.fr> > Content-Type: text/plain; charset=utf-8 > > Hi, > > @URGI, we have sucessfully tested connexion of galaxy instance with an > iRODS. > We have a pdf doc 67ko) explaining how we implement it if you want, but > don't know if i can send attachement on this mailinglist ? > > We (URGI) are also very interresting in data pull/push tools for > Galaxy-iRODS integration. Let us know when you will begin working on it. > > Regards, > > Mikael LOAEC > Systems and Networks Administrator > URGI INRA > > > ------------------------------ > > Message: 4 > Date: Wed, 9 Mar 2016 12:43:36 +0100 > From: Mohamed Kassam <k.mamoud(a)gmail.com> > To: Mikael Loaec <mikael.loaec(a)versailles.inra.fr> > Cc: galaxy-dev(a)lists.galaxyproject.org, Olivier Inizan > <Olivier.Inizan(a)versailles.inra.fr> > Subject: Re: [galaxy-dev] iRODS > Message-ID: > <CAJrYSO8e2dfb2JjdAqCB9CFkg=NELV8vrBCrwnQQ-FpKjU6Pkw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Thank you for your message, > > I sent to you Mikael an email if you can semd your pdf. > > It is a good idea to see if in The galaxy dev team they can integrate iRODS > for the next release. > > Best regards, > > Mohamed > > 2016-03-09 12:01 GMT+01:00 Mikael Loaec <mikael.loaec(a)versailles.inra.fr>: > >> Hi, >> >> @URGI, we have sucessfully tested connexion of galaxy instance with an >> iRODS. >> We have a pdf doc 67ko) explaining how we implement it if you want, but >> don't know if i can send attachement on this mailinglist ? >> >> We (URGI) are also very interresting in data pull/push tools for >> Galaxy-iRODS integration. Let us know when you will begin working on it. >> >> Regards, >> >> Mikael LOAEC >> Systems and Networks Administrator >> URGI INRA >> ___________________________________________________________ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > -------------- next part --------------

3 2

Re: [galaxy-dev] Problem uploading multigigabyte files via API
by Sytchev, Ilya 12 May '16

12 May '16

Just to follow up: we've implemented a workaround for uploading large files into libraries using upload_file_from_url(): https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.libraries.LibraryClient.upload_file_from_url Also, there is a corresponding function for uploading files into histories: https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend .galaxy.tools.ToolClient.put_url Ilya -----Original Message----- From: galaxy-dev <galaxy-dev-bounces(a)lists.galaxyproject.org> on behalf of Ilya Sytchev <isytchev(a)hsph.harvard.edu> Date: Wednesday, April 27, 2016 at 15:53 To: "galaxy-dev(a)lists.galaxyproject.org" <galaxy-dev(a)lists.galaxyproject.org> Subject: [galaxy-dev] Problem uploading multigigabyte files via API >Hi, > >I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came >across an issue where uploading files larger than ~3GB via the API into >histories and libraries fails. > >Steps to reproduce (using BioBlend 0.7.0): > >from bioblend.galaxy.objects import GalaxyInstance >gi = GalaxyInstance("galaxy-dev", "api_key") >h = gi.histories.create() >h.upload_dataset('/path/to/multigigabytefile') > >Observed results: > >ConnectionError: Unexpected response from galaxy: 504: <html> > ><head><title>504 Gateway Time-out</title></head> ><body bgcolor="white"> ><center><h1>504 Gateway Time-out</h1></center> ><hr><center>nginx/1.4.6 (Ubuntu)</center> ></body> ></html> > >It looks like Nginx is accepting the full incoming data transfer over the >network and saving it to disk successfully, and the error occurs during >the subsequent forwarding of the request to Galaxy. There doesn't seem to >be a particular threshold for file size (for example, 2GB and 2.7GB file >can be uploaded successfully). Also, large file uploads do work when >Galaxy is not running behind Nginx. > > >Relevant log entries and config details are attached. I'd appreciate any >help. > >Thanks, >Ilya > >

1 0

Problems getting Galaxy AMI running in AWS
by Quentin Spencer 12 May '16

12 May '16

Hi, I am having fairly fundamental problems getting the Galaxy Amazon Machine Image (AMI) running in Amazon Web Services (AWS). I followed the instructions at https://wiki.galaxyproject.org/CloudMan/AWS/GettingStarted but get the same errors trying to start up Galaxy each time. Please see https://biostar.usegalaxy.org/p/17582/ for full details of the errors I'm seeing. I have tried the last two public AMIs, Galaxy-CloudMan-1457720469 and Galaxy-CloudMan-1449500413 and am seeing the same errors with each of them. Thank you in advance for any help! Quentin Spencer

1 0

Data manager problem
by Scott Szakonyi 11 May '16

11 May '16

Hello, I'm having an odd problem after updating to the latest version of Galaxy. My users prepare data for analysis using data manager tools, such as "data_manager_bwa_mem_index_builder". When an attempt is made to run one of these tools, a pop up window opens stating "Tool request failed" and underneath "Uncaught error". I've pored over all the Galaxy system logs and can't find anything that might indicate what is happening. I've uninstalled and reinstalled the data manager tools, restarted both the Galaxy service and the server, to no avail. I get this same error with any installed data manager, yet all the standard tools are executing correctly and don't seem to be having any issues. If anyone can offer some insight or suggestions for resolving this issue, I'd be most appreciative. Best regards, -- Scott B. Szakonyi Research Programmer *Center for Research Computing* 107 Information Technology Center Notre Dame, IN 46556 http://crc.nd.edu

3 5

UTF8 problem with database "An error occured when getting updates from the server"
by julie dubois 11 May '16

11 May '16

Dear all, I have just changed my galaxy database encoding in utf8 to avoid error of loading histories. It works fine. But when I run a new job after this change, the bad characters re-appear in my output and the error "An error occured when getting updates from the server" re-appear too. When I've changed my galaxy database envoding, I've modified the /etc/mysql/my.cnf file and replace all of the bad charcater in the previous database before re-install it. Any idea of a change I omit ? Thanks. Julie

2 2

Galaxy not starting after upgrading to v16.01
by Tiago Macedo 10 May '16

10 May '16

Hi guys, after the upgrade to v16.01 I'm getting an error related to pbs-python when starting the job runners (the web runners seem fine). Here's the relevant part of the log file: galaxy.jobs.manager DEBUG 2016-05-10 15:52:49,972 Starting job handler galaxy.jobs INFO 2016-05-10 15:52:49,972 Handler 'job0' will load all configured runner plugins Traceback (most recent call last): File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/webapps/galaxy/buildapp.py", line 63, in paste_app_factory app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/app.py", line 150, in __init__ self.job_manager = manager.JobManager( self ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/jobs/manager.py", line 23, in __init__ self.job_handler = handler.JobHandler( app ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/jobs/handler.py", line 32, in __init__ self.dispatcher = DefaultJobDispatcher( app ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/jobs/handler.py", line 723, in __init__ self.job_runners = self.app.job_config.get_job_runner_plugins( self.app.config.server_name ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/jobs/__init__.py", line 671, in get_job_runner_plugins rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get( 'kwds', {} ) ) File "/afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/lib/galaxy/jobs/runners/pbs.py", line 94, in __init__ assert pbs is not None, PBS_IMPORT_MESSAGE AssertionError: The Python pbs-python package is required to use this feature, please install it or correct the following error: ImportError /afs/igc.gulbenkian.pt/folders/UBI/SERVICES/GALAXY/galaxy_dist/.venv/local/lib/python2.7/site-packages/pbs/_pbs.so: undefined symbol: job_log_mutex Removing PID file /var/run/galaxy/galaxy-job0.pid Any ideas on how to solve this? Thanks, Tiago Macedo

1 0

a quick question about picard
by Beginner TI 09 May '16

09 May '16

Hi All, I wrote a very simple wrapper for one picard function. Here is the xml file: <tool name="CreateSequenceDictionary" id="picard_CreateSequenceDictionary" version="1.126.0"> <description>Create Sequence Dictionary</description> <requirements> <requirement type="package" version="1.126.0">picard</requirement> </requirements> <macros> <import>picard_macros.xml</import> </macros> <command> @java_options@ java -jar \$JAVA_JAR_PATH/picard.jar CreateSequenceDictionary R="${ref_file}" O="${dict_file}" </command> <inputs> <param format="fa" name="ref_file" type="data" label="Using reference genome" help="-R,--reference_sequence" /> </inputs> <outputs> <data format="txt" name="dict_file" /> </outputs> <help> **Purpose** Create Sequence Dictionary from fa file </help> </tool> It's ugly, I know. :-( Just want to finish a quick test...however when I tried to run a test, it always says something like: Exception in thread "main" picard.PicardException: /media/4tb-ext/galaxy-data-store/database/files/008/dataset_8757.dat already exists. Delete this file and try again, or specify a different output file When I looked at that file, it has a size 0, but even I remove that file, I'll still see the same thing. Galaxy terminal executes: galaxy.jobs.runners.local DEBUG 2016-04-29 13:55:51,423 (3189) executing: GALAXY_SLOTS="1"; export GALAXY_SLOTS; _JAVA_OPTIONS=${_JAVA_OPTIONS:-'-Xmx2048m -Xms256m'} && export _JAVA_OPTIONS && java -jar $JAVA_JAR_PATH/picard.jar CreateSequenceDictionary R="/media/mybook/GATK_galaxyWrapper/hg38.fa" O="/media/4tb-ext/galaxy-data-store/database/files/008/dataset_8757.dat" Seems it did the right thing, but why that set_meta set state to ERROR? Oh there was indeed a warning: WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'eset', using datatype from /media/4tb-ext/galaxy-data-store/database/tmp/tmp_Uuw5Y. Have anyone seen this before? I'm quite confused since the same commandline runs smoothly in the terminal. Thanks, Rui

2 1

Adding Reference genome of Chicken and Duck in Tophat
by Vijay Kumar 09 May '16

09 May '16

Dear sir, Kindly add reference Genome of Chicken (Gallus gallus) and Duck (Anas platyrhynchos) in Tophat Reference genome section. *Regards,* *Dr. P. Vijayakumar,* *Assistant Professor,* *Instructional Livestock Farm Complex, * *Veterinary College and Research Institute,* *Orathanadu, **Thanjavur,* *Tamil Nadu,* *India- 614625*

2 1

Proper Galaxy Installation
by Durdevic, Marija 09 May '16

09 May '16

Dear all, I am currently working on Galaxy installation at our Center for Medical Research. As a login server, we are using VM based on SLES-12. We are planning to extend it on a Cluster later and we would like to use SLURM as DRM. Is there any recommendation or "step-by-step" recipe how to setup Galaxy in a proper way to avoid future issues? Are they any recommendation regarding the filesystem of directory management system that I should care about in advance? Any advice from experienced developers will be more than helpful. Thank you in advance. Regards, Mag. Marija Đurđević Core Facility Computational Bioanalytics Medical University of Graz Center for Medical Research Stiftingtalstraße 24, A-8010 Graz Austria Phone: +43 316/385-73024 Fax:+43 316/385-73009 Email: marija.djurdjevic(a)medunigraz.at Email: marija.djurdjevic(a)klinikum-graz.at Web: https://zmf.medunigraz.at/

2 2

Fwd: Error when uploading files (using Pulsar)
by Marius van den Beek 09 May '16

09 May '16

Ah, and if you want to figure out the ID of a tool, you can click the small info button of a dataset produced by the tool, so in the case of the upload tool it reads: Galaxy Tool ID: upload1 On 9 May 2016 at 12:25, Marius van den Beek <m.vandenbeek(a)gmail.com> wrote: > Puhh, you're right, and somewhere in the undepths of the wiki (I think) > this is documented, > but admittedly I didn't find it when searching for it. > If you want you could add a sample of your config to > https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample… > , > I believe this is something that the galaxy devteam carefully keeps up to > date. > You would definitely have my +1 for a Pull Request. If you don't have the > time I can also do this for you. > > Marius > > On 9 May 2016 at 12:02, Tiziano Flati <tiziano.flati(a)gmail.com> wrote: > >> Thank you very much, Marius, that worked! >> >> But how should I have known about the exact ID of the upload tool (i.e., >> upload1)? I think this is an extremely common pattern (upload data and >> execute jobs on a cluster) >> >> 2016-05-09 11:38 GMT+02:00 Marius van den Beek <m.vandenbeek(a)gmail.com>: >> >>> Hi Tiziano, >>> >>> I think you are correct in your assumption. >>> If you do not need to have uploads run on pulsar, you should be able to >>> specify a local destination for uploads (the upload tool id is upload1) >>> in your job_conf.xml. >>> There are some examples described here: >>> >>> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample… >>> If you can place a copy of galaxy and a virtualenv on your pulsar >>> server, you could also set this up as in >>> >>> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample… >>> >>> Note that I haven't tried this yet, but I think this is a good start. >>> Let us know if that works. >>> >>> Cheers, >>> Marius >>> >>> >>> On 9 May 2016 at 10:42, Tiziano Flati <tiziano.flati(a)gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I have succesfully setup a Galaxy-Pulsar architecture and I am able to >>>> run jobs over datasets* already uploaded on a history*. >>>> >>>> When I try to upload a new file to the history, though, the upload job >>>> fails with the following error: >>>> >>>> Traceback (most recent call last): >>>>> File "/home/flati/pulsar/files/staging/80/tool_files/upload.py", >>>>> line 18, in <module> >>>>> import galaxy.model # noqa >>>>> ImportError: No module named model >>>> >>>> >>>> Note: in job_conf.xml, Pulsar is the default destination: >>>> >>>> <destinations default="win_pulsar"> >>>> >>>> >>>> Does someone know what the problem is? >>>> I suspect that setting pulsar as the default destination causes the >>>> upload tool to run on Pulsar's side which, however, does not have access to >>>> Galaxy's lib directory (which contains the galaxy model module). >>>> >>>> Any help is very appreciated, >>>> Tiziano >>>> >>>> ___________________________________________________________ >>>> Please keep all replies on the list by using "reply all" >>>> in your mail client. To manage your subscriptions to this >>>> and other Galaxy lists, please use the interface at: >>>> https://lists.galaxyproject.org/ >>>> >>>> To search Galaxy mailing lists use the unified search at: >>>> http://galaxyproject.org/search/mailinglists/ >>>> >>> >>> >> >

1 0