- galaxy-dev - lists.galaxyproject.org

Problem with downloading datasets using wget or curl
by Ashok Varadharajan 22 Mar '17

22 Mar '17

Dear Team, We are running galaxy-16.07 behind the proxy using nginx, uwsgi and supervisor in scientific linux 7. I have problem with downloading datasets using wget. I am getting the following html for all my datasets in the galaxy when I try to download it using wget [for example: wget http:// {GALAXYURL}/galaxy/datasets/xxxxxxxxxxxx/display?to_ext=tabular] Can anyone help me to solve this issue ? <!DOCTYPE HTML> <html>  <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="viewport" content="maximum-scale=1.0"> <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1"> <title>Galaxy</title> <link rel="index" href="/galaxy/"/> <link href="/galaxy/static/style/jquery.rating.css?v=1489572328" media="screen" rel="stylesheet" type="text/css" /> <link href="/galaxy/static/style/jquery-ui/smoothness/jquery-ui.css?v=1489572328" media="screen" rel="stylesheet" type="text/css" /> <link href="/galaxy/static/style/base.css?v=1489572328" media="screen" rel="stylesheet" type="text/css" /> <link href="/galaxy/static/style/bootstrap-tour.css?v=1489572328" media="screen" rel="stylesheet" type="text/css" /> <script type="text/javascript"> // this is needed *before* the app code is loaded - many MVC access Galaxy.root for their url // TODO: change this by using a common Backbone.Model base class and url fn window.Galaxy = { root: '/galaxy/' }; </script> </head> <body scroll="no" class="full-content"> <div id="everything" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"> <div id="background"></div> <div id="masthead" class="navbar navbar-fixed-top navbar-inverse"></div> <div id="messagebox" style="display: none;"></div> <div id="inactivebox" class="panel-warning-message" style="display: none;"></div> </div> <div id='dd-helper' style="display: none;"></div> <noscript> <div class="overlay overlay-background noscript-overlay"> <div> <h3 class="title">Javascript Required for Galaxy</h3> <div> The Galaxy analysis interface requires a browser with Javascript enabled.<br> Please enable Javascript and refresh this page. </div> </div> </div> </noscript> <script src="/galaxy/static/scripts/bundled/libs.bundled.js?v=1489572328" type="text/javascript"></script> <script src="/galaxy/static/scripts/bundled/login.bundled.js?v=1489572328" type="text/javascript"></script> <script type="text/javascript"> app( {"redirect": "/galaxy/datasets/xxxxxxxxxxxxxxxx/display", "show_welcome_with_login": false, "user": {"quota_percent": null, "id": null, "total_disk_usage": 0.0, "nice_total_disk_usage": "0 bytes"}, "openid_providers": [], "root": "/galaxy/", "config": {"communication_server_host": "http://localhost", "ftp_upload_site": null, "mailing_lists": "https://wiki.galaxyproject.org/MailingLists", "nginx_upload_path": "/galaxy/api/tools", "terms_url": null, "enable_unique_workflow_defaults": false, "wiki_url": " https://wiki.galaxyproject.org/", "version_major": "16.07", "support_url": " https://wiki.galaxyproject.org/Support", "persistent_communication_rooms": [], "remote_user_logout_href": null, "message_box_content": null, "logo_src": "/galaxy/static/images/galaxyIcon_noText.png", "biostar_url_redirect": "http://{GALAXY_URL}/galaxy/biostar/biostar_redirect", "message_box_visible": false, "ftp_upload_dir": "{ftp_upload_dir}", "logo_url": "/galaxy/", "brand": "{BRAND}", "search_url": " http://galaxyproject.org/search/usegalaxy/", "screencasts_url": null, "ga_code": null, "require_login": true, "communication_server_port": 7070, "lims_doc_url": "https://usegalaxy.org/u/rkchak/p/sts", "inactivity_box_content": "Your account has not been activated yet. Feel free to browse around and see what's available, but you won't be able to upload data or run jobs until you have verified your email address.", "citation_url": "https://wiki.galaxyproject.org/CitingGalaxy", "allow_user_creation": false, "is_admin_user": false, "allow_user_dataset_purge": true, "server_startttime": 1489572328, "biostar_url": null, "datatypes_disable_auto": false, "use_remote_user": null, "message_box_class": "info"}, "welcome_url": "/galaxy/root/welcome"}, {} ); </script> </body> </html> *Thanks and Regards,* *Ashok Varadharajan*

1 0

bowtie2 indexes & errors
by Nikolaos Tur 21 Mar '17

21 Mar '17

Hi, I observe next error: [2017-03-21 10:30:56] Beginning TopHat run (v2.1.0) ----------------------------------------------- [2017-03-21 10:30:56] Checking for Bowtie Bowtie version: 2.2.5.0 [2017-03-21 10:30:56] Checking for Bowtie index files (genome).. Error: Could not find Bowtie 2 index files (/home/galaxy/tool-data/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84.*.bt2) And this error is clear: no index files with *.bt2. But we have for GRCm38_84: ../GRCm38_84/GRCm38_84.3.bt2l ../GRCm38_84/GRCm38_84.rev.1.bt2l ../GRCm38_84/GRCm38_84.1.bt2l ../GRCm38_84/GRCm38_84.rev.2.bt2l ../GRCm38_84/GRCm38_84.2.bt2l ../GRCm38_84/GRCm38_84.4.bt2l ../GRCm38_84/GRCm38_84.fa as I understand from the manual for bowtie2 "l" at the end of files means that indexes were created for large file. If it is correct way then why Galaxy tools do not understand this extension and how it should be solve? Okay I have simply created symbolic links to them with *.bt2 extensions (I don't know if *.tb2 and *.tb2l have the same format) and run job again: [2017-03-21 11:40:41] Beginning TopHat run (v2.1.0) ----------------------------------------------- [2017-03-21 11:40:41] Checking for Bowtie Bowtie version: 2.2.5.0 [2017-03-21 11:40:41] Checking for Bowtie index files (genome).. [2017-03-21 11:40:41] Checking for reference FASTA file [2017-03-21 11:40:41] Generating SAM header for /home/galaxy/tool-data/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84 [2017-03-21 11:40:50] Preparing reads left reads: min. length=101, max. length=101, 37099122 kept reads (11326 discarded) right reads: min. length=101, max. length=101, 36957824 kept reads (152624 discarded) [2017-03-21 12:11:25] Mapping left_kept_reads to genome GRCm38_84 with Bowtie2 [FAILED] Error running bowtie: Error reading _plen[] array: 895748346, 4294967292 Error: Encountered internal Bowtie 2 exception (#1) Command: /home/galaxy/tools/dependencies/bowtie2/2.2.5/iuc/package_bowtie_2_2_5/30bd7eaeddbf/bin/bowtie2-align-s --wrapper basic-0 -k 20 -D 15 -R 2 -N 0 -L 20 -i S,1,1.25 --gbar 4 --mp 6,2 --np 1 --rdg 5,3 --rfg 5,3 --score-min C,-14,0 -p 8 --sam-no-hd -x /home/galaxy/tooldata/GRCm38_84/bowtie2_index/GRCm38_84/GRCm38_84 - (ERR): bowtie2-align exited with value 1 If this is not an extension problem then what is an origin of this error and how to solve it? Thanx, Nikolaos

1 0

Conda dependency seemingly installed but failed to build job environment.
by Alexandre Defelicibus 20 Mar '17

20 Mar '17

Hi everyone, I'm trying to develop a suite of tools using planemo and conda to resolv the dependencies. However, I'm getting the following errors: PaddingError: Placeholder of length '80' too short in package /home/alexandre/miniconda2/envs/mulled-v1-7310548f515d1cc3a7cb34c2a7ebfca0f3d7105c7b326b09226b07ee94a5a142/bin/Rscript. The package must be rebuilt with conda-build > 2.0. Then, the process finishes with the error: DependencyException: Conda dependency seemingly installed but failed to build job environment. The last test output is attached. I'm using Galaxy 17.01 and I've tried to use the solution presented here <https://github.com/galaxyproject/galaxy/issues/3078>, but I got no success. I've set the Galaxy config but it seems that planemo hasn't using this values: conda_prefix = /home/alexandre/_conda conda_exec = /home/alexandre/_conda/bin/conda conda_debug = True conda_ensure_channels = iuc,bioconda,r,defaults,conda-forge conda_auto_install = True conda_auto_init = True use_cached_dependency_manager = True tool_dependency_cache_dir = /home/alexandre/_conda/_cache I'm running the planemo test with the line: planemo t --galaxy_root <galaxy_17.01_dir> Furthermore, the installation process of conda dependencies take so long every time I run the test. It is normal? The project is on Github on this link <https://github.com/adefelicibus/woland-galaxy/tree/galaxy-tools/tools/wolan…> . I don't know if I develop these tools using only conda or if I use TS to solve dependencies. I'll appreciate any help! Thank you. -- Alexandre Defelicibus, Me.

1 0

GSoC 2017: Project Publication Reference Tracking: Building a community
by Dave Clements 20 Mar '17

20 Mar '17

Hi All, As you may have heard the Open Genome Informatics proposal <http://gmod.org/wiki/GSoC> was accepted for GSoC 2017. That proposal covers all the GMOD projects, including Galaxy. One of projects in the proposal was: *Project Publication Reference Tracking (Galaxy/Reactome) <http://gmod.org/wiki/GSOC_Project_Ideas_2017>* Assuming that this item gets prioritized for funding, and that we find a great student, then this will be worked on this summer. *Hopefully we'll end up with a great tool that any open source community can use to keep track of publications that reference it.* *If having a tool that helps you track publications that use your resource interests you then please keep reading below. * If not, then thanks for your time, Dave C *Building a community* I've been reading the GSoC Mentoring Manual <http://write.flossmanuals.net/gsoc-mentoring/> (one of the authors is Duke Leto) and one of the things it suggests is getting the student engaged with the community. The community for this proposal isn't Galaxy or Reactome or GMOD per se (although GMOD is pretty close). The community is folks at open source projects that want to track references to their projects (which includes those who run bio DBs, public Galaxy Servers, and the various GMOD component projects). I'd like to build that community now. *If you are interested in this topic, and maybe in helping to provide guidance for a GSoC student then please let me know.** If I hear from enough people I'll create a mailing list. If not, we'll just use email (and probably a Gitter channel). Also, if anyone knows of tool that already does this, please let me know. I don't want to reinvent the wheel this summer. Galaxy does this using lots of email alerts, CiteULike, and some custom scripts (they work sort of, but oh the horror!). I've got a 6 page doc describing the current process that I've sent to every potential GSoC student who has contacted me. First thing I'll do is send that to those that are interested. Thanks, Dave C. * You can also post to the whole list, but I'm trying to cut down on noise in people's email. -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/

1 0

startup bug in lib/galaxy/tools/toolbox/base.py
by Matthias Bernt 16 Mar '17

16 Mar '17

Dear galaxy developers, I sent this originally to the galaxy-bugs list and was asked to forward this also to this list. What is the best way to report bugs? There is galaxy-bugs, galaxy-dev and also the issue tracker on galaxy. During startup of my galaxy 17.01 installation an error message pops up: galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,605 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/bwa/53646aaaafef/bwa/bwa.xml Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' Seems that tool_shed_repository is still None at this point. This happens for some tools. galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,764 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/bwa/53646aaaafef/bwa/bwa-mem.xml Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,768 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/fastqc/a00a6402d09a/fastqc/rgFastQC.xml Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,773 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/25c24379693a/… Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,779 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_quality/04a609b0fdf6/… Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,788 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer/e0cfb5a703ce/fastq_trimm… Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,793 Error reading tool from path: toolshed.g2.bx.psu.edu/repos/geert-vandeweyer/fastq_qc_trimmer/cba6282b5dc8… Traceback (most recent call last): File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560, in _load_tool_tag_set tool.tool_shed = tool_shed_repository.tool_shed AttributeError: 'NoneType' object has no attribute 'tool_shed' Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

2 2

Re: [galaxy-dev] Shared Data Library, large number of files
by Martin Čech 15 Mar '17

15 Mar '17

Can you please share your '/client/galaxy/scripts/layout/menu.js' file with me? (e.g. using https://gist.github.com or similar). thanks M. On Wed, Mar 8, 2017 at 1:34 PM D K <danielfortin86(a)gmail.com> wrote: > Hi Martin, > > I've tried this on both v16.07 and v16.10 > > D > > > > On Wed, Mar 8, 2017 at 10:29 AM, Martin Čech <marten(a)bx.psu.edu> wrote: > > Daniel: > > what version is your Galaxy? > > Martin > > On Wed, Mar 8, 2017 at 12:40 PM D K <danielfortin86(a)gmail.com> wrote: > > I've copied and pasted the javascript error below. Any help would be > greatly appreciated. Thanks! > > Uncaught TypeError: Cannot read property 'masthead' of undefined > at n.<anonymous> (menu.js:120) > at Function.each (jquery.js:370) > at HTMLDocument.<anonymous> (menu.js:117) > at l (jquery.js:3232) > at Object.add [as done] (jquery.js:3291) > at t.fn.init.vt.fn.ready (jquery.js:3542) > at callback (menu.js:116) > at Object.t.success (backbone.js:1051) > at l (jquery.js:3232) > at Object.fireWith [as resolveWith] (jquery.js:3362) > > > On Thu, Mar 2, 2017 at 10:43 AM, D K <danielfortin86(a)gmail.com> wrote: > > Hi Martin, > > I do get a javascript error, I've attached a screenshot of the relevant > window. I also tried logging in as a non-admin user and had the same issue. > In this Galaxy instance are 5 data libraries, each with about 150 > individual folders, and each folder has a variable number of fastq.gz files. > > Thanks! > > On Thu, Mar 2, 2017 at 10:13 AM, Martin Čech <marten(a)bx.psu.edu> wrote: > > Daniel: > > Could you please open browser console and look for javascript errors on > the page that fails to load for you? Additionally I assume you are an admin > when trying this (admins load all of the instance's libraries) could you > try loading the library list as a normal user? How many libraries do you > have on the instance? > > Thanks > > Martin > > On Thu, Mar 2, 2017 at 12:38 PM D K <danielfortin86(a)gmail.com> wrote: > > Hi Martin / Hans / Devon, > > I can confirm that I am using the new UI from the top menu. Right now when > I click on the link I often do not even get the list of libraries whereas > when I click on the admin Data Library link I at least see the list of > libraries (although I can't open any of them since they are so large). I've > tried refreshing the page, deleting the cookies for the session and using > Incognito mode, using chrome and firefox both from a linux and windows > machine and can confirm this behavior on all systems. However, occasionally > when clicking the Shared Data Library link from the top I do get a list of > libraries and can navigate them, but this has only happened once during my > tests and I cannot replicate this behavior. > > Thanks for the assistance! > > On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten(a)bx.psu.edu> wrote: > > Hi all, > > *Devon* is correct -the old data library UI under Admin menu is still in > place (because it has some features perceived as important to some admins). > However majority of users and most of the admins are using the interface > under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( > https://docs.galaxyproject.org/en/master/releases/15.10_announce.html) I > created a PR to adjust the label in the admin interface to make this clear: > https://github.com/galaxyproject/galaxy/pull/3694 > > The old UI is very slow for large libraries but the new UI under Shared > Data/Data Libraries in the top menu shouldn't have this kind of issues for > hundreds of folders/datasets that are mentioned here. > > *Daniel* are you using the new UI form the top menu? If so are you > experiencing this only with a specific folder full of linked datasets or > does it occur for any folder with many datasets in it? > > Martin > > > > On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan(a)dpryan.com> wrote: > > I observe the same thing on 16.07. As an example, I'm just opened a > data library under Admin->Data->Data Libraries and this took about a > minute and a half. It looks like this is related to the quantity of > data in this folder. The associated entry in the uwsgi log is as > follows: > > [pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} > [Thu Mar 2 08:40:19 2017] GET > > /library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff > => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers > in 73 bytes (14 switches on core 2) > > So this library is causing ~21 megs of data to be returned to the > browser, which I presume is the source of the issue. > > As an aside, the old data library API is still in use in the admin > section until at least 16.07. Perhaps this has been changed since > then, I hadn't planed to update our system until the end of the month. > For what it's worth, opening the same library via "shared data" takes > ~100 msec. > > Devon > -- > Devon Ryan, Ph.D. > Email: dpryan(a)dpryan.com > Data Manager/Bioinformatician > Max Planck Institute of Immunobiology and Epigenetics > Stübeweg 51 > 79108 Freiburg > Germany > > > On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh(a)fmi.ch> wrote: > > > > this sounds more like a browser issue to me > > which browser do you use? what happens when you use a different browser? > and > > what happens when you just reload the page > > > > > > Hans-Rudolf > > > > On 03/01/2017 06:17 PM, D K wrote: > >> > >> Actually, when I click on the Shared Data-> Data Libraries link I get > >> only the Galaxy banner and a blank page, so I'm not even seeing the data > >> libraries themselves. > >> > >> If I use the Admin link and then click Data libraries, I can see the > >> actual libraries. But when I try to enter one of the libraries with lots > >> of files it hangs. > >> > >> > >> On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86(a)gmail.com > >> <mailto:danielfortin86@gmail.com>> wrote: > >> > >> Hi Hans, > >> > >> I'm running 16.10. If I'm not mistaken, the old Data Library > >> interface is still accessible in the Admin menu isn't it? I have the > >> same issue when navigating to a Data Library either from the Admin > >> or Shared Data links. I also tried this in 16.07 with the same > >> result. If it makes a difference, I'm using PostgreSQL and Apache. > >> > >> Dwight > >> > >> > >> > >> On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh(a)fmi.ch > >> <mailto:hrh@fmi.ch>> wrote: > >> > >> Hi Daniel > >> > >> Just double checking which version of galaxy are you running? In > >> other words: are you still using the old Data Library interface? > >> (if I remember correctly, the old UI was deprecated in 15.10) > >> We had similar issues with the old UI. > >> > >> The new one works fine and is very fast. E.g. we have a 'MISEQ' > >> folder with more than 100 sub folders and a HISEQ folder with > >> more than 600 sub folders. Each containing up to 100plus fastq > >> and text files. > >> > >> Regards, Hans-Rudolf > >> > >> > >> > >> On 03/01/2017 12:13 AM, D K wrote: > >> > >> Hi, > >> > >> I've noticed that the more files / folders (think hundreds > >> of folders > >> each with 20+ fastq files) I add to a Shared Data Library, > >> the longer > >> and longer it takes to initially navigate the folder > >> structure (after > >> clicking on the specific Shared Data Library). Just opening > >> up the > >> Shared Data Library can take several minutes and often the > >> browser times > >> out. Is there a way to speed this up? > >> > >> If it's relevant, when I add shared data, it's linked rather > >> than copied > >> into Galaxy. Any suggestions would be greatly appreciated > >> > >> Thanks! > >> > >> > >> ___________________________________________________________ > >> Please keep all replies on the list by using "reply all" > >> in your mail client. To manage your subscriptions to this > >> and other Galaxy lists, please use the interface at: > >> https://lists.galaxyproject.org/ > >> <https://lists.galaxyproject.org/> > >> > >> To search Galaxy mailing lists use the unified search at: > >> http://galaxyproject.org/search/mailinglists/ > >> <http://galaxyproject.org/search/mailinglists/> > >> > >> > >> > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > > > > >

3 4

BioBlend: getting a list of datasets in a folder
by D K 15 Mar '17

15 Mar '17

I'm trying to get a list of all the datasets in a folder using BioBlend. I don't see a function for doing this in the API documentation. I can use "show_folder" to get some information regarding the contents of a folder, but not the dataset IDs themselves This is as far as I've gotten: s = gi.libraries.show_folder(LibID, folder_id = subfolderID) > print(s) {u'parent_library_id': u'd413a19dec13d11e', u'update_time': > u'2017-02-24T18:10:02.789955', u'description': > u'170124_M02550_0174_000000000-AV8FJ', u'name': > u'170124_M02550_0174_000000000-AV8FJ', u'deleted': False, u'item_count': 2, > u'parent_id': u'Ff1b9846ab84237e7', u'genome_build': u'?', u'model_class': > u'LibraryFolder', u'id': u'F84ffe6fca6c4fbda', u'library_path': [u'2017', > u'170124_M02550_0174_000000000-AV8FJ']} There are two items (item_count) as expected, how do I get the actual dataset IDs? Thanks!

1 2

galaxy user data on file system with limited file lifetime
by Matthias Bernt 15 Mar '17

15 Mar '17

Dear galaxy developers, The question in short: How can galaxy user data (e.g. file_path) be stored safely on a file system where files have a limited life time? Galaxy will run on a cluster (~2000 cores) head node where data is stored at three points: /home (Individual User Homes) with 50 GB quota /data (Research Group Directories) with 4 up to 100 TB quota /work (User Directories - Temporary File Area) 60 days life time We plan to store galaxy related data as follows: /work/galaxy/files <- file_path /work/galaxy/tmp <- new_file_path /work/galaxy/jobs <- job_working_directory As a note: Storing these in /data/ would undermine our quota system which our admins do not like. /data/galaxy will contain the galaxy installation including tool data (and I hope that we can just set the quotas high enough to never run out of space). Data libraries will be added using the "link mechanism" from /home/USER and /data/GROUP. I hope that I can automatize import and appropriate setting of permissions via the API / bioblend. Are there already scripts? Is this scheme reasonable? If yes: The main question is how I can guarantee that the life time of data of /work/ and the galaxy server play nice together. My idea consists of two parts: 1. Adapt cleanup_datasets.py (i.e. the function purge_histories) such that all histories (also those that have not been deleted) are purged which are at the file system life time. The modification seems to be to remove the test: app.model.History.table.c.deleted == true() At the same time the included data sets will be purged. 2. Using the API I will get the update time of each history or the update time of the youngest included data set (or is it the same anyway). For the files corresponding to the included data sets I will update the access times in the file systems. Such I will guarantee that only complete histories are purged. The script(s) can then be run via cron with a life time set to 1 day less than the file system life time (just to be sure). In theory jobs could run longer than 60 days. Therefore my idea would be to update access times of all files in job_working_directory daily. Thank you very much for any help. Best, Matthias -- ------------------------------------------- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.bernt(a)ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann -------------------------------------------

1 0

All Jobs Error Out
by Yip, Miu ki 15 Mar '17

15 Mar '17

Hi all, This afternoon, our local instance of Galaxy suddenly started producing errors for all jobs. This is the trace that gets printed out. Not sure what is happening because the error doesn’t give much information other than python not being able to find a module. Traceback (most recent call last): File "/localdata1/galaxy/database/jobs_directory/002/2931/set_metadata_LMpe2m.py", line 1, in from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/localdata1/galaxy/lib/galaxy_ext/metadata/set_metadata.py", line 25, in import galaxy.model.mapping # need to load this before we unpickle, in order to setup properties assigned by the mappers File "/localdata1/galaxy/lib/galaxy/model/mapping.py", line 19, in from galaxy.model.custom_types import JSONType, MetadataType, TrimmedString, UUIDType File "/localdata1/galaxy/lib/galaxy/model/custom_types.py", line 13, in from galaxy import app File "/localdata1/galaxy/lib/galaxy/app.py", line 17, in from galaxy.visualization.plugins.registry import VisualizationsRegistry File "/localdata1/galaxy/lib/galaxy/visualization/plugins/registry.py", line 15, in from galaxy.visualization.plugins import plugin as vis_plugins File "/localdata1/galaxy/lib/galaxy/visualization/plugins/plugin.py", line 13, in from galaxy.web.base import interactive_environments File "/localdata1/galaxy/lib/galaxy/web/base/interactive_environments.py", line 14, in from galaxy.tools.deps.docker_util import DockerVolume File "/localdata1/galaxy/lib/galaxy/tools/__init__.py", line 60, in import tool_shed.util.repository_util as repository_util File "/localdata1/galaxy/lib/tool_shed/util/repository_util.py", line 16, in from tool_shed.util import hg_util File "/localdata1/galaxy/lib/tool_shed/util/hg_util.py", line 11, in from mercurial.exchange import readbundle File "/lib64/python2.7/site-packages/mercurial/exchange.py", line 18, in from . import ( File "/lib64/python2.7/site-packages/mercurial/bundle2.py", line 168, in urlerr = util.urlerr AttributeError: 'module' object has no attribute 'urlerr' Could anyone provide any insight on the issue? Thanks!

2 2

View uploaded files
by pieter coussement 15 Mar '17

15 Mar '17

Hi, I just updated galaxy, and now I seem to be unable to view the contents of the files. So i'm able to upload, and can work with them and see the attributes. If however I want to open them ('the eye' on the left), I get an error (as do my collegues) I checked on the full path and the file is there. permission on the file: -rw-r--r-- 1 galaxy galaxy 16 mrt 1 12:10 dataset_140.dat nginx vhost ## # Galaxy ## upstream galaxy_app { server 127.0.0.1:8080; } proxy_next_upstream off; server { client_max_body_size 10G; # ... other server stuff ... location / { proxy_pass http://galaxy_app; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /static { alias /home/galaxy/Software/galaxy/static; expires 24h; } location /static/style { alias /home/galaxy/Software/galaxy/static/style/blue; expires 24h; } location /static/scripts { alias /home/galaxy/Software/galaxy/static/scripts; } location /favicon.ico { alias /home/galaxy/Software/galaxy/static/favicon.ico; } location /robots.txt { alias /home/galaxy/Software/galaxy/static/robots.txt; } # serve static content for visualization and interactive environment plugins location ~ ^/plugins/(?<plug_type>.+?)/(?<vis_name>.+?)/static/(?<static_file>.*?)$ { alias /home/galaxy/Software/galaxy/config/plugins/$plug_type/$vis_name/static/$static_file; } location /_x_accel_redirect/ { internal; alias /; } location ~ \.php$ { fastcgi_pass 127.0.0.1:9000; include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; fastcgi_intercept_errors on; error_page 404 /error/404.php; } } I would guess a permission error of some kind but I seem unable to find it. Nginx spits out: 2017/03/01 17:23:39 [error] 28783#0: *1457 open() "/usr/share/nginx/htmltrue/home/galaxy/Software/galaxy/database/files/000/dataset_14 0.dat" failed (2: No such file or directory), client: 172.19.40.126, server: , request: "GET /datasets/32d48e3506ae8cf7/display/?previ ew=True HTTP/1.1", upstream: "http://127.0.0.1:8080/datasets/32d48e3506ae8cf7/display/?preview=True", host: "172.19.32.42", referrer: "http://172.19.32.42/" the file is in home/galaxy/Software/galaxy/database/files/000/dataset_140.dat but the folder /usr/share/nginx/htmltrue does not exists... Any help on this one? Best regards, Pieter -- *ir. Pieter Coussement, PhD* pieter.coussement(a)inbiose.com <mailto:pieter.coussement@inbiose.com> +32 (0) 485 42 79 72 www.inbiose.com <http://www.inbiose.com> Registered address: Inbiose N.V., Technologiepark Zwijnaarde 3 – bus 41, B-9052 Gent, Belgium - rpr gent: 0534 887 593 Confidentiality notice: The information contained in this e-mail, including any attachments, is confidential and may be the subject of legal professional privilege. It is intended for the authorized use of the individual or entity addressed. If the receiver or reader of this message is not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is prohibited. If this email is received in error, please accept our apologies, delete all copies from your system, and notify the sender by return e-mail and delete this e-mail, including any attachments. Disclaimer: The author of this e-mail has taken every effort to ensure that the information and advice provided are accurate and up-to-date, based upon the information available at the time of writing. The information and advice can only be binding for Inbiose when this is clearly intended in the message and is contained in the information made available.

2 2