New subject: Shared Data Library, large number of files

8 Mar 2017

      Can you please share your '/client/galaxy/scripts/layout/menu.js' file with
me? (e.g. using https://gist.github.com or similar).
thanks
M.
On Wed, Mar 8, 2017 at 1:34 PM D K danielfortin86@gmail.com wrote:
...
Hi Martin,
I've tried this on both v16.07 and v16.10
D
On Wed, Mar 8, 2017 at 10:29 AM, Martin Čech marten@bx.psu.edu wrote:
Daniel:
what version is your Galaxy?
Martin
On Wed, Mar 8, 2017 at 12:40 PM D K danielfortin86@gmail.com wrote:
I've copied and pasted the javascript error below. Any help would be
greatly appreciated. Thanks!
Uncaught TypeError: Cannot read property 'masthead' of undefined
    at n.<anonymous> (menu.js:120)
    at Function.each (jquery.js:370)
    at HTMLDocument.<anonymous> (menu.js:117)
    at l (jquery.js:3232)
    at Object.add [as done] (jquery.js:3291)
    at t.fn.init.vt.fn.ready (jquery.js:3542)
    at callback (menu.js:116)
    at Object.t.success (backbone.js:1051)
    at l (jquery.js:3232)
    at Object.fireWith [as resolveWith] (jquery.js:3362)
On Thu, Mar 2, 2017 at 10:43 AM, D K danielfortin86@gmail.com wrote:
Hi Martin,
I do get a javascript error, I've attached a screenshot of the relevant
window. I also tried logging in as a non-admin user and had the same issue.
In this Galaxy instance are 5 data libraries, each with about 150
individual folders, and each folder has a variable number of fastq.gz files.
Thanks!
On Thu, Mar 2, 2017 at 10:13 AM, Martin Čech marten@bx.psu.edu wrote:
Daniel:
Could you please open browser console and look for javascript errors on
the page that fails to load for you? Additionally I assume you are an admin
when trying this (admins load all of the instance's libraries) could you
try loading the library list as a normal user? How many libraries do you
have on the instance?
Thanks
Martin
On Thu, Mar 2, 2017 at 12:38 PM D K danielfortin86@gmail.com wrote:
Hi Martin / Hans / Devon,
I can confirm that I am using the new UI from the top menu. Right now when
I click on the link I often do not even get the list of libraries whereas
when I click on the admin Data Library link I at least see the list of
libraries (although I can't open any of them since they are so large). I've
tried refreshing the page, deleting the cookies for the session and using
Incognito mode, using chrome and firefox both from a linux and windows
machine and can confirm this behavior on all systems. However, occasionally
when clicking the Shared Data Library link from the top I do get a list of
libraries and can navigate them, but this has only happened once during my
tests and I cannot replicate this behavior.
Thanks for the assistance!
On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech marten@bx.psu.edu wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in
place (because it has some features perceived as important to some admins).
However majority of users and most of the admins are using the interface
under 'Shared Data' I hope and the old UI is deprecated since 15.10 (
https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I
created a PR to adjust the label in the admin interface to make this clear:
https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared
Data/Data Libraries in the top menu shouldn't have this kind of issues for
hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you
experiencing this only with a specific folder full of linked datasets or
does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan dpryan@dpryan.com wrote:
I observe the same thing on 16.07. As an example, I'm just opened a
data library under Admin->Data->Data Libraries and this took about a
minute and a half. It looks like this is related to the quantity of
data in this folder. The associated entry in the uwsgi log is as
follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes}
[Thu Mar  2 08:40:19 2017] GET
/library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff
=> generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers
in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the
browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin
section until at least 16.07. Perhaps this has been changed since
then, I hadn't planed to update our system until the end of the month.
For what it's worth, opening the same library via "shared data" takes
~100 msec.
Devon
Devon Ryan, Ph.D.
Email: dpryan@dpryan.com
Data Manager/Bioinformatician
Max Planck Institute of Immunobiology and Epigenetics
Stübeweg 51
79108 Freiburg
Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz hrh@fmi.ch wrote:
...
this sounds more like a browser issue to me
which browser do you use? what happens when you use a different browser?
and
...
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
...
Actually, when I click on the Shared Data-> Data Libraries link I get
only the Galaxy banner and a blank page, so I'm not even seeing the data
libraries themselves.
If I use the Admin link and then click Data libraries, I can see the
actual libraries. But when I try to enter one of the libraries with lots
of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com
mailto:danielfortin86@gmail.com> wrote:
Hi Hans,

I'm running 16.10. If I'm not mistaken, the old Data Library
interface is still accessible in the Admin menu isn't it? I have the
same issue when navigating to a Data Library either from the Admin
or Shared Data links. I also tried this in 16.07 with the same
result.  If it makes a difference, I'm using PostgreSQL and Apache.

Dwight

On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch
<mailto:hrh@fmi.ch>> wrote:

    Hi Daniel

    Just double checking which version of galaxy are you running? In
    other words: are you still using the old Data Library interface?
    (if I remember correctly, the old UI was deprecated in 15.10)
    We had similar issues with the old UI.

    The new one works fine and is very fast. E.g. we have a 'MISEQ'
    folder with more than 100 sub folders and a HISEQ folder with
    more than 600 sub folders. Each containing up to 100plus fastq
    and text files.

    Regards, Hans-Rudolf

    On 03/01/2017 12:13 AM, D K wrote:

        Hi,

        I've noticed that the more files / folders (think hundreds
        of folders
        each with 20+ fastq files) I add to a Shared Data Library,
        the longer
        and longer it takes to initially navigate the folder
        structure (after
        clicking on the specific Shared Data Library). Just opening
        up the
        Shared Data Library can take several minutes and often the
        browser times
        out. Is there a way to speed this up?

        If it's relevant, when I add shared data, it's linked rather
        than copied
        into Galaxy. Any suggestions would be greatly appreciated

        Thanks!

        ___________________________________________________________
        Please keep all replies on the list by using "reply all"
        in your mail client.  To manage your subscriptions to this
        and other Galaxy lists, please use the interface at:
        https://lists.galaxyproject.org/
        <https://lists.galaxyproject.org/>

        To search Galaxy mailing lists use the unified search at:
        http://galaxyproject.org/search/mailinglists/
        <http://galaxyproject.org/search/mailinglists/>

Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Shared Data Library, large number of files

Devon

Devon

Devon

Devon