Re: [galaxy-dev] Shared Data Library, large number of files
this sounds more like a browser issue to me which browser do you use? what happens when you use a different browser? and what happens when you just reload the page Hans-Rudolf On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the data libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have the same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running? In other words: are you still using the old Data Library interface? (if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows: [pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET /library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2) So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue. As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec. Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different browser? and what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the data libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have the same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running? In other words: are you still using the old Data Library interface? (if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi all, *Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694 The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here. *Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it? Martin On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET
/library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different browser?
and
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the data libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have the same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running? In other words: are you still using the old Data Library interface? (if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Martin / Hans / Devon, I can confirm that I am using the new UI from the top menu. Right now when I click on the link I often do not even get the list of libraries whereas when I click on the admin Data Library link I at least see the list of libraries (although I can't open any of them since they are so large). I've tried refreshing the page, deleting the cookies for the session and using Incognito mode, using chrome and firefox both from a linux and windows machine and can confirm this behavior on all systems. However, occasionally when clicking the Shared Data Library link from the top I do get a list of libraries and can navigate them, but this has only happened once during my tests and I cannot replicate this behavior. Thanks for the assistance! On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET /library_common/browse_library?sort=name&f-description=All&f-name=All&f- deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the
data
libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have
browser? and the
same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running?
In
other words: are you still using the old Data Library
interface?
(if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked
rather
than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
_____________________________
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Daniel: Could you please open browser console and look for javascript errors on the page that fails to load for you? Additionally I assume you are an admin when trying this (admins load all of the instance's libraries) could you try loading the library list as a normal user? How many libraries do you have on the instance? Thanks Martin On Thu, Mar 2, 2017 at 12:38 PM D K <danielfortin86@gmail.com> wrote:
Hi Martin / Hans / Devon,
I can confirm that I am using the new UI from the top menu. Right now when I click on the link I often do not even get the list of libraries whereas when I click on the admin Data Library link I at least see the list of libraries (although I can't open any of them since they are so large). I've tried refreshing the page, deleting the cookies for the session and using Incognito mode, using chrome and firefox both from a linux and windows machine and can confirm this behavior on all systems. However, occasionally when clicking the Shared Data Library link from the top I do get a list of libraries and can navigate them, but this has only happened once during my tests and I cannot replicate this behavior.
Thanks for the assistance!
On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET
/library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different browser?
and
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the data libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have the same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running? In other words: are you still using the old Data Library interface? (if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Martin, I do get a javascript error, I've attached a screenshot of the relevant window. I also tried logging in as a non-admin user and had the same issue. In this Galaxy instance are 5 data libraries, each with about 150 individual folders, and each folder has a variable number of fastq.gz files. Thanks! On Thu, Mar 2, 2017 at 10:13 AM, Martin Čech <marten@bx.psu.edu> wrote:
Daniel:
Could you please open browser console and look for javascript errors on the page that fails to load for you? Additionally I assume you are an admin when trying this (admins load all of the instance's libraries) could you try loading the library list as a normal user? How many libraries do you have on the instance?
Thanks
Martin
On Thu, Mar 2, 2017 at 12:38 PM D K <danielfortin86@gmail.com> wrote:
Hi Martin / Hans / Devon,
I can confirm that I am using the new UI from the top menu. Right now when I click on the link I often do not even get the list of libraries whereas when I click on the admin Data Library link I at least see the list of libraries (although I can't open any of them since they are so large). I've tried refreshing the page, deleting the cookies for the session and using Incognito mode, using chrome and firefox both from a linux and windows machine and can confirm this behavior on all systems. However, occasionally when clicking the Shared Data Library link from the top I do get a list of libraries and can navigate them, but this has only happened once during my tests and I cannot replicate this behavior.
Thanks for the assistance!
On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET /library_common/browse_library?sort=name&f-description=All&f-name=All&f- deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the
data
libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have
browser? and the
same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running?
In
other words: are you still using the old Data Library
interface?
(if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked
rather
than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
_____________________________
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
I've copied and pasted the javascript error below. Any help would be greatly appreciated. Thanks! Uncaught TypeError: Cannot read property 'masthead' of undefined
at n.<anonymous> (menu.js:120) at Function.each (jquery.js:370) at HTMLDocument.<anonymous> (menu.js:117) at l (jquery.js:3232) at Object.add [as done] (jquery.js:3291) at t.fn.init.vt.fn.ready (jquery.js:3542) at callback (menu.js:116) at Object.t.success (backbone.js:1051) at l (jquery.js:3232) at Object.fireWith [as resolveWith] (jquery.js:3362)
On Thu, Mar 2, 2017 at 10:43 AM, D K <danielfortin86@gmail.com> wrote:
Hi Martin,
I do get a javascript error, I've attached a screenshot of the relevant window. I also tried logging in as a non-admin user and had the same issue. In this Galaxy instance are 5 data libraries, each with about 150 individual folders, and each folder has a variable number of fastq.gz files.
Thanks!
On Thu, Mar 2, 2017 at 10:13 AM, Martin Čech <marten@bx.psu.edu> wrote:
Daniel:
Could you please open browser console and look for javascript errors on the page that fails to load for you? Additionally I assume you are an admin when trying this (admins load all of the instance's libraries) could you try loading the library list as a normal user? How many libraries do you have on the instance?
Thanks
Martin
On Thu, Mar 2, 2017 at 12:38 PM D K <danielfortin86@gmail.com> wrote:
Hi Martin / Hans / Devon,
I can confirm that I am using the new UI from the top menu. Right now when I click on the link I often do not even get the list of libraries whereas when I click on the admin Data Library link I at least see the list of libraries (although I can't open any of them since they are so large). I've tried refreshing the page, deleting the cookies for the session and using Incognito mode, using chrome and firefox both from a linux and windows machine and can confirm this behavior on all systems. However, occasionally when clicking the Shared Data Library link from the top I do get a list of libraries and can navigate them, but this has only happened once during my tests and I cannot replicate this behavior.
Thanks for the assistance!
On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 (https://docs.galaxyproject.org/en/master/releases/15.10_ann ounce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET /library_common/browse_library?sort=name&f-description=All& f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id= 355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the
data
libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have
browser? and the
same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and
Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running?
In
other words: are you still using the old Data Library
interface?
(if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a
'MISEQ'
folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just
opening
up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked
rather
than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
_____________________________
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Daniel: what version is your Galaxy? Martin On Wed, Mar 8, 2017 at 12:40 PM D K <danielfortin86@gmail.com> wrote:
I've copied and pasted the javascript error below. Any help would be greatly appreciated. Thanks!
Uncaught TypeError: Cannot read property 'masthead' of undefined at n.<anonymous> (menu.js:120) at Function.each (jquery.js:370) at HTMLDocument.<anonymous> (menu.js:117) at l (jquery.js:3232) at Object.add [as done] (jquery.js:3291) at t.fn.init.vt.fn.ready (jquery.js:3542) at callback (menu.js:116) at Object.t.success (backbone.js:1051) at l (jquery.js:3232) at Object.fireWith [as resolveWith] (jquery.js:3362)
On Thu, Mar 2, 2017 at 10:43 AM, D K <danielfortin86@gmail.com> wrote:
Hi Martin,
I do get a javascript error, I've attached a screenshot of the relevant window. I also tried logging in as a non-admin user and had the same issue. In this Galaxy instance are 5 data libraries, each with about 150 individual folders, and each folder has a variable number of fastq.gz files.
Thanks!
On Thu, Mar 2, 2017 at 10:13 AM, Martin Čech <marten@bx.psu.edu> wrote:
Daniel:
Could you please open browser console and look for javascript errors on the page that fails to load for you? Additionally I assume you are an admin when trying this (admins load all of the instance's libraries) could you try loading the library list as a normal user? How many libraries do you have on the instance?
Thanks
Martin
On Thu, Mar 2, 2017 at 12:38 PM D K <danielfortin86@gmail.com> wrote:
Hi Martin / Hans / Devon,
I can confirm that I am using the new UI from the top menu. Right now when I click on the link I often do not even get the list of libraries whereas when I click on the admin Data Library link I at least see the list of libraries (although I can't open any of them since they are so large). I've tried refreshing the page, deleting the cookies for the session and using Incognito mode, using chrome and firefox both from a linux and windows machine and can confirm this behavior on all systems. However, occasionally when clicking the Shared Data Library link from the top I do get a list of libraries and can navigate them, but this has only happened once during my tests and I cannot replicate this behavior.
Thanks for the assistance!
On Thu, Mar 2, 2017 at 5:47 AM, Martin Čech <marten@bx.psu.edu> wrote:
Hi all,
*Devon* is correct -the old data library UI under Admin menu is still in place (because it has some features perceived as important to some admins). However majority of users and most of the admins are using the interface under 'Shared Data' I hope and the old UI is deprecated since 15.10 ( https://docs.galaxyproject.org/en/master/releases/15.10_announce.html). I created a PR to adjust the label in the admin interface to make this clear: https://github.com/galaxyproject/galaxy/pull/3694
The old UI is very slow for large libraries but the new UI under Shared Data/Data Libraries in the top menu shouldn't have this kind of issues for hundreds of folders/datasets that are mentioned here.
*Daniel* are you using the new UI form the top menu? If so are you experiencing this only with a specific folder full of linked datasets or does it occur for any folder with many datasets in it?
Martin
On Thu, Mar 2, 2017 at 3:49 AM Devon Ryan <dpryan@dpryan.com> wrote:
I observe the same thing on 16.07. As an example, I'm just opened a data library under Admin->Data->Data Libraries and this took about a minute and a half. It looks like this is related to the quantity of data in this folder. The associated entry in the uwsgi log is as follows:
[pid: 357|app: 0|req: 2059/5647] 10.1.6.77 () {44 vars in 1220 bytes} [Thu Mar 2 08:40:19 2017] GET
/library_common/browse_library?sort=name&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&operation=browse&id=355326571f7132ff => generated 21761790 bytes in 108644 msecs (HTTP/1.1 200) 2 headers in 73 bytes (14 switches on core 2)
So this library is causing ~21 megs of data to be returned to the browser, which I presume is the source of the issue.
As an aside, the old data library API is still in use in the admin section until at least 16.07. Perhaps this has been changed since then, I hadn't planed to update our system until the end of the month. For what it's worth, opening the same library via "shared data" takes ~100 msec.
Devon -- Devon Ryan, Ph.D. Email: dpryan@dpryan.com Data Manager/Bioinformatician Max Planck Institute of Immunobiology and Epigenetics Stübeweg 51 79108 Freiburg Germany
On Thu, Mar 2, 2017 at 8:30 AM, Hans-Rudolf Hotz <hrh@fmi.ch> wrote:
this sounds more like a browser issue to me which browser do you use? what happens when you use a different browser?
and
what happens when you just reload the page
Hans-Rudolf
On 03/01/2017 06:17 PM, D K wrote:
Actually, when I click on the Shared Data-> Data Libraries link I get only the Galaxy banner and a blank page, so I'm not even seeing the data libraries themselves.
If I use the Admin link and then click Data libraries, I can see the actual libraries. But when I try to enter one of the libraries with lots of files it hangs.
On Wed, Mar 1, 2017 at 9:03 AM, D K <danielfortin86@gmail.com <mailto:danielfortin86@gmail.com>> wrote:
Hi Hans,
I'm running 16.10. If I'm not mistaken, the old Data Library interface is still accessible in the Admin menu isn't it? I have the same issue when navigating to a Data Library either from the Admin or Shared Data links. I also tried this in 16.07 with the same result. If it makes a difference, I'm using PostgreSQL and Apache.
Dwight
On Tue, Feb 28, 2017 at 11:47 PM, Hans-Rudolf Hotz <hrh@fmi.ch <mailto:hrh@fmi.ch>> wrote:
Hi Daniel
Just double checking which version of galaxy are you running? In other words: are you still using the old Data Library interface? (if I remember correctly, the old UI was deprecated in 15.10) We had similar issues with the old UI.
The new one works fine and is very fast. E.g. we have a 'MISEQ' folder with more than 100 sub folders and a HISEQ folder with more than 600 sub folders. Each containing up to 100plus fastq and text files.
Regards, Hans-Rudolf
On 03/01/2017 12:13 AM, D K wrote:
Hi,
I've noticed that the more files / folders (think hundreds of folders each with 20+ fastq files) I add to a Shared Data Library, the longer and longer it takes to initially navigate the folder structure (after clicking on the specific Shared Data Library). Just opening up the Shared Data Library can take several minutes and often the browser times out. Is there a way to speed this up?
If it's relevant, when I add shared data, it's linked rather than copied into Galaxy. Any suggestions would be greatly appreciated
Thanks!
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ <http://galaxyproject.org/search/mailinglists/>
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
participants (4)
-
D K -
Devon Ryan -
Hans-Rudolf Hotz -
Martin Čech