Hello gents,
Hope everyone had a great holiday break! Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I've setup torque with some default and maximum amount of CPU and memory allocations. However, I have some worries when it comes to application (like tophat for example). By default, it takes half the CPU of a server unless specified otherwise. How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez System Administrator (Linux) Direction Générale des Technologies de l'Information et de la Communication * Bur. : 1-514-343-6111 poste 16568
Université de Montréal PAVILLON ROGER-GAUDRY, bureau X-218
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation. 1. Server redhat6 with galaxy behind an Apache HTTP Server proxy like described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar edgar.fernandez@umontreal.ca wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Edgar,
I'd suggest checking your Apache logs to see if the upload was denied by the server configuration (possibly for being too large).
Also, please try using the modal upload method to see whether the upload is actually completing. This can be found by clicking the icon of a box with an up arrow at the top of the tool panel.
There are multiple steps to an upload - the first is sending the data to the server. After that, Galaxy creates a job and detects metadata about the uploaded dataset. Prior to the modal upload, a (purple "uploading") history item would be created as soon as the upload started, but if the transfer to the server was interrupted the history item may remain "stuck" in that state without providing feedback to the user that the upload was no longer progressing. The modal provides an upload progress bar, and only once all the data has been sent to the server does it create a history item (in the gray "queued") state.
On Thu, Jan 22, 2015 at 8:50 AM, Fernandez Edgar < edgar.fernandez@umontreal.ca> wrote:
Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation.
- Server redhat6 with galaxy behind an Apache HTTP Server proxy like
described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar < edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your
help!!!
And yes I would like to hear your elaboration on my DRMAA runner which
is what I'm using.
So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing
with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to
interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually
isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots
allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar <
edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified
otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello Nate,
Thank you for answer so quickly. My apache logs (/var/log/httpd/galaxy-prod-access_log) looks like this ip_addr - - [22/Jan/2015:10:17:31 -0500] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 37 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 1401 ip_addr - - [22/Jan/2015:10:17:36 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:40 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:44 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:48 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235
And my galaxy logs (/home/galaxy/galaxy-prod/config/main.log): ip_addr - - [22/Jan/2015:10:17:31 -0400] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "POST /galaxy-prod/tool_runner/index HTTP/1.1" 500 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" Error - <type 'exceptions.OverflowError'>: signed integer is greater than maximum ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" URL: http://esiservera.esi.umontreal.ca:8081/galaxy-prod/tool_runner/index File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 185 in handle_request kwargs = trans.request.params.mixed() File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 900 in params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 892 in str_params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 818 in str_POST File '/usr/lib64/python2.6/cgi.py', line 508 in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File '/usr/lib64/python2.6/cgi.py', line 635 in read_multi headers = rfc822.Message(self.fp) File '/usr/lib64/python2.6/rfc822.py', line 108 in __init__ self.readheaders() File '/usr/lib64/python2.6/rfc822.py', line 155 in readheaders line = self.fp.readline() File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py', line 482 in readline data = self.file.readline(max_read) File '/usr/lib64/python2.6/socket.py', line 415 in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum
CGI Variables ------------- CONTENT_LENGTH: '2625522525' CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary9c9hOVrAGAXKJtdd' HTTP_ACCEPT: '*/*' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e' HTTP_HOST: 'esiservera.esi.umontreal.ca:8081' HTTP_ORIGIN: 'http://esiservera.esi.umontreal.ca:8081' HTTP_REFERER: 'http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36' HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' PATH_INFO: '/tool_runner/index' REMOTE_ADDR: 'ip_addr' REQUEST_METHOD: 'POST' SCRIPT_NAME: '/galaxy-prod' SERVER_NAME: 'esiservera.esi.umontreal.ca' SERVER_PORT: '7112' SERVER_PROTOCOL: 'HTTP/1.1'
WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x7fd1500e9b10> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e'>, 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x7fd1500e9550> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x7ef2850> paste.recursive.forward: <paste.recursive.Forwarder from /galaxy-prod> paste.recursive.include: <paste.recursive.Includer from /galaxy-prod> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy-prod> paste.recursive.script_name: '/galaxy-prod' paste.throw_errors: True request_id: 'c93de906a24911e4bd5d002264f83610' webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ ip_addr - - [22/Jan/2015:10:17:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:40 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:44 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:48 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:52 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:59 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:04 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:10 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:29 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
However, I still see on my galaxy webpage the document in blue with the arrow going up constantly.
Do you think it’s an apache configuration error or galaxy or both?
Cordialement / Regards, Edgar Fernandez
De : Nate Coraor [mailto:nate@bx.psu.edu] Envoyé : January-22-15 10:10 AM À : Fernandez Edgar Cc : John Chilton; galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
I'd suggest checking your Apache logs to see if the upload was denied by the server configuration (possibly for being too large).
Also, please try using the modal upload method to see whether the upload is actually completing. This can be found by clicking the icon of a box with an up arrow at the top of the tool panel.
There are multiple steps to an upload - the first is sending the data to the server. After that, Galaxy creates a job and detects metadata about the uploaded dataset. Prior to the modal upload, a (purple "uploading") history item would be created as soon as the upload started, but if the transfer to the server was interrupted the history item may remain "stuck" in that state without providing feedback to the user that the upload was no longer progressing. The modal provides an upload progress bar, and only once all the data has been sent to the server does it create a history item (in the gray "queued") state.
On Thu, Jan 22, 2015 at 8:50 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote: Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation. 1. Server redhat6 with galaxy behind an Apache HTTP Server proxy like described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111tel:1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hi Edgar,
Please make sure you have disabled the developer options `use_interactive` and `debug` in galaxy.ini, then restart your Galaxy server and try the upload again.
Also, please try out the modal upload dialog as I suggested, it should prevent a "stuck" upload in the future.
--nate
On Thu, Jan 22, 2015 at 10:25 AM, Fernandez Edgar < edgar.fernandez@umontreal.ca> wrote:
Hello Nate,
Thank you for answer so quickly.
My apache logs (/var/log/httpd/galaxy-prod-access_log) looks like this
*ip_addr - - [22/Jan/2015:10:17:31 -0500] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 37*
*ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235*
*ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 1401*
*ip_addr - - [22/Jan/2015:10:17:36 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235*
*ip_addr - - [22/Jan/2015:10:17:40 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235*
*ip_addr - - [22/Jan/2015:10:17:44 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235*
*ip_addr - - [22/Jan/2015:10:17:48 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235*
And my galaxy logs (/home/galaxy/galaxy-prod/config/main.log):
ip_addr - - [22/Jan/2015:10:17:31 -0400] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:32 -0400] "POST /galaxy-prod/tool_runner/index HTTP/1.1" 500 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
Error - <type 'exceptions.OverflowError'>: signed integer is greater than maximum
ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
URL: http://esiservera.esi.umontreal.ca:8081/galaxy-prod/tool_runner/index
File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__
app_iter = self.application(environ, sr_checker)
File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__
return self.application(environ, start_response)
File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__
return self.application(environ, start_response)
File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 132 in __call__
return self.handle_request( environ, start_response )
File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 185 in handle_request
kwargs = trans.request.params.mixed()
File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 900 in params
File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 892 in str_params
File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 818 in str_POST
File '/usr/lib64/python2.6/cgi.py', line 508 in __init__
self.read_multi(environ, keep_blank_values, strict_parsing)
File '/usr/lib64/python2.6/cgi.py', line 635 in read_multi
headers = rfc822.Message(self.fp)
File '/usr/lib64/python2.6/rfc822.py', line 108 in __init__
self.readheaders()
File '/usr/lib64/python2.6/rfc822.py', line 155 in readheaders
line = self.fp.readline()
File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py', line 482 in readline
data = self.file.readline(max_read)
File '/usr/lib64/python2.6/socket.py', line 415 in readline
bline = buf.readline(size)
OverflowError: signed integer is greater than maximum
CGI Variables
CONTENT_LENGTH: '2625522525'
CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary9c9hOVrAGAXKJtdd'
HTTP_ACCEPT: '*/*'
HTTP_ACCEPT_ENCODING: 'gzip, deflate'
HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8'
HTTP_CONNECTION: 'Keep-Alive'
HTTP_COOKIE: 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e'
HTTP_HOST: 'esiservera.esi.umontreal.ca:8081'
HTTP_ORIGIN: 'http://esiservera.esi.umontreal.ca:8081'
HTTP_REFERER: 'http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root'
HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36'
HTTP_X_REQUESTED_WITH: 'XMLHttpRequest'
PATH_INFO: '/tool_runner/index'
REMOTE_ADDR: 'ip_addr'
REQUEST_METHOD: 'POST'
SCRIPT_NAME: '/galaxy-prod'
SERVER_NAME: 'esiservera.esi.umontreal.ca'
SERVER_PORT: '7112'
SERVER_PROTOCOL: 'HTTP/1.1'
WSGI Variables
application: <paste.recursive.RecursiveMiddleware object at 0x7fd1500e9b10>
is_api_request: False
paste.cookies: (SimpleCookie: galaxysession='d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e', 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e')
paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>]
paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x7fd1500e9550>
paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x7ef2850>
paste.recursive.forward: <paste.recursive.Forwarder from /galaxy-prod>
paste.recursive.include: <paste.recursive.Includer from /galaxy-prod>
paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy-prod>
paste.recursive.script_name: '/galaxy-prod'
paste.throw_errors: True
request_id: 'c93de906a24911e4bd5d002264f83610'
webob._parsed_query_vars: (MultiDict([]), '')
wsgi process: 'Multithreaded'
ip_addr - - [22/Jan/2015:10:17:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:40 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:44 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:48 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:52 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:17:59 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:18:04 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:18:10 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:18:29 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
ip_addr - - [22/Jan/2015:10:18:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - " http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
However, I still see on my galaxy webpage the document in blue with the arrow going up constantly.
Do you think it’s an apache configuration error or galaxy or both?
Cordialement / Regards,
Edgar Fernandez
*De :* Nate Coraor [mailto:nate@bx.psu.edu] *Envoyé :* January-22-15 10:10 AM *À :* Fernandez Edgar *Cc :* John Chilton; galaxy-dev@bx.psu.edu
*Objet :* Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
I'd suggest checking your Apache logs to see if the upload was denied by the server configuration (possibly for being too large).
Also, please try using the modal upload method to see whether the upload is actually completing. This can be found by clicking the icon of a box with an up arrow at the top of the tool panel.
There are multiple steps to an upload - the first is sending the data to the server. After that, Galaxy creates a job and detects metadata about the uploaded dataset. Prior to the modal upload, a (purple "uploading") history item would be created as soon as the upload started, but if the transfer to the server was interrupted the history item may remain "stuck" in that state without providing feedback to the user that the upload was no longer progressing. The modal provides an upload progress bar, and only once all the data has been sent to the server does it create a history item (in the gray "queued") state.
On Thu, Jan 22, 2015 at 8:50 AM, Fernandez Edgar < edgar.fernandez@umontreal.ca> wrote:
Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation.
- Server redhat6 with galaxy behind an Apache HTTP Server proxy like
described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM
À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar < edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your
help!!!
And yes I would like to hear your elaboration on my DRMAA runner which
is what I'm using.
So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing
with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to
interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually
isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots
allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar <
edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified
otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello Nate,
I just tried the modal upload dialog and it says "File exceeds 2GB. Please use FTP client". Is there a way to increase that file size limitation?
Cordialement / Regards, Edgar Fernandez
De : Nate Coraor [mailto:nate@bx.psu.edu] Envoyé : January-22-15 12:02 PM À : Fernandez Edgar Cc : John Chilton; galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
Please make sure you have disabled the developer options `use_interactive` and `debug` in galaxy.ini, then restart your Galaxy server and try the upload again.
Also, please try out the modal upload dialog as I suggested, it should prevent a "stuck" upload in the future.
--nate
On Thu, Jan 22, 2015 at 10:25 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote: Hello Nate,
Thank you for answer so quickly. My apache logs (/var/log/httpd/galaxy-prod-access_log) looks like this ip_addr - - [22/Jan/2015:10:17:31 -0500] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 37 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 1401 ip_addr - - [22/Jan/2015:10:17:36 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:40 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:44 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:48 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235
And my galaxy logs (/home/galaxy/galaxy-prod/config/main.log): ip_addr - - [22/Jan/2015:10:17:31 -0400] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "POST /galaxy-prod/tool_runner/index HTTP/1.1" 500 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" Error - <type 'exceptions.OverflowError'>: signed integer is greater than maximum ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" URL: http://esiservera.esi.umontreal.ca:8081/galaxy-prod/tool_runner/index File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 185 in handle_request kwargs = trans.request.params.mixed() File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 900 in params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 892 in str_params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 818 in str_POST File '/usr/lib64/python2.6/cgi.py', line 508 in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File '/usr/lib64/python2.6/cgi.py', line 635 in read_multi headers = rfc822.Message(self.fp) File '/usr/lib64/python2.6/rfc822.py', line 108 in __init__ self.readheaders() File '/usr/lib64/python2.6/rfc822.py', line 155 in readheaders line = self.fp.readline() File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py', line 482 in readline data = self.file.readline(max_read) File '/usr/lib64/python2.6/socket.py', line 415 in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum
CGI Variables ------------- CONTENT_LENGTH: '2625522525tel:2625522525' CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary9c9hOVrAGAXKJtdd' HTTP_ACCEPT: '*/*' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e' HTTP_HOST: 'esiservera.esi.umontreal.ca:8081http://esiservera.esi.umontreal.ca:8081' HTTP_ORIGIN: 'http://esiservera.esi.umontreal.ca:8081' HTTP_REFERER: 'http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36' HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' PATH_INFO: '/tool_runner/index' REMOTE_ADDR: 'ip_addr' REQUEST_METHOD: 'POST' SCRIPT_NAME: '/galaxy-prod' SERVER_NAME: 'esiservera.esi.umontreal.cahttp://esiservera.esi.umontreal.ca' SERVER_PORT: '7112' SERVER_PROTOCOL: 'HTTP/1.1'
WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x7fd1500e9b10> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e'>, 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x7fd1500e9550> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x7ef2850> paste.recursive.forward: <paste.recursive.Forwarder from /galaxy-prod> paste.recursive.include: <paste.recursive.Includer from /galaxy-prod> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy-prod> paste.recursive.script_name: '/galaxy-prod' paste.throw_errors: True request_id: 'c93de906a24911e4bd5d002264f83610' webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ ip_addr - - [22/Jan/2015:10:17:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:40 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:44 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:48 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:52 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:59 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:04 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:10 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:29 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
However, I still see on my galaxy webpage the document in blue with the arrow going up constantly.
Do you think it’s an apache configuration error or galaxy or both?
Cordialement / Regards, Edgar Fernandez
De : Nate Coraor [mailto:nate@bx.psu.edumailto:nate@bx.psu.edu] Envoyé : January-22-15 10:10 AM À : Fernandez Edgar Cc : John Chilton; galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu
Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
I'd suggest checking your Apache logs to see if the upload was denied by the server configuration (possibly for being too large).
Also, please try using the modal upload method to see whether the upload is actually completing. This can be found by clicking the icon of a box with an up arrow at the top of the tool panel.
There are multiple steps to an upload - the first is sending the data to the server. After that, Galaxy creates a job and detects metadata about the uploaded dataset. Prior to the modal upload, a (purple "uploading") history item would be created as soon as the upload started, but if the transfer to the server was interrupted the history item may remain "stuck" in that state without providing feedback to the user that the upload was no longer progressing. The modal provides an upload progress bar, and only once all the data has been sent to the server does it create a history item (in the gray "queued") state.
On Thu, Jan 22, 2015 at 8:50 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote: Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation. 1. Server redhat6 with galaxy behind an Apache HTTP Server proxy like described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111tel:1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Hello guys,
I want to apologies. I didn’t see the note: Due to browser limitations, uploading files larger than 2GB is guaranteed to fail. To upload large files, use the URL method (below) or FTP (if enabled by the site administrator). and I forgot my file was that big.
Please accept my sincere apologies.
Cordialement / Regards, Edgar Fernandez
De : Fernandez Edgar Envoyé : January-22-15 12:35 PM À : 'Nate Coraor' Cc : John Chilton; galaxy-dev@bx.psu.edu Objet : RE: [galaxy-dev] galaxy and torque - resource allocation
Hello Nate,
I just tried the modal upload dialog and it says "File exceeds 2GB. Please use FTP client". Is there a way to increase that file size limitation?
Cordialement / Regards, Edgar Fernandez
De : Nate Coraor [mailto:nate@bx.psu.edu] Envoyé : January-22-15 12:02 PM À : Fernandez Edgar Cc : John Chilton; galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
Please make sure you have disabled the developer options `use_interactive` and `debug` in galaxy.ini, then restart your Galaxy server and try the upload again.
Also, please try out the modal upload dialog as I suggested, it should prevent a "stuck" upload in the future.
--nate
On Thu, Jan 22, 2015 at 10:25 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote: Hello Nate,
Thank you for answer so quickly. My apache logs (/var/log/httpd/galaxy-prod-access_log) looks like this ip_addr - - [22/Jan/2015:10:17:31 -0500] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 37 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:32 -0500] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 1401 ip_addr - - [22/Jan/2015:10:17:36 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:40 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:44 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235 ip_addr - - [22/Jan/2015:10:17:48 -0500] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 235
And my galaxy logs (/home/galaxy/galaxy-prod/config/main.log): ip_addr - - [22/Jan/2015:10:17:31 -0400] "POST /galaxy-prod/tool_runner/upload_async_create HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "POST /galaxy-prod/tool_runner/index HTTP/1.1" 500 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" Error - <type 'exceptions.OverflowError'>: signed integer is greater than maximum ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:32 -0400] "GET /galaxy-prod/tool_runner/upload_async_message HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" URL: http://esiservera.esi.umontreal.ca:8081/galaxy-prod/tool_runner/index File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/home/galaxy/galaxy-prod/lib/galaxy/web/framework/base.py', line 185 in handle_request kwargs = trans.request.params.mixed() File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 900 in params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 892 in str_params File 'build/bdist.linux-x86_64-ucs4/egg/webob/__init__.py', line 818 in str_POST File '/usr/lib64/python2.6/cgi.py', line 508 in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File '/usr/lib64/python2.6/cgi.py', line 635 in read_multi headers = rfc822.Message(self.fp) File '/usr/lib64/python2.6/rfc822.py', line 108 in __init__ self.readheaders() File '/usr/lib64/python2.6/rfc822.py', line 155 in readheaders line = self.fp.readline() File '/home/galaxy/galaxy-prod/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py', line 482 in readline data = self.file.readline(max_read) File '/usr/lib64/python2.6/socket.py', line 415 in readline bline = buf.readline(size) OverflowError: signed integer is greater than maximum
CGI Variables ------------- CONTENT_LENGTH: '2625522525tel:2625522525' CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary9c9hOVrAGAXKJtdd' HTTP_ACCEPT: '*/*' HTTP_ACCEPT_ENCODING: 'gzip, deflate' HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8' HTTP_CONNECTION: 'Keep-Alive' HTTP_COOKIE: 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e' HTTP_HOST: 'esiservera.esi.umontreal.ca:8081http://esiservera.esi.umontreal.ca:8081' HTTP_ORIGIN: 'http://esiservera.esi.umontreal.ca:8081' HTTP_REFERER: 'http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root' HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36' HTTP_X_REQUESTED_WITH: 'XMLHttpRequest' PATH_INFO: '/tool_runner/index' REMOTE_ADDR: 'ip_addr' REQUEST_METHOD: 'POST' SCRIPT_NAME: '/galaxy-prod' SERVER_NAME: 'esiservera.esi.umontreal.cahttp://esiservera.esi.umontreal.ca' SERVER_PORT: '7112' SERVER_PROTOCOL: 'HTTP/1.1'
WSGI Variables -------------- application: <paste.recursive.RecursiveMiddleware object at 0x7fd1500e9b10> is_api_request: False paste.cookies: (<SimpleCookie: galaxysession='d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e'>, 'galaxysession=d122f4f53e5da72f16add67ded507460da12cda8c8b0781476c9ddbea84bd40a1e299b3d9b7a9e3a; galaxysession=d122f4f53e5da72f439b77b098b6a77c8fe4c3cee6d2a1f587a8445290a468f305b3911dd27efa9e') paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>] paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 0x7fd1500e9550> paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x7ef2850> paste.recursive.forward: <paste.recursive.Forwarder from /galaxy-prod> paste.recursive.include: <paste.recursive.Includer from /galaxy-prod> paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from /galaxy-prod> paste.recursive.script_name: '/galaxy-prod' paste.throw_errors: True request_id: 'c93de906a24911e4bd5d002264f83610' webob._parsed_query_vars: (MultiDict([]), '') wsgi process: 'Multithreaded' ------------------------------------------------------------ ip_addr - - [22/Jan/2015:10:17:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:40 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:44 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:48 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:52 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:17:59 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:04 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:10 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:29 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36" ip_addr - - [22/Jan/2015:10:18:36 -0400] "GET /galaxy-prod/api/histories/5a0831c911c29227/contents HTTP/1.1" 200 - "http://esiservera.esi.umontreal.ca:8081/galaxy-prod/root" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36"
However, I still see on my galaxy webpage the document in blue with the arrow going up constantly.
Do you think it’s an apache configuration error or galaxy or both?
Cordialement / Regards, Edgar Fernandez
De : Nate Coraor [mailto:nate@bx.psu.edumailto:nate@bx.psu.edu] Envoyé : January-22-15 10:10 AM À : Fernandez Edgar Cc : John Chilton; galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu
Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Hi Edgar,
I'd suggest checking your Apache logs to see if the upload was denied by the server configuration (possibly for being too large).
Also, please try using the modal upload method to see whether the upload is actually completing. This can be found by clicking the icon of a box with an up arrow at the top of the tool panel.
There are multiple steps to an upload - the first is sending the data to the server. After that, Galaxy creates a job and detects metadata about the uploaded dataset. Prior to the modal upload, a (purple "uploading") history item would be created as soon as the upload started, but if the transfer to the server was interrupted the history item may remain "stuck" in that state without providing feedback to the user that the upload was no longer progressing. The modal provides an upload progress bar, and only once all the data has been sent to the server does it create a history item (in the gray "queued") state.
On Thu, Jan 22, 2015 at 8:50 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote: Hello John,
This is totally excellent! This gave me a great idea for my little torque installation.
I hope you don't mind but I have an unrelated issue with torque that I do not seem to grasp. My issue is that it take a rediculious amount of time to upload a 2Gb file. I tried many different configurations of apache to upload that 2Gb file without success. Galaxy keeps showing that arrow going up constantly (over 12h). And this is an upload from a file located on the server running galaxy.
Here is some information from my installation. 1. Server redhat6 with galaxy behind an Apache HTTP Server proxy like described on your website. 2. I have Apache HTTP Server running as the galaxy user with a. Serving Galaxy at a sub directory b. Compression and caching c. Sending files using Apache
You will see in attachment my apache's galaxy, deflate, expire and xsendfile configuration file. Furthermore, I've changed the following in galaxy.ini: apache_xsendfile = True upstream_gzip = False
Any suggestions? Any information I missed? PLEASE HELP!!!
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-21-15 9:34 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Was hoping someone with an actual torque cluster would respond. I think the way you would configure this might be for instance might be setting native_specification like "-l nodes=1:ppn=n". Depending on how things are configured or simply because I am ignorant - this may be wrong - but I think what you really want should look a lot like your arguments to qsub. So let say you have defined a job runner plugin called pbs_drmaa at the top of your job_conf.xml file. Then you could default everything to a single core by default and send "big" jobs to a destination with 8 cores with destinations and tool sections that look something like this...
...
<destinations default="singlecore"> <destination id="singlecore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=1</param> </destination> <destination id="multicore" runner="pbs_drmaa"> <param id="native_specification">-l nodes=1:ppn=8</param> </destination> </destinations>
and ...
<tools> <tool id="bowtie2" destination="multicore" /> <tool id="bowtie" destination="multicore" /> <tool id="deeptools" destination="multicore" /> <tool id="cufflinks"destination="multicore" /> <tool id="cuffdiff" destination="multicore" /> <tool id="cuffmerge" destination="multicore" /> <tool id="tophat2" destination="multicore" /> </tools>
Again - that native specification could be wrong - probably best to test it out locally (or maybe Nate or JJ will step in and correct me).
Hope this helps, -John
On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Once again I would like to convey my most sincere appreciation for your help!!! And yes I would like to hear your elaboration on my DRMAA runner which is what I'm using. So my installation looks like: galaxy --> pbs_drmaa --> torque
Cordialement / Regards, Edgar Fernandez
-----Message d'origine----- De : John Chilton [mailto:jmchilton@gmail.commailto:jmchilton@gmail.com] Envoyé : January-18-15 8:59 PM À : Fernandez Edgar Cc : galaxy-dev@bx.psu.edumailto:galaxy-dev@bx.psu.edu Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
Galaxy generally defers to the DRM (torque in your case) for dealing with these things. In your job_conf.xml you can specify limits for memory or CPUs and Galaxy will pass these along to the DRM at which point it is up to the DRM to enforce these - details depend on if you are using the PBS runner or the DRMAA runner (let me know which and I can try to elobrate if that would be useful).
In your particular case - I don't believe torque "schedules" RAM so things will generally only be... throttled... by CPUs counts. This is what I was told by the admins at MSI when I worked there anyway. If you want to place hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or switch over to a different DRM entirely like SLURM. Even for DRMs that deal more directly with memory - Galaxy doesn't provide a general mechanism for passing this along to the tool (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact with the environment variables your DRM sets .
This sounds really terrible in the abstract - but it reality it usually isn't an issue - most of Galaxy's multi-core mappers say have relatively low memory per CPU usage - and for things like assemblers where this is more important - one can usually just assign them to their own CPU or something like that to ensure they get all the memory available.
Unlike memory - Galaxy will attempt to pass the number of slots allocated to a job to the tools by setting the GALAXY_SLOTS environment variable. All of the multiple core devteam Galaxy tools at this point use this and so should work - as should probably most multi-core tools from the tool shed - at least the most popular ones.
Hope this helps,
-John
On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar <edgar.fernandez@umontreal.camailto:edgar.fernandez@umontreal.ca> wrote:
Hello gents,
Hope everyone had a great holiday break!
Wish you guys all the best for 2015!
I have a couple of questions about how resources (CPU and memory) is allocated when you have a galaxy and torque installation.
So I’ve setup torque with some default and maximum amount of CPU and memory allocations.
However, I have some worries when it comes to application (like tophat for example).
By default, it takes half the CPU of a server unless specified otherwise.
How is the CPU allocation is specified to application like tophat through galaxy?
Also, how does galaxy react if a job needs more memory than the limit set by torque?
Any information would help me a lot!
My sincere salutations to you all!!!
Cordialement / Regards,
Edgar Fernandez
System Administrator (Linux)
Direction Générale des Technologies de l'Information et de la Communication
( Bur. : 1-514-343-6111tel:1-514-343-6111 poste 16568
Université de Montréal
PAVILLON ROGER-GAUDRY, bureau X-218
Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
galaxy-dev@lists.galaxyproject.org
-
Fernandez Edgar -
John Chilton -
Nate Coraor