Re: [galaxy-dev] Problem with LWR (input size limitation)

Hi John, Thanks again for you help. So i place "sys.setdefaultencoding('utf-8')" at the top of the "lwr/web/framework.py", and i still have this bug. I use Python version "2.7 64 bits" to executes LWR on the windows server, and i encounter a "common" problem with the "setdefaultencoding" function which can't be loaded when doing "import sys" and "sys.setdefaultencoding('utf-8')". The hack that i found was to do "reload(sys)" then "sys.setdefaultencoding('utf-8')". But i still get the "utf8' codec can't decode byte" bug. Can it come from my input data which contains utf8 characters? Thank you very much, Bests regards, Misharl Monsoor Le 04/12/2014 06:25, John Chilton a écrit : > I have had one other report of this error previously - but I have not > been able to reproduce it. I have tried so hard to understand these > unicode/utf-8/Python 2 v 3/etc... issues and I must admit I simply > don't :(. > > Reading through http://farmdev.com/talks/unicode/ I found a hack that > might work, if you place: > > import sys > sys.setdefaultencoding('utf-8') > > at the top of lwr/web/framework.py does that fix the problem? > > -John > > On Wed, Nov 26, 2014 at 2:53 AM, Misharl Monsoor <mmonsoor(a)sb-roscoff.fr&gt; > wrote: >> >> Hi John, >> >> I can confirm you that your advice putting "LWR_CURL_TRANSPORT=1" works >> very >> fine, thanks a lot! Sorry, i haven't expressed myself correctly for my >> second question, thanks for answering, it helps me. >> I have another problem with LWR, when executing the jobs on the Windows >> Server, i have the following bug (it happens a lot of time, and i must >> restart the LWR server in order for it to works): >> >> File >> "C:\Python2764\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver >> py", line 287, in wsgi_execute >> self.wsgi_start_response) >> File "C:\Desktop\lwr_master\lwr\web\framework.py", l >> ne 37, in __call__ >> return controller(environ, start_response, **request_args) >> File "C:\Desktop\lwr_master\lwr\web\framework.py", l >> ne 141, in controller_replacement >> resp = self.__build_response(result) >> File "C:\Desktop\lwr_master\lwr\web\framework.py", l >> ne 129, in __build_response >> resp = Response(body=self.body(result)) >> File "C:\Desktop\lwr_master\lwr\web\framework.py", l >> ne 155, in body >> body = dumps(result) >> File "C:\Python2764\lib\json\__init__.py", line 243, in dumps >> return _default_encoder.encode(obj) >> File "C:\Python2764\lib\json\encoder.py", line 207, in encode >> chunks = self.iterencode(o, _one_shot=True) >> File "C:\Python2764\lib\json\encoder.py", line 270, in iterencode >> return _iterencode(o, 0) >> nicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 718: >> invalid >> start byte >> >> >> Bests regards and thank you very much for your help, >> >> Misharl Monsoor. >> >> >> >> Le 21/11/2014 17:44, John Chilton a écrit : >> >>> On Wed, Nov 19, 2014 at 12:58 PM, Misharl Monsoor >>> <mmonsoor(a)sb-roscoff.fr&gt; wrote: >>>> >>>> Hi John , >>>> >>>> First all, thank you very much for your quick reply. I will test your >>>> suggestion about pycurl, and tell you if it has worked. For the second >>>> question, i was wondering if it is possible with LWR to define a path >>>> (where >>>> are located the inputs) as an input (string input in the wrapper xml). >>> >>> Still not certain I understand - but I do not believe this is possible >>> for datasets - this is something that must be configured for the LWR >>> itself. If you have inputs to the tool that are not datasets - such as >>> reference or index data - those could potentially be specified this >>> way as just <data type="text"... in your tool. Those parameters will >>> be passed through on the command-line just like any other parameter. >>> >>> Hope this helps. >>> -John >>> >>>> Thank you again for your help, >>>> >>>> Misharl Monsoor. >>>> >>>> >>>> >>>> >>>> >>>> Le 19/11/2014 15:30, John Chilton a écrit : >>>> >>>>> If you install pycurl (globally or in a Galaxy's virtualenv) and set >>>>> LWR_CURL_TRANSPORT=1 in Galaxy's environment than the LWR will stream >>>>> large files with CURL instead of the mmap hack. >>>>> >>>>> I don't understand your second question - the LWR should stream >>>>> whatever data inputs are selected by the user. If you want the tool to >>>>> be able to take many inputs you can use multiple="true" on your data >>>>> parameter or use a repeat block. >>>>> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax). >>>>> >>>>> Hope this helps, >>>>> -John >>>>> >>>>> >>>>> On Mon, Nov 17, 2014 at 3:04 AM, Misharl Monsoor >>>>> <mmonsoor(a)sb-roscoff.fr&gt; >>>>> wrote: >>>>>> >>>>>> Hi everybody, >>>>>> >>>>>> In our lab, we are trying to connect our Galaxy instance to a Windows >>>>>> 7 >>>>>> 64 >>>>>> bits server, in order to executes programs that need to be run within >>>>>> Windows. However we have a problem concerning the size of the input >>>>>> that >>>>>> is >>>>>> uploaded to the Windows server, it doesn't accept inputs with a size >>>>>> > >>>>>> 1.3 >>>>>> Gb ? We have the following problem with mmap function: >>>>>> >>>>>> data = mmap.mmap(input.fileno(), 0, access=mmap.ACCESS_READ) >>>>>> error: [Errno 12] Cannot allocate memory >>>>>> >>>>>> >>>>>> I have another question concerning the inputs, is there a way to use >>>>>> several >>>>>> inputs that be uploaded to a Windows Server? >>>>>> >>>>>> Thank you very much in advance, >>>>>> >>>>>> Bests regards, >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Misharl Monsoor >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>>> >>>>