Re: [galaxy-dev] Problem with LWR (input size limitation)

I have had one other report of this error previously - but I have not been able to reproduce it. I have tried so hard to understand these unicode/utf-8/Python 2 v 3/etc... issues and I must admit I simply don't :(. Reading through http://farmdev.com/talks/unicode/ I found a hack that might work, if you place: import sys sys.setdefaultencoding('utf-8') at the top of lwr/web/framework.py does that fix the problem? -John On Wed, Nov 26, 2014 at 2:53 AM, Misharl Monsoor <mmonsoor(a)sb-roscoff.fr&gt; wrote: > Hi John, > > I can confirm you that your advice putting "LWR_CURL_TRANSPORT=1" works very > fine, thanks a lot! Sorry, i haven't expressed myself correctly for my > second question, thanks for answering, it helps me. > I have another problem with LWR, when executing the jobs on the Windows > Server, i have the following bug (it happens a lot of time, and i must > restart the LWR server in order for it to works): > > File > "C:\Python2764\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver > py", line 287, in wsgi_execute > self.wsgi_start_response) > File "C:\Desktop\lwr_master\lwr\web\framework.py", l > ne 37, in __call__ > return controller(environ, start_response, **request_args) > File "C:\Desktop\lwr_master\lwr\web\framework.py", l > ne 141, in controller_replacement > resp = self.__build_response(result) > File "C:\Desktop\lwr_master\lwr\web\framework.py", l > ne 129, in __build_response > resp = Response(body=self.body(result)) > File "C:\Desktop\lwr_master\lwr\web\framework.py", l > ne 155, in body > body = dumps(result) > File "C:\Python2764\lib\json\__init__.py", line 243, in dumps > return _default_encoder.encode(obj) > File "C:\Python2764\lib\json\encoder.py", line 207, in encode > chunks = self.iterencode(o, _one_shot=True) > File "C:\Python2764\lib\json\encoder.py", line 270, in iterencode > return _iterencode(o, 0) > nicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 718: > invalid > start byte > > > Bests regards and thank you very much for your help, > > Misharl Monsoor. > > > > Le 21/11/2014 17:44, John Chilton a écrit : > >> On Wed, Nov 19, 2014 at 12:58 PM, Misharl Monsoor >> <mmonsoor(a)sb-roscoff.fr&gt; wrote: >>> Hi John , >>> >>> First all, thank you very much for your quick reply. I will test your >>> suggestion about pycurl, and tell you if it has worked. For the second >>> question, i was wondering if it is possible with LWR to define a path >>> (where >>> are located the inputs) as an input (string input in the wrapper xml). >> Still not certain I understand - but I do not believe this is possible >> for datasets - this is something that must be configured for the LWR >> itself. If you have inputs to the tool that are not datasets - such as >> reference or index data - those could potentially be specified this >> way as just <data type="text"... in your tool. Those parameters will >> be passed through on the command-line just like any other parameter. >> >> Hope this helps. >> -John >> >>> Thank you again for your help, >>> >>> Misharl Monsoor. >>> >>> >>> >>> >>> >>> Le 19/11/2014 15:30, John Chilton a écrit : >>> >>>> If you install pycurl (globally or in a Galaxy's virtualenv) and set >>>> LWR_CURL_TRANSPORT=1 in Galaxy's environment than the LWR will stream >>>> large files with CURL instead of the mmap hack. >>>> >>>> I don't understand your second question - the LWR should stream >>>> whatever data inputs are selected by the user. If you want the tool to >>>> be able to take many inputs you can use multiple="true" on your data >>>> parameter or use a repeat block. >>>> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax). >>>> >>>> Hope this helps, >>>> -John >>>> >>>> >>>> On Mon, Nov 17, 2014 at 3:04 AM, Misharl Monsoor >>>> <mmonsoor(a)sb-roscoff.fr&gt; >>>> wrote: >>>>> Hi everybody, >>>>> >>>>> In our lab, we are trying to connect our Galaxy instance to a Windows 7 >>>>> 64 >>>>> bits server, in order to executes programs that need to be run within >>>>> Windows. However we have a problem concerning the size of the input >>>>> that >>>>> is >>>>> uploaded to the Windows server, it doesn't accept inputs with a size > >>>>> 1.3 >>>>> Gb ? We have the following problem with mmap function: >>>>> >>>>> data = mmap.mmap(input.fileno(), 0, access=mmap.ACCESS_READ) >>>>> error: [Errno 12] Cannot allocate memory >>>>> >>>>> >>>>> I have another question concerning the inputs, is there a way to use >>>>> several >>>>> inputs that be uploaded to a Windows Server? >>>>> >>>>> Thank you very much in advance, >>>>> >>>>> Bests regards, >>>>> >>>>> Cheers, >>>>> >>>>> Misharl Monsoor >>>>> ___________________________________________________________ >>>>> Please keep all replies on the list by using "reply all" >>>>> in your mail client. To manage your subscriptions to this >>>>> and other Galaxy lists, please use the interface at: >>>>> http://lists.bx.psu.edu/ >>>>> >>>>> To search Galaxy mailing lists use the unified search at: >>>>> http://galaxyproject.org/search/mailinglists/ >>>