Again on display on external UCSC
Hi again... I still have problems in visualizing on external UCSC sites using the display_application... I've implemented bigWig data format, and it works (not really the sniffer but who cares at the moment?).... What is important is that the display_applications gives me a link to the genome browser that is correct: http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... which points to http://genome.ifom-ieo-campus.it/cgi-bin/hgTracks?db=hg18&hgt.customText=http%3A%2F%2Fhost001.instruments.ifom-ieo-campus.it%3A8080%2Fdisplay_application%2Fcac7b9b902f807f2%2Fucsc_bigwig%2Fcampus%2F090e9746a4403c88%2Fparam%2Ftrack which has this link in hgt.customText http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... which says track type=bigWig name="wgEncodeYaleChIPseqRawDataRep2Helas3Pol2_fs.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... db=hg18 Everything is correct, apparently... Needles to say, the bigWig file is fully operational meaning that: 1- if I download the bigDataUrl to another machine and show that file through another apache, the genome browser can read it 2- the original file can be seen and loaded. BTW, the genome browser complains: http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... is not a chromosome id r-tree index file That probably means that there's something between the genome browser and the bigWig file (galaxy?) which passes wrong data/headers or possibly is passing something *before* the actual bigWig data stream... As I have issues with BAM files too, I believe the cause may be the same... Any help/hint/debug strategy is really appreciated! d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro@ifom-ieo-campus.it */
My attempts to understand why a bigwig file cannot be loaded by UCSC through galaxy's display_application ended up with tcpdump... If anybody would like to help me understanding what's wrong (and different) here I would really appreciate it. If needed I can make the tcpdupms available for analysis with wireshark... [ TCP Stream for a bigWig served outside galaxy] [stream 1: after setting hgt.customText in hgTracks equal to a path with bigWig track definition] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:50:29 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Fri, 11 Jun 2010 13:37:28 GMT ETag: "7140-9f-488c13d753600" Accept-Ranges: bytes Content-Length: 159 Connection: close Content-Type: text/plain track type=bigWig name="therightone.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it/ga-data/galaxy_dist/database/f... db=hg18 [stream 2 ] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:50:29 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT ETag: "9f-42e5586-488ae729691c0" Accept-Ranges: bytes Content-Length: 70145414 Connection: close Content-Type: chemical/x-mopac-input [stream 3] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:50:36 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT ETag: "9f-42e5586-488ae729691c0" Accept-Ranges: bytes Content-Length: 70145414 Connection: close Content-Type: chemical/x-mopac-input [stream 4: I believe this is after I ask for a different region on chromosome] HTTP/1.1 206 Partial Content Date: Fri, 11 Jun 2010 13:50:37 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT ETag: "9f-42e5586-488ae729691c0" Accept-Ranges: bytes Content-Length: 9819526 Content-Range: bytes 60325888-70145413/70145414 Connection: close Content-Type: chemical/x-mopac-input %.:....3...@...=>.} [cut, note that the bigWig data are not from the beginning of file] [stream 5] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:50:40 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT ETag: "9f-42e5586-488ae729691c0" Accept-Ranges: bytes Content-Length: 70145414 Connection: close Content-Type: chemical/x-mopac-input [ TCP Stream for the same bigWig served within galaxy] [stream 1: after hgt.customText is set to the path served by galaxy] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:46:29 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Fri, 11 Jun 2010 13:36:56 GMT ETag: "713f-f7-488c13b8cee00" Accept-Ranges: bytes Content-Length: 247 Connection: close Content-Type: text/plain track type=bigWig name="wgEncodeYaleChIPseqRawDataRep2Helas3Pol2_fs.bw" bigDataUrl=http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... db=hg18 [stream 2] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:46:29 GMT Server: PasteWSGIServer/0.5 Python/2.6.5 content-length: 70145414 content-type: chemical/x-mopac-input Set-Cookie: galaxysession=eb142648ac45b770ddabc752e2cd511e21664b5a654cabfbe79569d4ede184379f55c5dbc9000304; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1 Connection: close [stream 3: i get the error "http://host001.instruments.ifom-ieo-campus.it:8080/display_application/cac7b... is not a chromosome id r-tree index file" from the UCSC] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:46:29 GMT Server: PasteWSGIServer/0.5 Python/2.6.5 content-length: 70145414 content-type: chemical/x-mopac-input Set-Cookie: galaxysession=eb142648ac45b770233699f0ab65857e995c6e1b17d8e57d15b91a28b7b680af4cf210ffa8e6cc2a; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1 Connection: close &.......(......... [cut: note that bigWig data are served from the beginning, note that http headers are different] [stream 4: another attept to navigate on a different region of the chromosome] HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:46:30 GMT Server: PasteWSGIServer/0.5 Python/2.6.5 content-length: 70145414 content-type: chemical/x-mopac-input Set-Cookie: galaxysession=eb142648ac45b7709206aaeef08ccecdd9d6070c5f8aed59d9ddd2079322251934ece94b298127df; expires=Thu, 09-Sep-2010 15:46:30 GMT; Max-Age=7776000; Path=/; Version=1 Connection: close &.......(...............$. [cut: again, the bigWig data are sent entirely from the beginning] /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro@ifom-ieo-campus.it */
Looking at the headers, the big difference seems that galaxy webserver doesn't allow for "partial content"... On Jun 14, 2010, at 12:04 PM, Davide Cittaro wrote:
[stream 2 (working)]
HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:50:29 GMT Server: Apache/2.2.8 (Ubuntu) Last-Modified: Thu, 10 Jun 2010 15:12:15 GMT ETag: "9f-42e5586-488ae729691c0" Accept-Ranges: bytes Content-Length: 70145414 Connection: close Content-Type: chemical/x-mopac-input
[stream 2 (not working)]
HTTP/1.1 200 OK Date: Fri, 11 Jun 2010 13:46:29 GMT Server: PasteWSGIServer/0.5 Python/2.6.5 content-length: 70145414 content-type: chemical/x-mopac-input Set-Cookie: galaxysession=eb142648ac45b770ddabc752e2cd511e21664b5a654cabfbe79569d4ede184379f55c5dbc9000304; expires=Thu, 09-Sep-2010 15:46:29 GMT; Max-Age=7776000; Path=/; Version=1 Connection: close
Actually no "Accept-Ranges: bytes" nor the entity tag are present, I guess the hgTracks binary can't send a request for a specific section of the file and receives the whole stream in a single request. Looking at the w3c pages: 10.2.7 206 Partial Content The server has fulfilled the partial GET request for the resource. The request MUST have included a Range header field (section 14.35) indicating the desired range, and MAY have included an If-Range header field (section 14.27) to make the request conditional. The response MUST include the following header fields: - Either a Content-Range header field (section 14.16) indicating the range included with this response, or a multipart/byteranges Content-Type including Content-Range fields for each part. If a Content-Length header field is present in the response, its value MUST match the actual number of OCTETs transmitted in the message-body. - Date - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request - Expires, Cache-Control, and/or Vary, if the field-value might differ from that sent in any previous response for the same variant If the 206 response is the result of an If-Range request that used a strong cache validator (see section 13.3.3), the response SHOULD NOT include other entity-headers. If the response is the result of an If-Range request that used a weak validator, the response MUST NOT include other entity-headers; this prevents inconsistencies between cached entity-bodies and updated headers. Otherwise, the response MUST include all of the entity-headers that would have been returned with a 200 (OK) response to the same request. A cache MUST NOT combine a 206 response with other previously cached content if the ETag or Last-Modified headers do not match exactly, see 13.5.4. A cache that does not support the Range and Content-Range headers MUST NOT cache 206 (Partial) responses. The header fields listed above are not included when galaxy tries to serve the bigWig file. Is there a way to modify this behavior (possibly with the new datatype definition...). Thanks d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro@ifom-ieo-campus.it */
participants (1)
-
Davide Cittaro