• State Verified Answer
  • Replies 19 replies
  • Subscribers 26 subscribers
  • Views 2073 views
  • Users 0 members are here
Attachments (2) Download All
Nordic Case Info
  • Case ID: 259790
Options

AT#XHTTPCREQ serial_lte_modem does not handle chunked HTTP response

user95514

if a response to the http request is chunked (https://en.wikipedia.org/wiki/Chunked_transfer_encoding), it is not handled correctly by the serial_lte_modem.

Sometimes - e.g. if cloudflare is used, it is not possible to disable chunk encoding, so the serial_lte_modem app is unusable in these scenarios.

00> [00:03:22.707,031] <inf> httpc: Partial data received (743 bytes)
00> [00:03:22.707,061] <inf> httpc: All the data received (743 bytes)
00> [00:03:22.707,489] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  53 50 3a 37 34 33 2c 30 |#XHTTPCR SP:743,0
00> 0d 0a                                            |..               
00> [00:03:22.709,625] <inf> at_host: TX
00> 22 7d 5d 2c 22 67 72 6f  75 70 22 3a 22 63 66 2d |"}],"gro up":"cf-
00> 6e 65 6c 22 2c 22 6d 61  78 5f 61 67 65 22 3a 36 |nel","ma x_age":6
00> 30 34 38 30 30 7d 0d 0a  4e 45 4c 3a 20 7b 22 72 |04800}.. NEL: {"r
00> 65 70 6f 72 74 5f 74 6f  22 3a 22 63 66 2d 6e 65 |eport_to ":"cf-ne
00> 6c 22 2c 22 6d 61 78 5f  61 67 65 22 3a 36 30 34 |l","max_ age":604
00> 38 30 30 7d 0d 0a 53 65  72 76 65 72 3a 20 63 6c |800}..Se rver: cl
00> 6f 75 64 66 6c 61 72 65  0d 0a 43 46 2d 52 41 59 |oudflare ..CF-RAY
00> 3a 20 35 66 31 62 30 35  63 34 39 66 63 35 64 34 |: 5f1b05 c49fc5d4
00> 33 66 2d 48 41 4d 0d 0a  0d 0a 44 65 76 69 63 65 |3f-HAM.. ..Device
00> 20 31 38 32 30 65 66 66  64 20 77 61 73 20 6e 6f | 1820eff d was no
00> 74 20 66 6f 75 6e 64 64  34 39 37 36 63 30 32 39 |t foundd 4976c029
00> 37 37 37 30 38 64 39 66  62 35 36 33 31 30 34 65 |77708d9f b563104e
00> 36 62 30 61 31 31 31 34  31 36 30 35 32 39 37 30 |6b0a1114 16052970
00> 33 34 3b 20 65 78 70 69  72 65 73 3d 53 75 6e 2c |34; expi res=Sun,
00> 20 31 33 2d 44 65 63 2d  32 30 20 31 39 3a 35 30 | 13-Dec- 20 19:50
00> 3a 33 34 20 47 4d 54 3b  20 70 61 74 68 3d 2f 3b |:34 GMT;  path=/;
00> 20 64 6f 6d 61 69 6e 3d  2e 67 77 78 63 6c 6f 75 | domain= .gwxclou
00> 64 2e 63 6f 6d 3b 20 48  74 74 70 4f 6e 6c 79 3b |d.com; H ttpOnly;
00> 20 53 61 6d 65 53 69 74  65 3d 4c 61 78 0d 0a 43 | SameSit e=Lax..C
00> 46 2d 43 61 63 68 65 2d  53 74 61 74 75 73 3a 20 |F-Cache- Status: 
00> 44 59 4e 41 4d 49 43 0d  0a 63 66 2d 72 65 71 75 |DYNAMIC. .cf-requ
00> 65 73 74 2d 69 64 3a 20  30 36 36 34 63 31 65 65 |est-id:  0664c1ee
00> 64 63 30 30 30 30 64 34  33 66 36 38 38 37 30 30 |dc0000d4 3f688700
00> 30 30 30 30 30 30 30 31  0d 0a 52 65 70 6f 72 74 |00000001 ..Report
00> 2d 54 6f 3a 20 7b 22 65  6e 64 70 6f 69 6e 74 73 |-To: {"e ndpoints
00> 22 3a 5b 7b 22 75 72 6c  22 3a 22 68 74 74 70 73 |":[{"url ":"https
00> 3a 5c 2f 5c 2f 61 2e 6e  65 6c 2e 63 6c 6f 75 64 |:\/\/a.n el.cloud
00> 66 6c 61 72 65 2e 63 6f  6d 5c 2f 72 65 70 6f 72 |flare.co m\/repor
00> 74 3f 73 3d 4e 65 75 57  66 46 50 6b 75 47 25 32 |t?s=NeuW fFPkuG%2
00> 46 4e 61 61 4f 4f 43 65  69 70 41 51 4d 59 75 53 |FNaaOOCe ipAQMYuS
00> 75 61 69 39 36 25 32 46  7a 4e 6c 61 49 6d 50 49 |uai96%2F zNlaImPI
00> 74 72 52 44 67 7a 45 4f  55 61 4a 41 6e 78 66 51 |trRDgzEO UaJAnxfQ
00> 75 47 4f 74 34 75 34 73  52 48 46 5a 65 74 49 67 |uGOt4u4s RHFZetIg
00> 34 34 7a 34 74 6c 54 53  45 32 57 62 49 70 42 38 |44z4tlTS E2WbIpB8
00> 46 58 43 36 67 71 57 52  5a 4c 43 38 4f 69 41 41 |FXC6gqWR ZLC8OiAA
00> 44 43 7a 70 59 77 58 49  34 77 25 33 44 25 33 44 |DCzpYwXI 4w%3D%3D
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |........ ........
00> 00 00 00 00 00 00 00                             |.......

Parents
  • 0

    Hi, Michal!

    Thank you for reporting this. I will bring the info to the developers. 

    Best regards,
    Carl Richard

  • 0 in reply to user68044

    Hi Carl,

    do you have any info or ETA when this might be fixed? It is a big problem for us, basically making the serial_lte_modem unsusable in production and therefore delaying production of nrf91 based devices.

    Best regards, Michal V.

  • 0 in reply to user68044

    Hi again, Michal!

    A fix should have been pushed to the NCS master branch now. As described in the pull request the body was skipped. If you want to stay on 1.4.0 remove the following code from zephyr/subsys/net/lib/http/http_client.c : 

    if ((req->method == HTTP_PUT || req->method == HTTP_POST) &&
    req->internal.response.content_length == 0) {
	NET_DBG("No body expected");
	return 1;
}


    This should also fix the issue with the modem crashing, mentioned in your other ticket. Please report back there if it doesn't!

    Best regards,
    Carl Richard

  • 0 in reply to user68044

    Hello,

    Thank you for the fix - now it seems to me that the DBG message is gone, but the problem persists as the response still contains "headers" of the individual chunks.

    00> 23 58 48 54 54 50 43 52  45 51 3a 30 0d 0a       |#XHTTPCR EQ:0..  
00> [00:07:58.417,144] <dbg> net_http.http_client_req: (0x20021628): Sent 190 bytes
00> [00:07:59.161,193] <dbg> net_http.on_message_begin: (0x20021628): -- HTTP POST response (headers) --
00> [00:07:59.162,261] <dbg> net_http.on_status: (0x20021628): HTTP response status 200 OK
00> [00:07:59.163,055] <dbg> net_http.print_header_field: (0x20021628): [4] Date
00> [00:07:59.163,787] <dbg> net_http.print_header_field: (0x20021628): [29] Mon, 07 Dec 2020 17:04:48 GMT
00> [00:07:59.164,611] <dbg> net_http.print_header_field: (0x20021628): [12] Content-Type
00> [00:07:59.165,344] <dbg> net_http.print_header_field: (0x20021628): [31] application/json; charset=utf-8
00> [00:07:59.166,137] <dbg> net_http.print_header_field: (0x20021628): [17] Transfer-Encoding
00> [00:07:59.166,870] <dbg> net_http.print_header_field: (0x20021628): [7] chunked
00> [00:07:59.167,602] <dbg> net_http.print_header_field: (0x20021628): [10] Connection
00> [00:07:59.170,166] <dbg> net_http.print_header_field: (0x20021628): [10] keep-alive
00> [00:07:59.170,867] <dbg> net_http.print_header_field: (0x20021628): [10] Set-Cookie
00> [00:07:59.171,936] <dbg> net_http.print_header_field: (0x20021628): [127] __cfduid=d9f449204d08bfbd44f220c8051bf75ce1607360688; expires=Wed, 06-Jan-21 17:04:48 GMT; path=/; domain=.gwxcloud.com; HttpOn
00> [00:07:59.172,912] <dbg> net_http.print_header_field: (0x20021628): [4] Vary
00> [00:07:59.173,980] <dbg> net_http.print_header_field: (0x20021628): [15] Accept-Encoding
00> [00:07:59.175,018] <dbg> net_http.print_header_field: (0x20021628): [15] CF-Cache-Status
00> [00:07:59.175,933] <dbg> net_http.print_header_field: (0x20021628): [7] DYNAMIC
00> [00:07:59.176,513] <dbg> net_http.print_header_field: (0x20021628): [13] cf-request-id
00> [00:07:59.177,154] <dbg> net_http.print_header_field: (0x20021628): [32] 06dfc2c7880000d9055a0d0000000001
00> [00:07:59.177,795] <dbg> net_http.print_header_field: (0x20021628): [9] Report-To
00> [00:07:59.178,436] <dbg> net_http.print_header_field: (0x20021628): [127] {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=7ZqXcSGhkNFV%2FsOiuiL4QZsTPwu6xPX5iP6hLIhx6WF16taysh%2B0lic8osfS
00> [00:07:59.179,229] <dbg> net_http.print_header_field: (0x20021628): [3] NEL
00> [00:07:59.179,687] <dbg> net_http.print_header_field: (0x20021628): [39] {"report_to":"cf-nel","max_age":604800}
00> [00:07:59.186,004] <dbg> net_http.print_header_field: (0x20021628): [6] Server
00> [00:07:59.186,401] <dbg> net_http.print_header_field: (0x20021628): [6] cloudf
00> [00:07:59.224,151] <dbg> net_http.print_header_field: (0x20021628): [4] lare
00> [00:07:59.224,639] <dbg> net_http.print_header_field: (0x20021628): [6] CF-RAY
00> [00:07:59.255,035] <dbg> net_http.print_header_field: (0x20021628): [20] 5fdfd3ec0a7cd905-AMS
00> [00:07:59.255,798] <dbg> net_http.on_headers_complete: (0x20021628): Headers complete
00> [00:07:59.256,164] <dbg> net_http.on_body: (0x20021628): Processed 86 length 86
00> [00:07:59.256,500] <dbg> net_http.on_body: (0x20021628): Calling callback for partitioned 838 len data
00> [00:07:59.256,988] <dbg> net_http.on_message_complete: (0x20021628): -- HTTP POST response (complete) --
00> [00:07:59.257,843] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  53 50 3a 38 34 33 2c 30 |#XHTTPCR SP:843,0
00> 0d 0a                                            |..               
00> [00:07:59.260,345] <inf> at_host: TX
00> 48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d |HTTP/1.1  200 OK.
00> 0a 44 61 74 65 3a 20 4d  6f 6e 2c 20 30 37 20 44 |.Date: M on, 07 D
00> 65 63 20 32 30 32 30 20  31 37 3a 30 34 3a 34 38 |ec 2020  17:04:48
00> 20 47 4d 54 0d 0a 43 6f  6e 74 65 6e 74 2d 54 79 | GMT..Co ntent-Ty
00> 70 65 3a 20 61 70 70 6c  69 63 61 74 69 6f 6e 2f |pe: appl ication/
00> 6a 73 6f 6e 3b 20 63 68  61 72 73 65 74 3d 75 74 |json; ch arset=ut
00> 66 2d 38 0d 0a 54 72 61  6e 73 66 65 72 2d 45 6e |f-8..Tra nsfer-En
00> 63 6f 64 69 6e 67 3a 20  63 68 75 6e 6b 65 64 0d |coding:  chunked.
00> 0a 43 6f 6e 6e 65 63 74  69 6f 6e 3a 20 6b 65 65 |.Connect ion: kee
00> 70 2d 61 6c 69 76 65 0d  0a 53 65 74 2d 43 6f 6f |p-alive. .Set-Coo
00> 6b 69 65 3a 20 5f 5f 63  66 64 75 69 64 3d 64 39 |kie: __c fduid=d9
00> 66 34 34 39 32 30 34 64  30 38 62 66 62 64 34 34 |f449204d 08bfbd44
00> 66 32 32 30 63 38 30 35  31 62 66 37 35 63 65 31 |f220c805 1bf75ce1
00> 36 30 37 33 36 30 36 38  38 3b 20 65 78 70 69 72 |60736068 8; expir
00> 65 73 3d 57 65 64 2c 20  30 36 2d 4a 61 6e 2d 32 |es=Wed,  06-Jan-2
00> 31 20 31 37 3a 30 34 3a  34 38 20 47 4d 54 3b 20 |1 17:04: 48 GMT; 
00> 70 61 74 68 3d 2f 3b 20  64 6f 6d 61 69 6e 3d 2e |path=/;  domain=.
00> 67 77 78 63 6c 6f 75 64  2e 63 6f 6d 3b 20 48 74 |gwxcloud .com; Ht
00> 74 70 4f 6e 6c 79 3b 20  53 61 6d 65 53 69 74 65 |tpOnly;  SameSite
00> 3d 4c 61 78 0d 0a 56 61  72 79 3a 20 41 63 63 65 |=Lax..Va ry: Acce
00> 70 74 2d 45 6e 63 6f 64  69 6e 67 0d 0a 43 46 2d |pt-Encod ing..CF-
00> 43 61 63 68 65 2d 53 74  61 74 75 73 3a 20 44 59 |Cache-St atus: DY
00> 4e 41 4d 49 43 0d 0a 63  66 2d 72 65 71 75 65 73 |NAMIC..c f-reques
00> 74 2d 69 64 3a 20 30 36  64 66 63 32 63 37 38 38 |t-id: 06 dfc2c788
00> 30 30 30 30 64 39 30 35  35 61 30 64 30 30 30 30 |0000d905 5a0d0000
00> 30 30 30 30 30 31 0d 0a  52 65 70 6f 72 74 2d 54 |000001.. Report-T
00> 6f 3a 20 7b 22 65 6e 64  70 6f 69 6e 74 73 22 3a |o: {"end points":
00> 5b 7b 22 75 72 6c 22 3a  22 68 74 74 70 73 3a 5c |[{"url": "https:\
00> 2f 5c 2f 61 2e 6e 65 6c  2e 63 6c 6f 75 64 66 6c |/\/a.nel .cloudfl
00> 61 72 65 2e 63 6f 6d 5c  2f 72 65 70 6f 72 74 3f |are.com\ /report?
00> 73 3d 37 5a 71 58 63 53  47 68 6b 4e 46 56 25 32 |s=7ZqXcS GhkNFV%2
00> 46 73 4f 69 75 69 4c 34  51 5a 73 54 50 77 75 36 |FsOiuiL4 QZsTPwu6
00> 78 50 58 35 69 50 36 68  4c 49 68 78 36 57 46 31 |xPX5iP6h LIhx6WF1
00> 36 74 61 79 73 68 25 32  42 30 6c 69 63 38 6f 73 |6taysh%2 B0lic8os
00> 66 53 4a 73 34 38 48 68  6e 53 4b 5a 6d 62 6a 62 |fSJs48Hh nSKZmbjb
00> 41 44 25 32 42 47 76 30  62 31 41 57 36 34 4e 77 |AD%2BGv0 b1AW64Nw
00> 36 69 53 43 64 4f 59 72  38 57 4f 30 6a 5a 30 35 |6iSCdOYr 8WO0jZ05
00> 4f 47 51 4d 55 32 62 6d  48 67 25 33 44 25 33 44 |OGQMU2bm Hg%3D%3D
00> 22 7d 5d 2c 22 67 72 6f  75 70 22 3a 22 63 66 2d |"}],"gro up":"cf-
00> 6e 65 6c 22 2c 22 6d 61  78 5f 61 67 65 22 3a 36 |nel","ma x_age":6
00> 30 34 38 30 30 7d 0d 0a  4e 45 4c 3a 20 7b 22 72 |04800}.. NEL: {"r
00> 65 70 6f 72 74 5f 74 6f  22 3a 22 63 66 2d 6e 65 |eport_to ":"cf-ne
00> 6c 22 2c 22 6d 61 78 5f  61 67 65 22 3a 36 30 34 |l","max_ age":604
00> 38 30 30 7d 0d 0a 53 65  72 76 65 72 3a 20 63 6c |800}..Se rver: cl
00> 6f 75 64 66 6c 61 72 65  0d 0a 43 46 2d 52 41 59 |oudflare ..CF-RAY
00> 3a 20 35 66 64 66 64 33  65 63 30 61 37 63 64 39 |: 5fdfd3 ec0a7cd9
00> 30 35 2d 41 4d 53 0d 0a  0d 0a 35 36 0d 0a 7b 22 |05-AMS.. ..56..{"
00> 64 61 74 61 22 3a 22 33  33 34 36 61 62 32 36 65 |data":"3 346ab26e
00> 32 37 63 30 39 37 66 38  66 33 62 64 31 32 36 38 |27c097f8 f3bd1268
00> 66 35 39 37 30 30 34 35  65 34 66 35 30 64 34 37 |f5970045 e4f50d47
00> 37 34 38 37 66 63 34 33  61 62 34 37 37 61 36 35 |7487fc43 ab477a65
00> 30 61 63 32 34 66 38 22  2c 22 63 6d 64 22 3a 6e |0ac24f8" ,"cmd":n
00> 75 6c 6c 7d 0d 0a 30 0d  0a 0d 0a                |ull}..0. ...     
00> [00:07:59.302,124] <dbg> net_http.http_client_req: (0x20021628): Received 843 bytes

    In the end of the log two lines can be seen with the chunk headers:

    `00> 30 35 2d 41 4d 53 0d 0a  0d 0a 35 36 0d 0a 7b 22 |05-AMS.. ..56..{"`

    `00> 75 6c 6c 7d 0d 0a 30 0d  0a 0d 0a                |ull}..0. ...    `

    I believe the bytes `35 36 0d 0a` and `30 0d  0a 0d 0a` should not be part of the response, as they are just fragments of the http transport and have no meaning for the application communicating over HTTP. Also it would mean that the app has to parse again this response to eliminate those chunk headers.

    You can easily test this by calling:

    AT#XHTTPCCON=1,"gwxcloud.com",80

    AT#XHTTPCREQ="GET","/",""

    in the LTE Link Monitor app. In the following snippet, the ada is the chunk header.

    ....
Server: cloudflareCF-RAY: 5fdfcd499edefa5c-AMS
ada
<!DOCTYPE html>
...

    I haven't tested the modem crashing yet.

    Thank you for your help, Michal V.

  • 0 in reply to user95514

    Thanks for the update, Michal! Good to hear that there are some progress. I've made the developer aware that the issue with chunks persists and they are working on it!

    Best regards,
    Carl Richard

  • +1 in reply to user95514

    Hi Michal,

    I've added the handling of chunked payload in attached slm_at_httpc.c. Can you try it with your chunked response?

    Best Regards,

    Larry

  • 0 in reply to user26078

    Hi Larry,

    Thank you for the fix, I'll try it tomorrow and let you know.

    Best Regards, Michal

Reply Children
No Data
Related
Terms of service