rohc team mailing list archive
-
rohc team
-
Mailing list archive
-
Message #02241
Re: Oops in arm kernel 4.4.32 on d_tcp_parse_packet
-
To:
"rohc@xxxxxxxxxxxxxxxxxxx" <rohc@xxxxxxxxxxxxxxxxxxx>
-
From:
Yakir Matusovsky <yakir.matusovsky@xxxxxxxxxxx>
-
Date:
Fri, 23 Mar 2018 02:08:54 +0000
-
Accept-language:
en-NZ, en-US
-
Authentication-results:
spf=none (sender IP is ) smtp.mailfrom=yakir.matusovsky@xxxxxxxxxxx;
-
In-reply-to:
<20180322114143.496ad5c3@chimay.b2i-toulouse.prive>
-
Spamdiagnosticmetadata:
NSPM
-
Spamdiagnosticoutput:
1:99
-
Thread-index:
AQHTwZEUQrUVEXL1n0ywsuhz5A5uyqPcEZqAgAHcugA=
-
Thread-topic:
[Rohc] Oops in arm kernel 4.4.32 on d_tcp_parse_packet
Didier,
Applied your patches thanks. That problem is definitely fixed.
Something else here that I keep experiencing. I perform an SSHv2/TCP session over a link and it is inconsistent with TCP profile enabled.
Inconsistent - means, if I terminate the session and then re-open it, it stalls.
I get these (verbose level 2) on remote radio,
[ /decomp/d_tcp.c:1004 d_tcp_parse_ir_cr()] Base CID 0 does not exist, so it cannot be used for Context Replication by CID 2
[ 130.547529] [ /decomp/rohc_decomp.c:1375 rohc_decomp_decode_pkt()] failed to parse the IR-CR header
[ 130.563969] [ /decomp/rohc_decomp.c:1247 d_decode_header()] failed to decompress packet (code = 2)
[ 130.580353] [ /decomp/rohc_decomp.c:881 rohc_decompress3()] packet decompression failed: malformed packet (2)
[ 130.597712] ROHC-CMS (Err):RoHC error: MALFORMED
[ 131.558647] [ /decomp/d_tcp.c:1004 d_tcp_parse_ir_cr()] Base CID 0 does not exist, so it cannot be used for Context Replication by CID 2
[ 131.578544] [ /decomp/rohc_decomp.c:1375 rohc_decomp_decode_pkt()] failed to parse the IR-CR header
[ 131.594988] [ /decomp/rohc_decomp.c:1247 d_decode_header()] failed to decompress packet (code = 2)
[ 131.611346] [ /decomp/rohc_decomp.c:881 rohc_decompress3()] packet decompression failed: malformed packet (2)
[ 131.628700] ROHC-CMS (Err):RoHC error: MALFORMED
[ 133.470413] [ /decomp/d_tcp.c:1004 d_tcp_parse_ir_cr()] Base CID 0 does not exist, so it cannot be used for Context Replication by CID 2
[ 133.490288] [ /decomp/rohc_decomp.c:1375 rohc_decomp_decode_pkt()] failed to parse the IR-CR header
[ 133.506730] [ /decomp/rohc_decomp.c:1247 d_decode_header()] failed to decompress packet (code = 2)
[ 133.523091] [ /decomp/rohc_decomp.c:881 rohc_decompress3()] packet decompression failed: malformed packet (2)
[ 133.540457] ROHC-CMS (Err):RoHC error: MALFORMED
[ 137.445591] [ /decomp/rohc_decomp.c:1204 d_decode_header()] CID 2: packet 'IR-DYN' (1) does not carry static information, it cannot be received in No Context state
[ 137.467975] [ /decomp/rohc_decomp.c:881 rohc_decompress3()] packet decompression failed: malformed packet (2)
[ 137.485341] ROHC-CMS (Err):RoHC error: MALFORMED
PS. The prints with a string ROHC-CMS are my RoHC application wrapper error codes.
PS2. On the start of SSH session every time I do get a feedback sent from decompressor and received at compressor side.
PS3. No errors over the air, link is perfect.
PS4. It looks like something around the TCP compression; when run with all enabled except TCP it works fine.
Regards,
Yakir
On 22/03/18, 11:41 PM, "Rohc on behalf of Didier Barvaux" <rohc-bounces+yakir.matusovsky=mimomax.com@xxxxxxxxxxxxxxxxxxx on behalf of didier@xxxxxxxxxxx> wrote:
Yakir,
This is a known problem. I discovered it while fuzzing the decompressor
last month. I cherry-picked it for you on the 2.1.x bugfix branch along
with some other fixes.
Please give a try to the 2.1.x branch:
https://github.com/didier-barvaux/rohc/commits/2.1.x
All those fixes should be soon released in a future 2.1.1 release.
Regards,
Didier
Le Thu, 22 Mar 2018 03:51:38 +0000,
Yakir Matusovsky <yakir.matusovsky@xxxxxxxxxxx> a écrit :
> Following my concern here, I’ve continued to debug the ko and got the
> following to protect from oops,
>
> [ decomp/d_tcp.c:1009 d_tcp_parse_ir_cr()]
> context->decompressor->contexts[base_cid] == NULL
>
> Means, I’ve added a red code below in d_tcp.c,
>
> /* check whether the decoded base CID is allowed by the
> decompressor */ if(base_cid > context->decompressor->medium.max_cid)
> {
> rohc_decomp_warn(context, "unexpected Base
> CID %zu received: MAX_CID " "was set to %zu", base_cid,
> context->decompressor->medium.max_cid);
> goto error;
> }
>
> if(context->decompressor->contexts[base_cid] == NULL)
> {
> rohc_decomp_warn(context,
> "context->decompressor->contexts[base_cid] == NULL\n"); goto error;
> }
>
> base_context = context->decompressor->contexts[base_cid];
>
> Please advise the best way forward…
>
> Regards,
> Yakir
> From: Rohc
> <rohc-bounces+yakir.matusovsky=mimomax.com@xxxxxxxxxxxxxxxxxxx> on
> behalf of Yakir Matusovsky <yakir.matusovsky@xxxxxxxxxxx> Date:
> Thursday, 22 March 2018 at 4:19 PM To: "rohc@xxxxxxxxxxxxxxxxxxx"
> <rohc@xxxxxxxxxxxxxxxxxxx> Subject: [Rohc] Oops in arm kernel 4.4.32
> on d_tcp_parse_packet
>
> Hi
>
> I’ve experienced the problem below a few times already and want to
> get to the bottom of it. I have a data link and I initiate a remote
> SSH session (to the node over a link). I get this crash (my kernel
> panics on oops),
>
> > [ 458.076694] Internal error: Oops: 17 [#1] ARM
> [ 458.081321] Modules linked in: mdl_driver(O) rohc(O) fpga_driver(O)
> [ 458.087958] CPU: 0 PID: 505 Comm: kworker/0:2 Tainted: G
> O 4.4.32 #3 [ 458.095859] Hardware name: Generic AM33XX
> (Flattened Device Tree) [ 458.102485] Workqueue: ReceiveFramesQueue
> mac__frame_rx_task [mdl_driver] [ 458.109651] task: cc729100 ti:
> cc552000 task.ti: cc552000 [ 458.115691] PC is at
> d_tcp_parse_packet+0x64c/0x1194 [rohc] [ 458.121729] LR is at
> d_tcp_parse_packet+0x5e0/0x1194 [rohc] [ 458.127608] pc :
> [<bf077c34>] lr : [<bf077bc8>] psr: 80000013 [ 458.127608]
> sp : cc553940 ip : cc553940 fp : cc5539b4 [ 458.139709] r10:
> 00000000 r9 : 00000029 r8 : cc6a25e7 [ 458.145215] r7 : cc6a25e2
> r6 : 00000002 r5 : bf0ce8b4 r4 : cb175800 [ 458.152100] r3 :
> cc558080 r2 : cc580000 r1 : 00000000 r0 : 00000000 [ 458.158992]
> Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> [ 458.166513] Control: 10c5387d Table: 8b07c019 DAC: 00000051
> [ 458.172567] Process kworker/0:2 (pid: 505, stack limit =
> 0xcc552208) [ 458.179252] Stack: (0xcc553940 to 0xcc554000)
> [ 458.183851] 3940: bf0a1b32 bf09fa89 000003e1 bf07e554 00000002
> 00000008 c0097ac4 c00970c0 [ 458.192487] 3960: cb0f8000 cb175810
> cb175800 cb17580c cc55398c cc55398c cc5539bc cc553988 [ 458.201109]
> 3980: bf072ce4 bf0ce8c0 bf09fcac cc553b6c bf0775e8 cb175800 bf0ce8b4
> cc553b98 [ 458.209748] 39a0: cc580000 bf07e6c0 cc553ae4 cc5539c0
> bf054be8 bf0775f8 3880cb87 00000000 [ 458.218374] 39c0: 08c27650
> 00000000 cc6a25dd 00000033 00000005 0000002e 00000000 cc553b98
> [ 458.227008] 39e0: cb175810 cb178000 cc553a40 c00470dc c0056458
> c002f5b0 c06248c0 20070013 [ 458.235638] 3a00: cb175810 cc552000
> cc3714a4 cc552000 cc553b88 00000020 cb178000 cb168000 [ 458.244267]
> 3a20: 00000006 00000005 bf07e6c0 00000001 00000001 cc6a25dd 00000000
> 0000002e [ 458.252905] 3a40: c03f2ba4 c03f0b20 3880cb87 00000000
> 08c27650 00000000 cc139154 00000001 [ 458.261541] 3a60: cb0e46c0
> cb0e46c0 cc139154 7fffffff cc3714a4 cc552000 3880cb87 00000000
> [ 458.270174] 3a80: 08c27650 00000000 cc6a25dd 00000033 00000005
> 0000002e 3880cb87 00000000 [ 458.278805] 3aa0: 08c27650 00000000
> cc6a25dd 00000033 00000005 0000002e 000047f1 cc580000 [ 458.287441]
> 3ac0: cc553c98 cc590000 cc553c58 00000033 cc553c78 cc553bd0 cc553bcc
> cc553af0 [ 458.296072] 3ae0: bf055fc0 bf0532e0 3880cb87 00000000
> 08c27650 00000000 cc6a25dd 00000033 [ 458.304703] 3b00: 00000000
> 00000033 cc553c58 cc553c78 cc553b6c c05c5738 00000000 a0000093
> [ 458.313333] 3b20: 00000000 cc339400 cc339400 cc553c8c cc553b54
> cc553b40 c0047b50 c004c3ec [ 458.321967] 3b40: 00000000 a0000013
> 00000000 cc553cc0 60000013 cc339400 cc339400 cc553c8c [ 458.330606]
> 3b60: cc553b84 cc553b70 c004ee00 00000001 00000001 00000008 cc553b01
> cb175800 [ 458.339236] 3b80: 00000006 00000001 cc339400 00000001
> 00000000 00000000 00000020 c004f300 [ 458.347875] 3ba0: cc553bd4
> cc6a0000 cc6a25dd 0000000c cc580000 cc5c0000 cc6a0498 00000000
> [ 458.356507] 3bc0: cc553ce4 cc553bd8 bf0d02c4 bf055c5c 3880cb87
> 00000000 08c27650 00000000 [ 458.365148] 3be0: cc6a25dd 00000033
> 00000000 00000033 cc553c58 cc553c78 cc553c98 cc339400 [ 458.373784]
> 3c00: 00000000 00023e12 3880cb87 00000640 cc6a0ad8 00000000 cc6a25dd
> 0000000c [ 458.382413] 3c20: 00000000 0000000e 00000001 0000000c
> 00000001 00000000 3880cb87 00000000 [ 458.391049] 3c40: 08c27650
> 00000000 cc6a25dd 00000033 00000000 00000033 00000000 00000000
> [ 458.399682] 3c60: 00000000 00000000 cc6a04a6 00000632 00000000
> 00000000 00000000 00000000 [ 458.408307] 3c80: 00000000 00000000
> cc6a0ad8 000001f4 00000000 00000000 00000000 00000000 [ 458.416946]
> 3ca0: 00000000 00000000 cc6a0cdb 000001e5 00000000 00000000 c00649f0
> cc6a0000 [ 458.425581] 3cc0: bf0db600 cc6a2480 cc6a0480 cc6a25dd
> 00000001 cc6a0f90 cc553d3c cc553ce8 [ 458.434213] 3ce0: bf0c9d88
> bf0cfbe8 cc6a2c1e 00000640 cc553d08 00000000 00000008 00000000
> [ 458.442848] 3d00: ab7a1aa8 ab7a1aa8 3880cb87 00023e12 cc553d3c
> cc6a0000 bf0db600 cc6a15da [ 458.451479] 3d20: cc6a2c90 bf0d7f9b
> cc6a2000 bf0d8387 cc553d9c cc553d40 bf0ca5dc bf0c9b80 [ 458.460104]
> 3d40: 00000000 00000000 00000000 00000000 00000000 ab000000 00000000
> 00010037 [ 458.468744] 3d60: 00000037 00000037 00000000 00000000
> 00000000 bf0df358 00000001 00000038 [ 458.477368] 3d80: 00000000
> 00000000 bf0df77c 00000644 cc553dcc cc553da0 bf0d64f4 bf0c9fac
> [ 458.486009] 3da0: 00000000 00ab0000 00000000 bf0df77c 00000646
> 00000000 00000000 00000646 [ 458.494639] 3dc0: cc553e0c cc553dd0
> bf0d666c bf0d6348 00002010 000f4240 bf0db640 bf0df77c [ 458.503271]
> 3de0: 00000000 00000001 00000000 bf0deba6 00000037 00000000 00000000
> 00000037 [ 458.511902] 3e00: cc553e54 cc553e10 bf0d6984 bf0d6528
> 00000000 bf0deba6 00000037 00000000 [ 458.520536] 3e20: 8f489f0c
> 00000000 cc553e98 bf0deb9e 00000037 00000002 bf0df140 bf0db640
> [ 458.529167] 3e40: 00000000 00000008 cc553e84 cc553e58 bf0d7a50
> bf0d6904 00000037 00000000 [ 458.537791] 3e60: bf0deba6 00000037
> 08c1d53a bf0deb9e 0000003f bf0debdd cc553ec4 cc553e88 [ 458.546423]
> 3e80: bf0d3d80 bf0d79c0 00000044 ccd7f700 cc553eac cc6f3200 3880cb87
> 00023de8 [ 458.555058] 3ea0: 00000000 00000000 bf0deb9e 00000044
> bf0df140 bf0db640 cc553eec cc553ec8 [ 458.563693] 3ec0: bf0d3eec
> bf0d3bd8 cc6f3200 bf0de338 c05c4f8c 00000000 ccd7f700 00000000
> [ 458.572329] 3ee0: cc553f2c cc553ef0 c003eb64 bf0d3dc0 c05c4f8c
> c05c4f8c c05c70c0 c05c4f9c [ 458.580963] 3f00: cc6f3218 cc6f3200
> c05c4f8c c05c4f8c c05c70c0 c05c4f9c cc6f3218 00000008 [ 458.589596]
> 3f20: cc553f64 cc553f30 c003f694 c003e9b4 c003f3e0 00000000 00000000
> cc6f0600 [ 458.598224] 3f40: 00000000 cc6f3200 c003f3e0 00000000
> 00000000 00000000 cc553fac cc553f68 [ 458.606860] 3f60: c0043a8c
> c003f3ec cc6f0600 00000000 00000000 cc6f3200 00000000 cc553f7c
> [ 458.615498] 3f80: cc553f7c 00000000 cc553f88 cc553f88 cc6f0600
> c00439b8 00000000 00000000 [ 458.624133] 3fa0: 00000000 cc553fb0
> c000efb8 c00439c4 00000000 00000000 00000000 00000000 [ 458.632765]
> 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 [ 458.641397] 3fe0: 00000000 00000000 00000000 00000000
> 00000013 00000000 625f4449 735f7572 [ 458.650014] Backtrace:
> [ 458.652881] [<bf0775ec>] (d_tcp_parse_packet [rohc]) from
> [<bf054be8>] (d_decode_header+0x1918/0x2854 [rohc]) [ 458.663336]
> r10:bf07e6c0 r9:cc580000 r8:cc553b98 r7:bf0ce8b4 r6:cb175800
> r5:bf0775e8 [ 458.671617] r4:cc553b6c [ 458.674556] [<bf0532d4>]
> (d_decode_header [rohc]) from [<bf055fc0>]
> (rohc_decompress3+0x374/0x18dc [rohc]) [ 458.684747] r10:cc553bd0
> r9:cc553c78 r8:00000033 r7:cc553c58 r6:cc590000 r5:cc553c98
> [ 458.693027] r4:cc580000 [ 458.695946] [<bf055c50>]
> (rohc_decompress3 [rohc]) from [<bf0d02c4>]
> (rohc_cms_decompress+0x6e8/0xd10 [mdl_driver]) [ 458.706943]
> r10:00000000 r9:cc6a0498 r8:cc5c0000 r7:cc580000 r6:0000000c
> r5:cc6a25dd [ 458.715221] r4:cc6a0000 [ 458.718051] [<bf0cfbdc>]
> (rohc_cms_decompress [mdl_driver]) from [<bf0c9d88>]
> (rx_decompress+0x214/0x3dc [mdl_driver]) [ 458.729327] r10:cc6a0f90
> r9:00000001 r8:cc6a25dd r7:cc6a0480 r6:cc6a2480 r5:bf0db600
> [ 458.737603] r4:cc6a0000 [ 458.740426] [<bf0c9b74>]
> (rx_decompress [mdl_driver]) from [<bf0ca5dc>]
> (mdl_driver_rx_tasklet+0x63c/0x6ac [mdl_driver]) [ 458.751895]
> r10:bf0d8387 r9:cc6a2000 r8:bf0d7f9b r7:cc6a2c90 r6:cc6a15da
> r5:bf0db600 [ 458.760177] r4:cc6a0000 [ 458.763005] [<bf0c9fa0>]
> (mdl_driver_rx_tasklet [mdl_driver]) from [<bf0d64f4>]
> (mac__enqueue_data+0x1b8/0x1e0 [mdl_driver]) [ 458.774834]
> r10:00000644 r9:bf0df77c r8:00000000 r7:00000000 r6:00000038
> r5:00000001 [ 458.783116] r4:bf0df358 [ 458.785944] [<bf0d633c>]
> (mac__enqueue_data [mdl_driver]) from [<bf0d666c>]
> (mac_frag__newcue+0x150/0x1e4 [mdl_driver]) [ 458.797321]
> r9:00000646 r8:00000000 r7:00000000 r6:00000646 r5:bf0df77c
> r4:00000000 [ 458.805672] [<bf0d651c>] (mac_frag__newcue
> [mdl_driver]) from [<bf0d6984>] (mac_frag__process_frame+0x8c/0x11c
> [mdl_driver]) [ 458.817499] r10:00000037 r9:00000000 r8:00000000
> r7:00000037 r6:bf0deba6 r5:00000000 [ 458.825781] r4:00000001
> [ 458.828618] [<bf0d68f8>] (mac_frag__process_frame [mdl_driver])
> from [<bf0d7a50>] (extract_data_frame+0x9c/0xb4 [mdl_driver])
> [ 458.840529] r10:00000008 r9:00000000 r8:bf0db640 r7:bf0df140
> r6:00000002 r5:00000037 [ 458.848812] r4:bf0deb9e [ 458.851650]
> [<bf0d79b4>] (extract_data_frame [mdl_driver]) from [<bf0d3d80>]
> (rx_one_frame+0x1b4/0x1e8 [mdl_driver]) [ 458.862740] r6:bf0debdd
> r5:0000003f r4:bf0deb9e [ 458.867776] [<bf0d3bcc>] (rx_one_frame
> [mdl_driver]) from [<bf0d3eec>] (mac__frame_rx_task+0x138/0x17c
> [mdl_driver]) [ 458.878875] r8:bf0db640 r7:bf0df140 r6:00000044
> r5:bf0deb9e r4:00000000 [ 458.886052] [<bf0d3db4>]
> (mac__frame_rx_task [mdl_driver]) from [<c003eb64>]
> (process_one_work+0x1bc/0x2ec) [ 458.896326] r9:00000000
> r8:ccd7f700 r7:00000000 r6:c05c4f8c r5:bf0de338 r4:cc6f3200
> [ 458.904529] [<c003e9a8>] (process_one_work) from [<c003f694>]
> (worker_thread+0x2b4/0x3f0) [ 458.913157] r10:00000008 r9:cc6f3218
> r8:c05c4f9c r7:c05c70c0 r6:c05c4f8c r5:c05c4f8c [ 458.921443]
> r4:cc6f3200 [ 458.924132] [<c003f3e0>] (worker_thread) from
> [<c0043a8c>] (kthread+0xd4/0xec) [ 458.931746] r10:00000000
> r9:00000000 r8:00000000 r7:c003f3e0 r6:cc6f3200 r5:00000000
> [ 458.940029] r4:cc6f0600 [ 458.942719] [<c00439b8>] (kthread)
> from [<c000efb8>] (ret_from_fork+0x14/0x3c) [ 458.950346]
> r7:00000000 r6:00000000 r5:c00439b8 r4:cc6f0600 [ 458.956337] Code:
> e12fff35 ea000097 e5923018 e793a106 (e59a3008) [ 458.962893]
> ---[ end trace f6e6f365816638a5 ]--- [ 458.967792] Kernel panic -
> not syncing: Fatal exception in interrupt [ 458.974494] Rebooting in
> 30 seconds..
>
> Traces prior to crash show the following
>
> [ 485.966625] [ decomp/rohc_decomp.c:770 rohc_decompress3()]
> decompress the 51-byte packet #1
>
> [ 485.982491] [ decomp/rohc_decomp.c:3792
> rohc_decomp_parse_padding()] skip 4 byte(s) of padding
>
> [ 485.998560] [ decomp/rohc_decomp.c:1017 d_decode_header()]
> decompressor received 0 bytes of feedback for the same-side
> associated compressor
>
> [ 486.018811] [ decomp/rohc_decomp.c:3728 rohc_decomp_decode_cid()]
> add-CID present (0xe9) contains CID = 9
>
> [ 486.035899] [ decomp/rohc_decomp.c:3862
> rohc_decomp_find_context()] ROHC packet is an IR, IR-CR or IR-DYN
> packet
>
> [ 486.053606] [ decomp/rohc_decomp.c:3885
> rohc_decomp_find_context()] profile ID 0x0006 found in IR(-CR|-DYN)
> packet
>
> [ 486.071494] [ decomp/rohc_decomp.c:3901
> rohc_decomp_find_context()] context with CID 9 not found
>
> [ 486.087742] [ decomp/rohc_decomp.c:3964
> rohc_decomp_find_context()] create new context with CID 9 and profile
> 'IP/TCP' (0x0006)
>
> [ 486.106872] [ decomp/rohc_decomp.c:1162 d_decode_header()] decode
> packet with profile 'IP/TCP' (0x0006)
>
> [ 486.123763] [ decomp/d_tcp.c:596 tcp_detect_packet_type()] try to
> determine the header from first byte 0xfc
>
> [ 486.141014] [ decomp/rohc_decomp.c:1194 d_decode_header()] decode
> packet as 'IR-CR'
>
> [ 486.156079] [ decomp/rohc_decomp.c:1367 rohc_decomp_decode_pkt()]
> parse packet type 'IR-CR' (32)
>
> [ 486.172331] [ decomp/d_tcp.c:748 d_tcp_parse_packet()] rohc_length
> = 46, large_cid_len = 0
>
> [ 486.188024] [ decomp/d_tcp.c:937 d_tcp_parse_ir_cr()] B = 1 =>
> Base CID is present in packet
>
> [ 486.203905] [ decomp/d_tcp.c:939 d_tcp_parse_ir_cr()] CRC7 = 0x00
>
> [ 486.217307] [ decomp/d_tcp.c:969 d_tcp_parse_ir_cr()] 1-byte small
> base CID = 2
>
> [ 486.231980] [ decomp/d_tcp.c:993 d_tcp_parse_ir_cr()] IR-CR asks
> to replicate the Base CID 2 in the CID 9
>
> [ 486.249052] Unable to handle kernel NULL pointer dereference at
> virtual address 00000008
>
> It is well reproducible if I reboot the device (local or remote) and
> do from local ssh root@remoteIP right after boot.
>
> I use 2.1.0 rohc.ko loadable module, I wonder if anything like that
> happened before. Pretty sure it is the kernel fix branch.
>
> Thanks!
>
> Regards,
> Yakir Matusovsky
> MiMOMax Wireless Ltd.
>
_______________________________________________
Mailing list: https://launchpad.net/~rohc
Post to : rohc@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~rohc
More help : https://help.launchpad.net/ListHelp
Follow ups
References
