Skip to content

kernel panic on kpatch load with 4.9.88-1 #852

Closed
Closed
kernel panic on kpatch load with 4.9.88-1#852

Description

@rudis
rudis
opened on May 18, 2018

Hello,

I get the following kernel panic when applying a kpatch built with 0.5.0 (source patch), running kernel was 4.9.88-1 on Debian GNU/Linux Stretch:

[1164001.946575] general protection fault: 0000 [#1] SMP
[1164001.991105] Modules linked in: livepatch_4_9_88_1_20180518_1(OK+) livepatch_4_9_88_1_20180510_1(OK) tcp_diag udp_diag inet_diag unix_diag nf_conntrack_netlink nfnetlink ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common serpent_sse2_x86_64 serpent_generic cpuid ipt_MASQUERADE nf_nat_masquerade_ipv4 binfmt_misc drbg ansi_cprng authenc echainiv esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel cbc xfrm_user xfrm_algo ip_vti ip_tunnel tun livepatch_4_9_88_1_20180504_1(OK) nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 xt_owner xt_conntrack iptable_filter iptable_mangle xt_NETMAP
[1164002.606622]  xt_REDIRECT nf_nat_redirect xt_tcpudp iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_CT nf_conntrack xt_mark iptable_raw quota_v2 quota_tree iTCO_wdt iTCO_vendor_support evdev intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass intel_cstate intel_uncore intel_rapl_perf mgag200 pcspkr serio_raw ttm drm_kms_helper drm i2c_algo_bit hpilo hpwdt ie31200_edac sg lpc_ich mfd_core shpchp edac_core ipmi_si acpi_power_meter button pcc_cpufreq ipmi_devintf ipmi_msghandler coretemp loop ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb lrw glue_helper ablk_helper aes_x86_64 mbcache xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid10 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c
[1164003.205080]  crc32c_generic raid1 md_mod ses enclosure sd_mod scsi_transport_sas uas usb_storage ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel tg3 ehci_pci libata uhci_hcd ptp pps_core xhci_pci ghash_clmulni_intel cryptd xhci_hcd psmouse ehci_hcd scsi_mod libphy thermal usbcore usb_common
[1164003.424068] CPU: 1 PID: 17257 Comm: insmod Tainted: G           O  K 4.9.0-6-amd64 #1 Debian 4.9.88-1
[1164003.505823] Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 11/02/2015
[1164003.568970] task: ffff9ff3411a4480 task.stack: ffffac8f8271c000
[1164003.622925] RIP: 0010:[<ffffffff8ae2e1d0>]  [<ffffffff8ae2e1d0>] apply_paravirt+0xc0/0x140
[1164003.696246] RSP: 0018:ffffac8f8271f9a0  EFLAGS: 00010216
[1164003.744368] RAX: 00010749ffffffff RBX: ffffffffc0940658 RCX: 0000000000000085
[1164003.808033] RDX: 00000000bfebfbff RSI: ffffac8f8271f9a2 RDI: 0000000000000246
[1164003.871750] RBP: ffffac8f8271f9a2 R08: 0000000000000085 R09: ffffffff8ae5acb0
[1164003.935551] R10: 0000000000000001 R11: ffff9ff3544b4400 R12: ffffffffc0940660
[1164003.999178] R13: 0000000000000000 R14: ffff9ff3544b49c0 R15: ffff9ff3b43f0800
[1164004.062853] FS:  00007f04c1cea700(0000) GS:ffff9ff3ca640000(0000) knlGS:0000000000000000
[1164004.134673] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1164004.186437] CR2: 0000560cfd63e460 CR3: 00000001455c6000 CR4: 0000000000160670
[1164004.250434] Stack:
[1164004.270398]  401f0ff889486973 6172007172006b00 746c00650031312e 007265746e655f69
[1164004.336438]  74006e6f69007870 ffffac8f006e6f69 00ffac8f8271fa28 ffffffff8b13ae86
[1164004.402292]  ffffac8f8271fa68 ffffffffc09471ec ffffffff8b7da9eb 0000000affffffff
[1164004.468192] Call Trace:
[1164004.491969]  [<ffffffff8b13ae86>] ? vsscanf+0x4c6/0x800
[1164004.539197]  [<ffffffff8b13b20e>] ? sscanf+0x4e/0x70
[1164004.584207]  [<ffffffff8ae52be5>] ? arch_klp_init_object_loaded+0x105/0x130
[1164004.646577]  [<ffffffff8b13b0be>] ? vsscanf+0x6fe/0x800
[1164004.693747]  [<ffffffff8b13b20e>] ? sscanf+0x4e/0x70
[1164004.738656]  [<ffffffff8aee29e8>] ? klp_init_object_loaded+0xf8/0x210
[1164004.796141]  [<ffffffff8aee2d85>] ? klp_register_patch+0x285/0x390
[1164004.851443]  [<ffffffffc09491fa>] ? patch_init+0x1fa/0x1000 [livepatch_4_9_88_1_20180518_1]
[1164004.925209]  [<ffffffffc0949000>] ? 0xffffffffc0949000
[1164004.971569]  [<ffffffff8ae0218e>] ? do_one_initcall+0x4e/0x180
[1164005.023799]  [<ffffffff8afc87dd>] ? __vunmap+0x6d/0xc0
[1164005.070030]  [<ffffffff8afc87dd>] ? __vunmap+0x6d/0xc0
[1164005.116280]  [<ffffffff8af7eaa1>] ? do_init_module+0x5b/0x1ed
[1164005.167791]  [<ffffffff8af025a6>] ? load_module+0x2596/0x2ab0
[1164005.219206]  [<ffffffff8aefed50>] ? __symbol_put+0x60/0x60
[1164005.268346]  [<ffffffff8af02d06>] ? SYSC_finit_module+0xc6/0xf0
[1164005.321267]  [<ffffffff8ae03b7d>] ? do_syscall_64+0x8d/0xf0
[1164005.371234]  [<ffffffff8b41244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[1164005.432951] Code: 8d 7c 05 00 e8 62 f7 ff ff 0f b6 53 f9 48 8b 7b f0 48 89 ee e8 f2 f8 ff ff 49 39 dc 76 57 44 0f b6 43 09 41 80 f8 ff 75 84 0f 0b <48> 8b 10 48 8d 7d 08 48 83 e7 f8 48 89 55 00 89 ca 48 8b 74 10 
[1164005.598265] RIP  [<ffffffff8ae2e1d0>] apply_paravirt+0xc0/0x140
[1164005.651222]  RSP <ffffac8f8271f9a0>
[1164005.686679] ---[ end trace 128c0fa6efe85d9e ]---

Regards
Simon

Activity

  1. joe-lawrence commented on May 18, 2018

    @joe-lawrence
    joe-lawrence
    on May 18, 2018
    Contributor

    From a repro of the same issue:

    [ 2918.933946] RIP: 0010:[<ffffffffb602f37e>]  [<ffffffffb602f37e>] apply_paravirt+0xce/0x150
[ 2918.935078] RAX: 0001074effffffff

% scripts/faddr2line vmlinux apply_paravirt+0xce
apply_paravirt+0xce/0x14c:
apply_paravirt at arch/x86/kernel/alternative.c:585

585                 memcpy(insnbuf, p->instr, p->len);

Disassembly of section .text.apply_paravirt:

0000000000000000 <apply_paravirt>:
...
  ce:	48 8b 10             	mov    (%rax),%rdx

    (Summarizing @jpoimboe 's analysis on irc):
    RAX doesn't look like a kernel address and from the code/assembly, p->len derives its value from p = start, ie start as the first argument to apply_paravirt(). He then noticed strange relocation alignment, entries 0x10 apart, up until the second set, in the rela.klp.arch.vmlinux..parainstructions section:

    Relocation section [208] '.rela.klp.arch.vmlinux..parainstructions' for section [207] '.klp.arch.vmlinux..parainstructions' at offset 0x29dc78 contains 10 entries:
  Offset              Type            Value               Addend Name
  000000000000000000  X86_64_64       000000000000000000    +750 __get_user_pages
  0x0000000000000010  X86_64_64       000000000000000000    +823 __get_user_pages
  0x0000000000000020  X86_64_64       000000000000000000    +890 __get_user_pages
  0x0000000000000030  X86_64_64       000000000000000000    +941 __get_user_pages
  0x0000000000000040  X86_64_64       000000000000000000   +1631 __get_user_pages
  0x0000000000000050  X86_64_64       000000000000000000   +1671 __get_user_pages
  0x000000000000005c  X86_64_64       000000000000000000   +1245 handle_userfault
                  ^^
  0x000000000000006c  X86_64_64       000000000000000000   +1340 handle_userfault
  0x000000000000007c  X86_64_64       000000000000000000   +1417 handle_userfault
  0x000000000000008c  X86_64_64       000000000000000000   +1717 handle_userfault
  2. jpoimboe
    added 3 commits that reference this issue on May 18, 2018

    create-klp-module: Properly align merged .parainstructions sections

    aa9f5af

    create-klp-module: Properly align merged .parainstructions sections

    83529e7

    create-klp-module: Properly align merged .parainstructions sections

    3d04265
  3. joe-lawrence
    closed this as completedin #853on May 21, 2018
  4. joe-lawrence
    added a commit that references this issue on May 29, 2018

    create-klp-module: Properly align merged .parainstructions sections

    ba4b3d1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions