I have already filed an issue at https://bugzilla.kernel.org/show_bug.cgi?id=91661 but I wanted to see if anybody here has any idea about this. I have been experiencing GPFs when removing TPROXY rules in modern kernels for a while. This occurs on many different bare-metal x86 machines, plus KVM. The setup to reproduce is a bit specific: 1. TUN tunnel to remote server, with default route through tunnel (I have no idea whether this is important, but I haven't reproduced without it) 2. Transparent TCP proxy, using IP_TRANSPARENT to preserve src and dst addresses. (Basically a TCP accelerator) 3. TPROXY setup with the following rules: iptables -t nat -A OUTPUT -o tun10 -p tcp -m multiport --ports 80,443,8080 -j REDIRECT --to-ports 5080 iptables -t mangle -A bridge_existing -j MARK --set-xmark 0xf0/0xffffffff iptables -t mangle -A bridge_existing -j ACCEPT iptables -t mangle -A tcp_proxy -d 10.123.3.0/24 -p tcp -m socket -j bridge_existing iptables -t mangle -A tcp_proxy -d 10.123.3.1/32 -p tcp -j RETURN iptables -t mangle -A tcp_proxy -d 10.123.3.0/24 -p tcp -m multiport --dports 80,443,8080 -j TPROXY --on-port 5080 --on-ip 127.0.0.1 --tproxy-mark 0xf0/0xffffffff 4. As per the TPROXY documentation, the marked packets are sent to a separate routing table: local default dev lo scope host 5. Regular traffic through the box from outside. (I use 40 browser tabs accessing random pages every 10 seconds) Sometimes, when these rules are removed, the GPF occurs. To reproduce, I add and remove the rules every 3 seconds. It usually occurs within 10 minutes. Here is the info with a normal kernel (bare-metal): [ 262.017241] general protection fault: 0000 [#1] SMP [ 262.017436] Modules linked in: nf_conntrack_netlink nfnetlink netconsole configfs sch_sfq xt_connbytes xt_hashlimit xt_TPROXY xt_socket xt_length nf_defrag_ipv6 xt_REDIRECT nf_nat_redirect xt_multiport sch_htb xt_TCPMSS xt_CLASSIFY xt_dscp xt_mark xt_nat ipt_REJECT nf_reject_ipv4 xt_state xt_comment veth xt_CHECKSUM xt_tcpudp iptable_mangle iptable_filter xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables arptable_filter arp_tables x_tables tun nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc 8021q garp mrp joydev hid_generic usbhid hid bridge stp llc x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_pcm ast ghash_clmulni_intel snd_timer ttm aesni_intel iTCO_wdt evdev iTCO_vendor_support aes_x86_64 snd drm_kms_helper lrw gf128mul glue_helper soundcore ablk_helper drm cryptd i2c_i801 pcspkr tpm_tis battery tpm acpi_pad video button xhci_pci mei_me xhci_hcd mei processor lpc_ich shpchp mfd_core ipmi_watchdog ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod sg sd_mod crc32c_intel ahci libahci libata scsi_mod igb i2c_algo_bit ehci_pci i2c_core ehci_hcd dca ptp pps_core usbcore usb_common fan thermal thermal_sys [ 262.022561] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.19.0-rc4+ #2 [ 262.023149] Hardware name: Supermicro X10SLM+-LN4F/X10SLM+-LN4F, BIOS 2.00 04/24/2014 [ 262.023744] task: ffff8807fbecea00 ti: ffff8807fbee0000 task.ti: ffff8807fbee0000 [ 262.024370] RIP: 0010:[<ffffffff8140f553>] [<ffffffff8140f553>] __sk_free+0x13/0x130 [ 262.025027] RSP: 0018:ffff88081fd03b28 EFLAGS: 00010202 [ 262.025676] RAX: 3fa6000500003a98 RBX: ffff8806f2a80280 RCX: ffffffffa052c780 [ 262.026414] RDX: 6c6c616d560a026b RSI: ffff8807c8acc000 RDI: ffff8806f2a80280 [ 262.027133] RBP: ffff88078b5abc00 R08: ffff88078b45709c R09: 0000000000000001 [ 262.027838] R10: ffff88077c6e1c00 R11: 0000000000000001 R12: ffff880036c1bec0 [ 262.028559] R13: ffff880036fcd000 R14: ffff8807c8acc000 R15: 0000000000000000 [ 262.029314] FS: 0000000000000000(0000) GS:ffff88081fd00000(0000) knlGS:0000000000000000 [ 262.030096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 262.030887] CR2: ffffffffff600400 CR3: 0000000001816000 CR4: 00000000001407e0 [ 262.031677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 262.032495] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 262.033310] Stack: [ 262.034118] ffff8806f2a80280 ffff88078b5abc00 ffff880036c1bec0 ffffffffa0529113 [ 262.034993] ffff88078b5abc00 ffff8807c8acc000 ffff88078b5abc00 ffffffff818cf4d8 [ 262.035847] 00000000000005ac ffffffff8142630a 0000000000000000 ffff8807bc43dc00 [ 262.036752] Call Trace: [ 262.037637] <IRQ> [ 262.037705] [<ffffffffa0529113>] ? tun_net_xmit+0x143/0x390 [tun] [ 262.039532] [<ffffffff8142630a>] ? dev_hard_start_xmit+0x16a/0x3a0 [ 262.040440] [<ffffffff81425ec0>] ? validate_xmit_skb.isra.93.part.94+0x10/0x2f0 [ 262.041349] [<ffffffff81445da7>] ? sch_direct_xmit+0xc7/0x1d0 [ 262.042294] [<ffffffff81445f3e>] ? __qdisc_run+0x8e/0x1c0 [ 262.043271] [<ffffffff814267ca>] ? __dev_queue_xmit+0x28a/0x520 [ 262.044228] [<ffffffff81460477>] ? ip_finish_output2+0x137/0x3c0 [ 262.045203] [<ffffffff8145efff>] ? ip_fragment+0x2df/0xa70 [ 262.046170] [<ffffffff81460340>] ? ip_append_data.part.46+0xe0/0xe0 [ 262.047156] [<ffffffff81460770>] ? skb_set_owner_w+0x70/0x70 [ 262.048155] [<ffffffff81460bcf>] ? ip_finish_output+0x45f/0x850 [ 262.049147] [<ffffffff81424402>] ? __netif_receive_skb_core+0x552/0x7d0 [ 262.050159] [<ffffffff8101bca6>] ? native_sched_clock+0x26/0x90 [ 262.051140] [<ffffffff8101bd15>] ? sched_clock+0x5/0x10 [ 262.052140] [<ffffffff814253d2>] ? process_backlog+0xa2/0x130 [ 262.053157] [<ffffffff81424be1>] ? net_rx_action+0x201/0x340 [ 262.054148] [<ffffffff8106aecc>] ? __do_softirq+0x10c/0x280 [ 262.055132] [<ffffffff8106b195>] ? irq_exit+0x95/0xa0 [ 262.056076] [<ffffffff81014cda>] ? do_IRQ+0x4a/0xd0 [ 262.056984] [<ffffffff815178ed>] ? common_interrupt+0x6d/0x6d [ 262.057925] <EOI> [ 262.057936] [<ffffffff813f3a0c>] ? cpuidle_enter_state+0x5c/0x150 [ 262.059617] [<ffffffff813f39f9>] ? cpuidle_enter_state+0x49/0x150 [ 262.060425] [<ffffffff810a142d>] ? cpu_startup_entry+0x2fd/0x3a0 [ 262.061204] [<ffffffff810439a5>] ? start_secondary+0x155/0x180 [ 262.061961] Code: f0 29 9d 20 01 00 00 eb e8 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 41 54 55 53 48 8b 87 b0 02 00 00 48 89 fb 48 85 c0 74 02 <ff> d0 48 8b b3 e0 00 00 00 48 85 f6 74 13 48 89 df e8 f7 a2 02 [ 262.063621] RIP [<ffffffff8140f553>] __sk_free+0x13/0x130 [ 262.064359] RSP <ffff88081fd03b28> [ 262.065164] ---[ end trace 79ce0371f0ffc393 ]--- [ 262.761655] Kernel panic - not syncing: Fatal exception in interrupt [ 262.763647] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 262.764073] drm_kms_helper: panic occurred, switching back to text console [ 263.366508] ---[ end Kernel panic - not syncing: Fatal exception in interrupt If I enable lock debugging, I get a similar, but different result: [ 1080.464457] general protection fault: 0000 [#1] SMP [ 1080.469469] Modules linked in: xt_TPROXY xt_socket nf_defrag_ipv6 xt_REDIRECT nf_nat_redirect xt_multiport sch_htb tun xt_CLASSIFY xt_dscp xt_TCPMSS xt_mark xt_tcpudp tcp_yeah tcp_westwood tcp_veno tcp_vegas tcp_scalable tcp_lp tcp_illinois tcp_hybla tcp_htcp tcp_highspeed tcp_diag inet_diag tcp_bic xt_nat iptable_mangle iptable_nat nf_nat_ipv4 nf_nat 8021q garp mrp stp llc ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment arptable_filter arp_tables iptable_filter ip_tables x_tables nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc joydev hid_generic usbhid hid loop x86_pkg_temp_thermal intel_powerclamp intel_rapl iosf_mbi coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel evdev ast ttm drm_kms_helper snd_pcm aesni_intel aes_x86_64 drm lrw snd_timer gf128mul glue_helper snd ablk_helper cryptd soundcore i2c_i801 pcspkr ipmi_si battery tpm_tis ipmi_msghandler tpm video shpchp acpi_pad lpc_ich xhci_pci mfd_core mei_me xhci_hcd mei processor button ext4 crc16 mbcache jbd2 sg sd_mod crc32c_intel igb i2c_algo_bit ahci libahci i2c_core dca ehci_pci ehci_hcd libata fan thermal thermal_sys e1000e scsi_mod usbcore usb_common ptp pps_core [ 1080.581534] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0-rc5+ #19 [ 1080.587978] Hardware name: Supermicro X10SLM-F/X10SLM-F, BIOS 2.0 04/24/2014 [ 1080.595028] task: ffffffff81a1b520 ti: ffffffff81a00000 task.ti: ffffffff81a00000 [ 1080.602521] RIP: 0010:[<ffffffff810c01f0>] [<ffffffff810c01f0>] __lock_acquire+0x660/0x1ca0 [ 1080.610982] RSP: 0018:ffff88041fc03668 EFLAGS: 00010002 [ 1080.616302] RAX: 0000000000000000 RBX: 3ca096b028d7976b RCX: 0000000000000000 [ 1080.623447] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8804071f3220 [ 1080.630594] RBP: ffff88041fc03758 R08: 0000000000000001 R09: 0000000000000001 [ 1080.637740] R10: ffffffff81a1b520 R11: 0000000000000001 R12: 0000000000000000 [ 1080.644885] R13: ffff8804071f3220 R14: 0000000000000001 R15: 0000000000000000 [ 1080.652031] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [ 1080.660130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1080.665882] CR2: 00007fc91a21f000 CR3: 0000000001a14000 CR4: 00000000001407f0 [ 1080.673016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1080.680154] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1080.687292] Stack: [ 1080.689812] ffff8800d34ba390 ffffffff81076fe4 00000000f7167000 ffff8803f71090a0 [ 1080.697816] 000000001fc036a8 ffffffff81a1bda8 0000000000000007 ffffffff81a1bd58 [ 1080.705819] 0000000000000005 0000000000000001 000000000000002e 000000000000002e [ 1080.713823] Call Trace: [ 1080.716784] <IRQ> [ 1080.718715] [<ffffffff81076fe4>] ? __local_bh_enable_ip+0xa4/0xf0 [ 1080.726125] [<ffffffff810c1e35>] lock_acquire+0xe5/0x140 [ 1080.732040] [<ffffffff814ba99b>] ? skb_queue_tail+0x2b/0x60 [ 1080.738215] [<ffffffff815f679e>] _raw_spin_lock_irqsave+0x4e/0x70 [ 1080.744918] [<ffffffff814ba99b>] ? skb_queue_tail+0x2b/0x60 [ 1080.751102] [<ffffffff810c28a7>] ? trace_hardirqs_on_caller+0x1d7/0x210 [ 1080.758327] [<ffffffff814ba99b>] skb_queue_tail+0x2b/0x60 [ 1080.764347] [<ffffffff814bc6c8>] sock_queue_err_skb+0xe8/0x150 [ 1080.770802] [<ffffffff814ba70e>] ? __skb_clone+0x2e/0x140 [ 1080.776820] [<ffffffff814bcc03>] __skb_complete_tx_timestamp+0xe3/0x100 [ 1080.784060] [<ffffffff814bcc98>] __skb_tstamp_tx+0x78/0x90 [ 1080.790168] [<ffffffff814bccc4>] skb_tstamp_tx+0x14/0x20 [ 1080.796100] [<ffffffffa07d89a4>] tun_net_xmit+0x304/0x4c0 [tun] [ 1080.802641] [<ffffffffa07d86a5>] ? tun_net_xmit+0x5/0x4c0 [tun] [ 1080.809171] [<ffffffff814d0852>] dev_hard_start_xmit+0x3a2/0x4f0 [ 1080.815790] [<ffffffff814f64b9>] sch_direct_xmit+0xa9/0x1d0 [ 1080.821972] [<ffffffff814d0de4>] __dev_queue_xmit+0x444/0x7a0 [ 1080.828312] [<ffffffff814d0a00>] ? __dev_queue_xmit+0x60/0x7a0 [ 1080.834721] [<ffffffff814d1160>] dev_queue_xmit+0x10/0x20 [ 1080.840692] [<ffffffff814d8ed1>] neigh_direct_output+0x11/0x20 [ 1080.847073] [<ffffffff81516ad4>] ip_finish_output2+0x494/0x600 [ 1080.853449] [<ffffffff815167d8>] ? ip_finish_output2+0x198/0x600 [ 1080.859981] [<ffffffff810bd6ff>] ? __lock_is_held+0x4f/0x80 [ 1080.866068] [<ffffffff81517508>] ip_finish_output+0x8c8/0xa00 [ 1080.872313] [<ffffffff81518238>] ip_output+0x88/0xe0 [ 1080.877763] [<ffffffff815130a9>] ip_forward_finish+0xe9/0x150 [ 1080.883981] [<ffffffff8151348c>] ip_forward+0x37c/0x550 [ 1080.889663] [<ffffffff8151131e>] ip_rcv_finish+0x46e/0x580 [ 1080.895606] [<ffffffff815119ce>] ip_rcv+0x33e/0x3d0 [ 1080.900932] [<ffffffff814cccfe>] __netif_receive_skb_core+0x83e/0x950 [ 1080.907819] [<ffffffff814cc569>] ? __netif_receive_skb_core+0xa9/0x950 [ 1080.914784] [<ffffffff810edc75>] ? ktime_get_with_offset+0xb5/0x150 [ 1080.921485] [<ffffffff814cce67>] __netif_receive_skb+0x57/0x80 [ 1080.927748] [<ffffffff814ce218>] netif_receive_skb_internal+0x168/0x1e0 [ 1080.934798] [<ffffffff814cf080>] napi_gro_receive+0x70/0xf0 [ 1080.940817] [<ffffffffa01b0b19>] igb_poll+0xa89/0xe10 [igb] [ 1080.946830] [<ffffffff814cec10>] net_rx_action+0x140/0x340 [ 1080.952747] [<ffffffff814268da>] ? add_interrupt_randomness+0x3a/0x1e0 [ 1080.959700] [<ffffffff81076cd7>] __do_softirq+0x167/0x2f0 [ 1080.965532] [<ffffffff810770e7>] irq_exit+0x47/0xb0 [ 1080.970840] [<ffffffff815f9cdd>] do_IRQ+0xcd/0xf0 [ 1080.975975] [<ffffffff815f7932>] common_interrupt+0x72/0x72 [ 1080.981969] <EOI> [ 1080.983900] [<ffffffff8149598b>] ? cpuidle_enter_state+0xbb/0x190 [ 1080.990961] [<ffffffff81495984>] ? cpuidle_enter_state+0xb4/0x190 [ 1080.997465] [<ffffffff81495b37>] cpuidle_enter+0x17/0x20 [ 1081.003192] [<ffffffff810b78f6>] cpu_startup_entry+0x2c6/0x400 [ 1081.009438] [<ffffffff815e143d>] rest_init+0x12d/0x140 [ 1081.014990] [<ffffffff815e1315>] ? rest_init+0x5/0x140 [ 1081.020541] [<ffffffff81b4ccc3>] ? ftrace_init+0xc6/0x159 [ 1081.026352] [<ffffffff81b27129>] start_kernel+0x4b2/0x4bf [ 1081.032156] [<ffffffff81b269d7>] ? set_init_arg+0x57/0x57 [ 1081.037959] [<ffffffff81b26117>] ? early_idt_handlers+0x117/0x120 [ 1081.044457] [<ffffffff81b265f0>] x86_64_start_reservations+0x2a/0x2c [ 1081.051212] [<ffffffff81b26738>] x86_64_start_kernel+0x146/0x155 [ 1081.057619] Code: 81 48 c7 c2 b6 5e 7f 81 be 3d 03 00 00 48 c7 c7 7c 96 7f 81 31 c0 e8 90 24 fb ff e9 03 05 00 00 48 85 db 0f 84 fa 04 00 00 66 90 <3e> ff 83 98 01 00 00 8b 05 9b 6d 87 01 45 8b a2 68 07 00 00 85 [ 1081.078236] RIP [<ffffffff810c01f0>] __lock_acquire+0x660/0x1ca0 [ 1081.084695] RSP <ffff88041fc03668> [ 1081.088535] ---[ end trace 1aa64404a291b379 ]--- [ 1081.095085] Kernel panic - not syncing: Fatal exception in interrupt [ 1081.101800] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 1081.112336] drm_kms_helper: panic occurred, switching back to text console git bisect points to this changeset as the first bad one: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f... The previous changeset does not appear to exhibit this behaviour. I ran it a number of times, including overnight, with no issues. Obviously, that change can't be the true cause, but it does appear to trigger it. -- James Oakley jfunk@funktronics.ca