[Buildbot] #3456: NMI's on bhyve VM
Buildbot trac
trac at buildbot.net
Sun Feb 14 15:09:06 UTC 2016
#3456: NMI's on bhyve VM
---------------------+---------------------
Reporter: dustin | Owner: verm
Type: defect | Status: new
Priority: critical | Milestone: ongoing
Version: | Keywords:
---------------------+---------------------
These are killing Docker and all Docker containers :(
{{{
[451055.988542] INFO: rcu_sched detected stalls on CPUs/tasks: { 1}
(detected by 2, t=8293 jiffies, g=1234241, c=1234240, q=36)
[451055.990486] sending NMI to all CPUs:
[451055.990497] NMI backtrace for cpu 1
[451055.990502] CPU: 1 PID: 7273 Comm: docker Tainted: G C
3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u3
[451055.990503] Hardware name: BHYVE, BIOS 1.00 03/14/2014
[451055.990505] task: ffff880003a083d0 ti: ffff8800bb3ec000 task.ti:
ffff8800bb3ec000
[451055.990506] RIP: 0033:[<00000000004c1525>] [<00000000004c1525>]
0x4c1525
[451055.990539] RSP: 002b:00007f91793139e8 EFLAGS: 00000202
[451055.990540] RAX: 0000000000000059 RBX: 00007f917c517ac8 RCX:
0000000000000000
[451055.990541] RDX: 00000000000002ce RSI: 000000c208012000 RDI:
0000000000000000
[451055.990543] RBP: 0000000000000000 R08: 0000000001da5560 R09:
000000c208000000
[451055.990544] R10: 000000c207ffedff R11: 00000000000059c0 R12:
00007f917c51fec0
[451055.990545] R13: 0000000000000003 R14: 0000000000000002 R15:
000000000000000d
[451055.990547] FS: 00007f9179314700(0000) GS:ffff88013fc80000(0000)
knlGS:0000000000000000
[451055.990548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[451055.990550] CR2: 00007fd0000c9678 CR3: 00000000b91ec000 CR4:
00000000000406e0
[451055.990555] NMI backtrace for cpu 2
[451055.990558] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G C
3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u3
[451055.990559] Hardware name: BHYVE, BIOS 1.00 03/14/2014
[451055.990561] task: ffff88013af720d0 ti: ffff88013afc0000 task.ti:
ffff88013afc0000
[451055.990562] RIP: 0010:[<ffffffff810466dd>] [<ffffffff810466dd>]
default_send_IPI_mask_sequence_phys+0xad/0xe0
[451055.990566] RSP: 0018:ffff88013fd03e18 EFLAGS: 00000046
[451055.990568] RAX: 0000000000000400 RBX: 000000000000a0ea RCX:
0000000000000002
[451055.990569] RDX: 0000000000000002 RSI: 0000000000000200 RDI:
0000000000000300
[451055.990570] RBP: ffffffff818e2860 R08: ffffffff818e2860 R09:
0000000000000000
[451055.990571] R10: 0000000000000331 R11: ffff88013fd03b96 R12:
0000000000000400
[451055.990572] R13: 0000000000000096 R14: 0000000000000002 R15:
0000000000000002
[451055.990574] FS: 0000000000000000(0000) GS:ffff88013fd00000(0000)
knlGS:0000000000000000
[451055.990575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[451055.990577] CR2: 00007f5cb5f9db40 CR3: 0000000018349000 CR4:
00000000000406e0
[451055.990578] Stack:
[451055.990579] 0000000000000002 ffff88013fd0d660 ffffffff81853700
0000000000000002
[451055.990581] ffffffff818e2dc0 0000000000000024 ffffffff81853700
ffffffff81046963
[451055.990583] ffff88013fd0d660 ffffffff810c57ad 0000000000000001
0000000000000083
[451055.990585] Call Trace:
[451055.990586] <IRQ>
[451055.990590] [<ffffffff81046963>] ?
arch_trigger_all_cpu_backtrace+0xc3/0x140
[451055.990595] [<ffffffff810c57ad>] ? rcu_check_callbacks+0x60d/0x630
[451055.990599] [<ffffffff810cffa0>] ?
tick_sched_handle.isra.16+0x60/0x60
[451055.990604] [<ffffffff81074b80>] ? update_process_times+0x40/0x70
[451055.990607] [<ffffffff810cff60>] ?
tick_sched_handle.isra.16+0x20/0x60
[451055.990615] [<ffffffff810cffdc>] ? tick_sched_timer+0x3c/0x60
[451055.990619] [<ffffffff8108b147>] ? __run_hrtimer+0x67/0x1c0
[451055.990621] [<ffffffff8108b4f9>] ? hrtimer_interrupt+0xe9/0x220
[451055.990624] [<ffffffff81516beb>] ? smp_apic_timer_interrupt+0x3b/0x60
[451055.990628] [<ffffffff81514c7d>] ? apic_timer_interrupt+0x6d/0x80
[451055.990629] <EOI>
[451055.990632] [<ffffffff8101c8b0>] ? idle_notifier_unregister+0x20/0x20
[451055.990635] [<ffffffff81051c12>] ? native_safe_halt+0x2/0x10
[451055.990638] [<ffffffff8101c8c9>] ? default_idle+0x19/0xb0
[451055.990641] [<ffffffff810a8120>] ? cpu_startup_entry+0x340/0x400
[451055.990643] [<ffffffff81042c5f>] ? start_secondary+0x20f/0x2d0
[451055.990644] Code: 8b 0c 25 00 53 5f ff 80 e5 10 75 f2 44 89 f8 c1 e0
18 89 04 25 10 53 5f ff 41 83 fe 02 44 89 e0 41 0f 45 c6 89 04 25 00 53 5f
ff <eb> 91 4c 89 ef 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 5d 41 5c 41
[451055.990671] NMI backtrace for cpu 0
[451055.990675] CPU: 0 PID: 99 Comm: kworker/u6:4 Tainted: G C
3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u3
[451055.990676] Hardware name: BHYVE, BIOS 1.00 03/14/2014
[451055.990709] Workqueue: writeback bdi_writeback_workfn (flush-8:0)
[451055.990712] task: ffff880139f46150 ti: ffff88000395c000 task.ti:
ffff88000395c000
[451055.990714] RIP: 0010:[<ffffffffa00a0a4a>] [<ffffffffa00a0a4a>]
ahci_qc_issue+0x8a/0x110 [libahci]
[451055.990736] RSP: 0000:ffff88000395f760 EFLAGS: 00010006
[451055.990737] RAX: 0000000000002000 RBX: ffff8800038e0000 RCX:
000000000000000d
[451055.990739] RDX: ffffc9000067c100 RSI: 0000000000002000 RDI:
ffff8800038e0d20
[451055.990740] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000068
[451055.990741] R10: ffff880007d4d688 R11: 0000000000000000 R12:
ffff8800038e1e70
[451055.990742] R13: ffff88013a9fb098 R14: 0000000000000001 R15:
ffff8800038e0d20
[451055.990744] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000)
knlGS:0000000000000000
[451055.990745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[451055.990747] CR2: 00007fbb5e7fc000 CR3: 000000013a3fe000 CR4:
00000000000406f0
[451055.990748] Stack:
[451055.990749] ffff8800038e0000 ffffffffa005d73c 00000001038e0d20
ffff8800082e8e00
[451055.990751] ffffffff8182c8e0 0000000000000000 ffff8800038e0d20
ffff8800038e0000
[451055.990753] 0000000000000286 ffffffffa00632f0 ffff8800038e22a0
ffffffffa00662dd
[451055.990755] Call Trace:
[451055.990782] [<ffffffffa005d73c>] ? ata_qc_issue+0x16c/0x390 [libata]
[451055.990789] [<ffffffffa00632f0>] ? ata_scsi_invalid_field+0x40/0x40
[libata]
[451055.990795] [<ffffffffa00662dd>] ? ata_scsi_queuecmd+0x12d/0x410
[libata]
[451055.990816] [<ffffffffa0009394>] ? scsi_dispatch_cmd+0xb4/0x270
[scsi_mod]
[451055.990822] [<ffffffffa0011b8d>] ? scsi_request_fn+0x2fd/0x500
[scsi_mod]
[451055.990840] [<ffffffff81215c34>] ? kernfs_path_locked+0x34/0x80
[451055.990858] [<ffffffff8127d53f>] ? __blk_run_queue+0x2f/0x40
[451055.990878] [<ffffffff812a6d34>] ? cfq_insert_request+0x2f4/0x560
[451055.990882] [<ffffffff81281778>] ? blk_flush_plug_list+0x1d8/0x230
[451055.990885] [<ffffffff812819ca>] ? blk_queue_bio+0x1fa/0x360
[451055.990888] [<ffffffff8127d370>] ? generic_make_request+0xb0/0x100
[451055.990890] [<ffffffff8127d431>] ? submit_bio+0x71/0x150
[451055.990893] [<ffffffff81278b48>] ? bio_alloc_bioset+0x198/0x290
[451055.990896] [<ffffffff811d829a>] ? _submit_bh+0x12a/0x1f0
[451055.990899] [<ffffffff811dada1>] ?
__block_write_full_page.constprop.37+0x121/0x390
[451055.990901] [<ffffffff811db800>] ? I_BDEV+0x10/0x10
[451055.990916] [<ffffffff81173440>] ? page_referenced_one+0x160/0x160
[451055.990925] [<ffffffff811470ef>] ? __writepage+0xf/0x40
[451055.990928] [<ffffffff811479bb>] ? write_cache_pages+0x1db/0x3f0
[451055.990930] [<ffffffff811470e0>] ? global_dirtyable_memory+0x50/0x50
[451055.990961] [<ffffffffa00f1ae6>] ? ext4_writepages+0x446/0xd00 [ext4]
[451055.990964] [<ffffffff81147c13>] ? generic_writepages+0x43/0x60
[451055.990967] [<ffffffff811ceb79>] ?
__writeback_single_inode+0x39/0x220
[451055.990973] [<ffffffff811cf7e4>] ? writeback_sb_inodes+0x1a4/0x3e0
[451055.990976] [<ffffffff811cfab6>] ? __writeback_inodes_wb+0x96/0xc0
[451055.990979] [<ffffffff811cfd23>] ? wb_writeback+0x243/0x2d0
[451055.990982] [<ffffffff811d2061>] ? bdi_writeback_workfn+0x2a1/0x420
[451055.990986] [<ffffffff81081762>] ? process_one_work+0x172/0x420
[451055.990988] [<ffffffff81081df3>] ? worker_thread+0x113/0x4f0
[451055.990990] [<ffffffff81081ce0>] ? rescuer_thread+0x2d0/0x2d0
[451055.990993] [<ffffffff8108805d>] ? kthread+0xbd/0xe0
[451055.990996] [<ffffffff81087fa0>] ? kthread_create_on_node+0x180/0x180
[451055.991000] [<ffffffff81513c58>] ? ret_from_fork+0x58/0x90
[451055.991002] [<ffffffff81087fa0>] ? kthread_create_on_node+0x180/0x180
[451055.991004] Code: f0 ff ff 48 8b 36 8b 76 08 c1 e6 08 09 f1 89 4a 40
48 8b 4f 08 48 8b 09 8b 49 08 89 48 4c 8b 4f 58 b8 01 00 00 00 d3 e0 89 42
38 <48> 8b 47 08 48 8b 00 f6 80 d0 02 00 00 80 48 8b 10 8b 48 08 48
}}}
--
Ticket URL: <http://trac.buildbot.net/ticket/3456>
Buildbot <http://buildbot.net/>
Buildbot: build/test automation
More information about the bugs
mailing list