Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugcheck in ovpn_dco!OvpnTxBufferPoolGet #47

Open
lstipakov opened this issue Sep 5, 2023 · 2 comments
Open

Bugcheck in ovpn_dco!OvpnTxBufferPoolGet #47

lstipakov opened this issue Sep 5, 2023 · 2 comments
Assignees
@lstipakov
Labels
bug Something isn't working bugcheck

Comments

@lstipakov
Copy link
Member 

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000020, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, bitfield :
	bit 0 : value 0 = read operation, 1 = write operation
	bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff8030f6c15a4, address which referenced memory

Debugging Details:
------------------


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 2436

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 9260

    Key  : Analysis.Init.CPU.mSec
    Value: 17421

    Key  : Analysis.Init.Elapsed.mSec
    Value: 6418256

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 115

    Key  : WER.OS.Branch
    Value: ni_release_svc_prod3

    Key  : WER.OS.Timestamp
    Value: 2023-06-22T09:51:00Z

    Key  : WER.OS.Version
    Value: 10.0.22621.1928


FILE_IN_CAB:  090123-14234-01.dmp

TAG_NOT_DEFINED_202b:  *** Unknown TAG in analysis list 202b


BUGCHECK_CODE:  a

BUGCHECK_P1: 20

BUGCHECK_P2: 2

BUGCHECK_P3: 0

BUGCHECK_P4: fffff8030f6c15a4

READ_ADDRESS: Unable to get NonPagedPoolStart
Unable to get NonPagedPoolEnd
Unable to get PagedPoolStart
Unable to get PagedPoolEnd
 0000000000000020 

BLACKBOXBSD: 1 (!blackboxbsd)


BLACKBOXNTFS: 1 (!blackboxntfs)


BLACKBOXPNP: 1 (!blackboxpnp)


BLACKBOXWINLOGON: 1

CUSTOMER_CRASH_COUNT:  1

PROCESS_NAME:  System

TRAP_FRAME:  fffffe893a787260 -- (.trap 0xfffffe893a787260)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=00000000000002a0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8030f6c15a4 rsp=fffffe893a7873f0 rbp=fffffe893a787551
 r8=00000000000000d0  r9=0000000000000000 r10=0000000000000000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na po nc
nt!MmBuildMdlForNonPagedPool+0x14:
fffff803`0f6c15a4 488b7920        mov     rdi,qword ptr [rcx+20h] ds:00000000`00000020=????????????????
Resetting default scope

STACK_TEXT:  
fffffe89`3a787118 fffff803`0f8468a9     : 00000000`0000000a 00000000`00000020 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffffe89`3a787120 fffff803`0f841f34     : 00000000`00000000 fffff803`000000d0 00000000`00000002 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffffe89`3a787260 fffff803`0f6c15a4     : 00000000`00000008 00000000`000002b3 00000000`00000656 00000000`000000b8 : nt!KiPageFault+0x474
fffffe89`3a7873f0 fffff803`2a252e69     : 00000000`000002b3 fffffe89`3a787510 fffffe89`3a787510 fffffe89`3a787551 : nt!MmBuildMdlForNonPagedPool+0x14
fffffe89`3a787480 fffff803`2a257bf7     : ffff9285`41a262f0 fffff803`2a27294e fffffe89`3a7875bc 00000000`00000000 : ovpn_dco!OvpnTxBufferPoolGet+0x4d [c:\Users\lev\Projects\ovpn-dco-win\bufferpool.cpp @ 167] 
fffffe89`3a7874c0 fffff803`2a257a17     : ffff9285`41c6b398 ffff9285`41bd0a90 fffffe89`3a787610 fffffe89`3a7875f0 : ovpn_dco!OvpnTxProcessPacket+0x63 [c:\Users\lev\Projects\ovpn-dco-win\txqueue.cpp @ 49] 
fffffe89`3a7875b0 fffff803`2a28d8aa     : ffff9285`4462d0d0 fffffe89`3a787900 ffff9285`41bd0a90 fffff803`2a273995 : ovpn_dco!OvpnEvtTxQueueAdvance+0xe7 [c:\Users\lev\Projects\ovpn-dco-win\txqueue.cpp @ 154] 
fffffe89`3a787670 fffff803`2a29d915     : ffff9285`41c6b1a0 00000000`000005c0 ffff9285`41c6b1a0 ffff9285`41c6b640 : NetAdapterCx!NetClientQueueAdvance+0x2a [minio\netcx\adapter\nxqueue.cpp @ 50] 
fffffe89`3a7876a0 fffff803`2a29d4c6     : ffff9285`41c6b2c0 ffff9285`41c6b5c0 fffffe89`3a7879d0 ffff9285`41c6b5a0 : NetAdapterCx!NxTxXlat::YieldToNetAdapter+0xa5 [minio\netcx\translator\nxtxxlat.cpp @ 478] 
fffffe89`3a7876d0 fffff803`2a29d209     : 00000000`00000000 fffffe89`3a7879f0 ffff9285`41c6b1e8 00000000`00000000 : NetAdapterCx!NxTxXlat::TransmitNbls+0x196 [minio\netcx\translator\nxtxxlat.cpp @ 219] 
fffffe89`3a7878c0 fffff803`2a2a785a     : ffff9285`41c6b1e8 ffff9285`41c6c4d8 00000000`00000004 00000000`00000000 : NetAdapterCx!EvtTxPollQueueStarted+0x9 [minio\netcx\translator\nxtxxlat.cpp @ 185] 
fffffe89`3a7878f0 fffff803`2a2a7db1     : ffff9285`250fe040 fffff803`0f860c76 00000000`00000000 00000000`00000080 : NetAdapterCx!ExecutionContext::Poll+0x19e [minio\netcx\ec\lib\executioncontext.cpp @ 890] 
fffffe89`3a787a40 fffff803`2a2a7c47     : 00000000`00000003 00000000`00000080 ffff9285`2a1b7100 fffff780`00000014 : NetAdapterCx!ExecutionContext::RunLockHeld+0x13d [minio\netcx\ec\lib\executioncontext.cpp @ 1093] 
fffffe89`3a787a80 fffff803`2a2a7561     : ffff9285`41c6c010 ffff9285`41c6c010 ffff9285`254c9000 ffff9285`2a1b7100 : NetAdapterCx!ExecutionContext::RunLockNotHeld+0x3b [minio\netcx\ec\lib\executioncontext.cpp @ 1010] 
fffffe89`3a787ac0 fffff803`2a2c7639     : ffff9285`4497b040 fffff803`2a2c7630 00000000`00000000 00000000`0443315f : NetAdapterCx!ExecutionContext::WorkerThreadRoutine+0x65 [minio\netcx\ec\lib\executioncontext.cpp @ 762] 
fffffe89`3a787b00 fffff803`0f724e47     : ffff9285`4497b040 fffff803`2a2c7630 ffff9285`41c6c010 006fe47f`b19bbdff : NetAdapterCx!ExecutionContext::Thread+0x9 [minio\netcx\ec\lib\executioncontext.cpp @ 737] 
fffffe89`3a787b30 fffff803`0f8361b4     : ffffbd80`77051180 ffff9285`4497b040 fffff803`0f724df0 00000000`00000000 : nt!PspSystemThreadStartup+0x57
fffffe89`3a787b80 00000000`00000000     : fffffe89`3a788000 fffffe89`3a781000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x34


FAULTING_SOURCE_LINE:  c:\Users\lev\Projects\ovpn-dco-win\bufferpool.cpp

FAULTING_SOURCE_FILE:  c:\Users\lev\Projects\ovpn-dco-win\bufferpool.cpp

FAULTING_SOURCE_LINE_NUMBER:  167

FAULTING_SOURCE_CODE:  
   163: 
   164:     (*buffer)->Mdl = IoAllocateMdl(*buffer, ((OVPN_BUFFER_POOL_IMPL*)handle)->ItemSize, FALSE, FALSE, NULL);
   165:     MmBuildMdlForNonPagedPool((*buffer)->Mdl);
   166: 
>  167:     (*buffer)->Pool = handle;
   168: 
   169:     (*buffer)->Data = (*buffer)->Head + OVPN_BUFFER_HEADROOM;
   170:     (*buffer)->Tail = (*buffer)->Data;
   171: 
   172:     (*buffer)->Len = 0;


SYMBOL_NAME:  ovpn_dco!OvpnTxBufferPoolGet+4d

MODULE_NAME: ovpn_dco

IMAGE_NAME:  ovpn-dco.sys

STACK_COMMAND:  .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET:  4d

FAILURE_BUCKET_ID:  AV_ovpn_dco!OvpnTxBufferPoolGet

OS_VERSION:  10.0.22621.1928

BUILDLAB_STR:  ni_release_svc_prod3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {77c9c180-2793-eebf-3c3b-97fd67f64bbe}

Followup:     MachineOwner
---------
@lstipakov lstipakov self-assigned this Sep 5, 2023
@lstipakov lstipakov added the bug Something isn't working label Sep 5, 2023
@lstipakov
Copy link
Member Author

From what I see, TX pool is unreasonably large:

3: kd> dx ((ovpn_dco!OVPN_BUFFER_POOL_IMPL *)0xffff928541996b90)
((ovpn_dco!OVPN_BUFFER_POOL_IMPL *)0xffff928541996b90)                 : 0xffff928541996b90 [Type: OVPN_BUFFER_POOL_IMPL *]
    [+0x000] ListHead         [Type: _LIST_ENTRY]
    [+0x010] Lock             : 0x0 [Type: unsigned __int64]
    [+0x018] ItemSize         : 0x656 [Type: unsigned int]
    [+0x01c] PoolSize         : 1832047 [Type: long]
    [+0x020] Context          : 0xffff928541a262f0 [Type: void *]
    [+0x028] Tag              : 0xfffff8032a25b280 : "tx" [Type: char *]

@lstipakov
Copy link
Member Author

Portal incident id: 46108914

lstipakov added a commit to lstipakov/ovpn-dco-win that referenced this issue Sep 25, 2023
@lstipakov
Improve pool memory management
f8feaa2 
 - Handle case when IoAllocateMdl() and ExAllocatePool2()
 return NULL

 - Set max pool size (in-flight packets) to 100'000.

This might help with OpenVPN#47

By playing with Quick Edit in console window (which makes execution
stop) where iPerf3 was running I was able to get pool size growing
to unreasonable values (over 50k). My understanding is that normally it
should be <10k, but let's be on a safe side and cap it to 100k - in
kernel dump it was over 1mil and system ran out of memory.

Bump to 0.9.4.

Signed-off-by: Lev Stipakov <[email protected]>
@lstipakov lstipakov mentioned this issue Sep 25, 2023
Merged
lstipakov added a commit that referenced this issue Sep 26, 2023
@lstipakov
Improve pool memory management
74755eb 
 - Handle case when IoAllocateMdl() and ExAllocatePool2()
 return NULL

 - Set max pool size (in-flight packets) to 100'000.

This might help with #47

By playing with Quick Edit in console window (which makes execution
stop) where iPerf3 was running I was able to get pool size growing
to unreasonable values (over 50k). My understanding is that normally it
should be <10k, but let's be on a safe side and cap it to 100k - in
kernel dump it was over 1mil and system ran out of memory.

Bump to 0.9.4.

Signed-off-by: Lev Stipakov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lstipakov lstipakov

Labels
bug Something isn't working bugcheck
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lstipakov